Weblogic State and HealthState Monitoring with Email Notification

Configure Weblogic Mail Session (Optional. Only if you want to send email alert)
1. Login to weblogic console
2. Go to Mail Sessions
3. Click New
4. Enter details Name,JNDI Name, JavaMail Properties.
JavaMail Properties:
mail.port=25
mail.user=weblogic
mail.host=xx.xx.xx.xx
mail.transport.protocol=smtp
mail.from=weblogic

Configure Email Notifications in Weblogic Diagnostic
Go to notifications –> click new –> Select SMTP (E-Mail) –> Click Next –> Enter Notification Name & Check Enable Notification –> Got to SMTP Properties –> Select Mail session ->Enter Email Recipients

Server State Monitoring
1. Login to weblogic console
2. Go to Diagnostic Modules
3. Click on “
Module-FMWDFW” module
4. Go to Configuration –> Watches and Notifications –> Watches
5. Click New
6. Enter Watch Name, Watch Type: Collected Metrics
7. Click Next
8. Click on Add Expressions
9. Select ServerRunTIme –> Select weblogic.management.runtime.ServerLifeCycleRuntimeMBean –> Click Next –> Click Next
6. Select Message Attribute “State”
7. Select Operator “!=”
8. Enter value RUNNING
9. Below watch rule will be generated
(${ServerRuntime//[weblogic.management.runtime.ServerLifeCycleRuntimeMBean]//State} != ‘RUNNING’)
10. ServerRuntime dies along with the managed Server. Therefore we need to DomainRuntime.
Edit rule manually replace ServerRuntime with DomainRuntime
(${DomainRuntime//[weblogic.management.runtime.ServerLifeCycleRuntimeMBean]//State} != ‘RUNNING’)
11. Click Next
12. Select Alarm (if required. Used for not spamming email)
13. Select the notifications
14. Click Finish

Health State Monitoring
Stuck thread is very common issue with weblogic servers. Below is a good article on dealing with stuck threads.
http://oraclemiddlewareblog.com/2014/06/10/dealing-stuck-threads-weblogic/
We had a requirement to capture server healthstate for stuck thread. By default healthstate is not collected from diagnostic module. A harvester must be created to gather healthstate data.
1. Go to Weblogic Console –> Diagnostic Modules
2. Configuration tab –> Collected Metrics tab
3. Click new
4. Select ServerRuntime –> Select weblogic.management.runtime.ThreadPoolRuntimeMBean
5. Add Attribute Expression as “HealthState.State”  (without quotes)
6. Select the Server Instance
7. Click Finish

Now create a watch rule to compare harvested attribute value
8. Now go to Watches and Notifications tab –> Watches –> Click New
9. Enter Name, Watch Type: Collected Metrics
10. Add Watch Rule
(${ServerRuntime//[weblogic.management.runtime.ThreadPoolRuntimeMBean]com.bea:Name=ThreadPoolRuntime,ServerRuntime=osb_InstSvr_1a,Type=ThreadPoolRuntime//HealthState.State} != 0)
11. For creating above rule you can select Add Expressions –> ServerRuntime –> weblogic.management.runtime.ThreadPoolRuntimeMBean –> Select instance –> Attribute Expression: HealthState.State –> Operator: != –> Value: 0)
12. Select Alarm (if required. Used for not spamming email)
13. Select the notifications
14. Click Finish

Advertisements

Weblogic SSL WildCard Configuration

Weblogic 10.3.6 +
WildCard file server.pfx (format PKCS)Generate Java Keystore from WildCard

Generate Java Keystore from WildCard
1. Source environment
.setDomainEnv.sh

2. Use OpenSSL to check the pfx certificate’s content.
openssl pkcs12 -in server.pfx -out KEYSTORE.pem –nodes

3. Open KEYSTORE.pem file from step 2. You should find three certificates in it and the private key.
a. Private Key. To identify the private key, look for the following headings:
—–BEGIN RSA PRIVATE KEY—–
—–END RSA PRIVATE KEY—–

b. Root Certificate. To identify the Root Certificate, look for the following headings:
subject=/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
issuer=/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA

Subject and issuer must be the same. Save the content of it into a file called my_key_root.pem. Include all the content from BEGIN CERTIFICATE TO END CERTIFICATE.

c. Intermediate Certificate. To identify an Intermediate Certificate, look for the following heading:
subject=/C=US/O=GeoTrust, Inc./CN=GeoTrust SSL CA
issuer=/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA

Subject and issuer are different only on the CN. Save the content of it into a file called my_key_intermediate.pem. Include all the content from BEGIN CERTIFICATE TO END CERTIFICATE.

NOTE: This certificate is optional and there are some cases where it will not be present. If this is the case, go ahead and skip this step. In any other case, this needs to be added to the identity keystore jks file.

d. Server Certificate. To identify a Server Certificate, look for the following heading:
friendlyName: some.thing.com
subject=/serialNumber=sj6QjpTjKcpQGZ9QqWO-pFvsakS1t8MV/C=US/ST=Missouri/L=CHESTERFIELD/O=Oracle_Corp, Inc./OU=Oracle/CN=some.thing.com

A server certificate includes a heading called Friendly Name. Go ahead and save the content of it into a file called my_key_crt.pem. Include all the content from BEGIN CERTIFICATE TO END CERTIFICATE.

4. Create a Trust Keystore and import the Root certificate into it.
keytool -import -trustcacerts -file my_key_root.pem -alias my_key_root -keystore my_key_trust.jks -storepass <store_pass> -keypass <key_pass>

5. Generate an Identity Keystore and import the private key into it.
java utils.ImportPrivateKey -keystore my_key_identity.jks -storepass <store_pass> -storetype JKS -keypass <key_pass> -alias server_identity -certfile my_key_crt.pem -keyfile my_key_pk.pem -keyfilepass <pfx_password>

With these instructions, two jks files will be produced:
my_key_identity.jks & my_key_trust.jks

Configure WebLogic Server To Support Wildcard Certificates
1. Navigate to Home->Summary of Servers->ServerName
a. go to SSL tab
b. Click on Advanced
2. Lock and Edit
3. Set the Hostname Verification field to Custom Hostname Verifier.
4. Enter the name of the implementation of the weblogic.security.utils.SSLWLSWildcardHostnameVerifier interface in the Custom Hostname Verifier field.
5. Click Save.
6. To activate these changes, in the Change Center of the Administration Console, click Activate Changes.
7. Restart Server

Configure Weblogic SSL
Follow below steps to configure weblogic server to use above keystores:
1. Login to admin console
2. Navigate to servers>[server_name]>Configuration>Keystores
3. Select Custom Identity and Custom Trust and provide below details:
a. -Custom Identity Keystore: /path/to/my_key_identity.jks
b. -Custom Identity Keystore Type: jks
c. -Custom Identity Keystore Passphrase:<password>
d. -Confirm Custom Identity Keystore Passphrase:<password>
e. -Custom Trust Keystore: /path/to/my_key_trust.jks
f. -Custom Trust Keystore Type: jks
g. -Custom Trust Keystore Passphrase:<password>
h. -Confirm Custom Trust Keystore Passphrase:<password>
4. Then click on SSL tab next to Keystores and provide values for below parameters:
a. -Private Key Alias: server_identity
b. -Private Key Passphrase: <password>
c. -Confirm Private Key Passphrase: <password>
5. Then enable SSL port for that particular weblogic server by navigating servers>[server_name]>Configuration>General
6. Enable SSL port for that particular weblogic server by navigating servers>[server_name]>Configuration>General
7. Save and activate changes.

Note: If SSL port is enabled for first time you need to restart server.

Weblogic SSL Configuration

Steps to configure weblogic SSL

1. Create identity store

keytool -genkey -alias weblogicServer -keyalg RSA -keysize 1024 -keystore weblogic_identity.jks

Where Weblogic_identity.jks is name of identity store which will be created and weblogicServer is the private key alias name.

2. Create CSR

keytool -certreq -alias weblogicServer -file device.csr -keystore weblogic_identity.jks

Where device.csr is the certificate signing request created.

3. Submit Signing request

Now submit this CSR to any certification authorities to get public certificate and root/intermediate certificates.

If using internal CA using openssl execute below command to generate signed certificate from CSR.

openssl x509 -req -in device.csr -CA ca_root_cert.pem -CAkey ca_privkey.pem -CAcreateserial -out device.pem –days 3650

Where ca_root_cert.pem is CA root certificate and ca_privkey.pem is CA private key.

4. Create trust store and import root certificate to trust store

keytool -import -trustcacerts -alias myRoot -file /path/to/ca_root_cert.pem -keystore weblogic_trust.jks

Where ca_root_cert.pem is CA root certificate and myRoot is the root alias name for trust store.

5. Import root certificate to identity store

keytool -import -trustcacerts -alias entRoot -file /path/to/ca_root_cert.pem -keystore weblogic_identity.jks

Where ca_root_cert.pem is CA root certificate and entRoot is the root alias name for identity store.

6. Import signed certificate to identity store

keytool -import -trustcacerts -alias weblogicServer -file /path/to/device.pem -keystore weblogic_identity.jks

Where device.pem is the signed certificate and weblogicServer is private key alias name.

Follow below steps to configure weblogic server to use above keystores:

1. Login to admin console

2. Navigate to servers>[server_name]>Configuration>Keystores

3. Select Custom Identity and Custom Trust and provide below details:

a. -Custom Identity Keystore: /path/to/weblogic_identity.jks

b. -Custom Identity Keystore Type: jks

c. -Custom Identity Keystore Passphrase:<password>

d. -Confirm Custom Identity Keystore Passphrase:<password>

e. -Custom Trust Keystore: /path/to/weblogic_trust.jks

f. -Custom Trust Keystore Type: jks

g. -Custom Trust Keystore Passphrase:<password>

h. -Confirm Custom Trust Keystore Passphrase:<password>

4. Then click on SSL tab next to Keystores and provide values for below parameters:

a. -Private Key Alias: weblogicServer

b. -Private Key Passphrase: <password>

c. -Confirm Private Key Passphrase: <password>

5. Then enable SSL port for that particular weblogic server by navigating servers>[server_name]>Configuration>General

6. Enable SSL port for that particular weblogic server by navigating servers>[server_name]>Configuration>General

7. Save and activate changes.

Note: If SSL port is enabled for first time you need to restart server.

Weblogic Security Realm WLST import and export

Export
$DOMAIN_HOME/bin/setDomainEnv.sh
java weblogic.WLST
connect(‘weblogic’,’weblogic’, ‘t3://adminhostname:7001’)
domainRuntime()
cd(‘/DomainServices/DomainRuntimeService/DomainConfiguration/IDMDomain/SecurityConfiguration/IDMDomain/DefaultRealm/myrealm/AuthenticationProviders/DefaultAuthenticator’)
cmo.exportData(‘DefaultAtn’,’/u01/export/export.ldif’, Properties())

 

Import
$DOMAIN_HOME/bin/setDomainEnv.sh
java weblogic.WLST
connect(‘weblogic’,’weblogic’, ‘t3://adminhostname:7001’)
domainRuntime()
cd(‘/DomainServices/DomainRuntimeService/DomainConfiguration/IDMDomain/SecurityConfiguration/IDMDomain/DefaultRealm/myrealm/AuthenticationProviders/DefaultAuthenticator’)
cmo.importData(‘DefaultAtn’,’/u01/export/import.ldif’, Properties())

Weblogic Exit on out of memory

If weblogic server is crashing frequently with out of memory errors below work around can be helpful till root cause of issue is found.

  1. Configure node manager with CrashRecoveryEnabled=true
  2. Configure weblogic for exit on out of memory as given below

Navigate to $IDMDomain/bin
Add -XX:+ExitOnOutOfMemoryError parameter
a) Edit startManagedWebLogic.sh
Add below lines before export JAVA_OPTION
JAVA_OPTIONS=”-XX:+ExitOnOutOfMemoryError ${JAVA_OPTIONS}”
OR
b) Open weblogic console, navigate to Home >Summary of Servers >managed_server>Configuration>Server start
Add below line in argument
-XX:+ExitOnOutOfMemoryError
Restart managed server

So when ever there is out of memory error, on first occurrence weblogic server will exit and node manager will restart the server. However, you must investigate for root cause of out of memory error for permanent fix.