Weblogic State and HealthState Monitoring with Email Notification

Configure Weblogic Mail Session (Optional. Only if you want to send email alert)
1. Login to weblogic console
2. Go to Mail Sessions
3. Click New
4. Enter details Name,JNDI Name, JavaMail Properties.
JavaMail Properties:

Configure Email Notifications in Weblogic Diagnostic
Go to notifications –> click new –> Select SMTP (E-Mail) –> Click Next –> Enter Notification Name & Check Enable Notification –> Got to SMTP Properties –> Select Mail session ->Enter Email Recipients

Server State Monitoring
1. Login to weblogic console
2. Go to Diagnostic Modules
3. Click on “
Module-FMWDFW” module
4. Go to Configuration –> Watches and Notifications –> Watches
5. Click New
6. Enter Watch Name, Watch Type: Collected Metrics
7. Click Next
8. Click on Add Expressions
9. Select ServerRunTIme –> Select weblogic.management.runtime.ServerLifeCycleRuntimeMBean –> Click Next –> Click Next
6. Select Message Attribute “State”
7. Select Operator “!=”
8. Enter value RUNNING
9. Below watch rule will be generated
(${ServerRuntime//[weblogic.management.runtime.ServerLifeCycleRuntimeMBean]//State} != ‘RUNNING’)
10. ServerRuntime dies along with the managed Server. Therefore we need to DomainRuntime.
Edit rule manually replace ServerRuntime with DomainRuntime
(${DomainRuntime//[weblogic.management.runtime.ServerLifeCycleRuntimeMBean]//State} != ‘RUNNING’)
11. Click Next
12. Select Alarm (if required. Used for not spamming email)
13. Select the notifications
14. Click Finish

Health State Monitoring
Stuck thread is very common issue with weblogic servers. Below is a good article on dealing with stuck threads.
We had a requirement to capture server healthstate for stuck thread. By default healthstate is not collected from diagnostic module. A harvester must be created to gather healthstate data.
1. Go to Weblogic Console –> Diagnostic Modules
2. Configuration tab –> Collected Metrics tab
3. Click new
4. Select ServerRuntime –> Select weblogic.management.runtime.ThreadPoolRuntimeMBean
5. Add Attribute Expression as “HealthState.State”  (without quotes)
6. Select the Server Instance
7. Click Finish

Now create a watch rule to compare harvested attribute value
8. Now go to Watches and Notifications tab –> Watches –> Click New
9. Enter Name, Watch Type: Collected Metrics
10. Add Watch Rule
(${ServerRuntime//[weblogic.management.runtime.ThreadPoolRuntimeMBean]com.bea:Name=ThreadPoolRuntime,ServerRuntime=osb_InstSvr_1a,Type=ThreadPoolRuntime//HealthState.State} != 0)
11. For creating above rule you can select Add Expressions –> ServerRuntime –> weblogic.management.runtime.ThreadPoolRuntimeMBean –> Select instance –> Attribute Expression: HealthState.State –> Operator: != –> Value: 0)
12. Select Alarm (if required. Used for not spamming email)
13. Select the notifications
14. Click Finish