Upgrade JDK Used by Oracle WebLogic Server 11g

There may be multiple ways for achieving this, in this post I will be upgrading by installing new JDK home directory and replacing all the JAVA_HOME references from weblogic scripts to new JDK directory.

In this post I will upgrade JDK 1.7 Update 40 used by Weblogic server to JDK 1.7 Update 171

Check existing JDK version, for me its installed in /usr/java/jdk1.7.0_40

[oracle@oel1]$ /usr/java/jdk1.7.0_40/bin/java -fullversion

java full version “1.7.0_40-b43”

Download and unzip JDK 1.7 Update 171 : Patch 27334355

[oracle@oel1]$  mkdir /u01/java
[oracle@oel1]$  unzip p27334355_170171_Linux-x86-64.zip
[oracle@oel1]$ tar -xvf jdk-7u171-linux-x64.tar.gz
[oracle@oel1]$ /u01/java/jdk1.7.0_171/bin/java –fullversion

java full version “1.7.0_171-b31”

Stop all FMW processes from the current FMW home

Find all the files required change of JAVA_HOME (where /usr/java/jdk1.7.0_40 is my existing JAVA_HOME)

[oracle@oel1]$ cd $MW_HOME

[oracle@oel1]$ find . -type f -name “*.sh” -exec grep -il /usr/java/jdk1.7.0_40 {} \;
./utils/quickstart/quickstart.sh
./utils/bsu/bsu.sh
./utils/uninstall/uninstall.sh
./user_projects/domains/IDMDomain/bin/setDomainEnv.sh
./wlserver_10.3/common/bin/commEnv.sh

[oracle@oel1]$ find . -type f -name “*.properties” -exec grep -il /usr/java/jdk1.7.0_40 {} \;
./wlserver_10.3/.product.properties
./wlserver_10.3/common/nodemanager/nodemanager.properties
./coherence_3.7/.product.properties

Back up all above files and edit/replace the JAVA_HOME from old to new directory i.e from /usr/java/jdk1.7.0_40 to /u01/java/jdk1.7.0_171

In the case that you are using WLS 10.3.6 and the targetJDK is using a certified version of JAVA 7 (version 1.7.0_x) , you will need to copy manually some jar files:

[oracle@oel1]$ cd /u01/java/jdk1.7.0_171/jre/lib
[oracle@oel1]$ mkdir endorsed
[oracle@oel1]$ cd endorsed
[oracle@oel1]$ cp $MW_HOME/modules/javax.annotation_1.0.0.0_1-0.jar .
[oracle@oel1]$ cp $MW_HOME/modules/javax.xml.bind_2.1.1.jar .
[oracle@oel1]$ cp $MW_HOME/modules/javax.xml.ws_2.1.1.jar .

Start the processes

Advertisements

Avoid password prompt when using startComponent.sh–12c

With 12c release, identity management components like OID, OUD, OHS when installed and configured in collocated mode are started using startComponent.sh and stoped using stopComponent.sh scripts located in $DOMAIN_HOME/bin

However, when starting or stopping script prompts for node manager password.

image

Above prompt can be avoided by adding storeUserConfig option. This stores the password.

[oracle@oel1 bin]$ ./startComponent.sh oid1 storeUserConfig

image

Once the password is stored, you can execute start or stop without password prompt.

[oracle@oel1 bin]$ ./stopComponent.sh oid1
[oracle@oel1 bin]$ ./startComponent.sh oid1

image

Weblogic 12c Server State and Health State Monitoring with Email Notification

Configure Weblogic Mail Session (Optional. Only if you want to send email alert)
1. Login to weblogic console
2. Go to Mail Sessions
3. Click New
4. Enter details Name,JNDI Name, JavaMail Properties.

mail.port=25
mail.user=weblogic
mail.host=xx.xx.xx.xx
mail.transport.protocol=smtp
mail.from=weblogic

Configure Email Notifications in Weblogic Diagnostic
Go to Weblogic Console –> Diagnostic Modules –> Select Module “Module-FMWDFW” –> Configuration –> Policies and Action –> Actions
chrome_2018-02-19_12-09-33
Click new –> Select SMTP (E-Mail) –> Click Next –> Enter Notification Name & Check Enable Notification –> Got to SMTP Properties –> Select Mail session ->Enter Email Recipients

Server State Monitoring
1.Go to Weblogic Console –> Diagnostic Modules –> Select Module “Module-FMWDFW” –> Configuration –> Policies and Action –> Policies
2018-02-19_12-17-13
click New. Enter Details as below

Name: ServerStateWatch
Policy Type: Collected Metrics
Enable Policy: checked

Add rule expression as below, where slc.state is the state of server and slc.name is name of server

wls.domainRuntime.domain.serverLifeCycleRuntimes.stream().anyMatch(slc ->
((slc.state != ‘RUNNING’) and ( (slc.name == ‘AdminServer’) or (slc.name == ‘ManagedServer1’) ) ))

Then choose schedule and alarms in next screen as per requirement. In actions select email notification which was created in earlier step.
chrome_2018-02-19_12-23-11

Health State Monitoring
Go to Weblogic Console –> Diagnostic Modules –> Select Module “Module-FMWDFW” –> Configuration –> Policies and Action –> Policies
2018-02-19_12-17-13

click New. Enter Details as below

Name: ThreadPoolStateWatch
Policy Type: Collected Metrics
Enable Policy: checked

Add rule expression as below, where state is the health state of server and serverName is name of server

wls.domainRuntime.lookupServerRuntime(serverName=’AdminServer’).threadPoolRuntime.healthState.state != 0 or wls.domainRuntime.lookupServerRuntime(serverName=’ManagedServer1’).threadPoolRuntime.healthState.state != 0

Then choose schedule and alarms in next screen as per requirement. In actions select email notification which was created in earlier step.
chrome_2018-02-19_12-23-11

Weblogic State and HealthState Monitoring with Email Notification

Configure Weblogic Mail Session (Optional. Only if you want to send email alert)
1. Login to weblogic console
2. Go to Mail Sessions
3. Click New
4. Enter details Name,JNDI Name, JavaMail Properties.
JavaMail Properties:
mail.port=25
mail.user=weblogic
mail.host=xx.xx.xx.xx
mail.transport.protocol=smtp
mail.from=weblogic

Configure Email Notifications in Weblogic Diagnostic
Go to notifications –> click new –> Select SMTP (E-Mail) –> Click Next –> Enter Notification Name & Check Enable Notification –> Got to SMTP Properties –> Select Mail session ->Enter Email Recipients

Server State Monitoring
1. Login to weblogic console
2. Go to Diagnostic Modules
3. Click on “
Module-FMWDFW” module
4. Go to Configuration –> Watches and Notifications –> Watches
5. Click New
6. Enter Watch Name, Watch Type: Collected Metrics
7. Click Next
8. Click on Add Expressions
9. Select ServerRunTIme –> Select weblogic.management.runtime.ServerLifeCycleRuntimeMBean –> Click Next –> Click Next
6. Select Message Attribute “State”
7. Select Operator “!=”
8. Enter value RUNNING
9. Below watch rule will be generated
(${ServerRuntime//[weblogic.management.runtime.ServerLifeCycleRuntimeMBean]//State} != ‘RUNNING’)
10. ServerRuntime dies along with the managed Server. Therefore we need to DomainRuntime.
Edit rule manually replace ServerRuntime with DomainRuntime
(${DomainRuntime//[weblogic.management.runtime.ServerLifeCycleRuntimeMBean]//State} != ‘RUNNING’)
11. Click Next
12. Select Alarm (if required. Used for not spamming email)
13. Select the notifications
14. Click Finish

Health State Monitoring
Stuck thread is very common issue with weblogic servers. Below is a good article on dealing with stuck threads.
http://oraclemiddlewareblog.com/2014/06/10/dealing-stuck-threads-weblogic/
We had a requirement to capture server healthstate for stuck thread. By default healthstate is not collected from diagnostic module. A harvester must be created to gather healthstate data.
1. Go to Weblogic Console –> Diagnostic Modules
2. Configuration tab –> Collected Metrics tab
3. Click new
4. Select ServerRuntime –> Select weblogic.management.runtime.ThreadPoolRuntimeMBean
5. Add Attribute Expression as “HealthState.State”  (without quotes)
6. Select the Server Instance
7. Click Finish

Now create a watch rule to compare harvested attribute value
8. Now go to Watches and Notifications tab –> Watches –> Click New
9. Enter Name, Watch Type: Collected Metrics
10. Add Watch Rule
(${ServerRuntime//[weblogic.management.runtime.ThreadPoolRuntimeMBean]com.bea:Name=ThreadPoolRuntime,ServerRuntime=osb_InstSvr_1a,Type=ThreadPoolRuntime//HealthState.State} != 0)
11. For creating above rule you can select Add Expressions –> ServerRuntime –> weblogic.management.runtime.ThreadPoolRuntimeMBean –> Select instance –> Attribute Expression: HealthState.State –> Operator: != –> Value: 0)
12. Select Alarm (if required. Used for not spamming email)
13. Select the notifications
14. Click Finish

Weblogic SSL WildCard Configuration

Weblogic 10.3.6 +
WildCard file server.pfx (format PKCS)Generate Java Keystore from WildCard

Generate Java Keystore from WildCard
1. Source environment
.setDomainEnv.sh

2. Use OpenSSL to check the pfx certificate’s content.
openssl pkcs12 -in server.pfx -out KEYSTORE.pem –nodes

3. Open KEYSTORE.pem file from step 2. You should find three certificates in it and the private key.
a. Private Key. To identify the private key, look for the following headings:
—–BEGIN RSA PRIVATE KEY—–
—–END RSA PRIVATE KEY—–

b. Root Certificate. To identify the Root Certificate, look for the following headings:
subject=/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
issuer=/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA

Subject and issuer must be the same. Save the content of it into a file called my_key_root.pem. Include all the content from BEGIN CERTIFICATE TO END CERTIFICATE.

c. Intermediate Certificate. To identify an Intermediate Certificate, look for the following heading:
subject=/C=US/O=GeoTrust, Inc./CN=GeoTrust SSL CA
issuer=/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA

Subject and issuer are different only on the CN. Save the content of it into a file called my_key_intermediate.pem. Include all the content from BEGIN CERTIFICATE TO END CERTIFICATE.

NOTE: This certificate is optional and there are some cases where it will not be present. If this is the case, go ahead and skip this step. In any other case, this needs to be added to the identity keystore jks file.

d. Server Certificate. To identify a Server Certificate, look for the following heading:
friendlyName: some.thing.com
subject=/serialNumber=sj6QjpTjKcpQGZ9QqWO-pFvsakS1t8MV/C=US/ST=Missouri/L=CHESTERFIELD/O=Oracle_Corp, Inc./OU=Oracle/CN=some.thing.com

A server certificate includes a heading called Friendly Name. Go ahead and save the content of it into a file called my_key_crt.pem. Include all the content from BEGIN CERTIFICATE TO END CERTIFICATE.

4. Create a Trust Keystore and import the Root certificate into it.
keytool -import -trustcacerts -file my_key_root.pem -alias my_key_root -keystore my_key_trust.jks -storepass <store_pass> -keypass <key_pass>

5. Generate an Identity Keystore and import the private key into it.
java utils.ImportPrivateKey -keystore my_key_identity.jks -storepass <store_pass> -storetype JKS -keypass <key_pass> -alias server_identity -certfile my_key_crt.pem -keyfile my_key_pk.pem -keyfilepass <pfx_password>

With these instructions, two jks files will be produced:
my_key_identity.jks & my_key_trust.jks

Configure WebLogic Server To Support Wildcard Certificates
1. Navigate to Home->Summary of Servers->ServerName
a. go to SSL tab
b. Click on Advanced
2. Lock and Edit
3. Set the Hostname Verification field to Custom Hostname Verifier.
4. Enter the name of the implementation of the weblogic.security.utils.SSLWLSWildcardHostnameVerifier interface in the Custom Hostname Verifier field.
5. Click Save.
6. To activate these changes, in the Change Center of the Administration Console, click Activate Changes.
7. Restart Server

Configure Weblogic SSL
Follow below steps to configure weblogic server to use above keystores:
1. Login to admin console
2. Navigate to servers>[server_name]>Configuration>Keystores
3. Select Custom Identity and Custom Trust and provide below details:
a. -Custom Identity Keystore: /path/to/my_key_identity.jks
b. -Custom Identity Keystore Type: jks
c. -Custom Identity Keystore Passphrase:<password>
d. -Confirm Custom Identity Keystore Passphrase:<password>
e. -Custom Trust Keystore: /path/to/my_key_trust.jks
f. -Custom Trust Keystore Type: jks
g. -Custom Trust Keystore Passphrase:<password>
h. -Confirm Custom Trust Keystore Passphrase:<password>
4. Then click on SSL tab next to Keystores and provide values for below parameters:
a. -Private Key Alias: server_identity
b. -Private Key Passphrase: <password>
c. -Confirm Private Key Passphrase: <password>
5. Then enable SSL port for that particular weblogic server by navigating servers>[server_name]>Configuration>General
6. Enable SSL port for that particular weblogic server by navigating servers>[server_name]>Configuration>General
7. Save and activate changes.

Note: If SSL port is enabled for first time you need to restart server.

Weblogic SSL Configuration

Steps to configure weblogic SSL

1. Create identity store

keytool -genkey -alias weblogicServer -keyalg RSA -keysize 1024 -keystore weblogic_identity.jks

Where Weblogic_identity.jks is name of identity store which will be created and weblogicServer is the private key alias name.

2. Create CSR

keytool -certreq -alias weblogicServer -file device.csr -keystore weblogic_identity.jks

Where device.csr is the certificate signing request created.

3. Submit Signing request

Now submit this CSR to any certification authorities to get public certificate and root/intermediate certificates.

If using internal CA using openssl execute below command to generate signed certificate from CSR.

openssl x509 -req -in device.csr -CA ca_root_cert.pem -CAkey ca_privkey.pem -CAcreateserial -out device.pem –days 3650

Where ca_root_cert.pem is CA root certificate and ca_privkey.pem is CA private key.

4. Create trust store and import root certificate to trust store

keytool -import -trustcacerts -alias myRoot -file /path/to/ca_root_cert.pem -keystore weblogic_trust.jks

Where ca_root_cert.pem is CA root certificate and myRoot is the root alias name for trust store.

5. Import root certificate to identity store

keytool -import -trustcacerts -alias entRoot -file /path/to/ca_root_cert.pem -keystore weblogic_identity.jks

Where ca_root_cert.pem is CA root certificate and entRoot is the root alias name for identity store.

6. Import signed certificate to identity store

keytool -import -trustcacerts -alias weblogicServer -file /path/to/device.pem -keystore weblogic_identity.jks

Where device.pem is the signed certificate and weblogicServer is private key alias name.

Follow below steps to configure weblogic server to use above keystores:

1. Login to admin console

2. Navigate to servers>[server_name]>Configuration>Keystores

3. Select Custom Identity and Custom Trust and provide below details:

a. -Custom Identity Keystore: /path/to/weblogic_identity.jks

b. -Custom Identity Keystore Type: jks

c. -Custom Identity Keystore Passphrase:<password>

d. -Confirm Custom Identity Keystore Passphrase:<password>

e. -Custom Trust Keystore: /path/to/weblogic_trust.jks

f. -Custom Trust Keystore Type: jks

g. -Custom Trust Keystore Passphrase:<password>

h. -Confirm Custom Trust Keystore Passphrase:<password>

4. Then click on SSL tab next to Keystores and provide values for below parameters:

a. -Private Key Alias: weblogicServer

b. -Private Key Passphrase: <password>

c. -Confirm Private Key Passphrase: <password>

5. Then enable SSL port for that particular weblogic server by navigating servers>[server_name]>Configuration>General

6. Enable SSL port for that particular weblogic server by navigating servers>[server_name]>Configuration>General

7. Save and activate changes.

Note: If SSL port is enabled for first time you need to restart server.

Weblogic Security Realm WLST import and export

Export
$DOMAIN_HOME/bin/setDomainEnv.sh
java weblogic.WLST
connect(‘weblogic’,’weblogic’, ‘t3://adminhostname:7001’)
domainRuntime()
cd(‘/DomainServices/DomainRuntimeService/DomainConfiguration/IDMDomain/SecurityConfiguration/IDMDomain/DefaultRealm/myrealm/AuthenticationProviders/DefaultAuthenticator’)
cmo.exportData(‘DefaultAtn’,’/u01/export/export.ldif’, Properties())

 

Import
$DOMAIN_HOME/bin/setDomainEnv.sh
java weblogic.WLST
connect(‘weblogic’,’weblogic’, ‘t3://adminhostname:7001’)
domainRuntime()
cd(‘/DomainServices/DomainRuntimeService/DomainConfiguration/IDMDomain/SecurityConfiguration/IDMDomain/DefaultRealm/myrealm/AuthenticationProviders/DefaultAuthenticator’)
cmo.importData(‘DefaultAtn’,’/u01/export/import.ldif’, Properties())