Oracle Application Server: How to Bounce AS from One Location

In this post and some upcoming posts, I’m going to write more about Oracle application servers, a subject we have addressed too little on the Pythian blog.

In this post, I am addressing how to bounce a whole application server, including all tiers and databases from one location. The reason being, I have a request from a client to have the application server be bounced automatically during the weekend to release swaps and to address memory leaks. The application server on this client includes one Mid tier, one Infra tier, and one database (a metarepository database) in three different Oracle homes on two boxes.

As you know, for application server, tiers should be stopped and started up in a specific order. On startup, the sequence is like: database –> listener –> Infra tier –> Mid tier.

For shutdown, the sequence is vice versa. It is not safe to shutdown the database or Infra tier before making sure that the Mid tier is totally done.

So, in order to address client’s request, the basic plan was to have a script to shutdown Mid tier, then handshake with a tscript on the Infra tier to let it know that it is safe to shutdown the Infra tier. We would use the same approach for the Infra shutdown script and database shutdown script.

Is there a simpler way?

opmnctl, the main tool for startup/shutdown of application server components, is able to bounce the whole farm. However, first time you try to run opmnctl status @farm, you may just see the status of AS component only for single box rather than for the whole farm. Why?

For security reasons, in AS 10g, opmn requests (start, restart, and stop) to the farm work only if SSL is enabled in the opmn.xml file and a wallet file is configured. If neither SSL nor a wallet file are configured, OPMN will reject any remote process control request with HTTP code 403. Interestingly, OPMN in AS 10g is not SSL-enabled by default.

Here are the steps to enable SSL on OPMN and make it ready for a remote bounce.

1. Edit $OH/opmn/conf/opmn.xml on all tiers and add the following lines to the <notification-server> section.

   <ssl enabled="true" wallet-file="$ORACLE_HOME/opmn/conf/ssl.wlt/default"/>

2. Copy the existing wallet on each box to defined location for opmn. By default the wallet file is OH/Apache/Apache/conf/ssl.wlt/default. Copy this file to OH/opmn/conf/ssl.wlt/default.
3. Run the following command to pick up the changes in each Oracle Home.

   $ORACLE_HOME/dcm/bin/dcmctl updateconfig -ct opmn

4. Bounce the tiers.
5. Run the following statement from one location, and you should see status of all components in all tiers.

   opmnctl status  @farm
   opmnctl stopall @farm
   opmnctl startall @farm

   In my case:

   oracle(/home/oracle):opmnctl status @farm
   Processes in Instance: infraAS10g.##################
   -------------------+--------------------+---------+---------
   ias-component      | process-type       |     pid | status
   -------------------+--------------------+---------+---------
   
   HTTP_Server        | HTTP_Server        |  471476 | Alive
   
   LogLoader          | logloaderd         |  462866 | Alive
   
   dcm-daemon         | dcm-daemon         |  430186 | Alive
   
   OC4J               | OC4J_SECURITY      | 1089626 | Alive
   
   OC4J               | oca                | 1069158 | Alive
   
   DSA                | DSA                |     N/A | Down
   
   OID                | OID                | 1200560 | Alive
   
   Processes in Instance: Mid10gAS_###########################
   
   -------------------+--------------------+---------+---------
   
   ias-component      | process-type       |     pid | status
   
   -------------------+--------------------+---------+---------
   
   HTTP_Server        | HTTP_Server        |  978974 | Alive
   
   LogLoader          | logloaderd         |  471280 | Alive
   
   dcm-daemon         | dcm-daemon         |  962758 | Alive
   
   OC4J               | home               |  585772 | Alive
   
   OC4J               | OC4J_BI_Forms      |  946312 | Alive
   
   OC4J               | OC4J_Portal        |  790612 | Alive
   
   WebCache           | WebCache           |  704698 | Alive
   
   WebCache           | WebCacheAdmin      |  524378 | Alive
   
   Discoverer         | ServicesStatus     |  774258 | Alive
   
   Discoverer         | PreferenceServer   |  811190 | Alive
   
   wireless           | performance_server |     N/A | Down
   
   wireless           | messaging_server   |     N/A | Down
   
   wireless           | OC4J_Wireless      |     N/A | Down
   
   DSA                | DSA                |     N/A | Down

   When everything is ready, it is just a matter of having a shell script to be run on the InfraStructure tier and bouncing the whole AS environment.

   Here is the script I implemented for this client. In any steps if startup/shutdown fails, it will page the person on-call for manual intervention.

   ### Script to recycle IAS and DB ###
   ### 3 Envs should be set : INSTANCE_INFRA, INSTANCE_MIDAS,META_LISTENERAS_env=$1
   DB_env=$2
   ##################************************
   ##################************************
   ##################************************
   ##################************************
   ##################************************
   . ~/$AS_env
   export ENVIRONMENT=`hostname`
   export cnt=0
   ## Check number of AS component "
   echo ******************##################************************
   echo `date` "=> Checking number of AS components "
   echo ******************##################************************
   Mid_comps=`opmnctl @instance:$INSTANCE_MIDAS  status -noheaders | grep "|" |wc -l`
   Inf_comps=`opmnctl @instance:$INSTANCE_INFRA status -noheaders | grep "|" | wc -l`
   ## Starting AS/Database Recycle  ##
   echo ******************##################************************
   echo `date` "=>Shutting down the dcm-daemon and LogLoader first..."
   echo ******************##################************************
   opmnctl @farm stopproc ias-component=dcm-daemon
   opmnctl @farm stopproc ias-component=LogLoader
   ## Shutdown Mid AS on Foo ##
   echo ******************##################************************
   echo `date` "=>Shutting down mid tier AS ...."
   echo ******************##################************************
   opmnctl @instance:$INSTANCE_MIDAS stopproc
   ## Wait until the AS processes actually end up in a 'DOWN' state
   while opmnctl @instance:$INSTANCE_MIDAS status | grep -i "stopped"  > /dev/null
   do
   sleep 5
   ((cnt=cnt+1))
   if [ $cnt = 1000 ]; then
      break;
   fi
   echo `date` "=>Waiting for the  Mid Tier to go down.."
   done
   echo ******************##################************************
   echo `date` "=>Status of mid tier components..."
   echo ******************##################************************
   opmnctl @instance:$INSTANCE_MIDAS status
   Mid_Downcomps=`opmnctl @instance:$INSTANCE_MIDAS status -noheaders |grep "Down" |wc -l`
   if [ "$Mid_Downcomps" = "$Mid_comps" ]; then
      echo `date` "Mid-tier is down .... Safe to go for Infra"
   else
   /usr/sbin/sendmail oncall@example.com << EOF
   From: $ENVIRONMENT@example.com
   To: oncall@example.com
   Subject: Automatic AS bounce failed, It needs manual intervention.
   Message:
   Manual intevention is required to fix AS bounce
   EOF
   exit 4
   fi
   ## Shutdown Infra on Bar ##
   cnt=1
   echo ******************##################************************
   echo `date` "=>Shutting down infra ...."
   echo ******************##################************************
   opmnctl @instance:$INSTANCE_INFRA stopproc
   ## Wait until the AS processes actually end up in a 'DOWN' state
   while opmnctl @instance:$INSTANCE_INFRA status | grep -i "stopped" > /dev/null
   do
   sleep 5
   ((cnt=cnt+1))
   if [ $cnt = 1000 ]; then
      break;
   fi
   echo `date` "=>Waiting for the  Infra Tier to go down.."
   done
   echo ******************##################************************
   echo `date` "=>Status of infra components..."
   echo ******************##################************************
   opmnctl @instance:$INSTANCE_INFRA status
   Inf_Downcomps=`opmnctl @instance:$INSTANCE_INFRA status -noheaders |grep "Down" |wc -l`
   if [ "$Inf_Downcomps" = "$Inf_comps" ]; then
      echo `date` "Infra is down .... Safe to go for Database"
   else
   /usr/sbin/sendmail oncall@example.com << EOF
   From: $ENVIRONMENT.up@example.com
   To: oncall@example.com
   Subject: Automatic AS bounce failed, It needs manual intervention.
   Message:
   Manual intevention is required to fix AS bounce
   EOF
   exit 4
   fi
   ##################************************
   ##################************************
   ##################************************
   ##################************************
   . ~/$DB_env
   echo ******************##################************************
   echo `date` "=>Shutting down the listener $META_LISTENER ..."
   echo ******************##################************************
   lsnrctl stop $META_LISTENER
   echo ******************##################************************
   echo `date` "=>Bouncing the Database $ORACLE_SID ..."
   echo ******************##################************************
   sqlplus -s /nolog << EOF
   connect / as sysdba
   alter system flush shared_pool;
   shutdown immediate;
   startup;
   EOF
   echo ******************##################************************
   echo `date` "=>Startup up the listener $META_LISTENER ..."
   echo ******************##################************************
   lsnrctl start $META_LISTENER
   ##################************************
   ##################************************
   ##################************************
   ##################***********************
   . ~/$AS_env
   ## Startup Infra on Bar ##
   echo ******************##################************************
   echo `date` "=>Starting up infra ...."
   echo ******************##################************************
   opmnctl @instance:$INSTANCE_INFRA startproc
   while opmnctl @instance:$INSTANCE_INFRA status | grep -i "init"  > /dev/null
   do
   sleep 5
   echo `date` "=>Waiting for the Infra to come up completely..."
   done
   ## Startup Mid AS on Foo ##
   echo ******************##################************************
   echo `date` "=>Starting up mid tier AS on Foo ...."
   echo ******************##################************************
   opmnctl @instance:$INSTANCE_MIDAS startproc
   while  opmnctl @instance:$INSTANCE_MIDAS status  | grep -i "init"  > /dev/null
   do
   sleep 5
   echo `date` "=>Waiting for the  Mid Tier to come up completely.."
   done
   echo ******************##################************************
   echo `date` "=>Bringing up dcm-daemon and LogLoader for infra and mid tier"
   echo ******************##################************************
   opmnctl @farm startproc ias-component=dcm-daemon
   opmnctl @farm startproc ias-component=LogLoader
   echo ******************##################************************
   echo `date` "=>Status of mid tier components..."
   echo ******************##################************************
   opmnctl @instance:$INSTANCE_INFRA status
   echo ******************##################************************
   echo `date` "=>Status of mid tier components..."
   echo ******************##################************************
   opmnctl @instance:$INSTANCE_MIDAS status
   ##################************************
   ##################************************
   ##################************************
   ##################************************