Wednesday, August 8, 2012

Applying Grid Infrastructure Patch Set Update aka GI PSU to 11.2.0.2 Grid home and 11.2.0.2 RDBMS home


Applying Grid Infrastructure Patch Set Update aka GI PSU to 11.2.0.2 Grid home and 11.2.0.2 RDBMS home

Environment:
2 Node RAC running SUSE Linux Enterprise Server 10 (x86_64)
Oracle Grid Infrastructure 11.2.0.2
Oracle RAC rdbms 11.2.0.2


PSU - What/ When ??

Patch Set Updates (PSUs) are proactive cumulative patches containing recommended bug fixes that are released every quarter on the Tuesday closest to the 17th of January, April, July, and October.

Identifying the right PSU for your environment

-- First, find my version of Grid Infrastructure software.

> crsctl query crs softwareversion
Oracle Clusterware version on node [rac01] is [11.2.0.2.0]


-- Next, find the current recommended PSU for your version of software ?

Use support note, Oracle Recommended Patches -- Oracle Database [ID 756671.1]

As of last week of July 2012, the recommended patch for 11.2.0.2 is Patch 14192201

Grid Infrastructure
Document
Description
Rolling RAC
Patch Download
Note:14192201.8               
11.2.0.2.7 Grid Infrastructure Patch Set Update (GI PSU)
Yes
Patch:14192201

We are going one version behind in our environment and for environment consistency, I will be applying 11.2.0.2.6 in our environment which will be Patch 13696242 - 11.2.0.2.6 GI Patch Set Update (Includes Database PSU 11.2.0.2.6)

Obtain the Patch

Download the patch from support.oracle.com for the OS version that you are running. Place the downloaded files in a shared location that is accessible from both the nodes in the cluster configuration. This will make it easy as we will use the same source software for applying in all nodes of the RAC.


Patch Installation Prerequisites

To take a backup of the binaries, do the following tar of the binaries home.  Oracle also automatically takes a backup during opatch apply and stores it in “.patch_storage” under the $ORACLE_HOME where you apply the patch. This will be used during rollback, if the apply fails

As root

rac01:~ # export CRS_ORA_HOME=/export/11.2.0.2/grid
rac01:~ # export ORACLE_HOME=/export/oracle/product/11.2.0.2

cd $CRS_ORA_HOME
tar -cvpf /export/install/nels/qabkp/rac01/gi_bkp.tar ./

where
c - compress
v - verbose
p - preserve permissions
f - tar file
/export/install/nels/qabkp/rac01/gi_bkp.tar is the backup tar file

Similarly do for Oracle Home

cd $ORACLE_HOME
tar -cvpf /export/install/nels/qabkp/rac01/oh_bkp.tar ./


Do the above for each node in the RAC cluster

Note: Always use the readme.html from the patch source. It contains complete details required for applying the patch successfully.

Latest OPatch
               
OPatch is the utility used to apply the patch. Download the latest OPatch utility from My Oracle Support patch 6880880.

As software home owner, unzip the downloaded OPatch to the $CRS_ORA_HOME and $ORACLE_HOME

> unzip p6880880_112000_Linux-x86-64.zip -d $CRS_ORA_HOME

rac01:/export/install/nels/opatch # unzip p6880880_112000_Linux-x86-64.zip -d /export/11.2.0.2/grid
Archive:  p6880880_112000_Linux-x86-64.zip
   creating: /export/11.2.0.2/grid/OPatch/
   creating: /export/11.2.0.2/grid/OPatch/oplan/
  inflating: /export/11.2.0.2/grid/OPatch/oplan/README.html
  inflating: /export/11.2.0.2/grid/OPatch/oplan/README.txt
   creating: /export/11.2.0.2/grid/OPatch/oplan/jlib/
  inflating: /export/11.2.0.2/grid/OPatch/oplan/jlib/oplan.jar
  inflating: /export/11.2.0.2/grid/OPatch/oplan/jlib/oracle.oplan.classpath.jar
  inflating: /export/11.2.0.2/grid/OPatch/oplan/jlib/automation.jar
  inflating: /export/11.2.0.2/grid/OPatch/oplan/jlib/OsysModel.jar
  inflating: /export/11.2.0.2/grid/OPatch/oplan/jlib/EMrepoDrivers.jar
   creating: /export/11.2.0.2/grid/OPatch/oplan/jlib/apache-commons/
  inflating: /export/11.2.0.2/grid/OPatch/oplan/jlib/apache-commons/commons-cli-1.0.jar
   creating: /export/11.2.0.2/grid/OPatch/oplan/jlib/jaxb/
  inflating: /export/11.2.0.2/grid/OPatch/oplan/jlib/jaxb/activation.jar
  inflating: /export/11.2.0.2/grid/OPatch/oplan/jlib/jaxb/jaxb-api.jar
  inflating: /export/11.2.0.2/grid/OPatch/oplan/jlib/jaxb/jaxb-impl.jar
  inflating: /export/11.2.0.2/grid/OPatch/oplan/jlib/jaxb/jsr173_1.0_api.jar
  inflating: /export/11.2.0.2/grid/OPatch/oplan/jlib/osysmodel-utils.jar
  inflating: /export/11.2.0.2/grid/OPatch/oplan/jlib/CRSProductDriver.jar
  inflating: /export/11.2.0.2/grid/OPatch/oplan/oplan
   creating: /export/11.2.0.2/grid/OPatch/docs/
  inflating: /export/11.2.0.2/grid/OPatch/docs/FAQ
  inflating: /export/11.2.0.2/grid/OPatch/docs/Users_Guide.txt
  inflating: /export/11.2.0.2/grid/OPatch/docs/Prereq_Users_Guide.txt
   creating: /export/11.2.0.2/grid/OPatch/jlib/
  inflating: /export/11.2.0.2/grid/OPatch/jlib/opatch.jar
  inflating: /export/11.2.0.2/grid/OPatch/jlib/opatchsdk.jar
  inflating: /export/11.2.0.2/grid/OPatch/jlib/oracle.opatch.classpath.jar
  inflating: /export/11.2.0.2/grid/OPatch/jlib/oracle.opatch.classpath.unix.jar
  inflating: /export/11.2.0.2/grid/OPatch/jlib/oracle.opatch.classpath.windows.jar
   creating: /export/11.2.0.2/grid/OPatch/opatchprereqs/
   creating: /export/11.2.0.2/grid/OPatch/opatchprereqs/opatch/
  inflating: /export/11.2.0.2/grid/OPatch/opatchprereqs/opatch/opatch_prereq.xml
  inflating: /export/11.2.0.2/grid/OPatch/opatchprereqs/opatch/rulemap.xml
  inflating: /export/11.2.0.2/grid/OPatch/opatchprereqs/opatch/runtime_prereq.xml
   creating: /export/11.2.0.2/grid/OPatch/opatchprereqs/oui/
  inflating: /export/11.2.0.2/grid/OPatch/opatchprereqs/oui/knowledgesrc.xml
  inflating: /export/11.2.0.2/grid/OPatch/opatchprereqs/prerequisite.properties
  inflating: /export/11.2.0.2/grid/OPatch/opatch
  inflating: /export/11.2.0.2/grid/OPatch/opatch.bat
  inflating: /export/11.2.0.2/grid/OPatch/opatch.pl
  inflating: /export/11.2.0.2/grid/OPatch/opatch.ini
  inflating: /export/11.2.0.2/grid/OPatch/opatchdiag
  inflating: /export/11.2.0.2/grid/OPatch/opatchdiag.bat
  inflating: /export/11.2.0.2/grid/OPatch/emdpatch.pl
  inflating: /export/11.2.0.2/grid/OPatch/README.txt
   creating: /export/11.2.0.2/grid/OPatch/ocm/
   creating: /export/11.2.0.2/grid/OPatch/ocm/bin/
  inflating: /export/11.2.0.2/grid/OPatch/ocm/bin/emocmrsp
   creating: /export/11.2.0.2/grid/OPatch/ocm/doc/
   creating: /export/11.2.0.2/grid/OPatch/ocm/lib/
  inflating: /export/11.2.0.2/grid/OPatch/ocm/lib/emocmclnt-14.jar
  inflating: /export/11.2.0.2/grid/OPatch/ocm/lib/emocmclnt.jar
  inflating: /export/11.2.0.2/grid/OPatch/ocm/lib/emocmcommon.jar
  inflating: /export/11.2.0.2/grid/OPatch/ocm/lib/http_client.jar
  inflating: /export/11.2.0.2/grid/OPatch/ocm/lib/jcert.jar
  inflating: /export/11.2.0.2/grid/OPatch/ocm/lib/jnet.jar
  inflating: /export/11.2.0.2/grid/OPatch/ocm/lib/jsse.jar
  inflating: /export/11.2.0.2/grid/OPatch/ocm/lib/log4j-core.jar
  inflating: /export/11.2.0.2/grid/OPatch/ocm/lib/osdt_core3.jar
  inflating: /export/11.2.0.2/grid/OPatch/ocm/lib/osdt_jce.jar
  inflating: /export/11.2.0.2/grid/OPatch/ocm/lib/regexp.jar
  inflating: /export/11.2.0.2/grid/OPatch/ocm/lib/xmlparserv2.jar
 extracting: /export/11.2.0.2/grid/OPatch/ocm/ocm.zip
  inflating: /export/11.2.0.2/grid/OPatch/ocm/ocm_platforms.txt
   creating: /export/11.2.0.2/grid/OPatch/crs/
   creating: /export/11.2.0.2/grid/OPatch/crs/log/
 extracting: /export/11.2.0.2/grid/OPatch/crs/log/dummy
  inflating: /export/11.2.0.2/grid/OPatch/crs/auto_patch.pl
  inflating: /export/11.2.0.2/grid/OPatch/crs/crsconfig_lib.pm
  inflating: /export/11.2.0.2/grid/OPatch/crs/crsdelete.pm
  inflating: /export/11.2.0.2/grid/OPatch/crs/crspatch.pm
  inflating: /export/11.2.0.2/grid/OPatch/crs/installPatch.excl
  inflating: /export/11.2.0.2/grid/OPatch/crs/oracss.pm
  inflating: /export/11.2.0.2/grid/OPatch/crs/patch112.pl
  inflating: /export/11.2.0.2/grid/OPatch/crs/s_crsconfig_defs
  inflating: /export/11.2.0.2/grid/OPatch/crs/s_crsconfig_lib.pm
rac01:/export/install/nels/opatch # cd /export/11.2.0.2/grid
rac01:/export/11.2.0.2/grid # ls -ltr


> unzip p6880880_112000_Linux-x86-64.zip -d $ORACLE_HOME

rac01 |ORA1020| /export/install/nels/opatch
> unzip p6880880_112000_Linux-x86-64.zip -d $ORACLE_HOME
Archive:  p6880880_112000_Linux-x86-64.zip
   creating: /export/oracle/product/11.2.0.2/OPatch/
   creating: /export/oracle/product/11.2.0.2/OPatch/oplan/
  inflating: /export/oracle/product/11.2.0.2/OPatch/oplan/README.html
  inflating: /export/oracle/product/11.2.0.2/OPatch/oplan/README.txt
   creating: /export/oracle/product/11.2.0.2/OPatch/oplan/jlib/
  inflating: /export/oracle/product/11.2.0.2/OPatch/oplan/jlib/oplan.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/oplan/jlib/oracle.oplan.classpath.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/oplan/jlib/automation.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/oplan/jlib/OsysModel.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/oplan/jlib/EMrepoDrivers.jar
   creating: /export/oracle/product/11.2.0.2/OPatch/oplan/jlib/apache-commons/
  inflating: /export/oracle/product/11.2.0.2/OPatch/oplan/jlib/apache-commons/commons-cli-1.0.jar
   creating: /export/oracle/product/11.2.0.2/OPatch/oplan/jlib/jaxb/
  inflating: /export/oracle/product/11.2.0.2/OPatch/oplan/jlib/jaxb/activation.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/oplan/jlib/jaxb/jaxb-api.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/oplan/jlib/jaxb/jaxb-impl.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/oplan/jlib/jaxb/jsr173_1.0_api.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/oplan/jlib/osysmodel-utils.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/oplan/jlib/CRSProductDriver.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/oplan/oplan
   creating: /export/oracle/product/11.2.0.2/OPatch/docs/
  inflating: /export/oracle/product/11.2.0.2/OPatch/docs/FAQ
  inflating: /export/oracle/product/11.2.0.2/OPatch/docs/Users_Guide.txt
  inflating: /export/oracle/product/11.2.0.2/OPatch/docs/Prereq_Users_Guide.txt
   creating: /export/oracle/product/11.2.0.2/OPatch/jlib/
  inflating: /export/oracle/product/11.2.0.2/OPatch/jlib/opatch.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/jlib/opatchsdk.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/jlib/oracle.opatch.classpath.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/jlib/oracle.opatch.classpath.unix.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/jlib/oracle.opatch.classpath.windows.jar
   creating: /export/oracle/product/11.2.0.2/OPatch/opatchprereqs/
   creating: /export/oracle/product/11.2.0.2/OPatch/opatchprereqs/opatch/
  inflating: /export/oracle/product/11.2.0.2/OPatch/opatchprereqs/opatch/opatch_prereq.xml
  inflating: /export/oracle/product/11.2.0.2/OPatch/opatchprereqs/opatch/rulemap.xml
  inflating: /export/oracle/product/11.2.0.2/OPatch/opatchprereqs/opatch/runtime_prereq.xml
   creating: /export/oracle/product/11.2.0.2/OPatch/opatchprereqs/oui/
  inflating: /export/oracle/product/11.2.0.2/OPatch/opatchprereqs/oui/knowledgesrc.xml
  inflating: /export/oracle/product/11.2.0.2/OPatch/opatchprereqs/prerequisite.properties
  inflating: /export/oracle/product/11.2.0.2/OPatch/opatch
  inflating: /export/oracle/product/11.2.0.2/OPatch/opatch.bat
  inflating: /export/oracle/product/11.2.0.2/OPatch/opatch.pl
  inflating: /export/oracle/product/11.2.0.2/OPatch/opatch.ini
  inflating: /export/oracle/product/11.2.0.2/OPatch/opatchdiag
  inflating: /export/oracle/product/11.2.0.2/OPatch/opatchdiag.bat
  inflating: /export/oracle/product/11.2.0.2/OPatch/emdpatch.pl
  inflating: /export/oracle/product/11.2.0.2/OPatch/README.txt
   creating: /export/oracle/product/11.2.0.2/OPatch/ocm/
   creating: /export/oracle/product/11.2.0.2/OPatch/ocm/bin/
  inflating: /export/oracle/product/11.2.0.2/OPatch/ocm/bin/emocmrsp
   creating: /export/oracle/product/11.2.0.2/OPatch/ocm/doc/
   creating: /export/oracle/product/11.2.0.2/OPatch/ocm/lib/
  inflating: /export/oracle/product/11.2.0.2/OPatch/ocm/lib/emocmclnt-14.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/ocm/lib/emocmclnt.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/ocm/lib/emocmcommon.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/ocm/lib/http_client.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/ocm/lib/jcert.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/ocm/lib/jnet.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/ocm/lib/jsse.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/ocm/lib/log4j-core.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/ocm/lib/osdt_core3.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/ocm/lib/osdt_jce.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/ocm/lib/regexp.jar
  inflating: /export/oracle/product/11.2.0.2/OPatch/ocm/lib/xmlparserv2.jar
 extracting: /export/oracle/product/11.2.0.2/OPatch/ocm/ocm.zip
  inflating: /export/oracle/product/11.2.0.2/OPatch/ocm/ocm_platforms.txt
   creating: /export/oracle/product/11.2.0.2/OPatch/crs/
   creating: /export/oracle/product/11.2.0.2/OPatch/crs/log/
 extracting: /export/oracle/product/11.2.0.2/OPatch/crs/log/dummy
  inflating: /export/oracle/product/11.2.0.2/OPatch/crs/auto_patch.pl
  inflating: /export/oracle/product/11.2.0.2/OPatch/crs/crsconfig_lib.pm
  inflating: /export/oracle/product/11.2.0.2/OPatch/crs/crsdelete.pm
  inflating: /export/oracle/product/11.2.0.2/OPatch/crs/crspatch.pm
  inflating: /export/oracle/product/11.2.0.2/OPatch/crs/installPatch.excl
  inflating: /export/oracle/product/11.2.0.2/OPatch/crs/oracss.pm
  inflating: /export/oracle/product/11.2.0.2/OPatch/crs/patch112.pl
  inflating: /export/oracle/product/11.2.0.2/OPatch/crs/s_crsconfig_defs
  inflating: /export/oracle/product/11.2.0.2/OPatch/crs/s_crsconfig_lib.pm
rac01 |ORA1020| /export/install/nels/opatch
> 


Check the OPatch version

> opatch version
OPatch Version: 11.2.0.3.0

OPatch succeeded.


OCM Configuration

During patch apply, OPatch will prompt for the location of the OCM (Oracle Configuration Manager) response file. If you do not have one, run the following command to create it.

> $CRS_ORA_HOME/OPatch/opatch/ocm/bin/emocmrsp

Make a note of the response file location as you will need it.

Validation of Oracle Inventory

Before patch application, let check the consistency of the oracle inventory. If the below command succeeds, we are good.

> $CRS_ORA_HOME/OPatch/opatch lsinventory -detail -oh $CRS_ORA_HOME


Let’s do the same for the RDBMS home. I’m not listing the whole output again here.

> $ORACLE_HOME/OPatch/opatch lsinventory -detail -oh $ORACLE_HOME


One-off Patch Conflict Detection and Resolution

Check for conflicting patches by doing the following for each GI and RDBMS home you are planning to apply the patch on.

Use My Oracle Support Note 1061295.1 Patch Set Updates - One-off Patch Conflict Resolution to determine, for each conflicting patch, whether a conflict resolution patch is already available, and if you need to request a new conflict resolution patch or if the conflict may be ignored.

> opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./13696242
Oracle Interim Patch Installer version 11.2.0.3.0
Copyright (c) 2012, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /export/oracle/product/11.2.0.2
Central Inventory : /export/oracle/oraInventory
   from           : /export/oracle/product/11.2.0.2/oraInst.loc
OPatch version    : 11.2.0.3.0
OUI version       : 11.2.0.2.0
Log file location : /export/oracle/product/11.2.0.2/cfgtoollogs/opatch/opatch2012-07-28_00-10-27AM_1.log

Invoking prereq "checkconflictagainstohwithdetail"

Prereq "checkConflictAgainstOHWithDetail" passed.

OPatch succeeded.


Syntax

For each RAC home

>opatch prereq CheckConflictAgainstOHWithDetail -oh $ORACLE_HOME -phBaseDir <UNZIPPED_PATCH_LOCATION>/13696242

For each GI home

>opatch prereq CheckConflictAgainstOHWithDetail -oh $CRS_ORA_HOME -phBaseDir <UNZIPPED_PATCH_LOCATION>/13696242


Patch Installation

Note:

-- Stop EM agent if it is running on your server before you apply the patch. Our grid control agent is 10.2.

rac01 |AGENT| /export/oracle
> /export/oracle/product/10.2.0/agent10g/bin/emctl stop agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.
Stopping agent ... stopped.
rac01 |AGENT| /export/oracle


-- Make sure there is enough free space available in your GI and RDBMS home volume
-- If the available free space is less, try to use ‘opatch util cleanup’ to clean old backup files from previous patching. It is usually a hidden directory in GI_HOME or ORACLE_HOME called .patch_storage

Syntax:

#opatch auto <UNZIPPED_PATCH_LOCATION> -oh <path to GI home>

Apply OPatch auto as ROOT user on Grid Infrastructure home

During this process all services running on the node will be taken offline, patch applied and brought up online by Oracle

rac01 |ODSQA2| /export/oracle
> sudo su - root
rac01:~ # export CRS_ORA_HOME=/export/11.2.0.2/grid
rac01:~ # export ORACLE_HOME=/export/oracle/product/11.2.0.2
rac01:~ # export PATH=$CRS_ORA_HOME/OPatch:$PATH
rac01:~ # which opatch
/export/11.2.0.2/grid/OPatch/opatch
rac01:~ # opatch auto /export/install/nels/GI_PSU_11.2.0.2.6 -oh /export/11.2.0.2/grid
Executing /usr/bin/perl /export/11.2.0.2/grid/OPatch/crs/patch112.pl -patchdir /export/install/nels -patchn GI_PSU_11.2.0.2.6 -oh /export/11.2.0.2/grid -paramfile /export/11.2.0.2/grid/crs/install/crsconfig_params
opatch auto log file location is /export/11.2.0.2/grid/OPatch/crs/../../cfgtoollogs/opatchauto2012-07-28_00-45-16.log
Detected Oracle Clusterware install
Using configuration parameter file: /export/11.2.0.2/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/export/oracle/ocm.rsp
Successfully unlock /export/11.2.0.2/grid
patch /export/install/nels/GI_PSU_11.2.0.2.6/13696242  apply successful for home  /export/11.2.0.2/grid
patch /export/install/nels/GI_PSU_11.2.0.2.6/13696224  apply successful for home  /export/11.2.0.2/grid
ACFS-9459: ADVM/ACFS is not supported on this OS version: 'SLES10 SP2'
CRS-4123: Oracle High Availability Services has been started.

rac01:~ #
rac01:~ #

Apply OPatch auto as ROOT user on Oracle RDBMS home

rac01:~ # export PATH=$ORACLE_HOME/OPatch:$PATH
rac01:~ # which opatch
/export/oracle/product/11.2.0.2/OPatch/opatch
rac01:~ # opatch auto /export/install/nels/GI_PSU_11.2.0.2.6 -oh /export/oracle/product/11.2.0.2
Executing /usr/bin/perl /export/oracle/product/11.2.0.2/OPatch/crs/patch112.pl -patchdir /export/install/nels -patchn GI_PSU_11.2.0.2.6 -oh /export/oracle/product/11.2.0.2 -paramfile /export/11.2.0.2/grid/crs/install/crsconfig_params
opatch auto log file location is /export/oracle/product/11.2.0.2/OPatch/crs/../../cfgtoollogs/opatchauto2012-07-28_01-08-51.log
Detected Oracle Clusterware install
Using configuration parameter file: /export/11.2.0.2/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/export/oracle/ocm.rsp
patch /export/install/nels/GI_PSU_11.2.0.2.6/13696242/custom/server/13696242  apply successful for home  /export/oracle/product/11.2.0.2
patch /export/install/nels/GI_PSU_11.2.0.2.6/13696224  apply successful for home  /export/oracle/product/11.2.0.2
rac01:~ #
rac01:~ #

Check Applied Patch


> $CRS_ORA_HOME/OPatch/opatch lsinventory -bugs_fixed |grep -i 'GI PSU'
12311357   13696242  Sat Jul 28 01:17:55 GMT 2012   GI PSU 11.2.0.2.2 (INCLUDES DATABASE PSU 11.2.0.2.
12419353   13696242  Sat Jul 28 01:17:55 GMT 2012   GI PSU 11.2.0.2.3 (INCLUDES DATABASE PSU
12827731   13696242  Sat Jul 28 01:17:55 GMT 2012   GI PSU 11.2.0.2.4 (INCLUDES DATABASE PSU
12862012   13696242  Sat Jul 28 01:17:55 GMT 2012   11.2.0.2 GI PSU 4 HAS-CRS TRACKING BUG
13343447   13696242  Sat Jul 28 01:17:55 GMT 2012   GI PSU 11.2.0.2.5 (INCLUDES DATABASE PSU
13396479   13696242  Sat Jul 28 01:17:55 GMT 2012   11.2.0.2 GI PSU 5 HAS-CRS TRACKING BUG
13653086   13696242  Sat Jul 28 01:17:55 GMT 2012   GI PSU 11.2.0.2.5 (INCLUDES DATABASE PSU
13696242   13696242  Sat Jul 28 01:17:55 GMT 2012   GI PSU 11.2.0.2.6 (INCLUDES DATABASE PSU 11.2.0.2.
13768473   13696242  Sat Jul 28 01:17:55 GMT 2012   11.2.0.2 GI PSU 6 HAS-CRS TRACKING BUG


> $ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i 'DATABASE P'
10248523   13696224  Sat Jul 28 01:28:17 GMT 2012   DATABASE PSU 11.2.0.2.1
11724916   13696224  Sat Jul 28 01:28:17 GMT 2012   DATABASE PSU 11.2.0.2.2 (INCLUDES CPUAPR2011)
12419331   13696224  Sat Jul 28 01:28:17 GMT 2012   DATABASE PSU 11.2.0.2.3 (INCLUDES CPUJUL2011)
12827726   13696224  Sat Jul 28 01:28:17 GMT 2012   DATABASE PSU 11.2.0.2.4 (INCLUDES CPUOCT2011)
13343424   13696224  Sat Jul 28 01:28:17 GMT 2012   DATABASE PATCH SET UPDATE 11.2.0.2.5 (INCLUDES
13696224   13696224  Sat Jul 28 01:28:17 GMT 2012   DATABASE PATCH SET UPDATE 11.2.0.2.6 (INCLUDES
                                                    DISABLED IF 11201 DATABASE PRESENT
12311357   13696242  Sat Jul 28 01:17:55 GMT 2012   GI PSU 11.2.0.2.2 (INCLUDES DATABASE PSU 11.2.0.2.
12419353   13696242  Sat Jul 28 01:17:55 GMT 2012   GI PSU 11.2.0.2.3 (INCLUDES DATABASE PSU
12827731   13696242  Sat Jul 28 01:17:55 GMT 2012   GI PSU 11.2.0.2.4 (INCLUDES DATABASE PSU
13343447   13696242  Sat Jul 28 01:17:55 GMT 2012   GI PSU 11.2.0.2.5 (INCLUDES DATABASE PSU
13653086   13696242  Sat Jul 28 01:17:55 GMT 2012   GI PSU 11.2.0.2.5 (INCLUDES DATABASE PSU
13696242   13696242  Sat Jul 28 01:17:55 GMT 2012   GI PSU 11.2.0.2.6 (INCLUDES DATABASE PSU 11.2.0.2.

Perform all the above steps on the second node, one node at a time in a rolling fashion to install the patch in all the servers.

Node 2

rac02:~ # opatch auto /export/install/nels/GI_PSU_11.2.0.2.6 -oh /export/11.2.0.2/grid
Executing /usr/bin/perl /export/11.2.0.2/grid/OPatch/crs/patch112.pl -patchdir /export/install/nels -patchn GI_PSU_11.2.0.2.6 -oh /export/11.2.0.2/grid -paramfile /export/11.2.0.2/grid/crs/install/crsconfig_params
opatch auto log file location is /export/11.2.0.2/grid/OPatch/crs/../../cfgtoollogs/opatchauto2012-07-28_01-57-15.log
Detected Oracle Clusterware install
Using configuration parameter file: /export/11.2.0.2/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/export/oracle/ocm.rsp
Successfully unlock /export/11.2.0.2/grid
patch /export/install/nels/GI_PSU_11.2.0.2.6/13696242  apply successful for home  /export/11.2.0.2/grid
patch /export/install/nels/GI_PSU_11.2.0.2.6/13696224  apply successful for home  /export/11.2.0.2/grid
ACFS-9459: ADVM/ACFS is not supported on this OS version: 'SLES10 SP2'
CRS-4123: Oracle High Availability Services has been started.
rac02:~ #
rac02:~ #



rac02:~ # export PATH=$ORACLE_HOME/OPatch:$PATH
rac02:~ # which opatch
/export/oracle/product/11.2.0.2/OPatch/opatch
rac02:~ # opatch auto /export/install/nels/GI_PSU_11.2.0.2.6 -oh /export/oracle/product/11.2.0.2
Executing /usr/bin/perl /export/oracle/product/11.2.0.2/OPatch/crs/patch112.pl -patchdir /export/install/nels -patchn GI_PSU_11.2.0.2.6 -oh /export/oracle/product/11.2.0.2 -paramfile /export/11.2.0.2/grid/crs/install/crsconfig_params
opatch auto log file location is /export/oracle/product/11.2.0.2/OPatch/crs/../../cfgtoollogs/opatchauto2012-07-28_02-21-15.log
Detected Oracle Clusterware install
Using configuration parameter file: /export/11.2.0.2/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/export/oracle/ocm.rsp
patch /export/install/nels/GI_PSU_11.2.0.2.6/13696242/custom/server/13696242  apply successful for home  /export/oracle/product/11.2.0.2
patch /export/install/nels/GI_PSU_11.2.0.2.6/13696224  apply successful for home  /export/oracle/product/11.2.0.2
rac02:~ #
rac02:~ #


Check Applied Patch

rac02 |ODSQA1| /export/oracle
> $CRS_ORA_HOME/OPatch/opatch lsinventory -bugs_fixed |grep -i 'GI PSU'
12311357   13696242  Sat Jul 28 02:29:05 GMT 2012   GI PSU 11.2.0.2.2 (INCLUDES DATABASE PSU 11.2.0.2.
12419353   13696242  Sat Jul 28 02:29:05 GMT 2012   GI PSU 11.2.0.2.3 (INCLUDES DATABASE PSU
12827731   13696242  Sat Jul 28 02:29:05 GMT 2012   GI PSU 11.2.0.2.4 (INCLUDES DATABASE PSU
12862012   13696242  Sat Jul 28 02:29:05 GMT 2012   11.2.0.2 GI PSU 4 HAS-CRS TRACKING BUG
13343447   13696242  Sat Jul 28 02:29:05 GMT 2012   GI PSU 11.2.0.2.5 (INCLUDES DATABASE PSU
13396479   13696242  Sat Jul 28 02:29:05 GMT 2012   11.2.0.2 GI PSU 5 HAS-CRS TRACKING BUG
13653086   13696242  Sat Jul 28 02:29:05 GMT 2012   GI PSU 11.2.0.2.5 (INCLUDES DATABASE PSU
13696242   13696242  Sat Jul 28 02:29:05 GMT 2012   GI PSU 11.2.0.2.6 (INCLUDES DATABASE PSU 11.2.0.2.
13768473   13696242  Sat Jul 28 02:29:05 GMT 2012   11.2.0.2 GI PSU 6 HAS-CRS TRACKING BUG
rac02 |ODSQA1| /export/oracle


> $ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i 'DATABASE P'
10248523   13696224  Sat Jul 28 02:40:19 GMT 2012   DATABASE PSU 11.2.0.2.1
11724916   13696224  Sat Jul 28 02:40:19 GMT 2012   DATABASE PSU 11.2.0.2.2 (INCLUDES CPUAPR2011)
12419331   13696224  Sat Jul 28 02:40:19 GMT 2012   DATABASE PSU 11.2.0.2.3 (INCLUDES CPUJUL2011)
12827726   13696224  Sat Jul 28 02:40:19 GMT 2012   DATABASE PSU 11.2.0.2.4 (INCLUDES CPUOCT2011)
13343424   13696224  Sat Jul 28 02:40:19 GMT 2012   DATABASE PATCH SET UPDATE 11.2.0.2.5 (INCLUDES
13696224   13696224  Sat Jul 28 02:40:19 GMT 2012   DATABASE PATCH SET UPDATE 11.2.0.2.6 (INCLUDES
                                                    DISABLED IF 11201 DATABASE PRESENT
12311357   13696242  Sat Jul 28 02:29:05 GMT 2012   GI PSU 11.2.0.2.2 (INCLUDES DATABASE PSU 11.2.0.2.
12419353   13696242  Sat Jul 28 02:29:05 GMT 2012   GI PSU 11.2.0.2.3 (INCLUDES DATABASE PSU
12827731   13696242  Sat Jul 28 02:29:05 GMT 2012   GI PSU 11.2.0.2.4 (INCLUDES DATABASE PSU
13343447   13696242  Sat Jul 28 02:29:05 GMT 2012   GI PSU 11.2.0.2.5 (INCLUDES DATABASE PSU
13653086   13696242  Sat Jul 28 02:29:05 GMT 2012   GI PSU 11.2.0.2.5 (INCLUDES DATABASE PSU
13696242   13696242  Sat Jul 28 02:29:05 GMT 2012   GI PSU 11.2.0.2.6 (INCLUDES DATABASE PSU 11.2.0.2.
rac02 |ODSQA1| /export/oracle


Patch Post Installation Steps

For each database instance running on the Oracle home being patched, connect to the database using SQL*Plus. Connect as SYSDBA and run the catbundle.sql script


cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> @catbundle.sql psu apply
SQL> QUIT


Check the log files in $ORACLE_HOME/cfgtoollogs/catbundle for any errors


1 comment:

  1. Hi Samuel,

    Thanks for sharing such a wonderful post.

    Thanks

    ReplyDelete