This Saturday we were scheduled to patch eight oracle 11gR2 databases with the latest CPU/SPU set for one of our Virtual-DBA clients. The databases were running in a zone on Solaris 10. This is normally a pretty straight forward process, but this weekend we ran into an issue that we wanted to share.
We started by moving the patches to the server and unzipping them in a directory. We made a copy of the current $ORACLE_HOME by making a tar file out of it with the below command.
tar -cvf /u02/ora_home_before_patch.tar /u01/app/oracle/product/11.2.0/db_home1
We then gzipped it so it would take up less space on the drive. We then shutdown all 8 databases and the listeners that were running. We changed to the patch directory and attempted to apply the patch with the below command.
opatch napply -skip_subset -skip_duplicate
Opatch then threw an error complaining that the libclntsh.so.11.1 file was still in use and to make sure to stop all databases and listeners from the $ORACLE_HOME being patched. We checked all the oracle processes and no databases or listeners were started in the zone. We didn’t have root so I couldn’t use lsof if it was installed or pfiles. We got a Solaris admin involved and we couldn’t find anything holding the file open.
In the end we had to end of rebooting the zone. After the zone was rebooted we were able to apply the patch without any issues and restart the databases and run our catbundle.sql cpu apply command in sqlplus.