ASMCA fails to create 11.2 ASM instance when upgrading from 10.2 [ID 1125053.1]

In this Document
  Symptoms
  Cause
  Solution

Applies to:

Oracle Server - Enterprise Edition - Version: 11.2.0.1.0 to 11.2.0.1.0 - Release: 11.2 to 11.2
Information in this document applies to any platform.

Symptoms

While upgrading 10.2 CRS to 11.2 in a standalone installation, ASMCA fails to create 11.2 ASM instance.

Oracle 11gR2 Grid infrastructure installation reports the following errors:

INFO: Read: PRCR-1079 : Failed to start resource ora.asm
INFO: Read: CRS-2674: Start of ora.cssd on edprddb01 failed
INFO: Read: CRS-2678: ora.cssd on edprddb01 has experienced an unrecoverable failure
INFO: Read: CRS-0267: Human intervention required to resume its availability.

Installation invokes ASMCA to create ASM instance. ASMCA log shows ORA-1034 when starting ASM instance with "checkAndStartupInstance"

SEVERE:method oracle.sysman.assistants.usmca.backend.USMInstanceRac:checkAndStartupInstance
ORA-01034: ORACLE not available

oracle.sysman.assistants.util.sqlEngine.SQLFatalErrorException: ORA-01034: ORACLE not available

oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeImpl(SQLEngine.java:1650)
oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeQuery(SQLEngine.java:823)
oracle.sysman.assistants.usmca.backend.USMInstance.checkAndStartupInstance(USMInstance.java:602)
oracle.sysman.assistants.usmca.backend.USMInstanceRac.checkAndStartupInstance(USMInstanceRac.java:2668)
oracle.sysman.assistants.usmca.backend.USMInstanceRac.discoverASMInstance(USMInstanceRac.java:2383)
oracle.sysman.assistants.usmca.backend.USMInstance.initialize(USMInstance.java:248)
oracle.sysman.assistants.usmca.backend.USMInstanceRac.initialize(USMInstanceRac.java:275)
oracle.sysman.assistants.usmca.backend.USMInstance.getInstance(USMInstance.java:213)
oracle.sysman.assistants.usmca.service.UsmcaService.getASMStatus(UsmcaService.java:128)
oracle.sysman.assistants.usmca.Usmca.execute(Usmca.java:174)
oracle.sysman.assistants.usmca.Usmca.main(Usmca.java:333)

In the meantime, 11.2 CSS logs shows that CSS fails to listen with IPC protocol. Hence, 11.2 CSS cannot be started.

2010-02-21 10:06:46.569: [ CSSD][1030]clssgmclientOpenEndp: listening on clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_LclLstnr_localhost_1)(GIPCID=37f4a6e0-00000000-1740982))
2010-02-21 10:06:46.573: [GIPCXCPT][1030]gipcmodNetworkProcessBind: failed to bind endp 1111e3eb0 [0000000000000060] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_edprddb01_localhost)(GIPCID=00000000-00000000-0))', remoteAddr '', numPend
0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, flags 0x240712, usrFlags 0x14000 }, addr 1111e4370 [0000000000000062] { gipcAddress : name 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_edprddb01_localhost)(GIPCID=00000000-00000000-0))',
objFlags 0x0, addrFlags
2010-02-21 10:06:46.573: [GIPCXCPT][1030]gipcmodNetworkProcessBind: slos op : sgipcnDSBindHelper
2010-02-21 10:06:46.573: [GIPCXCPT][1030]gipcmodNetworkProcessBind: slos dep : Address already in use (67)
.
.
2010-02-21 10:06:46.968: [ CSSD][1030]###################################
2010-02-21 10:06:46.968: [ CSSD][1030]clssscExit: CSSD aborting from thread GMClientListener
2010-02-21 10:06:46.968: [ CSSD][1030]###################################

Cause

CSSD aborts itself as its listener thread can not be started. The error is "Address already in use". This is due to network socket files created by current CSS daemon (10.2).

Solution

1- Clean up network socket files

# rm -rf /tmp/.oracle /usr/tmp/.oracle /var/tmp/.oracle

Then reboot the server.

2- Deconfigure 11.2 ( Following example is for 11.2.0.1 / Use the equivalent commands for 11.2.0.2 and 11.2.0.3 ) CSS daemon:

As root, execute "$GRID_HOME/crs/install/roothas.pl -delete -force -verbose"

script /tmp/deconfigure.log
date
id
env

# perl $GRID_HOME/crs/install/roothas.pl -delete -force -verbose

exit

3- To reconfigure

script /tmp/configure.log
date
id
env

# perl roothas.pl

exit

4- Then run ASMCA manually to create ASM instance