Applies to:
Oracle Server - Enterprise Edition - Version: 11.2.0.1 and later [Release: 11.2 and later ]
Information in this document applies to any platform.
Purpose
11gR2 Grid Infrastructure clusterware (CRS) may become unhealthy if: filesystem becomes 100% full on "/" or mount point where clusterware home is installed, OS running out of memory, network not performing etc.
Generally speaking clusterware should automatically recover from this kind of situation but in some cases it may fail. The purpose of this document is to provide a list of troubleshooting actions in the event that clusterware auto recovery should fail.
Scope and Application
This document is intended for RAC Database Administrators and Oracle support engineers.
What to Do if 11gR2 Clusterware is Unhealthy
Common symptoms of unhealthy clusterware or srvctl or crsctl returning unexpected results or becoming unresponsive, include:
OS running out of space.
OS running out of memory.
OS running out of CPU resource.
NOTE: The following note provides a list of common cause for individual clusterware process failures:
Note 1050908.1 How to Troubleshoot Grid Infrastructure Startup Issues
1. Clusterware Process:
Once issue is identified and fixed, please wait for a few minutes, and verify clusterware processes state - all processes should show up as ONLINE.
1A. To find out clusterware processes state:
$GRID_HOME/bin/crsctl stat res -t -init
NAME TARGET STATE SERVER STATE_DETAILS
Cluster Resources
ora.asm 1 OFFLINE OFFLINE Instance Shutdown
ora.crsd 1 OFFLINE OFFLINE
ora.cssd 1 ONLINE ONLINE rac002f
ora.cssdmonitor 1 ONLINE ONLINE rac002f
ora.ctssd 1 ONLINE ONLINE rac002f OBSERVER
ora.diskmon 1 ONLINE ONLINE rac002f
ora.drivers.acfs 1 ONLINE ONLINE rac002f
ora.evmd 1 OFFLINE OFFLINE
ora.gipcd 1 ONLINE ONLINE rac002f
ora.gpnpd 1 ONLINE ONLINE rac002f
ora.mdnsd 1 ONLINE ONLINE rac002f
1B. In above example, ora.asm, ora.crsd and ora.evmd remained OFFLINE which means manual intervention is needed, to bring them up:
$GRID_HOME/bin/crsctl start res ora.crsd -init
CRS-2672: Attempting to start 'ora.asm' on 'rac002f'
CRS-2676: Start of 'ora.asm' on 'rac002f' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac002f'
CRS-2676: Start of 'ora.crsd' on 'rac002f' succeeded
As ora.crsd depend on ora.asm, ora.asm is started automatically when starting ora.crsd
To bring up ora.evmd:
$GRID_HOME/bin/crsctl start res ora.evmd -init
CRS-2672: Attempting to start 'ora.evmd' on 'rac001f'
CRS-2676: Start of 'ora.evmd' on 'rac001f' succeeded
1C. If process resource fails to start up, please refer to <> for troubleshooting steps; then try to stop it and restart it:
$GRID_HOME/bin/crsctl stop res ora.evmd -init
If this fails, try with "-f" option:
$GRID_HOME/bin/crsctl stop res ora.evmd -init -f
If stop fails with "-f" option, please refer to Appendix.
If the process is already stopped, the following errors will be reported:
CRS-2500: Cannot stop resource 'ora.evmd' as it is not running
CRS-4000: Command Stop failed, or completed with errors.
1D. If a critical clusterware process fails to start and there's no obvious reason, the next action is to restart clusterware on local node:
$GRID_HOME/bin/crsctl stop crs -f
1E. If above command fails, you may kill all clusterware processes by executing:
ps -ef | grep <keyword> | grep -v grep | awk '{print $2}' | xargs kill -9
1F. As a last resort, you can take out local node by rebooting it.
1G. If there's more than one node where clusterware is unhealthy, repeat the same procedure on all other nodes, once clusterware daemons are up on all nodes, next thing to verify is user resource.
2. Clusterware Exclusive Mode
Certain tasks requires clusterware to be in exclusive mode. To bring CRS in exclusive mode, shutdown CRS on all nodes (refer to above Step 1D, 1E and 1F), then as root, issue the following command on one node only:
$GRID_HOME/bin/crsctl start crs -excl
For 11.2.0.2 and above:
$GRID_HOME/bin/crsctl start crs -excl -nocrs
If cssd.bin fails to come up, as root, issue the following command:
$GRID_HOME/bin/crsctl start res ora.cssd -init -env "CSSD_MODE=-X"
3. User Resource:
3A. The crs_stat command has been deprecated in 11gR2, please do not use it anymore. Use the following command to query the resource state of all user resources:
$GRID_HOME/bin/crsctl stat res -t
NAME TARGET STATE SERVER STATE_DETAILS
Local Resources
ora.GI.dg ONLINE ONLINE rac001f ONLINE ONLINE rac002f
ora.LISTENER.lsnr ONLINE ONLINE rac001f ONLINE ONLINE rac002f
..
ora.gsd OFFLINE OFFLINE rac001f OFFLINE OFFLINE rac002f
ora.net1.network ONLINE ONLINE rac001f ONLINE ONLINE rac002f
Cluster Resources
ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE rac002f
ora.LISTENER_SCAN2.lsnr 1 ONLINE OFFLINE
ora.LISTENER_SCAN3.lsnr1 ONLINE OFFLINE
ora.b2.db1 ONLINE ONLINE rac001f2 ONLINE ONLINE rac002f Open
ora.b2.sb2.svc1 ONLINE ONLINE rac001f2 ONLINE ONLINE rac002f
ora.rac001f.vip1 ONLINE ONLINE rac001f ora.rac002f.vip1 ONLINE ONLINE rac002f
ora.oc4j1 OFFLINE OFFLINEora.scan1.vip1 ONLINE ONLINE rac002f
ora.scan2.vip1 ONLINE OFFLINE
ora.scan3.vip1 ONLINE OFFLINE
NOTE: ora.gsd is OFFLINE by default if there is no 9i database in the cluster. ora.oc4j is OFFLINE in 11.2.0.1 as Database Workload Management(DBWLM) is unavailable.
3B. In example above, resource ora.scan2.vip, ora.scan3.vip, ora.LISTENER_SCAN2.lsnr and ora.LISTENER_SCAN3.lsnr are OFFLINE.
To start it:
$GRID_HOME/bin/srvctl start scan
PRCC-1014 : scan1 was already running
3C. To start other OFFLINE resources:
$RESOURCE_HOME/bin/srvctl start resource_type <options>
$RESOURCE_HOME refers to location where the resource is running off, for example, vip in $GRID_HOME, 11.2 .db in 11.2 RDBMS home, and 11.1 .db in 11.1 RDBMS home.
For srvctl syntax, please refer to Server Control Utility Reference
3D. To stop user resource, try the following sequentially until resource stopped successfully:
$RESOURCE_HOME/bin/srvctl stop resource_type <options>
$RESOURCE_HOME/bin/srvctl stop resource_type <options> -f
$GRID_HOME/bin/crsctl stop res resource_name
$GRID_HOME/bin/crsctl stop res resource_name -f
Where resource_name is the name in "crsctl stat res" output.
Appendix
A. Process Resource Fails to Stop even with "-f" option:
$GRID_HOME/bin/crsctl stat res -w 'NAME = ora.ctssd' -t -init
ora.ctssd
1 ONLINE UNKNOWN node1 Wrong check return.
$GRID_HOME/bin/crsctl stop res ora.ctssd -init
CRS-2673: Attempting to stop 'ora.ctssd' on 'node1'
CRS-2675: Stop of 'ora.ctssd' on 'node1' failed
CRS-2679: Attempting to clean 'ora.ctssd' on 'node1'
CRS-2680: Clean of 'ora.ctssd' on 'node1' failed
Clean action for daemon aborted
$GRID_HOME/log/$HOST/agent/ohasd/orarootagent_root/orarootagent_root.log
2010-05-19 15:58:39.803: [ora.ctssd][1155352896] [check] PID will be looked for in /ocw/grid/ctss/init/node1.pid
2010-05-19 15:58:39.835: [ora.ctssd][1155352896] [check] PID which will be monitored will be 611
..
2010-05-19 15:58:40.016: [ COMMCRS][1239271744]clsc_connect: (0x2aaaac052ed0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=node1DBG_CTSSD))
[ clsdmc][1155352896]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=node1DBG_CTSSD)) with status 9
2010-05-19 15:58:40.016: [ora.ctssd][1155352896] [check] Error = error 9 encountered when connecting to CTSSD
..
2010-05-19 15:58:40.039: [ora.ctssd][1155352896] [check] Calling PID check for daemon
2010-05-19 15:58:40.039: [ora.ctssd][1155352896] [check] Trying to check PID = 611
..
2010-05-19 15:58:40.219: [ora.ctssd][1155352896] [check] PID check returned ONLINE CLSDM returned OFFLINE
2010-05-19 15:58:40.219: [ora.ctssd][1155352896] [check] Check error. Return = 5, state detail = Wrong check return.
2010-05-19 15:58:40.220: [ AGFW][1155352896] check for resource: ora.ctssd 1 1 completed with status: FAILED
2010-05-19 15:58:40.220: [ AGFW][1165842752] ora.ctssd 1 1 state changed from: UNKNOWN to: FAILED
ps -ef|grep 611|grep -v grep
root 611 7 0 May19 ? 00:00:00 [kmpathd/0]
cat /ocw/grid/ctss/init/node1.pid
611
In above example, stop of ora.ctssd fails as daemon pid file shows pid of octssd is 611, but "ps -ef" shows 611 is kmpathd which is not octssd.bin; also connects to ctssd via IPC key node1DBG_CTSSD fails.
To fix the issue, nullify ctssd pid file:
- /ocw/grid/ctss/init/node1.pid
Location of process resource pid can be $GRID_HOME/log/$HOST/$DAEMON/$HOST.pid or $GRID_HOME/$DAEMON/init/$HOST.pid
References
NOTE:1050908.1 - How to Troubleshoot Grid Infrastructure Startup Issues
NOTE:1053147.1 - 11gR2 Clusterware and Grid Home - What You Need to Know
NOTE:1069369.1 - How to Delete or Add Resource to OCR
NOTE:942166.1 - How to Proceed from Failed 11gR2 Grid Infrastructure (CRS) Installation
NOTE:969254.1 - How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on Linux/Unix
Producer --->> Kakfa server (Kafka Broker), cluster is a group of broker which has a toic - ---> consumer (request back for mo...
No comments:
Post a Comment