Oracle 无备份情况恢复ocr和olr
Oracle 11.2 GI中OLR损坏的修复方法
一、先关闭GI，命令不再贴出来了，参见：
http://blog.csdn.net/msdnchina/article/details/49402299
二、模拟OLR损坏
[root@hostb cdata]# ls -lrt total 2884 drwxr-xr-x 2 grid oinstall 4096 Oct 30 13:21 hostb drwxr-xr-x 2 grid oinstall 4096 Oct 30 19:06 localhost drwxrwxr-x 2 grid oinstall 4096 Dec 22 13:27 hosta-cluster -rw------- 1 root oinstall 272756736 Jan 8 13:38 hostb.olr [root@hostb cdata]# mv hostb.olr back_hostb.olr -----------> mv掉OLR [root@hostb cdata]# ls -lrt total 2884 drwxr-xr-x 2 grid oinstall 4096 Oct 30 13:21 hostb drwxr-xr-x 2 grid oinstall 4096 Oct 30 19:06 localhost drwxrwxr-x 2 grid oinstall 4096 Dec 22 13:27 hosta-cluster -rw------- 1 root oinstall 272756736 Jan 8 13:38 back_hostb.olr [root@hostb cdata]# /u01/app/11.2.0/grid/bin/crsctl start crs --------->起不来。 CRS-4124: Oracle High Availability Services startup failed. CRS-4000: command Start failed, or completed with errors. [root@hostb cdata]# pwd /u01/app/11.2.0/grid/cdata [root@hostb cdata]#
[oracle@hostb ohasd]$ pwd /u01/app/11.2.0/grid/log/hostb/ohasd [oracle@hostb ohasd]$ ls -lrt total 95720 -rw-r--r-- 1 root root 10540929 Nov 4 03:51 ohasd.l09 -rw-r--r-- 1 root root 10556558 Nov 8 16:28 ohasd.l08 -rw-r--r-- 1 root root 10558423 Nov 13 12:39 ohasd.l07 -rw-r--r-- 1 root root 10558697 Nov 18 05:44 ohasd.l06 -rw-r--r-- 1 root root 10557602 Nov 22 18:15 ohasd.l05 -rw-r--r-- 1 root root 10532670 Nov 27 08:24 ohasd.l04 -rw-r--r-- 1 root root 10556869 Dec 2 01:30 ohasd.l03 -rw-r--r-- 1 root root 10557883 Dec 6 23:49 ohasd.l02 -rw-r--r-- 1 root root 10542530 Dec 21 11:49 ohasd.l01 -rw-r--r-- 1 root root 6700 Jan 8 13:42 ohasdOUT.log -rw-r--r-- 1 root root 2986628 Jan 8 13:42 ohasd.log [oracle@hostb ohasd]$ vi ohasd.log 2016-01-08 13:42:59.562: [ default][3149834016] Initializing OLR 2016-01-08 13:42:59.564: [ OCROSD][3149834016]utopen:6m': failed in stat OCR file/disk /u01/app/11.2.0/grid/cdata/hostb.olr, errno=2, os err string=No such file or directory 2016-01-08 13:42:59.564: [ OCROSD][3149834016]utopen:7: failed to open any OCR file/disk, errno=2, os err string=No such file or directory 2016-01-08 13:42:59.564: [ OCRRAW][3149834016]proprinit: Could not open raw device 2016-01-08 13:42:59.564: [ OCRAPI][3149834016]a_init:16!: Backend init unsuccessful : [26] 2016-01-08 13:42:59.564: [ CRSOCR][3149834016] OCR context init failure. Error: PROCL-26: Error while accessing the physical storage Operating system error [No such file or directory] [2] 2016-01-08 13:42:59.565: [ default][3149834016] Created alert : (:OHAS00106:) : OLR initialization failed, error: PROCL-26: Error while accessing the physical storage Operating System error [No such file or directory] [2] 2016-01-08 13:42:59.565: [ default][3149834016][PANIC] OHASD exiting; Could not init OLR 2016-01-08 13:42:59.565: [ default][3149834016] Done.
开始对本地节点（主机名为hostb）进行OLR的recreate,参考文章：
Top 5 Grid Infrastructure Startup Issues (文档 ID 1368382.1) ，搜索关键字为"OLR"
[root@hostb bin]# ./crsctl stop crs CRS-4544: Unable to connect to OHAS CRS-4000: Command Stop failed, or completed with errors.
第一步：# /u01/app/11.2.0/grid/crs/install/rootcrs.pl -deconfig -force
[root@hostb install]# /u01/app/11.2.0/grid/crs/install/rootcrs.pl -deconfig -force Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type PRCR-1068 : Failed to query resources Cannot communicate with crsd PRCR-1070 : Failed to check if resource ora.gsd is registered Cannot communicate with crsd PRCR-1070 : Failed to check if resource ora.ons is registered Cannot communicate with crsd CRS-4535: Cannot communicate with Cluster Ready Services CRS-4000: Command Stop failed, or completed with errors. CRS-4544: Unable to connect to OHAS CRS-4000: Command Stop failed, or completed with errors. Successfully deconfigured Oracle clusterware stack on this node [root@hostb install]#
此时的/u01/app/11.2.0/grid/cdata目录情况：
[grid@hostb cdata]$ ls -lrt total 13124 drwxr-xr-x 2 grid oinstall 4096 Oct 30 13:21 hostb drwxr-xr-x 2 grid oinstall 4096 Oct 30 19:06 localhost drwxrwxr-x 2 grid oinstall 4096 Dec 22 13:27 hosta-cluster -rw------- 1 root oinstall 272756736 Jan 8 13:38 back_hostb.olr -rw-r--r-- 1 root root 10485760 Jan 12 18:08 hostb.olr [grid@hostb cdata]$
第二步：# /u01/app/11.2.0/grid/root.sh
[root@hostb install]# /u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-2672: Attempting to start 'ora.mdnsd' on 'hostb'
CRS-2676: Start of 'ora.mdnsd' on 'hostb' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'hostb'
CRS-2676: Start of 'ora.gpnpd' on 'hostb' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'hostb'
CRS-2672: Attempting to start 'ora.gipcd' on 'hostb'
CRS-2676: Start of 'ora.cssdmonitor' on 'hostb' succeeded
CRS-2676: Start of 'ora.gipcd' on 'hostb' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'hostb'
CRS-2672: Attempting to start 'ora.diskmon' on 'hostb'
CRS-2676: Start of 'ora.diskmon' on 'hostb' succeeded
CRS-2676: Start of 'ora.cssd' on 'hostb' succeeded
PRKO-2190 : VIP exists for node hostb, VIP name hostb-vip
PRCS-1037 : Single Client Access Name VIPs already exist
PRCS-1028 : Single Client Access Name listeners already exist
OC4J could not be created as it already exists
PRCR-1086 : resource ora.oc4j is already registered
PRCR-1086 : resource ora.cvu is already registered
PRCC-1014 : scan1 was already running
PRCR-1004 : Resource ora.scan1.vip is already running
PRCR-1079 : Failed to start resource ora.scan1.vip
CRS-5702: Resource 'ora.scan1.vip' is already running on 'hostb'
PRCC-1014 : LISTENER_SCAN1 was already running
PRCR-1004 : Resource ora.LISTENER_SCAN1.lsnr is already running
PRCR-1079 : Failed to start resource ora.LISTENER_SCAN1.lsnr
CRS-5702: Resource 'ora.LISTENER_SCAN1.lsnr' is already running on 'hostb'
PRCC-1014 : oc4j was already running
PRCR-1004 : Resource ora.oc4j is already running
PRCR-1079 : Failed to start resource ora.oc4j
CRS-5702: Resource 'ora.oc4j' is already running on 'hostb'
CRS-2672: Attempting to start 'ora.registry.acfs' on 'hostb'
CRS-2676: Start of 'ora.registry.acfs' on 'hostb' succeeded
PRCC-1014 : cvu was already running
PRCR-1004 : Resource ora.cvu is already running
PRCR-1079 : Failed to start resource ora.cvu
CRS-5702: Resource 'ora.cvu' is already running on 'hostb'
Preparing packages for installation...
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@hostb install]#
完成后的/u01/app/11.2.0/grid/cdata目录情况：
[grid@hostb cdata]$ ls -lrt
total 5612
drwxr-xr-x 2 grid oinstall 4096 Oct 30 19:06 localhost
drwxrwxr-x 2 grid oinstall 4096 Dec 22 13:27 hosta-cluster
-rw------- 1 root oinstall 272756736 Jan 8 13:38 back_hostb.olr
drwxr-xr-x 2 grid oinstall 4096 Jan 12 18:19 hostb
-rw------- 1 root oinstall 272756736 Jan 12 18:19 hostb.olr
[grid@hostb cdata]$
[grid@hostb cdata]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE hostb
ora.DATAFILE.dg
ONLINE ONLINE hostb
ora.FRA.dg
ONLINE ONLINE hostb
ora.LISTENER.lsnr
ONLINE ONLINE hostb
ora.asm
ONLINE ONLINE hostb Started
ora.gsd
OFFLINE OFFLINE hostb
ora.net1.Network ONLINE ONLINE hostb
ora.ons
ONLINE ONLINE hostb
ora.registry.acfs
ONLINE ONLINE hostb
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE hostb
ora.cvu
1 ONLINE ONLINE hostb
ora.orcl.db
1 ONLINE OFFLINE
2 ONLINE ONLINE hostb Open
ora.orcl.orclxa.svc
1 ONLINE ONLINE hostb
ora.orcla.vip
1 ONLINE INTERMEDIATE hostb FAILED OVER
ora.hostb.vip
1 ONLINE ONLINE hostb
ora.oc4j
1 ONLINE ONLINE hostb
ora.scan1.vip
1 ONLINE ONLINE hostb
[grid@hostb cdata]$
[grid@hostb cdata]$ lsnrctl status LSNRCTL for linux: version 11.2.0.3.0 - Production on 12-JAN-2016 18:21:34 Copyright (c) 1991, 2011, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))) STATUS of the LISTENER ------------------------ Alias LISTENER Version TNSLSNR for Linux: Version 11.2.0.3.0 - Production Start Date 12-JAN-2016 18:18:21 Uptime 0 days 0 hr. 3 min. 13 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /u01/app/11.2.0/grid/network/admin/listener.ora Listener Log File /u01/app/grid/diag/tnslsnr/hostb/listener/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.188.182)(PORT=1528))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.188.184)(PORT=1528))) Services Summary... Service "+ASM" has 1 instance(s). Instance "+ASM2", status READY, has 1 handler(s) for this service... Service "orcl" has 1 instance(s). Instance "orcl2", status READY, has 1 handler(s) for this service... Service "orclXDB" has 1 instance(s). Instance "orcl2", status READY, has 1 handler(s) for this service... Service "orclxa" has 1 instance(s). Instance "orcl2", status READY, has 1 handler(s) for this service... The command completed successfully [grid@hostb cdata]$
[oracle@hostb ~]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.3.0 Production on Tue Jan 12 18:29:52 2016 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options SQL> select open_mode from v$database; OPEN_MODE -------------------- READ WRITE
Oracle RAC无备份情况恢复ocr和olr
原文地址：http://www.2cto.com/database/201404/290110.html
今天是2014-04-01，忙碌了一天终于有时间写点东西了。前段时间写了RAc中ocr存在备份的情况下如何恢复，今天写一下在没有备份情况下如何去重建ocr和olr呢？
最大的悲剧莫过于数据库坏了备份没了。让我想起了前几年流行的一句话：“最大的悲剧是人活着，钱没了”。总之备份重于一切。
情景介绍：
在11G中ocr的asm磁盘组被用户不小心删了且所有备份都误删，olr呢也损坏了。至此悲剧重演。
恢复过程：（其实就是使用root.sh重建ocr的过程，重建之后可能需要重新注册相关资源如listener or database 实例等！）
1、清空所有节点cluster配置信息：（11G中存在rootcrs.sh脚本，该脚本默认存在于$GRID_HOME/crs/install）。
注，该脚本需要在root下执行，否则会有错误提示
[grid@rac-one install]$ ./rootcrs.pl
You must be logged in as root to run this script.
Log in as root and rerun this script.
2014-04-01 17:08:12: Not running as authorized user
Insufficient privileges to execute this script.
root or administrative privileges needed to run the script.
了解该脚本的功能可以查看-help。如升级、回退、重新配置等待》
[root@rac-two install]# ./rootcrs.pl -help
Usage:
rootcrs.pl [-verbose] [-upgrade [-force] | -patch]
[-paramfile ]
[-deconfig [-deinstall] [-keepdg] [-force] [-lastnode]]
[-downgrade -oldcrshome -version [-force] [-lastnode]]
[-unlock [-crshome ] [-nocrsstop]]
[-init]
Options:
-verbose Run this script in verbose mode
-upgrade Oracle HA is being upgraded from previous version
-patch Oracle HA is being upgraded to a patch version
-paramfile Complete path of file specifying HA parameter values
-lastnode Force the node this script is executing on to be considered
as the last node of deconfiguration or downgrade, and perform
actions associated with deconfiguring or downgrading the
last node
-downgrade Downgrade the clusterware
-version For use with downgrade; special handling is required if
downgrading to 9i. This is the old crs version in the format
A.B.C.D.E (e.g 11.1.0.6.0).
-deconfig Remove Oracle Clusterware to allow it to be uninstalled or reinstalled
-force Force the execution of steps in delete or dwongrade that cannot
be verified to be safe
-deinstall Reset the permissions on CRS home during de-configuration
-keepdg Keep existing diskgroups during de-configuration
-unlock Unlock CRS home
-crshome Complete path of crs home. Use with unlock option
-oldcrshome For use with downgrade. Complete path of the old crs home
-nocrsstop used with unlock option to reset permissions on an inactive grid home
-init Reset the permissions of all files and directories under CRS home
If neither -upgrade nor -patch is supplied, a new install is performed
To see the full manpage for this program, execute:
perldoc rootcrs.pl
[root@rac-two install]#
在root下清除所有节点cluster 信息：
节点2：
[grid@rac-one install]$ su
Password:
[root@rac-one install]# ./rootcrs.pl -deconfig -force
Using configuration parameter file: ./crsconfig_params
Network exists: 1/192.168.4.0/255.255.255.0/eth0, type static
VIP exists: /rac-one-vip/192.168.4.113/192.168.4.0/255.255.255.0/eth0, hosting node rac-one
VIP exists: /rac-two-vip/192.168.4.114/192.168.4.0/255.255.255.0/eth0, hosting node rac-two
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rac-one'
CRS-2677: Stop of 'ora.registry.acfs' on 'rac-one' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac-one'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac-one'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac-one'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rac-one'
CRS-2673: Attempting to stop 'ora.GIDG.dg' on 'rac-one'
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'rac-one'
CRS-2677: Stop of 'ora.DATADG.dg' on 'rac-one' succeeded
CRS-2677: Stop of 'ora.GIDG.dg' on 'rac-one' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac-one'
CRS-2677: Stop of 'ora.oc4j' on 'rac-one' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'rac-two'
CRS-2677: Stop of 'ora.asm' on 'rac-one' succeeded
CRS-2676: Start of 'ora.oc4j' on 'rac-two' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac-one' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac-one' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rac-one'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac-one'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac-one'
CRS-2673: Attempting to stop 'ora.asm' on 'rac-one'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac-one'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac-one'
CRS-2677: Stop of 'ora.crf' on 'rac-one' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac-one' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac-one' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac-one' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac-one' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac-one'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac-one' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac-one'
CRS-2677: Stop of 'ora.cssd' on 'rac-one' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac-one'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac-one' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'rac-one' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac-one'
CRS-2677: Stop of 'ora.gpnpd' on 'rac-one' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac-one' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Removing Trace File Analyzer
Successfully deconfigured Oracle clusterware stack on this node
[root@rac-one install]#
节点1，因为我的RAC是双节点的，那么在清除最后一个使用lastnode参数：
[grid@rac-two crs]$ cd install/
[grid@rac-two install]$ su
Password:
[root@rac-two install]# ./rootcrs.pl -deconfig -force -lastnode
Using configuration parameter file: ./crsconfig_params
CRS resources for listeners are still configured
Network exists: 1/192.168.4.0/255.255.255.0/eth0, type static
VIP exists: /rac-two-vip/192.168.4.114/192.168.4.0/255.255.255.0/eth0, hosting node rac-two
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rac-two'
CRS-2677: Stop of 'ora.registry.acfs' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.crsd' on 'rac-two'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac-two'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rac-two'
CRS-2673: Attempting to stop 'ora.GIDG.dg' on 'rac-two'
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'rac-two'
CRS-2677: Stop of 'ora.DATADG.dg' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.GIDG.dg' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac-two'
CRS-2677: Stop of 'ora.asm' on 'rac-two' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac-two' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac-two'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac-two'
CRS-2673: Attempting to stop 'ora.asm' on 'rac-two'
CRS-2677: Stop of 'ora.evmd' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac-two'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac-two'
CRS-2677: Stop of 'ora.cssd' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac-two'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac-two'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac-two'
CRS-2676: Start of 'ora.diskmon' on 'rac-two' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac-two' succeeded
CRS-4611: Successful deletion of voting disk +GIDG.
ASM de-configuration trace file location: /tmp/asmcadc_clean2014-04-01_05-14-52-PM.log
ASM Clean Configuration START
ASM Clean Configuration END
ASM with SID +ASM1 deleted successfully. Check /tmp/asmcadc_clean2014-04-01_05-14-52-PM.log for details.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac-two'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac-two'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac-two'
CRS-2673: Attempting to stop 'ora.asm' on 'rac-two'
CRS-2677: Stop of 'ora.mdnsd' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac-two'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac-two'
CRS-2677: Stop of 'ora.cssd' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rac-two'
CRS-2677: Stop of 'ora.crf' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac-two'
CRS-2677: Stop of 'ora.gipcd' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac-two'
CRS-2677: Stop of 'ora.gpnpd' on 'rac-two' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac-two' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Removing Trace File Analyzer
Successfully deconfigured Oracle clusterware stack on this node
[root@rac-two install]#
二、重建ocr和olr,使用root.sh脚本完成重建，其实这就是在安装RAC中执行的脚本，默认位置为：$GRID_HOME/;
eg：
节点1：
[root@rac-two grid]# ./root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac-two'
CRS-2676: Start of 'ora.mdnsd' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac-two'
CRS-2676: Start of 'ora.gpnpd' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac-two'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac-two'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac-two' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac-two'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac-two'
CRS-2676: Start of 'ora.diskmon' on 'rac-two' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac-two' succeeded
ASM created and started successfully.
Disk Group GIDG created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Successful addition of voting disk 498646ba39604f86bf697c9748a67697.
Successful addition of voting disk 2e1bd16f9e6d4f36bf93550dc8268725.
Successful addition of voting disk 3fbd31a0b2634feabfa1115a504cbbe6.
Successfully replaced voting disk group with +GIDG.
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 498646ba39604f86bf697c9748a67697 (/dev/asm-diske) [GIDG]
2. ONLINE 2e1bd16f9e6d4f36bf93550dc8268725 (/dev/asm-diskd) [GIDG]
3. ONLINE 3fbd31a0b2634feabfa1115a504cbbe6 (/dev/asm-diskf) [GIDG]
Located 3 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'rac-two'
CRS-2676: Start of 'ora.asm' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.GIDG.dg' on 'rac-two'
CRS-2676: Start of 'ora.GIDG.dg' on 'rac-two' succeeded
Preparing packages for installation...
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@rac-two grid]#
另外注意，如果asm磁盘损坏了那么需要重新修复磁盘，然后会自动重新创建之前的diskgroup.
至此olr和ocr创建成功
节点2：
[root@rac-one grid]# ./root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac-two, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Preparing packages for installation...
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@rac-one grid]#
3、检查资源信息：
现在呢开始使用srvctl和crsctl工具，这是oracle RAC 经常使用的，另外还有一个工具是oifcfg配置网卡信息等。不在 介绍了。
[root@rac-one bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@rac-one bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.GIDG.dg
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.asm
ONLINE ONLINE rac-one Started
ONLINE ONLINE rac-two Started
ora.gsd
OFFLINE OFFLINE rac-one
OFFLINE OFFLINE rac-two
ora.net1.network
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.ons
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.registry.acfs
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac-one
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac-two
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac-two
ora.cvu
1 ONLINE ONLINE rac-two
ora.oc4j
1 ONLINE ONLINE rac-two
ora.rac-one.vip
1 ONLINE ONLINE rac-one
ora.rac-two.vip
1 ONLINE ONLINE rac-two
ora.scan1.vip
1 ONLINE ONLINE rac-one
ora.scan2.vip
1 ONLINE ONLINE rac-two
ora.scan3.vip
1 ONLINE ONLINE rac-two
[root@rac-one bin]#
[root@rac-one bin]# su - grid
[grid@rac-one ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Tue Apr 1 17:40:53 2014
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> select name,state from v$asm_diskgroup;
NAME STATE
------------------------------ -----------
GIDG MOUNTED
DATADG DISMOUNTED
SQL> alter diskgroup datadg mount;
Diskgroup altered.
SQL>
4、添加资源
可以看到目前启动的缺少本地监听和database资源和实例，那么下面开始重新注册这些信息到ocr中，
注册listener：
注：使用grid用户：
[grid@rac-one ~]$ srvctl add listener -l listener
PRCN-2061 : Failed to add listener ora.LISTENER.lsnr
PRCN-2065 : Port(s) 1521 are not available on the nodes given
PRCN-2067 : Port 1521 is not available across node(s) "rac-two-vip"
[grid@rac-one ~]$
提示1521该端口被ora.rac-two.vip所使用，暂且跳过。
[grid@rac-one ~]$ crsctl stop resource ora.rac-two.vip
CRS-2673: Attempting to stop 'ora.rac-two.vip' on 'rac-two'
CRS-2677: Stop of 'ora.rac-two.vip' on 'rac-two' succeeded
[grid@rac-one ~]$ srvctl add listener -l listener
[grid@rac-one ~]$ crsctl start resource ora.rac-two.vip
CRS-2672: Attempting to start 'ora.rac-two.vip' on 'rac-two'
CRS-2676: Start of 'ora.rac-two.vip' on 'rac-two' succeeded
[grid@rac-one ~]$ srvctl config listener
Name: LISTENER
Network: 1, Owner: grid
Home:
End points: TCP:1521
[grid@rac-one ~]$
[grid@rac-one ~]$
[grid@rac-one ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
ONLINE ONLINE rac-one
OFFLINE OFFLINE rac-two
ora.GIDG.dg
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.LISTENER.lsnr
OFFLINE OFFLINE rac-one
OFFLINE OFFLINE rac-two
ora.asm
ONLINE ONLINE rac-one Started
ONLINE ONLINE rac-two Started
ora.gsd
OFFLINE OFFLINE rac-one
OFFLINE OFFLINE rac-two
ora.net1.network
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.ons
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.registry.acfs
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac-one
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac-two
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac-two
ora.cvu
1 ONLINE ONLINE rac-two
ora.oc4j
1 ONLINE ONLINE rac-two
ora.rac-one.vip
1 ONLINE ONLINE rac-one
ora.rac-two.vip
1 ONLINE ONLINE rac-two
ora.scan1.vip
1 ONLINE ONLINE rac-one
ora.scan2.vip
1 ONLINE ONLINE rac-two
ora.scan3.vip
1 ONLINE ONLINE rac-two
[grid@rac-one ~]$
至此监听添加到ocr完毕。下面开始添加db，注意使用oracle用户：
继续添加：
[grid@rac-one ~]$ su - oracle
Password:
[oracle@rac-one ~]$ srvctl add database -h
Adds a database configuration to the Oracle Clusterware.
Usage: srvctl add database -d -o [-c {RACONENODE | RAC | SINGLE} [-e ] [-i ] [-w ]] [-m ] [-p ] [-r {PRIMARY | PHYSICAL_STANDBY | LOGICAL_STANDBY | SNAPSHOT_STANDBY}] [-s ] [-t ] [-n ] [-y {AUTOMATIC | MANUAL | NORESTART}] [-g ""] [-x ] [-a ""] [-j ""]
-d Unique name for the database
-o ORACLE_HOME path
-c Type of database: RAC One Node, RAC, or Single Instance
-e Candidate server list for RAC One Node database
-i Instance name prefix for administrator-managed RAC One Node database (default first 12 characters of )
-w Online relocation timeout in minutes
-x Node name. -x option is specified for single-instance databases
-m Domain for database. Must be set if database has DB_DOMAIN set.
-p Server parameter file path
-r Role of the database (primary, physical_standby, logical_standby, snapshot_standby)
-s Startup options for the database. Examples of startup options are OPEN, MOUNT, or 'READ ONLY'.
-t Stop options for the database. Examples of shutdown options are NORMAL, TRANSACTIONAL, IMMEDIATE, or ABORT.
-n Database name (DB_NAME), if different from the unique name given by the -d option
-y Management policy for the database (AUTOMATIC, MANUAL, or NORESTART)
-g "" Comma separated list of database server pool names
-a "" Comma separated list of disk groups
-j "" Comma separated list of ACFS paths where database's dependency will be set
-h Print usage
[oracle@rac-one ~]$ srvctl add database -d Rac -o /u01/app/oracle/product/11.2.0/db_1/ -c RAC
[oracle@rac-one ~]$
[oracle@rac-one ~]$ srvctl add instance -h
Adds a database instance configuration to the Oracle Clusterware.
Usage: srvctl add instance -d -i -n [-f]
-d Unique name for the database
-i Instance name
-n Node name
-f Force the add operation even though some resource(s) will be stopped
-h Print usage
[oracle@rac-one ~]$ srvctl add instance -d Rac -i Rac1 -n rac-two
[oracle@rac-one ~]$ srvctl add instance -d Rac -i Rac2 -n rac-one
[oracle@rac-one ~]$
[oracle@rac-one ~]$ srvctl config database -d Rac
Database unique name: Rac
Database name:
Oracle home: /u01/app/oracle/product/11.2.0/db_1/
Oracle user: oracle
Spfile:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: Rac
Database instances: Rac2,Rac1
Disk Groups:
Mount point paths:
Services:
Type: RAC
Database is administrator managed
[oracle@rac-one ~]$
重启crs。
[root@rac-one bin]# ./crsctl start cluster -all
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac-one'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac-two'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac-one' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac-two'
CRS-2672: Attempting to start 'ora.cssd' on 'rac-one'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac-one'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac-two'
CRS-2676: Start of 'ora.diskmon' on 'rac-one' succeeded
CRS-2676: Start of 'ora.diskmon' on 'rac-two' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac-two' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac-one' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac-two'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac-one'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac-one'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac-two'
CRS-2676: Start of 'ora.ctssd' on 'rac-one' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rac-one'CRS-2676: Start of 'ora.ctssd' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rac-two'
CRS-2676: Start of 'ora.evmd' on 'rac-one' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac-two' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac-one' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac-one'
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac-two'
CRS-2676: Start of 'ora.asm' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac-two'
CRS-2676: Start of 'ora.crsd' on 'rac-two' succeeded
CRS-2676: Start of 'ora.asm' on 'rac-one' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac-one'
CRS-2676: Start of 'ora.crsd' on 'rac-one' succeeded
再次查看（等2分钟）：
[root@rac-one bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.GIDG.dg
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.LISTENER.lsnr
ONLINE ONLINE rac-one
ONLINE INTERMEDIATE rac-two Not All Endpoints R
egistered
ora.asm
ONLINE ONLINE rac-one Started
ONLINE ONLINE rac-two Started
ora.gsd
OFFLINE OFFLINE rac-one
OFFLINE OFFLINE rac-two
ora.net1.network
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.ons
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.registry.acfs
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac-one
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac-two
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac-two
ora.cvu
1 ONLINE ONLINE rac-two
ora.oc4j
1 ONLINE ONLINE rac-two
ora.rac-one.vip
1 ONLINE ONLINE rac-one
ora.rac-two.vip
1 ONLINE ONLINE rac-two
ora.rac.db
1 ONLINE ONLINE rac-two Open
2 ONLINE ONLINE rac-one Open
ora.scan1.vip
1 ONLINE ONLINE rac-one
ora.scan2.vip
1 ONLINE ONLINE rac-two
ora.scan3.vip
1 ONLINE ONLINE rac-two
为什么会存在一个监听在rac-two上无法注册呢？
查看该节点监听：
[grid@rac-two admin]$ ps -ef | grep LISTENER
grid 6120 1 0 17:02 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid 8511 1 0 18:28 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
grid 11010 1 0 18:42 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid 11246 9191 0 18:44 pts/1 00:00:00 grep LISTENER
原来是启动了两个，全部停掉再次重启：
[grid@rac-two admin]$ id
uid=1100(grid) gid=1000(oinstall) groups=1000(oinstall),1100(asmadmin),1200(dba),1300(asmdba),1301(asmoper)
[grid@rac-two admin]$ lsnrctl stop
LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 01-APR-2014 18:45:11
Copyright (c) 1991, 2013, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
The command completed successfully
[grid@rac-two admin]$ ps -ef | grep LISTENER
grid 6120 1 0 17:02 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid 8511 1 0 18:28 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
grid 11452 9191 0 18:45 pts/1 00:00:00 grep LISTENER
[grid@rac-two admin]$ exit
logout
[root@rac-two bin]# kill -9 6120
[root@rac-two bin]# ps -ef | grep LISTENER
grid 8511 1 0 18:28 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
root 11518 11280 0 18:46 pts/1 00:00:00 grep LISTENER
[root@rac-two bin]# su - grid
[grid@rac-two ~]$ crsctl status res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.GIDG.dg
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.LISTENER.lsnr
ONLINE ONLINE rac-one
OFFLINE OFFLINE rac-two
ora.asm
ONLINE ONLINE rac-one Started
ONLINE ONLINE rac-two Started
ora.gsd
OFFLINE OFFLINE rac-one
OFFLINE OFFLINE rac-two
ora.net1.network
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.ons
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.registry.acfs
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac-two
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac-one
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac-one
ora.cvu
1 ONLINE ONLINE rac-one
ora.oc4j
1 ONLINE ONLINE rac-one
ora.rac-one.vip
1 ONLINE ONLINE rac-one
ora.rac-two.vip
1 ONLINE ONLINE rac-two
ora.rac.db
1 ONLINE ONLINE rac-two Open
2 ONLINE ONLINE rac-one Open
ora.scan1.vip
1 ONLINE ONLINE rac-two
ora.scan2.vip
1 ONLINE ONLINE rac-one
ora.scan3.vip
1 ONLINE ONLINE rac-one
[grid@rac-two ~]$ crsctl start resource ora.LISTENER.lsnr
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'rac-two'
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'rac-two' succeeded
[grid@rac-two ~]$
[grid@rac-two ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.GIDG.dg
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.LISTENER.lsnr
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.asm
ONLINE ONLINE rac-one Started
ONLINE ONLINE rac-two Started
ora.gsd
OFFLINE OFFLINE rac-one
OFFLINE OFFLINE rac-two
ora.net1.network
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.ons
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.registry.acfs
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac-two
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac-one
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac-one
ora.cvu
1 ONLINE ONLINE rac-one
ora.oc4j
1 ONLINE ONLINE rac-one
ora.rac-one.vip
1 ONLINE ONLINE rac-one
ora.rac-two.vip
1 ONLINE ONLINE rac-two
ora.rac.db
1 ONLINE ONLINE rac-two Open
2 ONLINE ONLINE rac-one Open
ora.scan1.vip
1 ONLINE ONLINE rac-two
ora.scan2.vip
1 ONLINE ONLINE rac-one
ora.scan3.vip
1 ONLINE ONLINE rac-one
[grid@rac-two ~]$
至此所有问题得到彻底解决。
Oracle database 11g rac损坏ocr和votedisk恢复实验
本人的操作环境：oracle database rdbms 11g rac on OEL5.5
查看一下表决磁盘和ocr的信息：
[root@rac1 bin]# pwd
/u01/app/11.2.0/grid/bin
[root@rac1 bin]# ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 5122b184495d4fe9bf1fad29647807ba (ORCL:VOL1) [OCRVOTI]
Located 1 voting disk(s).
[root@rac1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2700
Available space (kbytes) : 259420
ID : 268167937
Device/File Name : +OCRVOTI
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
查看当前ocr备份情况，ocr
[root@rac1 bin]# ./ocrconfig -showbackup
rac2 2014/03/25 12:04:28 /u01/app/11.2.0/grid/cdata/rac-cluster/backup00.ocr
rac2 2014/03/21 16:16:32 /u01/app/11.2.0/grid/cdata/rac-cluster/backup01.ocr
rac2 2014/03/21 12:16:31 /u01/app/11.2.0/grid/cdata/rac-cluster/backup02.ocr
rac2 2014/03/25 12:04:28 /u01/app/11.2.0/grid/cdata/rac-cluster/day.ocr
rac2 2014/03/19 14:26:16 /u01/app/11.2.0/grid/cdata/rac-cluster/week.ocr
可以进行手工备份：
[root@rac1 bin]# ./ocrconfig -local -manualbackup
rac1 2014/03/25 14:33:39 /u01/app/11.2.0/grid/cdata/rac1/backup_20140325_143339.olr
rac1 2014/03/25 10:34:33 /u01/app/11.2.0/grid/cdata/rac1/backup_20140325_103433.olr
rac1 2014/03/25 09:53:32 /u01/app/11.2.0/grid/cdata/rac1/backup_20140325_095332.olr
rac1 2014/03/25 09:53:18 /u01/app/11.2.0/grid/cdata/rac1/backup_20140325_095318.olr
rac1 2014/03/18 10:57:51 /u01/app/11.2.0/grid/cdata/rac1/backup_20140318_105751.olr
在asmcmd的md_backup命令备份磁盘组，顺便查看该磁盘组都存放什么？？
[grid@rac1 ~]$ asmcmd -p
ASMCMD [+] > md_backup /home/grid/ocrvote2.bak -G OCRVOTI
Disk group metadata to be backed up: OCRVOTI
Current alias directory path: rac-cluster
Current alias directory path: rac-cluster/ASMPARAMETERFILE
Current alias directory path: rac-cluster/OCRFILE
也可以手工导出ocr内容
[root@rac1 bin]# ./ocrconfig -export /home/grid/ocr2.bak
我们可以破坏存放ocr的设备文件
[root@rac1 bin]# dd if=/dev/zero of=/dev/sdg bs=1024k count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.002366 seconds, 443 MB/s
然哈停止集群：
[root@rac1 bin]# ./crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'
CRS-2673: Attempting to stop 'ora.OCRVOTI.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rac1'
CRS-2673: Attempting to stop 'ora.test.db' on 'rac1'
CRS-2673: Attempting to stop 'ora.gsd' on 'rac1'
CRS-2677: Stop of 'ora.gsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' su 忽略。。。。。
我们在启动clusterware 发现无法启动了
[root@rac1 bin]# ./crsctl start has
CRS-4123: Oracle High Availability Services has been started.
[root@rac1 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
ocr和vote disk损坏恢复步骤大致如下：
1）停止所有节点clusterware
# crsctl stop crs
# crsctl stop crs -f
2）以root用户在其中一个节点独占模式启动clusterware
# crsctl start crs -excl -nocrs
备注：如果发现crsd在运行，那么通过如下命令将之停止。
# crsctl stop resource ora.crsd -init
3）创建新的存放ocr和vote disk的磁盘组，磁盘组名和原有的一致(如果想改变位置，需修改/etc/oracle/ocr.loc文件)
备注：如发现无法创建等情况，可以采用如下删除磁盘组等排错思路
SQL> drop diskgroup disk_group_name force including contents;
4）还原ocr，并检查
# ocrconfig -restore file_name
# ocrcheck
5）恢复表决磁盘，并检查
# crsctl replace votedisk +asm_disk_group
# crsctl query css votedisk
6）停止独占模式运行的clusterware
# crsctl stop crs -f
7）所有节点正常启动clusterware
# crsctl start crs
8）CVU验证所有RAC节点OCR的完整性
$ cluvfy comp ocr -n all -verbose
About Me
...............................................................................................................................● 本文整理自网络
● 本文在itpub（http://blog.itpub.net/26736162）、博客园（http://www.cnblogs.com/lhrbest）和个人微信公众号（xiaomaimiaolhr）上有同步更新
● 本文itpub地址：http://blog.itpub.net/26736162/abstract/1/
● 本文博客园地址：http://www.cnblogs.com/lhrbest
● 本文pdf版及小麦苗云盘地址：http://blog.itpub.net/26736162/viewspace-1624453/
● 数据库笔试面试题库及解答：http://blog.itpub.net/26736162/viewspace-2134706/
● QQ群：230161599 微信群：私聊
● 联系我请加QQ好友(646634621)，注明添加缘由
● 于 2017-06-02 09:00 ~ 2017-06-30 22:00 在魔都完成
● 文章内容来源于小麦苗的学习笔记，部分整理自网络，若有侵权或不当之处还请谅解
● 版权所有，欢迎分享本文，转载请保留出处
...............................................................................................................................
拿起手机使用微信客户端扫描下边的左边图片来关注小麦苗的微信公众号：xiaomaimiaolhr，扫描右边的二维码加入小麦苗的QQ群，学习最实用的数据库技术。
