前段时间配的一套11203 RAC ADG on EXADATA Machine的环境，在ADG 的standby side的node2 通过dblink查询时提示ora-1089错误，但是在node1 查询正常，DG的recover 进程是在node1 上，后确认是个bug 简单的记录。
-- on standby node2 ,but on standby node1 was worked
select count(*) from dbmt.dbmt_tableinfo@LINK_WEEJAR_A2.ANBOB.COM;
ORA-01089: immediate shutdown in progress - no operations are permitted
Process ID: 771
Session ID: 1517 Serial number: 4211
-- diag
alter session set events '10046 trace name context forever,level 12:1089 trace name errorstack level 3';
select count(*) from dbmt.dbmt_tableinfo@LINK_WEEJAR_A2.ANBOB.COM;
--error
Call Stacke in trace file
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
skdstdst()+36 call kgdsdst() 000000000 ? 000000000 ?
7FFFB26EA518 ? 000000001 ?
000000001 ? 000000002 ?
ksedst1()+98 call skdstdst() 000000000 ? 000000000 ?
7FFFB26EA518 ? 000000001 ?
000000000 ? 000000002 ?
ksedst()+34 call ksedst1() 000000000 ? 000000001 ?
7FFFB26EA518 ? 000000001 ?
000000000 ? 000000002 ?
dbkedDefDump()+2741 call ksedst() 000000000 ? 000000001 ?
7FFFB26EA518 ? 000000001 ?
000000000 ? 000000002 ?
ksedmp()+36 call dbkedDefDump() 000000003 ? 000000000 ?
7FFFB26EA518 ? 000000001 ?
000000000 ? 000000002 ?
dbkdaKsdActDriver() call ksedmp() 000000003 ? 000000000 ?
+1960 7FFFB26EA518 ? 000000001 ?
000000000 ? 000000002 ?
dbgdaExecuteAction( call dbkdaKsdActDriver() 7F86B4312710 ? 7FFFB26F11A0 ?
)+1065 7FFFB26EA518 ? 000000001 ?
000000000 ? 000000002 ?
dbgdaRunAction()+81 call dbgdaExecuteAction( 7F86B4312710 ? 00A1B8DA0 ?
0 ) 0020C0003 ? 7FFFB26F11A0 ?
000000001 ? 000000002 ?
dbgdRunActions()+59 call dbgdaRunAction() 7F86B4312710 ? 000000005 ?
0020C0003 ? 7FFFB26F11A0 ?
000000001 ? 000000002 ?
dbgdProcessEventAct call dbgdRunActions() 7F86B4312710 ? 000000005 ?
ions()+651 0020C0003 ? 7FFFB26F11A0 ?
000000001 ? 000000002 ?
dbgdChkEventKgErr() call dbgdProcessEventAct 7F86B4312710 ? 00BC1D9A0 ?
在MOS中Bug 17162712相符，该bug 影响11.2.0.3/4 在12.2 (Future Release) 12.1.0.2 (Server Patch Set) 版本中才修改了bug;或在不升级到12C前的版本有提供ONE patch 小补丁修复; 或重启standby db 的方式绕过这个bug,如下
This bug is only relevant when using Real Application Clusters (RAC) and Database Link / Distributed
If the recovering instance of a RAC standby fails, one of the remaining instances will do special instance recovery.
After this recovery one of the shutdown flags is not cleared, triggering the ORA-1089:”immediate shutdown in progress”
when a database link is used.
During the recovery an ORA-1089 is normal, the problem is that the error is raised even after the recovery
STACK TRACE:
————
skdstdst <- ksedst1 <- ksedst <- dbkedDefDump <- ksedmp <- dbkdaKsdActDriver
<- dbgdaExecuteAction <- dbgdaRunAction <- dbgdRunActions <-
dbgdProcessEventActions <- dbgdChkEventKgErr <- dbkdChkEventRdbmsEr <- ksfpec
<- dbgePostErrorKGE <- 1129 <- dbkePostKGE_kgsf <- kgeselv <- ksesecl0 <-
k2gInsert <- k2lbeg <- k2sibg <- npibeg <- kpnpre <- upirtrc <- kpurcsc <-
kpuexec <- OCIStmtExecute <- OCIKGetDescInfo <- ddfnetCFull <- ddfnet2Normal
<- kkmfcbrm <- kkmpfcbk <- qcsprfro <- qcsprfro_tree <- qcsprfro_tree <-
qcspafq <- qcspqbDescendents <- qcspqb <- kkmdrv <- opiSem <- opiprs <-
kksParseChildCursor <- rpiswu2 <- kksLoadChild <- kxsGetRuntimeLock <- kksfbc
<- kkspsc0 <- kksParseCursor <- opiosq0 <- kpooprx <- kpoal8 <- opiodr <-
ttcpip <- opitsk <- opiino <- opiodr <- opidrv <- sou2o <- opimai_real <-
ssthrdmain <- main <- libc_start_main <- start
# 重启STANDBY INSTANCE
[oracle@qdexa1db01 (orarpt1)oracle]$ srvctl config database
rptstby
[oracle@qdexa1db01 (orarpt1)oracle]$ srvctl stop database -d rptstby
[oracle@qdexa1db01 (orarpt1)oracle]$ srvctl start database -d rptstby
SQL> select name,open_mode,database_role from gv$database;
NAME OPEN_MODE DATABASE_ROLE
--------- -------------------- ----------------
ORARPT MOUNTED PHYSICAL STANDBY
ORARPT MOUNTED PHYSICAL STANDBY
# open all standby instances
SQL> alter database open;
SQL> select name,open_mode,database_role from gv$database;
NAME OPEN_MODE DATABASE_ROLE
--------- -------------------- ----------------
ORARPT READ ONLY PHYSICAL STANDBY
ORARPT READ ONLY PHYSICAL STANDBY
SQL> alter database recover managed standby database PARALLEL 48 using current logfile disconnect from session;
Database altered.
# in both instances run same queries over dblink not with ora-1089 error also Now.
SQL> select count(*) from dbmt.dbmt_tableinfo@LINK_WEEJAR_A2.ANBOB.COM;
COUNT(*)
----------
317