[20150529]ORA-16664.txt
--今天在例行检查中.执行 dgmgrl 出现ora-16664错误.
--检查drcdbcn1.log文件,发现如下错误.
05/25/2015 16:34:58
Site dbcndg returned ORA-16664.
Site dbcndg2 returned ORA-16664.
Data Guard Broker Status Summary:
Type Name Severity Status
Configuration dbcn Warning ORA-16607
Primary Database dbcn Success ORA-00000
Physical Standby Database dbcndg Error ORA-16664
Physical Standby Database dbcndg2 Error ORA-16664
$ oerr ora 16664
16664, 0000, "unable to receive the result from a database"
// *Cause: During execution of a command, a database in the Data Guard
// broker configuration failed to return a result.
// *Action: Check Data Guard broker logs for the details of the failure.
// Ensure network communication is working properly amongst the
// members of the configuration. Fix any possible network problems
// and reissue the command.
--检查相应dg的drcdbcndg2.log文件:
05/29/2015 15:18:40
Connection to database dbcn returns ORA-01017.
Please check database dbcn is using a remote password file,
its remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and the SYS password is the same as this database.
Failed to connect to remote database dbcn. Error is ORA-01017
Failed to send message to site dbcn. Error code is ORA-01017.
--很明显按照提示,数据库的sys口令被修改了.而实际上的情况是dg的日志应用是正常的,why什么呢?
--实际上口令根本没有改变.
--只好建立新的口令文件,再检查问题消失!(我一般不建立口令文件,而是从服务器拷贝)
$ xxd -c 16 orapwdbcndg2.xxx >| /tmp/a1.txt
$ xxd -c 16 orapwdbcndg2 >| /tmp/a2.txt
$ diff /tmp/a1.txt /tmp/a2.txt
43,45c43,45
---
> 00002a0: 0000 0000 1000 0000 1f6c 6a1d 373c 0d87 .........lj.7> 00002b0: 341a a3ce a467 0b30 319c 0026 9f40 8dcb 4..?g.01..&.@.?
> 00002c0: 7759 8363 f8eb 2c00 5359 5300 0000 0000 wY.c?,.SYS.....
50,51c50,51
---
> 0000310: 1f7c e9a6 313b f7cf 3149 da0b b16d 01a3 .|?1;.?I?.m..
> 0000320: 9cdb 8e8c 20ad 5153 2695 e166 c132 d300 .?. .QS&.醘??
--才想起来11G可能前几天修改profile,也修改了口令.
--恢复后我重新修改sys的口令(还是原来的密码),我发现即使我修改成原来的口令,马上出现:
DGMGRL> show configuration
Configuration - dbcn
Protection Mode: MaxPerformance
Databases:
dbcn - Primary database
dbcndg - Physical standby database
Error: ORA-16664: unable to receive the result from a database
dbcndg2 - Physical standby database
Error: ORA-16664: unable to receive the result from a database
Fast-Start Failover: DISABLED
Configuration Status:
ERROR
$ ll orapwdbcn1
-rw-r----- 1 oracle oinstall 1536 2015-05-29 15:53:40 orapwdbcn1
$ ll orapwdbcn2
-rw-r----- 1 oracle oinstall 1536 2014-10-28 17:04:04 orapwdbcn2
$ stat orapwdbcn1
File: `orapwdbcn1'
Size: 1536 Blocks: 8 IO Block: 4096 regular file
Device: fc03h/64515d Inode: 10698784 Links: 1
Access: (0640/-rw-r-----) Uid: ( 1001/ oracle) Gid: ( 1001/oinstall)
Access: 2015-05-29 16:03:03.000000000 +0800
Modify: 2015-05-29 15:53:40.000000000 +0800
Change: 2015-05-29 16:03:03.000000000 +0800
$ stat orapwdbcn2
File: `orapwdbcn2'
Size: 1536 Blocks: 8 IO Block: 4096 regular file
Device: fc03h/64515d Inode: 5996564 Links: 1
Access: (0640/-rw-r-----) Uid: ( 1001/ oracle) Gid: ( 1001/oinstall)
Access: 2015-05-29 16:03:40.000000000 +0800
Modify: 2014-10-28 17:04:04.000000000 +0800
Change: 2015-05-29 16:03:40.000000000 +0800
--还发现一个bug,这样修改口令文件,仅仅在一个实例上修改.oracle真tmd的变态!
--再做一个检查就知道了,拷贝口令文件到dg后:
--在RAC实例1上执行:
DGMGRL> show configuration
Configuration - dbcn
Protection Mode: MaxPerformance
Databases:
dbcn - Primary database
dbcndg - Physical standby database
dbcndg2 - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS
--在RAC实例2上执行:
DGMGRL> show configuration
Configuration - dbcn
Protection Mode: MaxPerformance
Databases:
dbcn - Primary database
dbcndg - Physical standby database
Error: ORA-16664: unable to receive the result from a database
dbcndg2 - Physical standby database
Error: ORA-16664: unable to receive the result from a database
Fast-Start Failover: DISABLED
Configuration Status:
ERROR
--我把实例1的口令文件拷贝到实例2:
--在RAC实例2上执行:
DGMGRL> show configuration
Configuration - dbcn
Protection Mode: MaxPerformance
Databases:
dbcn - Primary database
dbcndg - Physical standby database
dbcndg2 - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS
--我把dg在使用原来的口令文件,问题再次出现!
--看来千万不要乱改sys口令.
--在各个机器检查一遍,dgmgrl显示都是正常了.
--这个问题到底是什么原因引起的,要找测试环境来测试看看.