ORACLE的告警日志里面包含许多有用的信息,尤其是一些ORACLE的ORA错误信息,所以有必要及时归档、监控数据库告警日志的ORA错误,及时提醒数据库管理员DBA处理这些错误信息,那么我们首先来看看告警日志的内容片断:
Thread 1 advanced to log sequence 37749 (LGWR switch)
Current log# 6 seq# 37749 mem# 0: /u01/oradata/SCM2/redo06.log
Thu Jun 27 15:02:30 2013
Thread 1 advanced to log sequence 37750 (LGWR switch)
Current log# 2 seq# 37750 mem# 0: /u01/oradata/SCM2/redo02.log
Thu Jun 27 15:13:43 2013
Thread 1 advanced to log sequence 37751 (LGWR switch)
Current log# 3 seq# 37751 mem# 0: /u01/oradata/SCM2/redo03.log
Thu Jun 27 15:25:30 2013
Thread 1 advanced to log sequence 37752 (LGWR switch)
Current log# 4 seq# 37752 mem# 0: /u01/oradata/SCM2/redo04.log
Thu Jun 27 15:32:20 2013
ORA-00060: Deadlock detected. More info in file /u01/app/oracle/admin/SCM2/bdump/scm2_s001_14052.trc.
Thu Jun 27 15:35:05 2013
Thread 1 advanced to log sequence 37753 (LGWR switch)
Current log# 5 seq# 37753 mem# 0: /u01/oradata/SCM2/redo05.log
Thu Jun 27 15:43:11 2013
Thread 1 advanced to log sequence 37754 (LGWR switch)
Current log# 1 seq# 37754 mem# 0: /u01/oradata/SCM2/redo01.log
Thu Jun 27 15:49:58 2013
Thread 1 advanced to log sequence 37755 (LGWR switch)
Current log# 6 seq# 37755 mem# 0: /u01/oradata/SCM2/redo06.log
Thu Jun 27 16:01:25 2013
Thread 1 advanced to log sequence 37756 (LGWR switch)
Current log# 2 seq# 37756 mem# 0: /u01/oradata/SCM2/redo02.log
Thu Jun 27 16:12:14 2013
Thread 1 advanced to log sequence 37757 (LGWR switch)
Current log# 3 seq# 37757 mem# 0: /u01/oradata/SCM2/redo03.log
Thu Jun 27 16:24:10 2013
Thread 1 advanced to log sequence 37758 (LGWR switch)
归档告警日志文件
告警日志文件如果不加管理的话,那么文件会持续增长,有时候文件会变得非常大,不利于读写。一般建议将告警日志按天归档,归档文件保留三个月(视情况而定),下面来看看将告警日志文件归档的两个Shell脚本:
alert_log_archive.sh version 1
- #*************************************************************************
- # FileName :alert_log_archive.sh
- #*************************************************************************
- # Author :Kerry
- # CreateDate :2013-07-02
- # blogs :www.cnblogs.com/kerrycode
- # Description :this script is made the alert log archived every day
- #*************************************************************************
- #! /bin/bash
- date=`date +%Y%m%d`
- alert_log_path="$ORACLE_BASE/admin/$ORACLE_SID/bdump"
- alert_log_file="alert_$ORACLE_SID.log"
- alert_arc_file="alert_$ORACLE_SID.log""."${date}
- cd ${alert_log_path};
- if [ ! -e "${alert_log_file}" ]; then
- echo "the alert log didn't exits, please check file path is correct!";
- exit;
- fi
- if [ -e ${alert_arc_file} ];then
- echo "the alert log file have been archived!"
- else
- cat ${alert_log_file} >> ${alert_arc_file}
- cat /dev/null > ${alert_log_file}
- fi
其实脚本1和脚本差别不大,仅仅是mv与cat >>的区别
alert_log_archive.sh version 2
- #*************************************************************************
- # FileName :alert_log_archive.sh
- #*************************************************************************
- # Author :Kerry
- # CreateDate :2013-07-02
- # blogs :www.cnblogs.com/kerrycode
- # Description :this script is made the alert log archived every day
- #*************************************************************************
- #! /bin/bash
- date=`date +%Y%m%d`
- alert_log_path="$ORACLE_BASE/admin/$ORACLE_SID/bdump"
- alert_log_file="alert_$ORACLE_SID.log"
- alert_arc_file="alert_$ORACLE_SID.log""."${date}
- cd ${alert_log_path};
- if [ ! -e "${alert_log_file}" ]; then
- echo "the alert log didn't exits, please check file path is correct!";
- exit;
- fi
- if [ -e ${alert_arc_file} ];then
- echo "the alert log file have been archived!"
- else
- mv ${alert_log_file} ${alert_arc_file}
- cat /dev/null > ${alert_log_file}
- fi
然后在crontab定时任务里面加上下面语句,每天23点59对告警日志进行归档。
[oracle@DB-Server scripts]$ crontab -l
# the alert log archived every day Add by kerry 2013-07-02
59 23 * * * /home/oracle/scripts/alert_log_archive.sh >/dev/null 2>$1
细心的朋友可能已经发现上面的脚本、配置错误了,我在部署测试的过程中,是指定二
十分钟执行一次,但是等了四十分钟,发现定时任务一次都没有执行,手工执行上面脚本是完全没有问题的,最后仔细的检查一遍,居然发现悲剧的发现时自己一时
粗心将&符号写成了$,真是很二的一个错误
59 23 * * * /home/oracle/scripts/alert_log_archive.sh >/dev/null 2>$1
59 23 * * * /home/oracle/scripts/alert_log_archive.sh >/dev/null 2>&1
接下来测试发现脚本执行有问题,在crontab
里执行该shell脚本时,获取不到ORACLE的环境变量,这是因为crontab环境变量问题,Crontab的环境默认情况下并不包含系统中当前用
户的环境。所以,你需要在shell脚本中添加必要的环境变量的设置,修改的脚本如下:
alert_log_archive.sh V1
- #*************************************************************************
- # FileName :alert_log_archive.sh
- #*************************************************************************
- # Author :Kerry
- # CreateDate :2013-07-02
- # blogs :www.cnblogs.com/kerrycode
- # Description :this script is made the alert log archived every day
- #*************************************************************************
- #! /bin/bash
- # these solved the oracle variable problem.
- export ORACLE_SID=gps
- export ORACLE_BASE=/u01/app/oracle
- date=`date +%Y%m%d`
- alert_log_path="$ORACLE_BASE/admin/$ORACLE_SID/bdump"
- alert_log_file="alert_$ORACLE_SID.log"
- alert_arc_file="alert_$ORACLE_SID.log""."${date}
- cd ${alert_log_path};
- if [ ! -e "${alert_log_file}" ]; then
- echo "the alert log didn't exits, please check file path is correct!";
- exit;
- fi
- if [ -e ${alert_arc_file} ];then
- echo "the alert log file have been archived!"
- else
- cat ${alert_log_file} >> ${alert_arc_file}
- cat /dev/null > ${alert_log_file}
- fi
alert_log_archive.sh V2
- #*************************************************************************
- # FileName :alert_log_archive.sh
- #*************************************************************************
- # Author :Kerry
- # CreateDate :2013-07-0
- # blogs :www.cnblogs.com/kerrycode
- # Description :this script is made the alert log archived every day
- #*************************************************************************
- #! /bin/bash
- # these solved the oracle variable problem.
- export ORACLE_SID=gps
- export ORACLE_BASE=/u01/app/oracle
- date=`date +%Y%m%d`
- alert_log_path="$ORACLE_BASE/admin/$ORACLE_SID/bdump"
- alert_log_file="alert_$ORACLE_SID.log"
- alert_arc_file="alert_$ORACLE_SID.log""."${date}
- cd ${alert_log_path};
- if [ ! -e "${alert_log_file}" ]; then
- echo "the alert log didn't exits, please check file path is correct!";
- exit;
- fi
- if [ -e ${alert_arc_file} ];then
- echo "the alert log file have been archived!"
- else
- mv ${alert_log_file} ${alert_arc_file}
- cat /dev/null > ${alert_log_file}
- fi
监控告警日志文件
接下来看看如何监控告警日志文件的ORA错误,这里是采用Perl结合Shell的方式,因为Shell获取错误的时间、行数等不如Perl操作字符串方便。
monitoring_alert_log.pl
- #**********************************************************************************
- # FileName :monitoring_alert_log.pl
- #**********************************************************************************
- # Author :Kerry
- # CreateDate :2013-07-01
- # blogs :www.cnblogs.com/kerrycode
- # Description :check the alert log and find out the ora error
- #**********************************************************************************
- # Modified Date Modified User Version Modified Reason
- # 2013-07-02 Kerry V01.0.1 add comment for this script
- #***********************************************************************************
- #! /usr/bin/perl
- use strict;
- my($argv) = @ARGV;
- if ( @ARGV != 1)
- {
- print '
- Parameter error: you must assined the alert log file as a input parameter or the number of prarameter is not right.
- ';
- exit
- }
- if( ! -e $argv )
- {
- print '
- Usage: monitoring_alert_log.pl
- $ cat alert_[sid].log | monitoring_alert_log.pl
- $ tail -f alert_[sid].log | monitoring_alert_log.pl
- $ monitoring_alert_log.pl alert_[sid].log
- ';
- exit;
- }
- my $err_regex = '^(\w+ \w+ \d{2} \d{2}:\d{2}:\d{2} \d{4})|(ORA-\d+:.+)$';
- my $date = "";
- my $line_counter = 0;
- while ( <> )
- {
- $line_counter++;
- if( m/$err_regex/oi )
- {
- if ($1)
- {
- $date = $1;
- next;
- }
- print "$line_counter | $date | $2 \n" if ($2);
- }
- }
monitoring_alert_log.sh
- #**********************************************************************************
- # FileName : monitoring_alert_log.sh
- #**********************************************************************************
- # Author : Kerry
- # CreateDate : 2013-07-01
- # blogs : www.cnblogs.com/kerrycode
- # Description: check the alert log and find out the ora error
- #**********************************************************************************
- # Modified Date Modified User Version Modified Reason
- # 2013-07-02 Kerry V01.0.1 add comment and modified script
- #***********************************************************************************
- #!/bin/bash
- # these solved the oracle variable problem.
- export ORACLE_SID=gsp
- export ORACLE_BASE=/u01/app/oracle
- logfile="/home/oracle/scripts/alter_err_log.txt"
- pl_monitoring_alert="/home/oracle/scripts/monitoring_alert_log.pl"
- pl_sendmail="/home/oracle/scripts/sendmail.pl"
- alert_logfile="$ORACLE_BASE/admin/$ORACLE_SID/bdump/alert_$ORACLE_SID.log"
- #delete the old alter error log file
- rm -f${logfile}
- rm -f${pl_sendmail}
- #run the perl and check if exists the ora error
- perl ${pl_monitoring_alert} ${alert_logfile}> ${logfile}
- #if have no error in alert log then exit the program
- if [[ -e "${logfile}" && ! -s "${logfile}" ]]; then
- exit;
- fi
- date_today=`date +%Y_%m_%d`
- subject="Monitoring the Oracle Alert logs and find ora errors"
- content="Dear All,
- The
Instance ${ORACLE_SID}\' alert log occured the ora errors ,please see
the detail in attachment and take action for it. many thanks! - Oracle Alert Services
- "
- echo "#!/usr/bin/perl" >> ${pl_sendmail}
- echo "use Mail::Sender;" >> ${pl_sendmail}
- echo "\$sender = new Mail::Sender {smtp => '10.xxx.xxx.xxx', from => 'xxxx@xxxx.com'}; ">> ${pl_sendmail}
- echo "\$sender->MailFile({to => 'kerry@xxxxx.com',">> ${pl_sendmail}
- echo "cc=>'konglb@esquel.com'," >> ${pl_sendmail}
- echo "subject => '$subject',">> ${pl_sendmail}
- echo "msg => '$content',">> ${pl_sendmail}
- echo "file => '$logfile'});">> ${pl_sendmail}
- perl ${pl_sendmail}
*/20 6-21 * * * /home/oracle/scripts/monitoring_alert_log.sh >/dev/null 2>&1
问题/优化脚本:Crontab
定时任务配置每二十分钟执行一次,结果,又有麻烦事情来了,假如8点发生了ORA错误,之后到下午6点都没有发生ORA错误,上面的脚本会每隔二十分钟发
送一次邮件,重复发送,感觉比较烦人,而我需要的是:只有当新的ORA错误出现,才给DBA发送邮件,否则就不要发送,其次,感觉二十分钟的时间段太长
了,如果出现了严重错误,二十分钟后才去处理,就显得时延比较滞后,但是如果你频率短的话,
基于第一个bug,你回收到N多邮件,那么我们继续改写,优化下面脚本吧
Code Snippet
- #****************************************************************************************************
- # FileName :monitoring_alert_log.sh
- #****************************************************************************************************
- # Author :Kerry
- # CreateDate :2013-07-01
- # Description :check the alert log and find out the ora error
- #****************************************************************************************************
- # Modified Date Modified User Version Modified Reason
- # 2013-07-02 Kerry V01.0.1 add comment and modified script
- # 2013-07-02 Kerry V01.0.2 Solved the email repated send problems, only
- # the new ora error occured then send the email.
- #****************************************************************************************************
- #!/bin/bash
- # these solved the oracle variable problem.
- export ORACLE_SID=gsp
- export ORACLE_BASE=/u01/app/oracle
- new_log_file="/home/oracle/scripts/new_err_log.txt"
- old_log_file="/home/oracle/scripts/old_err_log.txt"
- pl_monitoring_alert="/home/oracle/scripts/monitoring_alert_log.pl"
- pl_sendmail="/home/oracle/scripts/sendmail.pl"
- alert_logfile="$ORACLE_BASE/admin/$ORACLE_SID/bdump/alert_${ORACLE_SID}.log"
- #delete the old alter error log file
- #rm -f${new_log_file}
- rm -f${old_log_file}
- mv ${new_log_file}${old_log_file}
- rm -f${pl_sendmail}
- #run the perl and check if exists the ora error
- perl ${pl_monitoring_alert} ${alert_logfile}> ${new_log_file}
- #if have no error in alert log then exit the program
- if [[ -e "${new_log_file}" && ! -s "${new_log_file}" ]]; then
- exit;
- fi
- new_err_num=`cat ${new_log_file} | wc -l`
- old_err_num=`cat ${old_log_file} | wc -l`
- if [ ${new_err_num} -le ${old_err_num} ]; then
- exit
- fi
- date_today=`date +%Y_%m_%d`
- subject="xxx (192.168.xxx.xxx) Monitoring the Oracle Alert logs and find ora errors"
- content="Dear All,
- The Instance ${ORACLE_SID}\' alert
log occured the ora errors ,please see the detail in attachment and take
action for it. many thanks! - Oracle Alert Services
- "
- echo "#!/usr/bin/perl" >> ${pl_sendmail}
- echo "use Mail::Sender;" >> ${pl_sendmail}
- echo "\$sender = new Mail::Sender {smtp => '10.xxx.xxx.xxx', from => 'xxxx@xxxx.com'}; ">> ${pl_sendmail}
- echo "\$sender->MailFile({to => 'kerry@xxxxxx.com',">> ${pl_sendmail}
- echo "cc=>'xxxxx@xxxxx.com'," >> ${pl_sendmail}
- echo "subject => '$subject',">> ${pl_sendmail}
- echo "msg => '$content',">> ${pl_sendmail}
- echo "file => '${new_log_file}'});">> ${pl_sendmail}
- perl ${pl_sendmail}
但是我在部署过程中,由于环境问题(多台ORACLE服务器,不同的操作系统、不同的环境),发送邮件的部分出现改动,又有下面两个小版本的改动
Code Snippet
- #**********************************************************************************
- # FileName : monitoring_alert_log.sh
- #**********************************************************************************
- # Author : Kerry
- # CreateDate : 2013-07-01
- # Description: check the alert log and find out the ora error
- #***********************************************************************************
- # Modified Date Modified User Version Modified Reason
- # 2013-07-02 Kerry V01.0.1 add comment and modified script
- # 2013-07-02 Kerry V01.0.2 Solved the email repated send problems, only
- # the new ora error occured then send the email
- #***********************************************************************************
- #!/bin/bash
- new_log_file="/home/oracle/scripts/new_err_log.txt"
- old_log_file="/home/oracle/scripts/old_err_log.txt"
- pl_monitoring_alert="/home/oracle/scripts/monitoring_alert_log.pl"
- email_content="/home/oracle/scripts/sendmail.txt"
- alert_logfile="$ORACLE_BASE/admin/$ORACLE_SID/bdump/alert_$ORACLE_SID.log"
- #delete the old alter error log file
- rm -f${old_log_file}
- mv ${new_log_file} ${old_log_file}
- rm -f${pl_sendmail}
- #run the perl and check if exists the ora error
- perl ${pl_monitoring_alert} ${alert_logfile}> ${new_log_file}
- #if have no error in alert log then exit the program
- if [[ -e "${new_log_file}" && ! -s "${new_log_file}" ]]; then
- exit;
- fi
- new_err_num=`cat ${new_log_file} | wc -l`
- old_err_num=`cat ${old_log_file} | wc -l`
- if [ ${new_err_num} -le ${old_err_num} ]; then
- exit
- fi
- date_today=`date +%Y_%m_%d`
- subject="Monitoring the Oracle Alert logs and find ora errors"
- content="Dear All,
- The
Instance ${ORACLE_SID}\' alert log occured the ora errors ,please see
the detail in attachment and take action for it. many thanks! - The Error is blow :
- "
- echo 'Content-Type: text/html' > ${email_content}
- echo 'To: xxxxx@xxxxx.com' >> ${email_content}
- echo ${subject} >> ${email_content}
- echo '<pre style="font-family: courier; font-size: 9pt">' >> ${email_content}
- echo ${content} >> ${email_content}
- cat ${new_log_file} >>${email_content} 2>&1
- echo 'Oracle Alert Services' >> ${email_content}
- /usr/sbin/sendmail -t -f ${subject} < ${email_content}
- rm -f ${email_content}
Code Snippet
- #**********************************************************************************
- # FileName : monitoring_alert_log.sh
- #**********************************************************************************
- # Author : Kerry
- # CreateDate :2013-07-01
- # Description: check the alert log and find out the ora error
- #***********************************************************************************
- # Modified Date Modified User Version Modified Reason
- # 2013-07-02 Kerry V01.0.1 add comment and modified script
- # 2013-07-02 Kerry V01.0.2 Solved the email repated send problems, only
- # the new ora error occured then send the email
- #***********************************************************************************
- #!/bin/bash
- new_log_file="/home/oracle/scripts/new_err_log.txt"
- old_log_file="/home/oracle/scripts/old_err_log.txt"
- pl_monitoring_alert="/home/oracle/scripts/monitoring_alert_log.pl"
- email_content="/home/oracle/scripts/sendmail.pl"
- alert_logfile="$ORACLE_BASE/admin/$ORACLE_SID/bdump/alert_$ORACLE_SID.log"
- reportname="alert_log_err.txt"
- #delete the old alter error log file
- rm -f${old_log_file}
- mv ${new_log_file} ${old_log_file}
- rm -f${pl_sendmail}
- #run the perl and check if exists the ora error
- perl ${pl_monitoring_alert} ${alert_logfile}> ${new_log_file}
- #if have no error in alert log then exit the program
- if [[ -e "${new_log_file}" && ! -s "${new_log_file}" ]]; then
- exit;
- fi
- date_today=`date +%Y_%m_%d`
- subject="Monitoring the Oracle Alert logs and find ora errors"
- content="Dear All,
- The
Instance ${ORACLE_SID}\' alert log occured the ora errors ,please see
the detail in attachment and take action for it. many thanks! - Oracle Alert Services
- "
- ( ${content} ; uuencode ${new_log_file} ${reportname} ) | /bin/mail -s ${subject} xxxx@xxxx.com xxxxx@xxx.com
- /bin/mail