1、关于MHA
MHA(Master High Availability)目前在MySQL高可用方面是一个相对成熟的解决方案,它由日本DeNA公司youshimaton(现就职于Facebook公司)开发,是一套优秀的作为MySQL高可用性环境下故障切换和主从提升的高可用软件。在MySQL故障切换过程中,MHA能做到在0~30秒之内自动完成数据库的故障切换操作,并且在进行故障切换的过程中,MHA能在最大程度上保证数据的一致性,以达到真正意义上的高可用。
该软件由两部分组成:MHA Manager(管理节点)和MHA Node(数据节点)。MHA Manager可以单独部署在一台独立的机器上管理多个master-slave集群,也可以部署在一台slave节点上。MHA Node运行在每台MySQL服务器上,MHA Manager会定时探测集群中的master节点,当master出现故障时,它可以自动将最新数据的slave提升为新的master,然后将所有其他的slave重新指向新的master。整个故障转移过程对应用程序完全透明。
在MHA自动故障切换过程中,MHA试图从宕机的主服务器上保存二进制日志,最大程度的保证数据的不丢失,但这并不总是可行的。例如,如果主服务器硬件故障或无法通过ssh访问,MHA没法保存二进制日志,只进行故障转移而丢失了最新的数据。使用MySQL 5.5的半同步复制,可以大大降低数据丢失的风险。MHA可以与半同步复制结合起来。如果只有一个slave已经收到了最新的二进制日志,MHA可以将最新的二进制日志应用于其他所有的slave服务器上,因此可以保证所有节点的数据一致性。
目前MHA主要支持一主多从的架构,要搭建MHA,要求一个复制集群中必须最少有三台数据库服务器,一主二从,即一台充当master,一台充当备用master,另外一台充当从库,因此至少需要三台服务器。
2、MHA组件说明
Manager节点:
-masterha_check_ssh:MHA依赖的SSH环境检测工具;
-masterha_check_repl:MySQL复制环境检测工具;
-masterha_manager:MHA服务主程序;
-masterha_check_status:MHA运行状态探测工具;
-masterha_master_monitor:MySQL master节点可用性检测工具;
-masterha_master_switch:master节点切换工具;
-masterha_conf_host:添加或删除配置的节点;
-masterha_stop:关闭MHA服务的工具;
Node节点:
-save_binary_logs:保存和复制master的二进制日志;
-apply_diff_relay_logs:识别差异的中继日志事件并用于其他slave;
-fiter_mysqlbinlog:去除不必要的ROLLBACK事件(MHA已不再使用这个工具);
-purge_relay_logs:清除中继日志(不会阻塞SQL线程);
自定义扩展:
-secondary_check_script:通过多条网络路由检测master的可用性;
-master_ip_failover_script:更新appliction使用的masterip;
-shutdown_script:强制关闭master节点;
-report_script:发送报告;
-init_conf_load_script:加载初始配置参数;
-master_ip_online_change_script:更新master节点ip地址
3、部署及测试
1.环境介绍
my01节点为主,my02、my03节点为从,其中my02为备选主库,my03为mhamanager节点(manager也可以放到其他机器)
10.20.30.101 my01 10.20.30.102 my02 bak01 10.20.30.103 my03 mhamanager |
2.配置ssh互信
3.配置一主两从(基于GTID的增强半同步)
GRANT ALL ON *.* TO 'mhauser'@'10.20.30.%' IDENTIFIED BY 'mhapass'; |
4.配置MHA
MHA在发生切换的过程中,从库的恢复过程中依赖于relay log的相关信息,所以这里要将relay log的自动清除设置为OFF,
采用手动清除relay log的方式。在默认情况下,从服务器上的中继日志会在SQL线程执行完毕后被自动删除。
在MHA环境中,这些中继日志在恢复其他从服务器时可能会被用到,因此需要禁用中继日志的自动删除功能。定期清除中继日志需要考虑到复制延时的问题。在ext3的文件系统下,删除大的文件需要一定的时间,会导致严重的复制延时。为了避免复制延时,需要暂时为中继日志创建硬链接,因为在linux系统中通过硬链接删除大文件速度会很快。(在mysql数据库中,删除大表时,通常也采用建立硬链接的方式)
MHA节点中包含了pure_relay_logs命令工具,它可以为中继日志创建硬链接,执行SET GLOBAL relay_log_purge=1,等待几秒钟以便SQL线程切换到新的中继日志,再执行SET GLOBAL relay_log_purge=0.这是此工具的原理.
参考《MHA使用purge_relay_logs清理relay logs》设置定期清理脚本
从库执行
set global relay_log_purge=0; set global read_only=1; |
先安装node,再安装manager
mha4mysql-manager
为依赖而安装:
perl-Config-Tiny
perl-Email-Date-Format
perl-Log-Dispatch
perl-MIME-Lite
perl-MIME-Types
perl-Mail-Sender
perl-Mail-Sendmail
perl-MailTools
perl-Parallel-ForkManager
perl-Params-Validate
perl-TimeDate
manager管理节点配置
# tree /mha/ # 存放mha的相关配置与脚本
/mha/
├── app1 # 日志目录
│ ├── app1.log # 日志文件
│ └── app1.master_status.health # 检查文件
├── app1.cnf # mha配置文件
├── bin
│ └── mhaCmd.sh # mha启动脚本,自己编写
├── master_ip_failover # mha检测失败后执行的自动切换脚本
└── sendEmail # mha切换后的发送告警邮件的脚本
mkdir -p /mha/{app1,bin,tmp}
/mha/app1.cnf
[server default] manager_log=/mha/app1/app1.log manager_workdir=/mha/app1/ master_binlog_dir=/data/mysql/mysql3306/logs master_ip_failover_script=/mha/master_ip_failover master_ip_online_change_script=/mha/master_ip_failover report_script=/mha/sendEmail remote_workdir=/mha/tmp ping_interval=5 user=mhauser password=mhapass repl_user=repl repl_password=repl4slave ssh_port=22 ssh_user=root secondary_check_script=masterha_secondary_check -s 10.20.30.102 -s 10.20.30.103
[server1] hostname=10.20.30.101 port=3306 candidate_master=1 [server2] hostname=10.20.30.102 port=3306 candidate_master=1 [server3] hostname=10.20.30.103 port=3306 |
mhaCmd.sh
chmod +x bin/mhaCmd.sh
#!/bin/bash . /etc/profile . ~/.bash_profile . ~/.bashrc run_num=$(ps -ef|grep masterha_manager |grep -v grep|wc -l) pid_file='/mha/app1/app1.master_status.health' start() { if [[ $run_num < 1 ]];then args="--conf=/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover" nohup masterha_manager $args < /dev/null > /mha/app1/app1.log 2>&1 & else echo 'mha is already running...' fi } stop() { if [[ $run_num < 1 ]];then echo 'mha not running ...' exit 64 else ps -ef|grep masterha_manager |grep -v grep|awk '{print $2}'|xargs kill -9 rm -f $pid_file echo 'mha stop...' fi } status() { masterha_check_status --conf=/mha/app1.cnf } case "$1" in start) start ;; stop) stop ;; status) status ;; *) echo 'mhaCmd {stop|start|status}' ;; esac |
master_ip_failover
chmod +x master_ip_failover
#!/usr/bin/env perl use strict; use warnings FATAL => 'all'; use Getopt::Long; my ( $command, $ssh_user, $orig_master_host, $orig_master_ip, $orig_master_port, $new_master_host, $new_master_ip, $new_master_port ); my $vip = '10.20.30.110/24'; # Virtual IP my $key = "1"; my $ssh_start_vip = "/sbin/ifconfig eth0:$key $vip"; my $ssh_stop_vip = "/sbin/ifconfig eth0:$key down"; my $exit_code = 0; GetOptions( 'command=s' => \$command, 'ssh_user=s' => \$ssh_user, 'orig_master_host=s' => \$orig_master_host, 'orig_master_ip=s' => \$orig_master_ip, 'orig_master_port=i' => \$orig_master_port, 'new_master_host=s' => \$new_master_host, 'new_master_ip=s' => \$new_master_ip, 'new_master_port=i' => \$new_master_port, ); exit &main(); sub main { print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n"; if ( $command eq "stop" || $command eq "stopssh" ) { my $exit_code = 1; eval { print "Disabling the VIP on old master: $orig_master_host \n"; &stop_vip(); $exit_code = 0; }; if ($@) { warn "Got Error: $@\n"; exit $exit_code; } exit $exit_code; } elsif ( $command eq "start" ) { my $exit_code = 10; eval { print "Enabling the VIP - $vip on the new master - $new_master_host \n"; &start_vip(); $exit_code = 0; }; if ($@) { warn $@; exit $exit_code; } exit $exit_code; } elsif ( $command eq "status" ) { print "Checking the Status of the script.. OK \n"; exit 0; } else { &usage(); exit 1; } } sub start_vip() { `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`; } sub stop_vip() { return 0 unless ($ssh_user); `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`; } sub usage { "Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n"; } |
sendEmail
chmod +x sendEmail
#!/usr/bin/perl # Copyright (C) 2011 DeNA Co.,Ltd. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA ## Note: This is a sample script and is not complete. Modify the script based on your environment. use strict; use warnings FATAL => 'all'; use Mail::Sender; use Getopt::Long; #new_master_host and new_slave_hosts are set only when recovering master succeeded my ( $dead_master_host, $new_master_host, $new_slave_hosts, $subject, $body ); my $smtp='smtp.163.com'; my $mail_from='xxx@163.com'; my $mail_user='xxx@163.com'; my $mail_pass='xxxxxx'; my $mail_to=['xxx@xxx.cn','xxxxx@xxx.cn']; GetOptions( 'orig_master_host=s' => \$dead_master_host, 'new_master_host=s' => \$new_master_host, 'new_slave_hosts=s' => \$new_slave_hosts, 'subject=s' => \$subject, 'body=s' => \$body, ); mailToContacts($smtp,$mail_from,$mail_user,$mail_pass,$mail_to,$subject,$body); sub mailToContacts { my ( $smtp, $mail_from, $user, $passwd, $mail_to, $subject, $msg ) = @_; open my $DEBUG, "> /tmp/monitormail.log" or die "Can't open the debug file:$!\n"; my $sender = new Mail::Sender { ctype => 'text/plain; charset=utf-8', encoding => 'utf-8', smtp => $smtp, from => $mail_from, auth => 'LOGIN', TLS_allowed => '0', authid => $user, authpwd => $passwd, to => $mail_to, subject => $subject, debug => $DEBUG }; $sender->MailMsg( { msg => $msg, debug => $DEBUG } ) or print $Mail::Sender::Error; return 1; } # Do whatever you want here exit 0; |
启动前检测
ssh互信配置
masterha_check_ssh --conf=/mha/app1.cnf
[info] All SSH connection tests passed successfully.
复制检查
masterha_check_repl --conf=/mha/app1.cnf
MySQL Replication Health is OK.
如果报错,建立软链接解决(因为yum安装默认安装了一个mysql,让系统定位我们安装的mysql的PATH)
Can't exec "mysqlbinlog": 没有那个文件或目录 at /usr/share/perl5/vendor_perl/MHA/BinlogManager.pm line 106.或者Testing mysql connection and privileges..sh: mysql: command not found
ln -s /usr/local/mysql/bin/mysqlbinlog /usr/bin/mysqlbinlog
ln -s /usr/local/mysql/bin/mysql /usr/bin
ln -s /usr/bin/masterha_secondary_check /usr/local/bin/masterha_secondary_check
第一次VIP设置
因为是脚本方式设置,那么第一次的VIP需要手工到master上指定
ifconfig eth0:1 10.20.30.110/24 |
启动MHA
/mha/bin/mhaCmd.sh start
查看MHA状态
/mha/bin/mhaCmd.sh status
app1 (pid:6159) is running(0:PING_OK), master:10.20.30.101
测试MHA
关掉原来my01节点,后台正常切换到了my02了
后台程序还是会报错,这跟设置的ping时长有关
MHA切换后也会自动停止
切换MHA后,需要重新添加原来节点回集群
把原来宕机的节点重新配置成新的从库,然后重新配置(配置文件中server部分会删掉宕掉的节点,需要重新配置),然后重新启动MHA,即可恢复