在删除mon节点前, 请务必确保删除后的mon节点可以达到健康状态, 例如从5个节点删除到4个节点, 并且4个节点里面有3个或以上mon是健康的, 这样的状态ceph storage cluster才是健康的.
删除mon节点分两种情况, 1种情况是删除一个健康的mon节点, 另一种情况是删除一个不健康的mon节点.
首选来看看如何删除一个健康的mon节点 :
连接到该mon节点, 关闭服务, 或(停止ceph_mon服务)
[root@mon5 ~]# ps -ewf|grep mon
root 127 1 0 Dec09 pts/0 00:00:24 ceph-mon -i mon5 --public-addr 172.17.0.10 --mon-data /data01/ceph/mon/ceph-5
root 202 28 0 11:24 pts/0 00:00:00 grep --color=auto mon
[root@mon5 ~]# kill 127
[root@mon5 ~]# ps -ewf|grep mon
root 204 28 0 11:24 pts/0 00:00:00 grep --color=auto mon关闭ceph-mon进程后
[root@mon5 ~]# ceph -s
cluster f649b128-963c-4802-ae17-5a76f36c4c76
health HEALTH_WARN 1 mons down, quorum 0,1,2,3 mon1,mon2,mon3,mon4
monmap e3: 5 mons at {mon1=172.17.0.2:6789/0,mon2=172.17.0.3:6789/0,mon3=172.17.0.4:6789/0,mon4=172.17.0.9:6789/0,mon5=172.17.0.10:6789/0}, election epoch 16, quorum 0,1,2,3 mon1,mon2,mon3,mon4
osdmap e29: 4 osds: 4 up, 4 in
pgmap v2384: 128 pgs, 1 pools, 0 bytes data, 0 objects
41210 MB used, 1596 GB / 1636 GB avail
128 active+clean使用命令删除节点
[root@mon5 ~]# ceph mon remove mon5
removed mon.mon5 at 172.17.0.10:6789/0, there are now 4 monitors
[root@mon5 ~]# ceph -s
cluster f649b128-963c-4802-ae17-5a76f36c4c76
health HEALTH_OK
monmap e4: 4 mons at {mon1=172.17.0.2:6789/0,mon2=172.17.0.3:6789/0,mon3=172.17.0.4:6789/0,mon4=172.17.0.9:6789/0}, election epoch 20, quorum 0,1,2,3 mon1,mon2,mon3,mon4
osdmap e29: 4 osds: 4 up, 4 in
pgmap v2384: 128 pgs, 1 pools, 0 bytes data, 0 objects
41210 MB used, 1596 GB / 1636 GB avail
128 active+clean建议修改所有mon节点的配置文件, 把删掉的节点从mon members 配置和host配置去除 .
(修改ceph.conf只是为了好看, 其实mon节点之间是通过monmap来发现其他mon节点的, 所以配置文件不是必须的)
/etc/ceph/ceph.conf
mon initial members = mon1, mon2, mon3, mon4
mon host = 172.17.0.2, 172.17.0.3, 172.17.0.4, 172.17.0.9接下来让ceph storage cluster不健康, 关掉2个节点, 剩余2个节点.
[root@localhost ceph]# ssh 172.17.0.9
root@172.17.0.9's password:
Last login: Tue Dec 9 17:26:03 2014 from 172.17.42.1
[root@mon4 ~]# ps -ewf|grep mon
root 223 1 0 Dec09 pts/0 00:00:24 ceph-mon -i mon4 --public-addr 172.17.0.9 --mon-data /data01/ceph/mon/ceph-4
root 278 261 0 11:29 pts/1 00:00:00 grep --color=auto mon
[root@mon4 ~]# kill 223
[root@mon4 ~]# ceph -s
cluster f649b128-963c-4802-ae17-5a76f36c4c76
health HEALTH_WARN 1 mons down, quorum 0,1,2 mon1,mon2,mon3
monmap e4: 4 mons at {mon1=172.17.0.2:6789/0,mon2=172.17.0.3:6789/0,mon3=172.17.0.4:6789/0,mon4=172.17.0.9:6789/0}, election epoch 22, quorum 0,1,2 mon1,mon2,mon3
osdmap e29: 4 osds: 4 up, 4 in
pgmap v2392: 128 pgs, 1 pools, 0 bytes data, 0 objects
41219 MB used, 1596 GB / 1636 GB avail
128 active+clean
[root@localhost ceph]# ssh 172.17.0.4
root@172.17.0.4's password:
Last login: Tue Dec 9 17:26:32 2014 from 172.17.42.1
[root@mon3 ~]# service ceph -a stop mon.mon3
=== mon.mon3 ===
Stopping Ceph mon.mon3 on mon3...kill 903...done
[root@mon3 ~]# ceph -s
2014-12-10 11:33:47.004145 7f24641a0700 0 -- :/1001200 >> 172.17.0.4:6789/0 pipe(0x7f2460023190 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f2460023420).fault
^CError connecting to cluster: InterruptedOrTimeoutError因为集群中有4个mon节点, 已经有2个被关闭了, 所以现在ceph 集群是不健康的状态, 为了达到健康, 我们可以启动这两个节点, 或从集群中把这两个节点删掉. 让集群只有2个mon节点.
接下来要从因为mon节点选举数不够导致集群不健康的情况, 删除mon节点 :
(因为集群已经不健康了, 所以不能直接操作来删除, 必须通过dump和inject来删除不健康节点, 使ceph集群的mon选举数足够多来恢复集群健康)
首选要确认剩余的健康的mon节点是哪些? 本例是mon1, mon2.
mon3, mon4都关闭了, 并且假设不可恢复, 或者不需要他们了.
首先要关闭所有的要留下的mon, 本例为mon1, mon2
[root@mon1 tmp]# ps -efw|grep mon
root 560 1 0 Dec09 ? 00:01:22 /usr/bin/ceph-mon -i mon1 --pid-file /var/run/ceph/mon.mon1.pid -c /etc/ceph/ceph.conf --cluster ceph
root 1302 1228 0 11:38 pts/1 00:00:00 grep --color=auto mon
[root@mon1 tmp]# kill 560
[root@mon2 ~]# ps -ewf|grep mon
root 220 1 0 Dec09 ? 00:00:28 /usr/bin/ceph-mon -i mon2 --pid-file /var/run/ceph/mon.mon2.pid -c /etc/ceph/ceph.conf --cluster ceph
root 684 665 0 11:39 pts/0 00:00:00 grep --color=auto mon
[root@mon2 ~]# kill 220然后连接到mon节点, dump monmap, 并从dump出来的monmap中删除不需要的节点(mon3, mon4)
[root@mon2 ~]# ceph-mon -i mon2 --extract-monmap /tmp/monmap
2014-12-10 11:39:39.775643 7ff70b6af880 -1 wrote monmap to /tmp/monmap
[root@mon2 ~]# monmaptool /tmp/monmap --rm mon3
monmaptool: monmap file /tmp/monmap
monmaptool: removing mon3
monmaptool: writing epoch 4 to /tmp/monmap (3 monitors)
[root@mon2 ~]# monmaptool /tmp/monmap --rm mon4
monmaptool: monmap file /tmp/monmap
monmaptool: removing mon4
monmaptool: writing epoch 4 to /tmp/monmap (2 monitors)使用修改后的monmap导入当前mon节点的monmap数据中.
[root@mon2 ~]# ceph-mon -i mon2 --inject-monmap /tmp/monmap启动
[root@mon2 ~]# /usr/bin/ceph-mon -i mon2 --pid-file /var/run/ceph/mon.mon2.pid -c /etc/ceph/ceph.conf --cluster ceph另一个节点也做同样的操作 :
[root@mon1 ~]# ceph-mon -i mon1 --extract-monmap /tmp/monmap
2014-12-10 11:43:16.959514 7fe2231b5880 -1 wrote monmap to /tmp/monmap
[root@mon1 ~]# monmaptool /tmp/monmap --rm mon3
monmaptool: monmap file /tmp/monmap
monmaptool: removing mon3
monmaptool: writing epoch 4 to /tmp/monmap (3 monitors)
[root@mon1 ~]# monmaptool /tmp/monmap --rm mon4
monmaptool: monmap file /tmp/monmap
monmaptool: removing mon4
monmaptool: writing epoch 4 to /tmp/monmap (2 monitors)
[root@mon1 ~]# ceph-mon -i mon1 --inject-monmap /tmp/monmap
[root@mon1 ~]# /usr/bin/ceph-mon -i mon1 --pid-file /var/run/ceph/mon.mon1.pid -c /etc/ceph/ceph.conf --cluster ceph现在的ceph storage cluster只有2个mon节点, 而且都up了, 所以集群健康.
[root@mon1 ~]# ceph -s
cluster f649b128-963c-4802-ae17-5a76f36c4c76
health HEALTH_OK
monmap e5: 2 mons at {mon1=172.17.0.2:6789/0,mon2=172.17.0.3:6789/0}, election epoch 26, quorum 0,1 mon1,mon2
osdmap e29: 4 osds: 4 up, 4 in
pgmap v2395: 128 pgs, 1 pools, 0 bytes data, 0 objects
41223 MB used, 1596 GB / 1636 GB avail
128 active+clean如果被删除的节点以后都不需要加进来了, 建议修改一下当前剩余节点mon1, mon2的配置文件, 把他们去除.
(修改ceph.conf只是为了好看, 其实mon节点之间是通过monmap来发现其他mon节点的, 所以配置文件不是必须的)
/etc/ceph/ceph.conf
......
mon initial members = mon1, mon2
mon host = 172.17.0.2, 172.17.0.3删除mon的操作讲完了, 接下来可以用上一篇讲的方法把mon3, mon4, mon5加入进来.
因为mon3,4,5的信息都在, 所以加进来很简单.
[root@mon1 ~]# ceph mon add mon3 172.17.0.4
port defaulted to 6789; added mon.mon3 at 172.17.0.4:6789/0
[root@mon1 ~]# ceph -s
cluster f649b128-963c-4802-ae17-5a76f36c4c76
health HEALTH_WARN 1 mons down, quorum 0,1 mon1,mon2
monmap e6: 3 mons at {mon1=172.17.0.2:6789/0,mon2=172.17.0.3:6789/0,mon3=172.17.0.4:6789/0}, election epoch 28, quorum 0,1 mon1,mon2
osdmap e29: 4 osds: 4 up, 4 in
pgmap v2410: 128 pgs, 1 pools, 0 bytes data, 0 objects
41234 MB used, 1596 GB / 1636 GB avail
128 active+clean
[root@mon1 ~]# ssh 172.17.0.4
root@172.17.0.4's password:
Last login: Wed Dec 10 11:31:54 2014 from 172.17.42.1
[root@mon3 ~]# service ceph start mon.mon3
=== mon.mon3 ===
Starting Ceph mon.mon3 on mon3...
Starting ceph-create-keys on mon3...
[root@mon3 ~]# ceph -s
cluster f649b128-963c-4802-ae17-5a76f36c4c76
health HEALTH_OK
monmap e6: 3 mons at {mon1=172.17.0.2:6789/0,mon2=172.17.0.3:6789/0,mon3=172.17.0.4:6789/0}, election epoch 30, quorum 0,1,2 mon1,mon2,mon3
osdmap e29: 4 osds: 4 up, 4 in
pgmap v2411: 128 pgs, 1 pools, 0 bytes data, 0 objects
41234 MB used, 1596 GB / 1636 GB avail
128 active+clean但是务必保证加进来后即使未启动集群也是健康的, 所以有3个节点健康后, 可以一次加2个进来.
[root@mon3 ~]# ceph mon add mon4 172.17.0.9
port defaulted to 6789; added mon.mon4 at 172.17.0.9:6789/0
[root@mon3 ~]# ceph mon add mon5 172.17.0.10
port defaulted to 6789; added mon.mon5 at 172.17.0.10:6789/0
[root@mon4 ~]# /usr/bin/ceph-mon -i mon4 --pid-file /var/run/ceph/mon.mon4.pid -c /etc/ceph/ceph.conf --cluster ceph --mon-data /data01/ceph/mon/ceph-4
[root@mon4 ~]# ceph -s
cluster f649b128-963c-4802-ae17-5a76f36c4c76
health HEALTH_WARN 1 mons down, quorum 0,1,2,3 mon1,mon2,mon3,mon4
monmap e8: 5 mons at {mon1=172.17.0.2:6789/0,mon2=172.17.0.3:6789/0,mon3=172.17.0.4:6789/0,mon4=172.17.0.9:6789/0,mon5=172.17.0.10:6789/0}, election epoch 36, quorum 0,1,2,3 mon1,mon2,mon3,mon4
osdmap e29: 4 osds: 4 up, 4 in
pgmap v2417: 128 pgs, 1 pools, 0 bytes data, 0 objects
41241 MB used, 1596 GB / 1636 GB avail
128 active+clean
[root@mon5 ~]# /usr/bin/ceph-mon -i mon5 --pid-file /var/run/ceph/mon.mon5.pid -c /etc/ceph/ceph.conf --cluster ceph --mon-data /data01/ceph/mon/ceph-5
[root@mon5 ~]# ceph -s
cluster f649b128-963c-4802-ae17-5a76f36c4c76
health HEALTH_OK
monmap e8: 5 mons at {mon1=172.17.0.2:6789/0,mon2=172.17.0.3:6789/0,mon3=172.17.0.4:6789/0,mon4=172.17.0.9:6789/0,mon5=172.17.0.10:6789/0}, election epoch 38, quorum 0,1,2,3,4 mon1,mon2,mon3,mon4,mon5
osdmap e29: 4 osds: 4 up, 4 in
pgmap v2418: 128 pgs, 1 pools, 0 bytes data, 0 objects
41243 MB used, 1596 GB / 1636 GB avail
128 active+clean时间： 2024-11-05 12:16:34