about ceph monitor's consistency

    ceph 的mon节点维护了整个ceph 存储集群的map状态.

    因为mon节点是多个的, 所以要求所有mon节点的map状态一致, mon使用paxos算法(分布式一致性算法)来修改map的状态, 确保所有mon节点的map数据一致.

    因此mon节点并不是通过集群配置文件如ceph.conf来保持map一致的, 而是map文件.

    每次map文件变更前, 需要多数monitor节点同意, 确保多数或所有(quorum)节点的map文件一致, 并且map文件是增量更新的, 每次更新产生一个新的version, 所以当mon节点离线再加入时, 需要同步增量部分的信息. 

    monitor相互之间是如何发现的呢? 

    同样是通过map文件, 而不是静态的集群配置文件如ceph.conf.

    我们来dump一个map文件的信息看看 ?

[root@mon5 ~]# ceph mon getmap -o /tmp/monmap
got monmap epoch 8

打印map文件内容

[root@mon5 ~]# monmaptool --print /tmp/monmap
monmaptool: monmap file /tmp/monmap
epoch 8
fsid f649b128-963c-4802-ae17-5a76f36c4c76
last_changed 2014-12-10 11:52:43.961159
created 2014-12-09 15:24:33.339570
0: 172.17.0.2:6789/0 mon.mon1
1: 172.17.0.3:6789/0 mon.mon2
2: 172.17.0.4:6789/0 mon.mon3
3: 172.17.0.9:6789/0 mon.mon4
4: 172.17.0.10:6789/0 mon.mon5

所有mon节点的信息应该是一致的.

[参考]
1. http://docs.ceph.com/docs/master/rados/operations/add-or-rm-mons/

Monitors are critical components of a Ceph cluster, and they need to maintain a quorum for the whole system to work properly. To establish a quorum, the monitors need to discover each other. Ceph has strict requirements for discovering monitors.

Ceph clients and other Ceph daemons use ceph.conf to discover monitors. However, monitors discover each other using the monitor map, not ceph.conf. For example, if you refer to Adding a Monitor (Manual) you will see that you need to obtain the current monmap for the cluster when creating a new monitor, as it is one of the required arguments of ceph-mon -i {mon-id} --mkfs. The following sections explain the consistency requirements for Ceph monitors, and a few safe ways to change a monitor’s IP address.

CONSISTENCY REQUIREMENTS

A monitor always refers to the local copy of the monmap when discovering other monitors in the cluster. Using the monmap instead of ceph.conf avoids errors that could break the cluster (e.g., typos in ceph.conf when specifying a monitor address or port). Since monitors use monmaps for discovery and they share monmaps with clients and other Ceph daemons, the monmap provides monitors with a strict guarantee that their consensus is valid.

Strict consistency also applies to updates to the monmap. As with any other updates on the monitor, changes to the monmap always run through a distributed consensus algorithm called Paxos. The monitors must agree on each update to the monmap, such as adding or removing a monitor, to ensure that each monitor in the quorum has the same version of the monmap. Updates to the monmap are incremental so that monitors have the latest agreed upon version, and a set of previous versions, allowing a monitor that has an older version of the monmap to catch up with the current state of the cluster.

If monitors discovered each other through the Ceph configuration file instead of through the monmap, it would introduce additional risks because the Ceph configuration files aren’t updated and distributed automatically. Monitors might inadvertently use an older ceph.conf file, fail to recognize a monitor, fall out of a quorum, or develop a situation where Paxos isn’t able to determine the current state of the system accurately. Consequently, making changes to an existing monitor’s IP address must be done with great care.