Ceph cluster configure introduce - 2

一般配置 :

初始化时的监控节点配置, 包含(非FQDN)hostname -s, IP.
[global]
mon_initial_members = ceph1
mon_host = 10.0.0.1

如果是手工部署, 可以写3个mon节点的信息, 如 :
mon initial members = mon1, mon2, mon3
mon host = 172.17.0.2, 172.17.0.3, 172.17.0.4

如果是使用chef, ceph-deploy来配置, 建议配置时只使用一个节点, 后面再添加, 如 :
mon_initial_members = ceph1
mon_host = 10.0.0.1

网络配置 : 

生产环境建议至少应该将public网络和cluster网络分开.

public网络中有ceph客户端的流量, 有ceph 监控的流量, 还有ceph mds的流量.

集群网络的话, 主要包含osd节点之间的数据复制流量, 心跳流量.

OSD节点的交互包括 : 

OSD节点之间的复制交互, 心跳交互

OSD节点和客户端, 监控节点的交互

网络配置举例 : 

http://ceph.com/docs/master/rados/configuration/network-config-ref

[mon.a]
        host = {hostname}
        mon addr = {ip-address}:6789
[osd.0]
        public addr = {ip-address}/{netmask} [, {ip-address}/{netmask}]  192.168.0.0/24
        cluster addr = {ip-address}/{netmask} [, {ip-address}/{netmask}]  10.0.0.0/24

监控配置 : 

http://ceph.com/docs/master/rados/configuration/mon-config-ref

监控节点主要配置地址, 数据目录, 例如 :

[mon]
        mon host = hostname1,hostname2,hostname3
        mon addr = 10.0.0.10:6789,10.0.0.11:6789,10.0.0.12:6789
        mon data = /var/lib/ceph/mon/$cluster-$id
[mon.a]
        host = hostname1
        mon addr = 10.0.0.10:6789

认证配置 : 

http://ceph.com/docs/master/rados/operations/authentication

http://ceph.com/docs/master/rados/configuration/auth-config-ref

升级CEPH版本前, 建议关闭认证, 升级完再打开, 配置例如 :

auth cluster required = cephx
auth service required = cephx
auth client required = cephx

key配置未列出.

OSD配置

http://ceph.com/docs/master/rados/configuration/osd-config-ref

http://blog.163.com/digoal@126/blog/static/163877040201411711835765/

OSD一般需要配置journal大小路径, 数据目录路径, 文件系统的XATTR是否使用OMAP存储(ext4必须使用omap), 等, 例如 :

[osd]
    osd journal = /var/lib/ceph/osd/$cluster-$id/journal
    osd journal size = 1024
    osd data = /var/lib/ceph/osd/$cluster-$id
    filestore xattr use omap = true

心跳配置

http://docs.ceph.com/docs/master/rados/configuration/mon-osd-interaction/

首先看看心跳流程 : 

OSD1检测到OSD2异常.

间隔 [osd] osd heartbeat interval = 秒.

OSD1 向monitor报告其他节点异常.

[mon] mon osd min down reports = 多少次mon将其标记为down状态

OSD 报告peer异常.

[osd] osd mon heartbeat interval = 多少秒

osd间隔默认30秒向mon请求最新的map, 并检测map中的其他osd的状态. 向mon报告.

OSD1 向montiro报告自身异常.

当mon osd report timeout时间之后未收到OSD主动发起的状态信息报告, MON将标记该OSD DOWN.

那么OSD多久给MON发一次报告呢? 除了状态改变引起的报告, 还可以设置间隔.

[osd]
osd mon report interval min = 5秒 默认
osd mon report interval max = 120秒 默认

日志,debug配置

http://docs.ceph.com/docs/master/rados/troubleshooting/log-and-debug/

例子

DEBUG参考配置 :

[global]
        debug ms = 1/5

[mon]
        debug mon = 20
        debug paxos = 1/5
        debug auth = 2

[osd]
        debug osd = 1/5
        debug filestore = 1/5
        debug journal = 1
        debug monc = 5/20

[mds]
        debug mds = 1
        debug mds balancer = 1
        debug mds log = 1
        debug mds migrator = 1

日志文件rotate参考配置

rotate 7
weekly
compress
sharedscripts
Modify it by adding a size setting.

rotate 7
weekly
size 500M
compress
sharedscripts

Then, start the crontab editor for your user space.

crontab -e
Finally, add an entry to check the etc/logrotate.d/ceph file.

30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph >/dev/null 2>&1

日志参考配置 : 

log file = /var/log/ceph/$cluster-$name.log
mon cluster log file = /var/log/ceph/$cluster.log

[参考]

1. http://blog.163.com/digoal@126/blog/static/1638770402014118103927428/

2. http://ceph.com/docs/master/rados/configuration/ceph-conf/