MongoDB replicSet 1.8.1 产品部署推荐:
1. 文件系统加载时使用参数noatime
2. no VM PAGEs
3. 推荐使用逻辑卷,文件系统推荐ext4或xfs
4. 3个full nodes 或 2个full nodes+1个arbiter node (最好是奇数个物理服务器,否则仲裁会有问题，例如两台物理机，两个mongod进程，相互网络不通的话，任何一台都无法达到majority，因此都无法成为primary。那就是只读了.因此本例的物理服务器只有2台是不合理的。)
5. 推荐使用auth,
6. keyFile建议权限400
7. 推荐关闭http访问
8. 建议开启journal , 注意，开启journal后一个逻辑写将产生最多4个物理写
(1main,1journal,1local,1journal)
但是由于IO是异步的，所以一般不会有4个物理写这么严重。
本例环境:
2个full nodes + 1个arbiter node
member1 : 192.168.175.67:5281
member2 : 192.168.175.70:5281
member3(arbiter Only) : 192.168.175.70:5282
详细配置:
1. 操作系统版本 Red Hat Enterprise Linux Server release 5.6 (Tikanga) 64位
2. sshd配置
vi /etc/ssh/sshd_config
PubkeyAuthentication no
UseDNS no
3. ssh配置
vi /etc/ssh/ssh_config
GSSAPIAuthentication no
4. root用户 crontab配置
8 * * * * /usr/sbin/ntpdate asia.pool.ntp.org && /sbin/hwclock --systohc
1 * * * * /usr/local/bin/monitor_entry.sh disk
5. ntpd配置
vi /etc/sysconfig/ntpd
SYNC_HWCLOCK=yes
6. rc.local配置
vi /etc/rc.local
sysctl -w net.ipv4.ip_conntrack_max=655360
sysctl -w net.ipv4.tcp_timestamps=0
7. 服务配置
chkconfig --level 35 cmirror off
chkconfig --level 35 rhnsd off
chkconfig --level 35 ricci off
8. 更新网卡驱动(RHEL5.6不需要更新)
9. sysctl.conf配置
vi /etc/sysctl.conf
kernel.shmmni = 4096
kernel.sem = 50100 64128000 50100 1280
fs.file-max = 7672460
net.ipv4.ip_local_port_range = 9000 65000
net.core.rmem_default = 1048576
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.core.netdev_max_backlog = 10000
net.ipv4.ip_conntrack_max = 655360
fs.aio-max-nr = 1048576
net.ipv4.tcp_timestamps = 0
vm.overcommit_memory = 0
10. vi /etc/pam.d/login
session required pam_limits.so
11. vi /etc/security/limits.conf
* soft nofile 131072
* hard nofile 131072
* soft nproc 131072
* hard nproc 131072
* soft core unlimited
* hard core unlimited
* soft memlock 50000000
* hard memlock 50000000
12. 主机名配置
hostname db-192-168-175-67.sky-mobi.com.hz.sandun
vi /etc/sysconfig/network
HOSTNAME=db-192-168-175-67.sky-mobi.com.hz.sandun
13. vi /etc/resolv.conf
search sky-mobi.com.hz.sandun
nameserver 211.140.188.188
14. 主机名配置
vi /etc/hosts
127.0.0.1 localhost.localdomain localhost
192.168.175.67 db-192-168-175-67.sky-mobi.com.hz.sandun db-192-168-175-67
192.168.175.70 db-192-168-175-70.sky-mobi.com.hz.sandun db-192-168-175-70
15. 密码配置
passwd root
passwd mongo
16. mongo用户profile
vi .bash_profile
export PS1="$USER@`/bin/hostname -s`-> "
export MONGO_HOME=/opt/mongo
export PATH=$MONGO_HOME/bin:$PATH:.
umask 022
alias rm='rm -i'
alias ll='ls -lh'
17. 下载解压最新的稳定版
wget mongodb-linux-x86_64-1.8.1.tar
tar -xvf mongodb-linux-x86_64-1.8.1.tar
mv mongodb-linux-x86_64-1.8.1 /opt/mongo
chown -R mongo:mongo /opt/mongo
18. 建立日志目录
mkdir /var/log/mongo
chown -R mongo:mongo /var/log/mongo
19. 建立数据文件目录和配置文件目录
mkdir -p /opt/mongodata/conf
chown -R mongo:mongo /opt/mongodata
19.1 192.168.175.70上需要多建立一个arbiter的数据目录和配置文件目录
mkdir -p /database/mongodb/data1/mongodata/conf
chown -R mongo:mongo /database/mongodb/data1/mongodata
20. 配置密钥文件:
1.8.1版本开始增加了replicaSet的auth支持,但是replicaSet的member之间通讯认证需要用到keyFile,确保所有的member服务器上都有一个同样的keyFile,确保权限是400的.类似一个密钥文件.
member1 :
echo "this is a key file created by digoal zhou at 20110518 used to auth by replica set members each OTHER" > /opt/mongodata/conf/keyFile
chmod 400 /opt/mongodata/conf/keyFile
member2 :
echo "this is a key file created by digoal zhou at 20110518 used to auth by replica set members each OTHER" > /opt/mongodata/conf/keyFile
chmod 400 /opt/mongodata/conf/keyFile
member3 :
echo "this is a key file created by digoal zhou at 20110518 used to auth by replica set members each OTHER" > /database/mongodb/data1/mongodata/conf/keyFile
chmod 400 /database/mongodb/data1/mongodata/conf/keyFile
20.1 配置启动文件:
member1 & member2 :
vi /opt/mongodata/conf/mongod.conf
member3 :
vi /database/mongodb/data1/mongodata/conf/mongod.conf
logpath=/var/log/mongo/mongod5281.log
logappend=true
fork = true
port = 5281
dbpath=/opt/mongodata
auth = true
nohttpinterface = true
nssize = 128
directoryperdb = true
maxConns = 1500
oplogSize = 10240
keyFile=/opt/mongodata/conf/keyFile
journal=true
profile=1
slowms=100
replSet=blss
logpath=/var/log/mongo/mongod5281.log
logappend=true
fork = true
port = 5281
dbpath=/opt/mongodata
auth = true
nohttpinterface = true
nssize = 128
directoryperdb = true
maxConns = 1500
oplogSize = 10240
keyFile=/opt/mongodata/conf/keyFile
journal=true
profile=1
slowms=100
replSet=blss
logpath=/var/log/mongo/mongod5282.log
logappend=true
fork = true
port = 5282
dbpath=/database/mongodb/data1/mongodata
auth = true
nohttpinterface = true
nssize = 128
directoryperdb = true
maxConns = 1500
oplogSize = 10240
keyFile=/database/mongodb/data1/mongodata/conf/keyFile
journal=true
profile=1
slowms=100
replSet=blss
21. 启动所有节点并初始化replicaSet
member1 & member2
mongod -f /opt/mongodata/conf/mongod.conf
member3 :
mongod -f /database/mongodb/data1/mongodata/conf/mongod.conf
# 初始化(只需要连到一个节点操作)
db.runCommand({replSetInitiate : {
_id : "blss",
members: [
{
_id : 0,
host : "192.168.175.67:5281" ,
arbiterOnly : false
}
,
{
_id : 1,
host : "192.168.175.70:5281" ,
arbiterOnly : false
}
,
{
_id : 2,
host : "192.168.175.70:5282" ,
arbiterOnly : true
}
]
}})
# 等待local初始完成(确保所有节点都正常),添加用户
blss:PRIMARY> rs.status()
{
"set" : "blss",
"date" : ISODate("2011-05-17T10:22:47Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "192.168.175.67:5281",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"optime" : {
"t" : 1305625603000,
"i" : 1
},
"optimeDate" : ISODate("2011-05-17T09:46:43Z"),
"self" : true
},
{
"_id" : 1,
"name" : "192.168.175.70:5281",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 2990,
"optime" : {
"t" : 1305625603000,
"i" : 1
},
"optimeDate" : ISODate("2011-05-17T09:46:43Z"),
"lastHeartbeat" : ISODate("2011-05-17T10:22:46Z")
},
{
"_id" : 2,
"name" : "192.168.175.70:5282",
"health" : 1,
"state" : 7,
"stateStr" : "ARBITER",
"uptime" : 2994,
"optime" : {
"t" : 0,
"i" : 0
},
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2011-05-17T10:22:46Z")
}
],
"ok" : 1
}
# 状态正常,新建用户
mongo 127.0.0.1:5281/admin
db.addUser("xxx","xxxxx");
db.auth("xxx","xxxxx");
# 新增业库blss务用户
use digoal
db.addUser("digoal","Fdigoal-")
22. 查看日志
23. 其他管理命令
rs.?
24. 切换,自动
25. 连接到Replica Sets环境的驱动配置
Connecting Drivers to Replica Sets :
Ideally a MongoDB driver can connect to a cluster of servers which represent a , and automatically find the right set member with which replica set to communicate. Failover should be automatic too. The general steps are:
1. The user, when opening the connection, specifies host[:port] for one or more members of the set. Not all members need be specified -- in fact the exact members of the set might change over time. This list for the connect call is the . seed list
2. The driver then connects to all servers on the seed list, perhaps in parallel to minimize connect time. Send an ismaster command to each server.
3. When the server is in replSet mode, it will return a field with all members of the set that are potentially eligible to serve data. The hosts client should cache this information. Ideally this refreshes too, as the set's config could change over time.
4. Choose a server with which to communicate.
If ismaster == true, that server is primary for the set. This server can be used for writes and immediately consistent reads.
If secondary == true, that server is not primary, but is available for eventually consistent reads. In this case, you can use the field to see which server the master should be. primary
4. If an error occurs with the current connection, find the new primary and resume use there.
26. 配置iptables
27. 配置监控
OTHERs:
关于JAVa连接MONGODB replica set的一个例子 :
Now that we have a replica set, it's time to use it with the Java driver. First, we can connect to the replica set. We can connect to any instance, the driver will fecth the list of other instances and other informations like who is the master. But, it's a good practice to have a list of several nodes to connect to, so if one node we connect to is down, we can fetch the nodes list from the other:
String url = "192.168.175.67:5281,192.168.175.70:5281";
ArrayList<ServerAddress> addr = new ArrayList<ServerAddress>();
for (String s: url.split(",")) {
addr.add(new ServerAddress(s));
}
Mongo mongo = new Mongo(addr);
控制是否要把READ请求发给slave
Then, you can use the driver normally. By default, it will send all the requests, reads and writes, to the master. Bit you can configure the driver to send only writes to the master, the reads will be dispatched on the slaves. And it's only one line of code to do this:
mongo.slaveOk();
28. 扩容和去除节点测试
去掉192.168.175.70:5282 member
增加192.168.175.71:5281 member
1. 首先把192.168.175.71:5281配置好,mongod起来
2. 去掉192.168.175.70:5282
连接到主节点
mongo 127.0.0.1:5281/admin
db.auth("digoal","pwd")
blss:PRIMARY> rs.conf()
{
"_id" : "blss",
"version" : 1,
"members" : [
{
"_id" : 0,
"host" : "192.168.175.67:5281"
},
{
"_id" : 1,
"host" : "192.168.175.70:5281"
},
{
"_id" : 2,
"host" : "192.168.175.70:5282",
"arbiterOnly" : true
}
]
}
blss:PRIMARY> rs.remove("192.168.175.70:5282")
完成后会断开重连
Fri May 20 09:29:06 trying reconnect to 127.0.0.1:5281
Fri May 20 09:29:06 reconnect 127.0.0.1:5281 ok
因此需要重新认证
blss:PRIMARY> db.auth("digoal","pwd")
blss:PRIMARY> rs.conf()
{
"_id" : "blss",
"version" : 2,
"members" : [
{
"_id" : 0,
"host" : "192.168.175.67:5281"
},
{
"_id" : 1,
"host" : "192.168.175.70:5281"
}
]
}
等待192.168.175.71:5281 member节点起来后
blss:PRIMARY> rs.add({"_id" : 2,"host" : "192.168.175.71:5281"})
完成后会断开重连
Fri May 20 09:31:44 trying reconnect to 127.0.0.1:5281
Fri May 20 09:31:44 reconnect 127.0.0.1:5281 ok
因此需要重新认证
blss:PRIMARY> db.auth("digoal","pwd")
blss:PRIMARY> rs.conf()
{
"_id" : "blss",
"version" : 3,
"members" : [
{
"_id" : 0,
"host" : "192.168.175.67:5281"
},
{
"_id" : 1,
"host" : "192.168.175.70:5281"
},
{
"_id" : 2,
"host" : "192.168.175.71:5281"
}
]
}
blss:PRIMARY> rs.status()
{
"set" : "blss",
"date" : ISODate("2011-05-20T01:32:56Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "192.168.175.67:5281",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"optime" : {
"t" : 1305855176000,
"i" : 507
},
"optimeDate" : ISODate("2011-05-20T01:32:56Z"),
"self" : true
},
{
"_id" : 1,
"name" : "192.168.175.70:5281",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 70,
"optime" : {
"t" : 1305855174000,
"i" : 817
},
"optimeDate" : ISODate("2011-05-20T01:32:54Z"),
"lastHeartbeat" : ISODate("2011-05-20T01:32:54Z")
},
{
"_id" : 2,
"name" : "192.168.175.71:5281",
"health" : 1,
"state" : 3,
"stateStr" : "RECOVERING",
"uptime" : 66,
"optime" : {
"t" : 0,
"i" : 0
},
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2011-05-20T01:32:54Z")
}
],
"ok" : 1
}
由于数据量较大,RECOVERING可能需要很长时间.
1.8.1支持从SECONDARY同步,因此对主节点压力不大.