使用 docker 后出现的网络异常现象

硬件:

[root@sh-storage-128204 ~]# dmidecode -t system
 dmidecode 2.12
SMBIOS 2.7 present.

Handle 0x0001, DMI type 1, 27 bytes
System Information
        Manufacturer: To be filled by O.E.M.
        Product Name: Tecal RH2288H V2-24S
        Version: V100R002
        Serial Number: 2102310QPE10E9000146
        UUID: 4A190814-D21D-B211-8DC0-000000821800
        Wake-up Type: Power Switch
        SKU Number: Type1Sku0
        Family: Type1Family

环境:

[root@sh-storage-128204 ~]# docker info
Containers: 4
Images: 153
Storage Driver: devicemapper
 Pool Name: docker-8:17-1075308946-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 3.478 GB
 Data Space Total: 107.4 GB
 Data Space Available: 103.9 GB
 Metadata Space Used: 6.689 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.141 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2015-10-14)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 24
Total Memory: 125.7 GiB
Name: sh-storage-128204.sh.vclound.com
ID: QV4X:VHQE:EMOI:4TBJ:FZ6K:3N3C:A64Y:PRYR:X2QZ:HHUB:OTND:ZSFF

用法:
用于搭建 docker private registry
架构:

                  nginx    <- 本地 docker
                    |
    |---------------+-------------------|
   registry       registry        registry  <- 本地 docker
    |---------------+-------------------|
                    |
                ceph 集群 (rados) <- 另外一个集群

问题:
搭建了 docker, 启动成功后, 发现上传镜像时偶尔会出现连接卡住问题
检测与 mon 连接发现:

[root@sh-storage-128204 ~]# nmap -p 6789 10.198.128.200
Starting Nmap 6.40 ( http://nmap.org ) at 2016-01-20 09:47 CST
sendto in send_ip_packet_sd: sendto(5, packet, 44, 0, 10.198.128.200, 16) => Operation not permitted
Offending packet: TCP 10.198.128.204:43301 > 10.198.128.200:6789 S ttl=48 id=11619 iplen=44  seq=2271879518 win=1024 <mss 1460>
sendto in send_ip_packet_sd: sendto(5, packet, 44, 0, 10.198.128.200, 16) => Operation not permitted
Offending packet: TCP 10.198.128.204:43302 > 10.198.128.200:6789 S ttl=45 id=9421 iplen=44  seq=2271945055 win=1024 <mss 1460>
Nmap scan report for sh-storage-128200.sh.vclound.com (10.198.128.200)
Host is up (0.000091s latency).
PORT     STATE    SERVICE
6789/tcp filtered ibm-db2-admin
MAC Address: 90:E2:BA:85:21:28 (Intel Corporate)
Nmap done: 1 IP address (1 host up) scanned in 0.29 seconds

[root@sh-storage-128204 ~]# nmap -p 6789 10.198.128.200
Starting Nmap 6.40 ( http://nmap.org ) at 2016-01-20 09:47 CST
sendto in send_ip_packet_sd: sendto(5, packet, 44, 0, 10.198.128.200, 16) => Operation not permitted
Offending packet: TCP 10.198.128.204:48544 > 10.198.128.200:6789 S ttl=54 id=17389 iplen=44  seq=1275261731 win=1024 <mss 1460>
sendto in send_ip_packet_sd: sendto(5, packet, 44, 0, 10.198.128.200, 16) => Operation not permitted
Offending packet: TCP 10.198.128.204:48545 > 10.198.128.200:6789 S ttl=53 id=40168 iplen=44  seq=1275327266 win=1024 <mss 1460>
Nmap scan report for sh-storage-128200.sh.vclound.com (10.198.128.200)
Host is up (0.000092s latency).
PORT     STATE    SERVICE
6789/tcp filtered ibm-db2-admin
MAC Address: 90:E2:BA:85:21:28 (Intel Corporate)
Nmap done: 1 IP address (1 host up) scanned in 0.28 seconds

[root@sh-storage-128204 ~]# nmap -p 6789 10.198.128.200
Starting Nmap 6.40 ( http://nmap.org ) at 2016-01-20 09:47 CST
Nmap scan report for sh-storage-128200.sh.vclound.com (10.198.128.200)
Host is up (0.00039s latency).
PORT     STATE SERVICE
6789/tcp open  ibm-db2-admin
MAC Address: 90:E2:BA:85:21:28 (Intel Corporate)
Nmap done: 1 IP address (1 host up) scanned in 0.21 seconds

与 mon 连接时, 会经常出现 6789/tcp filtered ibm-db2-admin 异常现象

另外, 参考系统日志
会出现下面异常 (过滤了大部分, 只显示重要部分信息

/var/log/messages
Jan 20 09:49:16 sh-storage-128204 kernel: nf_conntrack: table full, dropping packet
Jan 20 09:49:16 sh-storage-128204 kernel: nf_conntrack: table full, dropping packet
Jan 20 09:49:16 sh-storage-128204 kernel: nf_conntrack: table full, dropping packet
....
Jan 20 09:49:44 sh-storage-128204 kernel: net_ratelimit: 137 callbacks suppressed
Jan 20 09:49:49 sh-storage-128204 kernel: net_ratelimit: 166 callbacks suppressed
Jan 20 09:49:54 sh-storage-128204 kernel: net_ratelimit: 64 callbacks suppressed
Jan 20 09:50:01 sh-storage-128204 kernel: net_ratelimit: 132 callbacks suppressed

ping 也会出现异常现场

[root@sh-storage-128204 ~]# ping localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
ping: sendmsg: Operation not permitted

原因: 当前的网络连接数过大, 导致内核无法响应

解决方法:
修改 /et/sysctl.conf 进行内核参数优化

[root@sh-storage-128204 ~]# sysctl  -p
vm.swappiness = 10
net.ipv4.ip_forward = 1
net.ipv4.tcp_max_syn_backlog = 8192
net.core.netdev_max_backlog = 8192
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.netfilter.nf_conntrack_max = 1048576
net.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_tcp_timeout_established = 54000
net.netfilter.nf_conntrack_generic_timeout = 120