use process's network device namespace on CentOS 6.5+ x64 by openstack modified iproute package

在阅读docker 高级网络时, 发现原来还可以设置进程级别的网络设备namespace.

我这里的环境是CentOS 6.5 x64, 这个版本的iproute包还比较老, 不支持ip netns指令, 所以需要更新一下, 参考本文末尾,

如果你使用的也是CentOS 6.5, 请务必更新iproute后再来做这个实验.

使用ip netns自定义docker container的网络配置例子.

1. 启动2个container, 并且使用--net=none, 即不分配网络设备.

# Start up two containers in two terminal windows

[root@db-172-16-3-221 ~]# docker run -i -t --rm --net=none --name a centos:centos6 /bin/bash
bash-4.1# 

[root@db-172-16-3-221 ~]# docker run -i -t --rm --net=none --name b centos:centos6 /bin/bash
bash-4.1# 

启动后, 我们使用ip link 只能看到回环口

bash-4.1# ip link show
24: lo: <LOOPBACK,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

2. 接下来要获取两个container的pid, 一会要给这两个进程设置网络设备.

# Learn the container process IDs
# and create their namespace entries

[root@db-172-16-3-221 ~]# docker inspect -f '{{.State.Pid}}' a
39162
[root@db-172-16-3-221 ~]# docker inspect -f '{{.State.Pid}}' b
39221

3. 创建netns进程空间目录. 并将进程的net目录软链接过来, 命名为进程号.

[root@db-172-16-3-221 ~]# mkdir -p /var/run/netns
[root@db-172-16-3-221 ~]# ln -s /proc/39162/ns/net /var/run/netns/39162
[root@db-172-16-3-221 ~]# ln -s /proc/39221/ns/net /var/run/netns/39221

4. 在本地创建2个pointopoint端口的链路

# Create the "peer" interfaces and hand them out

[root@db-172-16-3-221 ~]# ip link add A type veth peer name B

可以看到, 当前多了A,B两个接口

[root@db-172-16-3-221 ~]# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 5c:f3:fc:94:12:00 brd ff:ff:ff:ff:ff:ff
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 5c:f3:fc:94:12:02 brd ff:ff:ff:ff:ff:ff
4: usb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 5e:f3:fc:95:12:03 brd ff:ff:ff:ff:ff:ff
5: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
32: B: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether ea:86:64:96:5f:a5 brd ff:ff:ff:ff:ff:ff
33: A: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 6e:81:ef:3f:9e:5a brd ff:ff:ff:ff:ff:ff

5. 接下来的操作因为centos 6.5 不支持ip netns控制, (仅仅在6.5中添加了namespace支持), 所以需要添加openstack改过的包, 

具体的安装步骤参考本文末尾的参考部分.

在ubuntu 14.04中是默认支持ip netns的.

将本地端口A,B分别指派给container进程, 并配置IP地址, 路由, 启动端口.

[root@db-172-16-3-221 ~]# ip link set A netns 39162
[root@db-172-16-3-221 ~]# ip netns exec 39162 ip addr add 10.1.1.1/32 dev A
[root@db-172-16-3-221 ~]# ip netns exec 39162 ip link set A up
[root@db-172-16-3-221 ~]# ip netns exec 39162 ip route add 10.1.1.2/32 dev A

[root@db-172-16-3-221 ~]# ip link set B netns 39221
[root@db-172-16-3-221 ~]# ip netns exec 39221 ip addr add 10.1.1.2/32 dev B
[root@db-172-16-3-221 ~]# ip netns exec 39221 ip link set B up
[root@db-172-16-3-221 ~]# ip netns exec 39221 ip route add 10.1.1.1/32 dev B

将端口分配给container后, 在本地就看不到这两个端口了, 所以必须使用ip netns exec来操作这两个端口.

[root@db-172-16-3-221 ~]# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 5c:f3:fc:94:12:00 brd ff:ff:ff:ff:ff:ff
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 5c:f3:fc:94:12:02 brd ff:ff:ff:ff:ff:ff
4: usb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 5e:f3:fc:95:12:03 brd ff:ff:ff:ff:ff:ff
5: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff

配置完后, 在container中可以查询到端口信息, 相互也能ping通 : 

bash-4.1# ifconfig
A         Link encap:Ethernet  HWaddr 0A:42:B9:F9:EA:59
          inet addr:10.1.1.1  Bcast:0.0.0.0  Mask:255.255.255.255
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

bash-4.1# ifconfig
B         Link encap:Ethernet  HWaddr E6:92:25:AB:A0:68
          inet addr:10.1.1.2  Bcast:0.0.0.0  Mask:255.255.255.255
          inet6 addr: fe80::e492:25ff:feab:a068/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:468 (468.0 b)  TX bytes:468 (468.0 b)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

bash-4.1# ping 10.1.1.2
PING 10.1.1.2 (10.1.1.2) 56(84) bytes of data.
64 bytes from 10.1.1.2: icmp_seq=1 ttl=64 time=0.025 ms
^C
--- 10.1.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 802ms
rtt min/avg/max/mdev = 0.025/0.025/0.025/0.000 ms
bash-4.1# ping 10.1.1.1
PING 10.1.1.1 (10.1.1.1) 56(84) bytes of data.
64 bytes from 10.1.1.1: icmp_seq=1 ttl=64 time=0.026 ms
^C
--- 10.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 518ms
rtt min/avg/max/mdev = 0.026/0.026/0.026/0.000 ms
bash-4.1# ip route
10.1.1.1 dev B  scope link 

[参考]

1. man ip

ip link netns 相关操作说明 :
       ip link set DEVICE { up | down | arp { on | off } |
               promisc { on | off } |
               allmulticast { on | off } |
               dynamic { on | off } |
               multicast { on | off } |
               txqueuelen PACKETS |
               name NEWNAME |
               address LLADDR | broadcast LLADDR |
               mtu MTU |
               netns PID |
               netns NETNSNAME |
               alias NAME |
               vf NUM [ mac LLADDR ] [ vlan VLANID [ qos VLAN-QOS ] ] [ rate TXRATE ]  }
       netns PID
              move the device to the network namespace associated with the process PID.
       netns NETNSNAME
              move the device to the network namespace associated with name NETNSNAME.

ip netns 相关操作说明 :
       ip netns { list | monitor }
       ip netns { add | delete } NETNSNAME
       ip netns exec NETNSNAME command ...
ip netns - process network namespace management
       A network namespace is logically another copy of the network stack, with it’s own routes, firewall  rules,  and
       network devices.

       By  convention  a  named  network  namespace  is an object at /var/run/netns/NAME that can be opened.  The file
       descriptor resulting from opening /var/run/netns/NAME refers to the specified network namespace.  Holding  that
       file descriptor open keeps the network namespace alive.  The file descriptor can be used with the setns(2) sys-
       tem call to change the network namespace associated with a task.

       The convention for network namespace aware applications is to look for global network configuration files first
       in /etc/netns/NAME/ then in /etc/.  For example, if you want a different version of /etc/resolv.conf for a net-
       work namespace used to isolate your vpn you would name it /etc/netns/myvpn/resolv.conf.

       ip netns exec automates handling of this configuration, file convention for network namespace unaware  applica-
       tions,  by  creating  a mount namespace and bind mounting all of the per network namespace configure files into
       their traditional location in /etc.

   ip netns list - show all of the named network namespaces
   ip netns monitor - report when network namespace names are created and destroyed
   ip netns add NAME - create a new named network namespace
   ip netns delete NAME - delete the name of a network namespace
   ip netns exec NAME cmd ... - Run cmd in the named network namespace

2. man ip-netns  (ubuntu)

3. https://github.com/amotoki/openvnet-test-tools/blob/master/README.md

我在写本文的时候, juno下已经没有epel6的目录了, 所以选择了havana, 如下

https://repos.fedorapeople.org/repos/openstack/openstack-havana/epel-6/

你需要在yum里面修改一下, baseurl里juno改成 havana.

[root@db-172-16-3-221 yum.repos.d]# vi rdo-release.repo
[openstack-juno]
name=OpenStack Juno Repository
baseurl=http://repos.fedorapeople.org/repos/openstack/openstack-havana/epel-6/
enabled=1
skip_if_unavailable=0
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-RDO-Juno

升级iproute包

[root@db-172-16-3-221 yum.repos.d]# yum upgrade iproute
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Upgrade Process
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * epel: mirrors.ustc.edu.cn
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
openstack-juno                                                                                               | 2.9 kB     00:00
openstack-juno/primary_db                                                                                    | 975 kB     02:13
updates                                                                                                      | 3.4 kB     00:00
updates/primary_db                                                                                           | 188 kB     00:00
Resolving Dependencies
--> Running transaction check
---> Package iproute.x86_64 0:2.6.32-31.el6 will be updated
---> Package iproute.x86_64 0:2.6.32-130.el6ost.netns.2 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================================================
 Package                  Arch                    Version                                     Repository                       Size
====================================================================================================================================
Updating:
 iproute                  x86_64                  2.6.32-130.el6ost.netns.2                   openstack-juno                  367 k

Transaction Summary
====================================================================================================================================
Upgrade       1 Package(s)

Total download size: 367 k
Is this ok [y/N]: y

Network Namespace

In CentOS 6.x, network namespace is not supported, so we need to additional packages to use network namespace. OpenStack RDO provides network namespace aware kernel and iproute.

CentOS 6.5

In CentOS 6.5, kernel itself supports network namespace, but iproute does not support network namespace (no "netns" subcommand).

The following commands setup OpenStack RDO release repository to /etc/yum.repos.d and "yum update" installs a new version of iproute (which supports network namespace),

# yum install http://rdo.fedorapeople.org/rdo-release.rpm
# yum install iproute

or you can install iproute package explicitly.

# yum install yum install http://rdo.fedorapeople.org/rdo-release.rpm

After the installation, check the appropriate iproute package is installed and it works.

# rpm -qa | grep iproute
iproute-2.6.32-130.el6ost.netns.2.x86_64
# ip netns
# ip netns add testns1
# ip netns
testns1
# ip netns delete testns1

CentOS 6.4 or older

I have not tested these version, but ROD provides network namespace aware kernel. The packages are available at:http://repos.fedorapeople.org/repos/openstack/openstack-havana/epel-6/