It’s becoming more common to have both Openstack and Kubernetes clusters in our data centers.
On one side, we have virtual machines managed by Openstack while, on the other side, we have containerized workloads controlled by Kubernetes.
Realistically, it might happen that virtual machines and containers need to talk to each other. If so, we need, somehow to enable communications between two separate clusters.
Being two distinct clusters, each of them has its own networking plane: Openstack will probably rely on OVS while Kubernetes will deploy one of the many available CNIs.
As each cluster is independent from the other, communications must be enabled on a third “element”. This third element might be the ip fabric.
Virtual machines will exit the compute via a provider network and will present themselves on the ip fabric within a vlan. Containers will do something similar.
As a result, we might end up with two vlans and we will allow them to talk to each other by configuring the ip fabric leaf (or spine) as a L2/L3 gw. Alternatively, the L3 gw role might be delegated to an upper layer device like the SDN GW (in this case the ipfabric leaf/spine will only be L2GW).
A key takeaway here is that virtual/containerized workloads become part of the underlay, making service plane (workloads) and network plane (ip fabric) interlaced.
Provisioning a service will also need configuration on the ip fabric underlay. This is a concept I had already talked about here when exploring the advantages of having Contrail in your datacenter.
The same consideration applies here with the addition that we do not only need to connect virtual machines but containers as well.
Let’s assume we need to deploy an application whose building blocks are both virtual machines and containers. This means that we will need to deploy some objects inside Openstack and some other objects inside Kubernetes. From a networking point of view we need to act on Openstack, Kubernetes and the ip fabric. It would be great if this could be simplified. And here comes Contrail!
Contrail can work with both Openstack and Kubernetes (Contrail as a CNI).
Of course, some parts of the code are “orchestrator specific”. By this I mean that Contrail needs all the logic to interact with Neutron/Nova on one side and with Kubernetes API and other K8s components on the other side.
Anyhow, apart from this, the core of the solution is still the same! Virtual networks are still VRFs. Workloads are still interfaces! No matter whether we have a virtual machine or a container behind an interface; to Contrail, it is just an interface to be placed into a VRF. Same holds for overlay (MPLSoUDP, MPLSoGRE, VXLAN) between computes and towards the SDN gateways.
As a result, letting a virtual machine to communicate with a container or viceversa is just a matter of placing interfaces within the same vrf (virtual network). This can be achieved by having a single entity “controlling” the virtual networking of both Openstack and Kubernetes clusters. This entity is our beloved Contrail Controller.
Our datacenter would look like this:
As you can see, from a configuration point of view, clusters are still distinct. Kubernetes master will control containers while Openstack controller will be in charge of Nova VMs.
The key change here is the presence of a Contrail controller that will interact (still using XMPP) with both Openstack compute nodes and Kubernetes workers. This way we provide our clusters a single networking plane.
As Contrail woks at the overlay level, we no longer need our ip fabric to “see” virtual machines and containers. Workloads traffic is hidden inside overlay tunnels (either between compute nodes or towards sdn gateways…exactly like any contrail cluster).
Having a single networking plane brings other advantages. For example, Contrail will take care of ip address allocation. Let’s assume we have a virtual network with address 192.168.1.0/24. If we create a VM on that VN, Contrail will assign address 192.168.1.3. Later, if we create a container in kubernetes and connect it to that same VN, Contrail will assign address 192.168.1.4 as it knows .3 is already in use. With two distinct clusters, achieving this would require additional “tools” (e.g. configure static allocation pools or have a higher level orchestrator acting as IPAM manager). Contrail simplifies network operations!
Now, enough with the theory. Let’s see an example!
I built the above topology in a virtual lab. To deploy Contrail, I used the Ansible deployer which allows us to provision both Openstack and Kubernetes clusters. I will not go into the details of how installing contrail with ansible deployer (here is an example to deploy a K8s cluster); I assume previous knowledge about this tool.
As you know, the core of the ansible deployer is the instances.yaml file. Here is the one I used:
###FILE USED TO DESCRIBE THE CONTRAIL CLOUD THAT ANSIBLE DEPLOYER HAS TO BUILD
global_configuration:
CONTAINER_REGISTRY: hub.juniper.net/contrail
CONTAINER_REGISTRY_USERNAME: xxx
CONTAINER_REGISTRY_PASSWORD: yyy
provider_config:
bms:
ssh_user: root
ssh_pwd: Embe1mpls
ntpserver: 10.102.255.254
domainsuffix: multiorch.contrail
instances:
cnt-control:
provider: bms
ip: 10.102.240.183
roles:
config:
config_database:
control:
webui:
analytics:
analytics_database:
analytics_alarm:
os-control:
provider: bms
ip: 10.102.240.178
roles:
openstack:
os-compute:
provider: bms
ip: 10.102.240.171
roles:
vrouter:
VROUTER_GATEWAY: 192.168.200.1
openstack_compute:
k8s-master:
provider: bms
roles:
k8s_master:
kubemanager:
ip: 10.102.240.174
k8s-worker:
provider: bms
roles:
vrouter:
VROUTER_GATEWAY: 192.168.200.1
k8s_node:
ip: 10.102.240.172
contrail_configuration:
CLOUD_ORCHESTRATOR: openstack
CONTRAIL_CONTAINER_TAG: 2008.121
CONTRAIL_VERSION: 2008.121
OPENSTACK_VERSION: queens
ENCAP_PRIORITY: "VXLAN,MPLSoUDP,MPLSoGRE"
BGP_ASN: 65100
CONFIG_NODEMGR__DEFAULTS__minimum_diskGB: 2
DATABASE_NODEMGR__DEFAULTS__minimum_diskGB: 2
CONFIG_DATABASE_NODEMGR__DEFAULTS__minimum_diskGB: 2
KUBERNETES_CLUSTER_PROJECT: {}
KUBERNETES_API_NODES: 192.168.200.12
KUBERNETES_API_SERVER: 192.168.200.12
KUBEMANAGER_NODES: 192.168.200.12
RABBITMQ_NODE_PORT: 5673
KEYSTONE_AUTH_URL_VERSION: /v3
VROUTER_GATEWAY: 192.168.200.1
CONTROLLER_NODES: 10.102.240.183
CONTROL_NODES: 192.168.200.10
JVM_EXTRA_OPTS: "-Xms1g -Xmx2g"
PHYSICAL_INTERFACE: "ens3f1"
kolla_config:
kolla_globals:
enable_haproxy: no
enable_ironic: no
enable_swift: no
enable_barbican: no
kolla_passwords:
keystone_admin_password: contrail123
metadata_secret: meta123
The instances section includes 5 servers:
- openstack controller
- kubernetes master
- contrail controller
- openstack compute
- kubernetes worker
If you look at the “contrail_configuration” section, you will notice we configure contrail so to interact with both openstack controller (KEYSTONE_AUTH_URL_VERSION) and kubernetes master (KUBERNETES_API_NODES).
Once all the nodes are ready for installation, run these commands from the “deployer” node (which can be the contrail controller itself):
ansible-playbook -e orchestrator=openstack -i inventory/ playbooks/configure_instances.yml
ansible-playbook -e orchestrator=openstack -i inventory/ playbooks/install_openstack.yml
ansible-playbook -e orchestrator=openstack -i inventory/ playbooks/install_k8s.yml
ansible-playbook -e orchestrator=openstack -i inventory/ playbooks/install_contrail.ymlIf everything goes well, we should have our cluster up and running.
Let’s connect to Contrail (Tungsten Fabric) GUI:
We see two virtual routers: os-compute and k8s-worker!
We look at control nodes:
There is one single controller! This our key concept of “single networking plane” turned into reality đŸ™‚
Next, I created a virtual network:
- FQ name: default-domain:k8s-contrail:seamless
- CIDR: 192.168.1.0/24
I launch a VM connected to that VN:
nova boot --image cirros2 --flavor cirros --nic net-id=<seamless_uuid> vm
(kolla-toolbox)[ansible@os-control /]$ nova list
+--------------------------------------+------+--------+------------+-------------+------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+------------------------+
| 3cf82185-5261-4b35-87bf-4eaa9de3caaf | vm | ACTIVE | - | Running | seamless=192.168.100.3 |
+--------------------------------------+------+--------+------------+-------------+------------------------+Next, I create a container attached to that VN:
[root@k8s-master ~]# cat cn.yaml
---
kind: Namespace
apiVersion: v1
metadata:
name: seamless
annotations:
'opencontrail.org/network' : '{"domain":"default-domain", "project": "k8s-contrail", "name":"seamless"}'
labels:
name: seamless
---
apiVersion: v1
kind: Pod
metadata:
name: cont
namespace: seamless
spec:
containers:
- name: cont
image: alpine
command: ["tail"]
args: ["-f", "/dev/null"]
kubectl apply -f vn.yaml
[root@k8s-master ~]# kubectl get pod -n seamless -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cont 1/1 Running 0 74s 192.168.100.4 k8s-worker <none> <none>
[root@k8s-master ~]# kubectl get -f cn.yaml -o wide
NAME STATUS AGE
namespace/seamless Active 31m
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/cont 1/1 Running 0 31m 192.168.100.4 k8s-worker <none> <none>As you can see, pod was assigned ip 192.168.100.4 as 192.168.100.3 was already taken by the VM. This is one of the advantages of having a single networking plane for multiple clusters.
Let’s check routing table from GUI:
We have both IPs! We have just connected a virtual machine to a worload…and this is totally transparent to the underlay!
Let’s move to the worker and check vrouter agent:
(vrouter-agent)[root@k8s-worker /]$ vif --get 3
Vrouter Interface Table
vif0/3 OS: tapeth0-11ccbe NH: 51
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:192.168.100.4
Vrf:4 Mcast Vrf:4 Flags:PL3L2DEr QOS:-1 Ref:6
RX packets:67 bytes:2814 errors:0
TX packets:87 bytes:3654 errors:0
Drops:67
(vrouter-agent)[root@k8s-worker /]$ rt --get 192.168.100.3/32 --vrf 4
Match 192.168.100.3/32 in vRouter inet4 table 0/4/unicast
Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP
vRouter inet4 routing table 0/4/unicast
Destination PPL Flags Label Nexthop Stitched MAC(Index)
192.168.100.3/32 0 LP 25 39 2:85:bf:53:6b:67(96024)
(vrouter-agent)[root@k8s-worker /]$ nh --get 39
Id:39 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:8 Vrf:0
Flags:Valid, MPLSoUDP, Etree Root,
Oif:0 Len:14 Data:56 68 a6 6f 05 ff 56 68 a6 6f 06 f7 08 00
Sip:192.168.200.14 Dip:192.168.200.13All the routing information is there! Container can reach the virtual machine through a MPLSoUPD tunnel connecting the kubernetes worker with the openstack compute.
Is this enough to allow communications? Not yet!
Remember…security groups are still there!
VM belongs to an Openstack project (admin project here) while container belongs to an other project (mapped to a kubernetes namespace). Each project has is own security group. By default, security groups only allow ingress traffic from someone belonging to the same security group. As the two workloads have their interface assigned to different security groups, a dialogue between them is not allowed!
To overcome this, we need to act on security groups.
One easy way is to allow ingress traffic from 0.0.0.0/0 on both security groups (this is the container sec groups but same can be done for the vm sec group):
Alternatively, we can allow a specific security group in the ingress direction.
For example, on k8s-seamless-default-sg (sec group of the namespace/project to which container belongs) we allow default sec group (sec group of the openstack project to which vm belongs to):
Same can be done on the security group assigned to vm interface:
Now, we can access our container and ping the vm:
[root@k8s-worker ~]# docker ps | grep seaml
18c3898a09ac alpine "tail -f /dev/null" 9 seconds ago Up 8 seconds k8s_cont_cont_seamless_e4a7ed6d-38e9-11eb-b8fe-5668a66f06f8_0
6bb1c3b40300 k8s.gcr.io/pause:3.1 "/pause" 17 seconds ago Up 15 seconds k8s_POD_cont_seamless_e4a7ed6d-38e9-11eb-b8fe-5668a66f06f8_0
[root@k8s-worker ~]# docker exec -it 18c38 sh
/ # ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
24: eth0@if25: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
link/ether 02:e4:e5:52:ce:38 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.4/24 scope global eth0
valid_lft forever preferred_lft forever
/ # ping 192.168.100.3
PING 192.168.100.3 (192.168.100.3): 56 data bytes
64 bytes from 192.168.100.3: seq=0 ttl=64 time=3.813 ms
64 bytes from 192.168.100.3: seq=1 ttl=64 time=1.802 ms
64 bytes from 192.168.100.3: seq=2 ttl=64 time=2.260 ms
64 bytes from 192.168.100.3: seq=3 ttl=64 time=1.945 ms
^C
--- 192.168.100.3 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 1.802/2.455/3.813 ms
/ #There it is! VM and container talking to each other!
No matter if your workload is a virtual machine or a container…at least from a networking point of view, everything will be under the Contrail umbrella!
Ciao
IoSonoUmberto
IoSonoUnRouter 