Mirroring traffic to local and remote DPIs

When something is not working, we might desire a copy of the traffic to analyze it and see if there is anything wrong.

In other situations, we might want to send a copy of all the traffic towards a DPI appliance.

In both cases, we need to mirror traffic.

I’ve built a very small lab to learn how mirroring on MX series works. I’ve focused on two use-cases:

  • mirroring to a locally connected DPI/analyzer
  • mirroring to a remote DPI/analyzer using GRE tunnel

This is the lab topology

The idea is to catch traffic from client and mirror it towards the DPI, either via direct link (local use-case) or via GRE tunnel (remote use-case).

Let’s start from the local use-case.

We configure the interface towards the client:

set interfaces ge-0/0/0 unit 0 family inet address 192.168.1.1/24

and the one towards the DPI:

set interfaces ge-0/0/1 unit 0 family inet address 192.168.2.1/31

Next, we define a port-mirroring instance:

set forwarding-options port-mirroring instance direct input rate 1
set forwarding-options port-mirroring instance direct input run-length 1
set forwarding-options port-mirroring instance direct family inet output interface ge-0/0/2.0 next-hop 192.168.5.2

Within that instance,we tell junos to mirror every single packet (rate 1, run-length 1) and send mirrored traffic out of interface ge-0/0/2. We also need to specify the next-hop as we cannot perform usual lookup for mirrored traffic (that most likely would be sent through another interface).

The port mirroring instance is associated to a FPC:

set chassis fpc 0 port-mirror-instance gre

Then, we define a firewall filter to accept and mirror traffic:

set firewall family inet filter mirror term mirror then count mirrored
set firewall family inet filter mirror term mirror then port-mirror-instance gre
set firewall family inet filter mirror term mirror then accept

and we apply the filter to the client facing interface:

set interfaces ge-0/0/0 unit 0 family inet filter input mirror
set interfaces ge-0/0/0 unit 0 family inet filter output mirror

We configure the same filter in both directions in order to mirror both upstream and downstream packets.

We verify mirroring is working by checking on the dpi if we receive mirrored traffic.

[root@dpi labtopo_versioning]# ifconfig ens3f1
ens3f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.5.2  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::5468:a6ff:fe6a:14e3  prefixlen 64  scopeid 0x20<link>
        ether 56:68:a6:6a:14:e3  txqueuelen 1000  (Ethernet)
        RX packets 330  bytes 142886 (139.5 KiB)
        RX errors 15  dropped 0  overruns 0  frame 15
        TX packets 85  bytes 11024 (10.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@dpi labtopo_versioning]# tcpdump -i ens3f1 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3f2, link-type EN10MB (Ethernet), capture size 262144 bytes
07:09:18.184931 IP 192.168.1.2 > 75.1.2.100: ICMP echo request, id 30407, seq 1437, length 1008
07:09:18.185930 IP 192.168.1.1 > 192.168.1.2: ICMP net 75.1.2.100 unreachable, length 36
07:09:19.186083 IP 192.168.1.2 > 75.1.2.100: ICMP echo request, id 30407, seq 1438, length 1008
07:09:19.381347 IP 192.168.1.1 > 192.168.1.2: ICMP net 75.1.2.100 unreachable, length 36

We also have a verification command on junos:

root@mirror# run show forwarding-options port-mirroring
Instance Name: direct
  Instance Id: 2
  Input parameters:
    Rate                  : 1
    Run-length            : 1
    Maximum-packet-length : 0
  Output parameters:
    Family              State     Destination          Next-hop
    inet                up        ge-0/0/2.0           192.168.5.2

Now, let’s move to the remote use-case.

We configure a gre tunnel from mirror to r2 (through r1). To allow this, I configured ospf between those machines so to exchange loopback addresses. I omit this configuration.

This is gre interface configuration:

set chassis fpc 0 pic 0 tunnel-services bandwidth 10g
set interfaces gr-0/0/0 unit 1 tunnel source 3.3.3.3
set interfaces gr-0/0/0 unit 1 tunnel destination 2.2.2.2
set interfaces gr-0/0/0 unit 1 family inet address 10.10.10.0/31

I create another mirroring instance:

set forwarding-options port-mirroring instance gre input rate 1
set forwarding-options port-mirroring instance gre input run-length 1
set forwarding-options port-mirroring instance gre family inet output interface gr-0/0/0.1
set chassis fpc 0 port-mirror-instance gre

We edit the filter so to match this new instance:

set firewall family inet filter mirror term mirror then count mirrored
set firewall family inet filter mirror term mirror then port-mirror-instance gre
set firewall family inet filter mirror term mirror then accept

We start traffic and check on dpi (this time on interface ens2f1):

[root@dpi labtopo_versioning]# tcpdump -i ens3f2 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3f1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

Nothing!

Why that?

Well, mirrored traffic travel through the GRE tunnel and land on r2. Things break on r2 as that router does not know it has to send traffic to the DPI.

How to solve it? We need to configure mirroring on r2 as well!

This is easy now!

First, mirroring instance (this time we use global instance):

set forwarding-options port-mirroring input rate 1
set forwarding-options port-mirroring input run-length 1
set forwarding-options port-mirroring family inet output interface ge-0/0/1.0 next-hop 192.168.4.2
set interfaces ge-0/0/1 unit 0 family inet address 192.168.4.1/24

Second, we create a mirror filter and apply it to the gre interface:

set firewall family inet filter mirror term mirror then count mirrored
set firewall family inet filter mirror term mirror then port-mirror
set firewall family inet filter mirror term mirror then accept
set interfaces gr-0/0/0 unit 0 tunnel source 2.2.2.2
set interfaces gr-0/0/0 unit 0 tunnel destination 3.3.3.3
set interfaces gr-0/0/0 unit 0 family inet filter input mirror
set interfaces gr-0/0/0 unit 0 family inet address 10.10.10.1/31

This time, we only need input direction as we are already receiving both upstream and downstream traffic as the result of mirroring performed on the router connected to the client.

That’s it! Now mirroring works!

Are we done? Not yet. There is a third use-case.

Assume we want to send mirrored traffic to multiple DPIs. This can be done by using so-called next-hop-groups. Here, due to lab limitation I only have one DPI so i will configure mirroring to send traffic through two paths but, ultimately, they will end up on the same DPI (via different interfaces).

This is how to achieve it on router mirror:

set forwarding-options port-mirroring instance double input rate 1
set forwarding-options port-mirroring instance double input run-length 1
set forwarding-options port-mirroring instance double family inet output next-hop-group mgroup
set forwarding-options next-hop-group mgroup group-type inet
set forwarding-options next-hop-group mgroup interface gr-0/0/0.1
set forwarding-options next-hop-group mgroup interface ge-0/0/2.0 next-hop 192.168.5.2

Mirroring instance now references a next-hop-group.

Next-hop-group then lists two output interfaces:

  • a local interface (the one we used for local use-case)
  • a gre tunnel (the one we used for remote use-case)

We verify mirroring is up:

root@mirror# run show forwarding-options port-mirroring
Instance Name: double
  Instance Id: 4
  Input parameters:
    Rate                  : 1
    Run-length            : 1
    Maximum-packet-length : 0
  Output parameters:
    Family              State     Destination          Next-hop
    inet                up        mgroup

Of course, we update the firewall filter:

set firewall family inet filter mirror term mirror then port-mirror-instance gre

and we are done!

No changes are needed on r2 as it will keep mirroring traffic arriving on its gre interface.

Let’s go to the DPI and see what traffic is coming:

[root@dpi labtopo_versioning]# tcpdump -n -i any icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
07:27:46.640517 IP 192.168.1.2 > 75.1.2.100: ICMP echo request, id 30407, seq 2544, length 1008
07:27:46.640715 IP 192.168.1.1 > 192.168.1.2: ICMP net 75.1.2.100 unreachable, length 36
07:27:46.641508 IP 192.168.1.2 > 75.1.2.100: ICMP echo request, id 30407, seq 2544, length 1008
07:27:46.641524 IP 192.168.1.1 > 192.168.1.2: ICMP net 75.1.2.100 unreachable, length 36

As you may notice, we are receiving the same icmp packet (seq 2544) twice: one from interface ens3f1 (local connection to mirror router) and one from ens3f2 (remote connection to mirror router via r2 and a gre tunnel).

Now we are done!

Go and mirror 🙂

Ciao
IoSonoUmberto

Leave a comment