NX-OSv 9K and Programmable Fabric with VXLAN BGP EVPN - Part I
Nowadays, where all workloads and applications are being deployed in virtual environment and demand for workload mobility really high, the traditional designs for data centers are no longer sufficient to address all the requirements. Data Center design evolved from STP based approach to vPC based design that addresses some of major issues, like convergence time, unused links, ECMP and so on. vPC approach was awesome it fixed some STP drawbacks but not all of them and it was not enough to accommodate all the requirements of the evolving data centers. That's when Clos Topology Fabric with VXLAN BGP EVPN came on stage. For those who are not familiar with VXLAN BGP EVPN fabric here is useful URLs:
Deploy a VXLAN Network with an MP-BGP EVPN Control Plane White Paper
Cisco Programmable Fabric with VXLAN BGP EVPN Configuration Guide
If you like learning something new and building network simulations like I do you will be excited to hear that Cisco released version 7.0(3)I6(1) of NX-OSv and VXLAN BGP EVPN supported in this version. (Release Notes)Deploy a VXLAN Network with an MP-BGP EVPN Control Plane White Paper
Cisco Programmable Fabric with VXLAN BGP EVPN Configuration Guide
Well, if we can do it why not to play with VXLAN. Let's create some basic EVPN Fabric. Here is my setup parameters.
- Fabric IGP - ISIS for Underlay
- Fabric BGP AS 65010
- VTEP IP Range: 10.0.0.0/24
- Router-ID Range 10.1.0.0/24
- L3 (Tenant1) VNI : 900001
- L2 (Tenant1) VNI: 200991
- L2 (Tenant1) VNI: 200992
- L3 (Tenant2) VNI : 900002
- L2 (Tenant2) VNI: 200993
Topology Diagram:
First of all let's talk briefly about L2/L3 VNs and routing principles used in VXLAN BGP EVPN fabric. Layer-2 virtual network (VN) represent L2 segment and nodes in that segment are able to communicate between each other. Communication between servers or end hosts of the same Layer-2 virtual network is bridged like if they located in the same VLAN. L2 VNI is used for VXLAN tagging when packet destined to the node that resides in the same L2 virtual network segment. Communication between end hosts that belong to different Layer-2 virtual networks represents Layer-3 communication(routing), and L3 VNI is used to tag VXLAN packet routed between different L2 VNI. You can see it below:
Note that L2 and L3 VNIs are unique and global significant across the VXLAN BGP EVPN fabric. L2 VNI and L3 VNI can't overlap cause they are using the same VNI field in VXLAN encapsulated packet. All the Layer-2 virtual networks of a single VRF or tenant are associated with a unique Layer-3 VNI (VRF/Tenant VNI).
To make out virtual setup working we need to establish basic connectivity, enable and configure IGP(IS-IS), BGP L2VPN EVNP and multicast routing for MDT(Mutli-Destination Traffic). SPINEs will serve as a BGP RR and anycast RP Nodes.
First of all we need to create underlay for future fabric. I like IS-IS so I will use it as an IGP protocol for my underlay.
I am going to use ip unnumbered feature in order to save IP address space and simplify management.
SPINE Switches Underlay configuration:
hostname SPINE1
feature pim
feature isis
boot nxos bootflash:nxos.7.0.3.I6.1.bin
! Configuring Level-2 Only ISIS
router isis UNDERLAY
net 47.0001.0100.0000.0001.00
is-type level-2
log-adjacency-changes
address-family ipv4 unicast
passive-interface default level-2
! Router ID Loopback interface
interface loopback1
description : RID
ip address 10.1.0.1/32
ip router isis UNDERLAY
ip pim sparse-mode
! VTEP Source Interface (Optional for SPINE. Added for consistency)
interface loopback0
description : VTEP
ip address 10.0.0.1/32
ip router isis UNDERLAY
ip pim sparse-mode
! Anycast RP interface
interface loopback254
description : Anycast RP
ip address 10.0.0.254/32
ip router isis UNDERLAY
ip pim sparse-mode
! RP Configuration
ip pim rp-address 10.0.0.254 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8
ip pim anycast-rp 10.0.0.254 10.0.0.1
ip pim anycast-rp 10.0.0.254 10.0.0.2
! Fabric interfaces
interface Ethernet1/1 - 3
no switchport
mtu 9000
medium p2p
ip unnumbered loopback1
ip router isis UNDERLAY
no isis passive-interface level-2
ip pim sparse-mode
no shutdown
LEAF Switches Underlay configuration:
hostname LEAF01
feature pim
feature isis
boot nxos bootflash:nxos.7.0.3.I6.1.bin
! Configuring Level-2 Only ISIS
router isis UNDERLAY
net 47.0004.0100.0000.0004.00
is-type level-2
log-adjacency-changes
address-family ipv4 unicast
passive-interface default level-2
! Router ID loopback
interface loopback1
description : RID
ip address 10.1.0.4/32
ip router isis UNDERLAY
ip pim sparse-mode
! VTEP Source Interface
interface loopback0
description : VTEP
ip address 10.0.0.4/32
ip router isis UNDERLAY
ip pim sparse-mode
! Define PIM RP
ip pim rp-address 10.0.0.254 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8
! Fabric interfaces
interface Ethernet1/1 - 2
no switchport
mtu 9000
medium p2p
ip unnumbered loopback1
ip router isis UNDERLAY
no isis passive-interface level-2
ip pim sparse-mode
no shutdown
ISIS is up, basic connectivity established and loopbacks are advertised by IGP and we can reach leaf4 from leaf1:
SPINE02# sh isis adjacency
IS-IS process: UNDERLAY VRF: default
IS-IS adjacency database:
Legend: '!': No AF level connectivity in given topology
System ID SNPA Level State Hold Time Interface
LEAF01 N/A 2 UP 00:00:24 Ethernet1/1
LEAF02 N/A 2 UP 00:00:26 Ethernet1/2
LEAF03 N/A 2 UP 00:00:29 Ethernet1/3
SPINE2# sh ip ro isis-UNDERLAY
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF
10.0.0.1/32, ubest/mbest: 3/0
*via 10.1.0.4, Eth1/1, [115/81], 00:01:18, isis-UNDERLAY, L2
*via 10.1.0.5, Eth1/2, [115/81], 00:00:24, isis-UNDERLAY, L2
*via 10.1.0.6, Eth1/3, [115/81], 00:00:58, isis-UNDERLAY, L2
10.0.0.4/32, ubest/mbest: 1/0
*via 10.1.0.4, Eth1/1, [115/41], 00:01:18, isis-UNDERLAY, L2
via 10.0.0.4, Eth1/1, [250/0], 00:05:27, am
10.0.0.5/32, ubest/mbest: 1/0
*via 10.1.0.5, Eth1/2, [115/41], 00:00:24, isis-UNDERLAY, L2
via 10.0.0.5, Eth1/2, [250/0], 00:04:23, am
10.0.0.6/32, ubest/mbest: 1/0
*via 10.1.0.6, Eth1/3, [115/41], 00:00:58, isis-UNDERLAY, L2
via 10.0.0.6, Eth1/3, [250/0], 00:04:05, am
10.1.0.1/32, ubest/mbest: 3/0
*via 10.1.0.4, Eth1/1, [115/81], 00:01:18, isis-UNDERLAY, L2
*via 10.1.0.5, Eth1/2, [115/81], 00:00:24, isis-UNDERLAY, L2
*via 10.1.0.6, Eth1/3, [115/81], 00:00:58, isis-UNDERLAY, L2
10.1.0.4/32, ubest/mbest: 1/0
*via 10.1.0.4, Eth1/1, [115/80], 00:01:18, isis-UNDERLAY, L2
via 10.1.0.4, Eth1/1, [250/0], 00:01:18, am
10.1.0.5/32, ubest/mbest: 1/0
*via 10.1.0.5, Eth1/2, [115/80], 00:00:24, isis-UNDERLAY, L2
via 10.1.0.5, Eth1/2, [250/0], 00:00:24, am
10.1.0.6/32, ubest/mbest: 1/0
*via 10.1.0.6, Eth1/3, [115/80], 00:00:58, isis-UNDERLAY, L2
via 10.1.0.6, Eth1/3, [250/0], 00:00:58, am
SPINE2#
LEAF01# ping 10.0.0.6 source 10.0.0.4
PING 10.0.0.6 (10.0.0.6) from 10.0.0.4: 56 data bytes
64 bytes from 10.0.0.6: icmp_seq=0 ttl=253 time=34.387 ms
64 bytes from 10.0.0.6: icmp_seq=1 ttl=253 time=8.032 ms
64 bytes from 10.0.0.6: icmp_seq=2 ttl=253 time=7.66 ms
64 bytes from 10.0.0.6: icmp_seq=3 ttl=253 time=6.233 ms
64 bytes from 10.0.0.6: icmp_seq=4 ttl=253 time=6.343 ms
--- 10.0.0.6 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 6.233/12.53/34.387 ms
LEAF01#
Our next step is to configure BGP L2VPN EVPN.
SPINE Switches BGP configuration:
SPINE Switches BGP configuration:
nv overlay evpn
feature bgp
feature nv overlay
router bgp 65010
router-id 10.0.0.1
template peer Leaf
remote-as 65010
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 10.0.0.4
inherit peer Leaf
neighbor 10.0.0.5
inherit peer Leaf
neighbor 10.0.0.6
inherit peer Leaf
LEAF Switches BGP configuration:
nv overlay evpn
feature bgp
feature nv overlay
router bgp 65010
router-id 10.0.0.4
template peer RR
remote-as 65010
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.0.0.1
inherit peer RR
neighbor 10.0.0.2
inherit peer RR
Let's confirm that BGP is working:
SPINE01# sh bgp l2vpn evpn summary
BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 10.0.0.1, local AS number 65010
BGP table version is 13, L2VPN EVPN config peers 3, capable peers 3
6 network entries and 6 paths using 1296 bytes of memory
BGP attribute entries [3/468], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [0/0]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.0.4 4 65010 43 44 13 0 0 00:36:25 0
10.0.0.5 4 65010 43 44 13 0 0 00:36:23 0
10.0.0.6 4 65010 43 44 13 0 0 00:36:25 0
SPINE02# sh bgp l2vpn evpn summary
BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 10.0.0.2, local AS number 65010
BGP table version is 26, L2VPN EVPN config peers 3, capable peers 3
6 network entries and 6 paths using 1296 bytes of memory
BGP attribute entries [3/468], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [0/0]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.0.4 4 65010 73 77 26 0 0 01:06:25 0
10.0.0.5 4 65010 73 77 26 0 0 01:04:12 0
10.0.0.6 4 65010 74 81 26 0 0 01:04:06 0
SPINE02#
Finally our EVPN Fabric is up and running. Now we can and add tenant-VRF, VNIs and endpoints. We have to enable required features, configure vn-segments, nve interface, SVIs and EVPN itself.
LEAF Switches EVPN VNI configuration:
feature interface-vlan
feature vn-segment-vlan-based
feature fabric forwarding
system fabric dynamic-vlans 301-2400
system fabric core-vlans 100-300
! Define mac address for anycast gateway feature
fabric forwarding anycast-gateway-mac 0000.2222.2a21
! Create L3 VNI for inter-vni L3 forwarding and assign VNI ID
vlan 101
vn-segment 900001
vlan 102
vn-segment 900002
! Create L2 VNI and assign VNI ID
vlan 991
vn-segment 200991
vlan 992
vn-segment 200992
vlan 993
vn-segment 200993
route-map DIRECT permit 10
match tag 12345
! Configure Tenant VRFs
vrf context Tenant1:l3-900001
vni 900001
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
!
vrf context Tenant2:l3-900002
vni 900002
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
! Create L3 SVI for inter-VNI forwarding
interface Vlan101
no shutdown
mtu 9000
vrf member Tenant1:l3-900001
no ip redirects
ip forward
!
interface Vlan102
no shutdown
mtu 9000
vrf member Tenant2:l3-900002
no ip redirects
ip forward
!
! Anycast GW SVI 991-992 for VNI
interface Vlan991
no shutdown
vrf member Tenant1:l3-900001
no ip redirects
ip address 1.1.1.1/24 tag 12345
fabric forwarding mode anycast-gateway
!
interface Vlan992
no shutdown
vrf member Tenant1:l3-900001
no ip redirects
ip address 1.2.2.1/24 tag 12345
fabric forwarding mode anycast-gateway
!
interface Vlan993
no shutdown
vrf member Tenant2:l3-900002
no ip redirects
ip address 1.3.3.1/24 tag 12345
fabric forwarding mode anycast-gateway
! Create NVE interface
interface nve1
source-interface loopback0
host-reachability protocol bgp
! Assign L2 and L3 VNIs
member vni 900001 associate-vrf
member vni 900002 associate-vrf
member vni 200991
suppress-arp
mcast-group 225.4.0.1
member vni 200992
suppress-arp
mcast-group 225.4.0.1
member vni 200993
suppress-arp
mcast-group 225.4.0.2
no shutdown
! L2 Access interface towards Endpoints
interface Ethernet1/5
switchport mode trunk
switchport trunk allowed vlan 991-993
! Tenant vrf Definition under BPG
router bgp 65010
vrf Tenant1:l3-900001
address-family ipv4 unicast
advertise l2vpn evpn
redistribute direct route-map DIRECT
vrf Tenant2:l3-900002
address-family ipv4 unicast
advertise l2vpn evpn
redistribute direct route-map DIRECT
! EVPN
evpn
vni 200991 l2
rd auto
route-target import auto
route-target export auto
vni 200992 l2
rd auto
route-target import auto
route-target export auto
vni 200993 l2
rd auto
route-target import auto
route-target export auto
So far we do not have any endpoints connected to the fabric and we are able to see only type-5 BGP routes redistributed from directly connected:
SPINE1# sh bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 22, local router ID is 10.0.0.1
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10.0.0.4:3
*>i[5]:[0]:[0]:[24]:[1.1.1.0]:[0.0.0.0]/224
10.0.0.4 0 100 0 ?
*>i[5]:[0]:[0]:[24]:[1.2.2.0]:[0.0.0.0]/224
10.0.0.4 0 100 0 ?
Route Distinguisher: 10.0.0.4:4
*>i[5]:[0]:[0]:[24]:[1.3.3.0]:[0.0.0.0]/224
10.0.0.4 0 100 0 ?
Route Distinguisher: 10.0.0.5:3
*>i[5]:[0]:[0]:[24]:[1.1.1.0]:[0.0.0.0]/224
10.0.0.5 0 100 0 ?
*>i[5]:[0]:[0]:[24]:[1.2.2.0]:[0.0.0.0]/224
10.0.0.5 0 100 0 ?
Route Distinguisher: 10.0.0.5:4
*>i[5]:[0]:[0]:[24]:[1.3.3.0]:[0.0.0.0]/224
10.0.0.5 0 100 0 ?
Route Distinguisher: 10.0.0.6:3
*>i[5]:[0]:[0]:[24]:[1.1.1.0]:[0.0.0.0]/224
10.0.0.6 0 100 0 ?
*>i[5]:[0]:[0]:[24]:[1.2.2.0]:[0.0.0.0]/224
10.0.0.6 0 100 0 ?
Route Distinguisher: 10.0.0.6:4
*>i[5]:[0]:[0]:[24]:[1.3.3.0]:[0.0.0.0]/224
10.0.0.6 0 100 0 ?
SPINE1#
and at this point we are able to see NVE peers on the device:
LEAF01# sh nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1 10.0.0.5 Up CP 00:02:49 5000.0002.0007
nve1 10.0.0.6 Up CP 00:01:35 5000.0003.0007
LEAF01# sh nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 200991 225.4.0.1 Up CP L2 [991] SA
nve1 200992 225.4.0.1 Up CP L2 [992] SA
nve1 200993 225.4.0.2 Up CP L2 [993] SA
nve1 900001 n/a Up CP L3 [Tenant1:l3-900001]
nve1 900002 n/a Up CP L3 [Tenant2:l3-900002]
LEAF01#
I've added few linux boxes to my simulation and tried to ping, mac addresses learned and I start seeing Type-2 EVPN routes and ping was successful! (even between VNIs inside of Tenant1). Keep in mind that because of virtualized image you have to use some workarounds for "sh mac address-table"
LEAF01# sh system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 992 5000.0007.0000 dynamic 00:00:55 F F Eth1/5
* 993 5000.000e.0000 static - F F (0x47000002) nve-peer2 10.0.0.
* 992 5000.0009.0000 static - F F (0x47000001) nve-peer1 10.0.0.
* 991 0000.2222.2a21 dynamic 00:04:51 F F Eth1/5
* 991 5000.000b.0000 static - F F (0x47000001) nve-peer1 10.0.0.
* 991 5000.0006.0000 dynamic 00:08:15 F F Eth1/5
* 992 5000.000d.0000 static - F F (0x47000002) nve-peer2 10.0.0.
* 993 5000.0008.0000 dynamic 00:00:58 F F Eth1/5
G 991 5000.0001.0007 static - F F sup-eth1(R)
* 993 5000.000a.0000 static - F F (0x47000001) nve-peer1 10.0.0.
G 992 5000.0001.0007 static - F F sup-eth1(R)
G 993 5000.0001.0007 static - F F sup-eth1(R)
* 991 5000.000c.0000 static - F F (0x47000002) nve-peer2 10.0.0.
1 1 -00:00:22:22:2a:21 - 1
root@LinuxVNI991-1:~# arp -a
? (1.1.1.20) at 50:00:00:0b:00:00 [ether] on eth0
? (1.1.1.30) at 50:00:00:0c:00:00 [ether] on eth0
? (1.1.1.1) at 00:00:22:22:2a:21 [ether] on eth0
root@LinuxVNI991-1:~#
EVPN Type-2 Example:
LEAF01# sh bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 183, local router ID is 10.0.0.4
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10.0.0.4:33758 (L2VNI 200991)
*>l[2]:[0]:[0]:[48]:[0000.2222.2a21]:[0]:[0.0.0.0]/216
10.0.0.4 100 32768 i
*>l[2]:[0]:[0]:[48]:[5000.0006.0000]:[0]:[0.0.0.0]/216
10.0.0.4 100 32768 i
*>i[2]:[0]:[0]:[48]:[5000.000b.0000]:[0]:[0.0.0.0]/216
10.0.0.5 100 0 i
*>i[2]:[0]:[0]:[48]:[5000.000c.0000]:[0]:[0.0.0.0]/216
10.0.0.6 100 0 i
*>l[2]:[0]:[0]:[48]:[5000.0006.0000]:[32]:[1.1.1.10]/272
10.0.0.4 100 32768 i
*>i[2]:[0]:[0]:[48]:[5000.000b.0000]:[32]:[1.1.1.20]/272
10.0.0.5 100 0 i
*>i[2]:[0]:[0]:[48]:[5000.000c.0000]:[32]:[1.1.1.30]/272
10.0.0.6 100 0 i
Route Distinguisher: 10.0.0.4:33759 (L2VNI 200992)
*>l[2]:[0]:[0]:[48]:[5000.0007.0000]:[0]:[0.0.0.0]/216
10.0.0.4 100 32768 i
*>i[2]:[0]:[0]:[48]:[5000.0009.0000]:[0]:[0.0.0.0]/216
10.0.0.5 100 0 i
*>i[2]:[0]:[0]:[48]:[5000.000d.0000]:[0]:[0.0.0.0]/216
10.0.0.6 100 0 i
*>i[2]:[0]:[0]:[48]:[5000.0009.0000]:[32]:[1.2.2.20]/272
10.0.0.5 100 0 i
*>i[2]:[0]:[0]:[48]:[5000.000d.0000]:[32]:[1.2.2.30]/272
10.0.0.6 100 0 i
LEAF01# sh bgp l2vpn evpn 1.1.1.20
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.0.0.4:33758 (L2VNI 200991)
BGP routing table entry for [2]:[0]:[0]:[48]:[5000.000b.0000]:[32]:[1.1.1.20]/272, version 156
Paths: (1 available, best #1)
Flags: (0x000312) on xmit-list, is in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop, in rib
Imported from 10.0.0.5:33758:[2]:[0]:[0]:[48]:[5000.000b.0000]:[32]:[1.1.1.20]/144
AS-Path: NONE, path sourced internal to AS
10.0.0.5 (metric 81) from 10.0.0.1 (10.0.0.1)
Origin IGP, MED not set, localpref 100, weight 0
Received label 200991 900001
Extcommunity: RT:65010:200991 RT:65010:900001 ENCAP:8 MAC Mobility Sequence:00:1 Router MAC:5000.0002.0007
Originator: 10.0.0.5 Cluster list: 10.0.0.1
Path-id 1 not advertised to any peer
Route Distinguisher: 10.0.0.5:33758
BGP routing table entry for [2]:[0]:[0]:[48]:[5000.000b.0000]:[32]:[1.1.1.20]/272, version 151
Paths: (2 available, best #2)
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is not in HW, is locked
Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
AS-Path: NONE, path sourced internal to AS
10.0.0.5 (metric 81) from 10.0.0.2 (10.0.0.2)
Origin IGP, MED not set, localpref 100, weight 0
Received label 200991 900001
Extcommunity: RT:65010:200991 RT:65010:900001 ENCAP:8 MAC Mobility Sequence:00:1 Router MAC:5000.0002.0007
Originator: 10.0.0.5 Cluster list: 10.0.0.2
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
AS-Path: NONE, path sourced internal to AS
10.0.0.5 (metric 81) from 10.0.0.1 (10.0.0.1)
Origin IGP, MED not set, localpref 100, weight 0
Received label 200991 900001
Extcommunity: RT:65010:200991 RT:65010:900001 ENCAP:8 MAC Mobility Sequence:00:1 Router MAC:5000.0002.0007
Originator: 10.0.0.5 Cluster list: 10.0.0.1
Path-id 1 not advertised to any peer
Route Distinguisher: 10.0.0.4:3 (L3VNI 900001)
BGP routing table entry for [2]:[0]:[0]:[48]:[5000.000b.0000]:[32]:[1.1.1.20]/272, version 150
Paths: (1 available, best #1)
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported from 10.0.0.5:33758:[2]:[0]:[0]:[48]:[5000.000b.0000]:[32]:[1.1.1.20]/144
AS-Path: NONE, path sourced internal to AS
10.0.0.5 (metric 81) from 10.0.0.1 (10.0.0.1)
Origin IGP, MED not set, localpref 100, weight 0
Received label 200991 900001
Extcommunity: RT:65010:200991 RT:65010:900001 ENCAP:8 MAC Mobility Sequence:00:1 Router MAC:5000.0002.0007
Originator: 10.0.0.5 Cluster list: 10.0.0.1
Path-id 1 not advertised to any peer
LEAF01#
sh ip route vrf
LEAF01# sh ip ro vrf Tenant1:l3-900001
IP Route Table for VRF "Tenant1:l3-900001"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF
1.1.1.0/24, ubest/mbest: 1/0, attached
*via 1.1.1.1, Vlan991, [0/0], 00:53:40, direct, tag 12345
1.1.1.1/32, ubest/mbest: 1/0, attached
*via 1.1.1.1, Vlan991, [0/0], 00:53:40, local, tag 12345
1.1.1.10/32, ubest/mbest: 1/0, attached
*via 1.1.1.10, Vlan991, [190/0], 00:52:41, hmm
1.1.1.20/32, ubest/mbest: 1/0
*via 10.0.0.5%default, [200/0], 00:34:13, bgp-65010, internal, tag 65010 (evpn) segid: 900001 tunnelid: 0xa000005 encap: VXLAN
1.1.1.30/32, ubest/mbest: 1/0
*via 10.0.0.6%default, [200/0], 00:32:53, bgp-65010, internal, tag 65010 (evpn) segid: 900001 tunnelid: 0xa000006 encap: VXLAN
1.2.2.0/24, ubest/mbest: 1/0, attached
*via 1.2.2.1, Vlan992, [0/0], 00:53:41, direct, tag 12345
1.2.2.1/32, ubest/mbest: 1/0, attached
*via 1.2.2.1, Vlan992, [0/0], 00:53:41, local, tag 12345
1.2.2.20/32, ubest/mbest: 1/0
*via 10.0.0.5%default, [200/0], 00:41:50, bgp-65010, internal, tag 65010 (evpn) segid: 900001 tunnelid: 0xa000005 encap: VXLAN
1.2.2.30/32, ubest/mbest: 1/0
*via 10.0.0.6%default, [200/0], 00:32:29, bgp-65010, internal, tag 65010 (evpn) segid: 900001 tunnelid: 0xa000006 encap: VXLAN
If we capture packets we can see that when we ping between endpoints connected to the same VNI we are using L2 VNI to tag VXLAN packet:
when we ping between endpoints connected to the different VNIs inside of one VRF we are using L3 VRF VNI to tag VXLAN packet:
If we add mutual import/export between Tenant1 and Tenant2 and do ping between VRFs we can see that in case of different VRFs NX-OS uses destination L3 VNI to encapsulate VXLAN packet from SRC to DST.
when we ping between endpoints connected to the different VNIs inside of one VRF we are using L3 VRF VNI to tag VXLAN packet:
If we add mutual import/export between Tenant1 and Tenant2 and do ping between VRFs we can see that in case of different VRFs NX-OS uses destination L3 VNI to encapsulate VXLAN packet from SRC to DST.
vrf context Tenant1:l3-900001
address-family ipv4 unicast
route-target both 65010:900002
route-target both 65010:900002 evpn
!
vrf context Tenant2:l3-900002
address-family ipv4 unicast
route-target both 65010:900001
route-target both 65010:900001 evpn
Here we go! EVPN Fabric up and running and we can move forward and create external connectivity.
There are few options how we can do WAN handoff. I will describe them in Part II.
Very nice an concise walk-through, very well written.
ReplyDeleteLooking forward to Part II.