Zenn
📝

[SONiC-VS] EVPN-VXLANv4 w/ BGP unnumbered

2022/08/16に公開
network
network
bgp
cml
sonic
tech

【概要】

以前構築したSONiC on CMLの環境で、BGP unnumberedと、EVPN-VXLANとの組み合わせを試してみました。

今回は最低限の疎通確認を念頭に構成したところ、virtual switch版のSONiC.202205を用いて、BGP unnumbered用いたIPv4転送及びEVPN-VXLANを用いたL2転送が可能な結果となりました。

以降はその記録となります。

参考にしていた主な情報は以下になります。

[Edgecore SONiC] BGP Unnumbered
https://support.edge-core.com/hc/en-us/articles/900002377366--Edgecore-SONiC-BGP-Unnumbered

[Edgecore SONiC] EVPN L2 VxLAN
https://support.edge-core.com/hc/en-us/articles/900002720523--Edgecore-SONiC-EVPN-L2-VxLAN

[Edgecore SONiC] VLAN & Inter-VLAN Routing
https://support.edge-core.com/hc/en-us/articles/900000210426-VLAN-Inter-VLAN-Routing

SONiC USER MANUAL
https://github.com/sonic-net/SONiC/blob/master/doc/SONiC-User-Manual.md

SONiC Command Line Interface Guide
https://github.com/sonic-net/sonic-utilities/blob/master/doc/Command-Reference.md

Vxlan SONiC
https://github.com/sonic-net/SONiC/blob/master/doc/vxlan/Vxlan_hld.md

また、仮想化環境のSONiCを用いたVXLANやEVPN-VXLANについて、先達が既に情報を公開されています。

SONiCとVXLAN - Qiita
https://qiita.com/masru0714/items/71f2d8e0efa4c28f991d

SONiCでVXLANを使ってみた - Qiita
https://qiita.com/masru0714/items/140c1fb301e4503325a2

SONiCとCumulus LinuxでBGP/EVPN/VXLAN - メモのページ - チラシの裏メモ 3枚目
https://debslink.hatenadiary.jp/entry/20210902/1630584311

【詳細】

構成図(トポロジ)

ネットワーク情報

<トランスポート関連>

node hostname Loopback0 ASN
sonicvsvm-0 R001 10.0.0.1/32 65001
sonicvsvm-1 R002 10.0.0.2/32 65001
sonicvsvm-2 R011 10.0.0.11/32 65000
sonicvsvm-3 R012 10.0.0.12/32 65000
sonicvsvm-4 R013 10.0.0.13/32 65000
sonicvsvm-5 R014 10.0.0.14/32 65000
sonicvsvm-6 R101 10.0.0.101/32 65001
sonicvsvm-7 R102 10.0.0.102/32 65001

<サービス関連>

node hostname VLAN VNI
sonicvsvm-6 R101 2 5002
sonicvsvm-7 R102 2 5002
node interface MAC address VLAN IP address VRF
csr1000v-0 Gi2 0000.0000.0101 untag 192.168.2.101 101-2
Gi3 0000.0000.0102 untag 192.168.2.102 102-2

SONiC OS設定の抜粋

<全般>

  • 保存＆再起動（適宜実施）
node command
sonicvsvm-0~7 sudo config save -y
sudo config reload -fy

<トランスポート関連>

  • トランジットリンク：IPv6 link-local
node command
sonicvsvm-0~5 sudo config ipv6 enable link-local
sonicvsvm-6/7 sudo config interface ipv6 enable use-link-local-only Ethernet0
sudo config interface ipv6 enable use-link-local-only Ethernet4
  • ループバック：IPv4
node command
sonicvsvm-0 sudo config interface ip add Loopback0 10.0.0.1/32
sonicvsvm-1 sudo config interface ip add Loopback0 10.0.0.2/32
sonicvsvm-2 sudo config interface ip add Loopback0 10.0.0.11/32
sonicvsvm-3 sudo config interface ip add Loopback0 10.0.0.12/32
sonicvsvm-4 sudo config interface ip add Loopback0 10.0.0.13/32
sonicvsvm-5 sudo config interface ip add Loopback0 10.0.0.14/32
sonicvsvm-6 sudo config interface ip add Loopback0 10.0.0.101/32
sudo config interface ip add Loopback0 10.255.0.101/32 secondary
sonicvsvm-7 sudo config interface ip add Loopback0 10.0.0.102/32
sudo config interface ip add Loopback0 10.255.0.102/32 secondary

<サービス関連>

  • VLAN
node command
sonicvsvm-6/7 sudo config vlan add 2
sudo config vlan member add -u 2 Ethernet8
  • VXLAN（EVPN対応）
node command
sonicvsvm-6 sudo config vxlan add VXLAN 10.255.0.101
sonicvsvm-7 sudo config vxlan add VXLAN 10.255.0.102
sonicvsvm-6/7 sudo config vxlan evpn_nvo add NVO VXLAN
sudo config vxlan map add VXLAN 2 5002

FRRouting設定の抜粋

sonicvsvm-0/1
no router bgp 65100
router bgp 65001
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 neighbor SERVICE peer-group
 neighbor SERVICE remote-as internal
 neighbor SERVICE update-source Loopback0
 neighbor TRANSPORT peer-group
 neighbor TRANSPORT remote-as external
 neighbor TRANSPORT capability extended-nexthop
 neighbor Ethernet0 interface peer-group TRANSPORT
 neighbor Ethernet4 interface peer-group TRANSPORT
 neighbor Ethernet8 interface peer-group TRANSPORT
 neighbor Ethernet12 interface peer-group TRANSPORT
 bgp listen range 10.0.0.0/8 peer-group SERVICE
 address-family ipv4 unicast
  redistribute connected
  neighbor TRANSPORT activate
  neighbor TRANSPORT allowas-in
 exit-address-family
 address-family l2vpn evpn
  neighbor SERVICE activate
  neighbor SERVICE route-reflector-client
 exit-address-family
exit
sonicvsvm-2/3/4/5
no router bgp 65100
router bgp 65000
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 neighbor TRANSPORT peer-group
 neighbor TRANSPORT remote-as external
 neighbor TRANSPORT capability extended-nexthop
 neighbor Ethernet0 interface peer-group TRANSPORT
 neighbor Ethernet4 interface peer-group TRANSPORT
 neighbor Ethernet8 interface peer-group TRANSPORT
 address-family ipv4 unicast
  redistribute connected
  neighbor TRANSPORT activate
  neighbor TRANSPORT allowas-in
 exit-address-family
exit
sonicvsvm-6/7
no router bgp 65100
router bgp 65001
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 neighbor SERVICE peer-group
 neighbor SERVICE remote-as internal
 neighbor SERVICE update-source Loopback0
 neighbor TRANSPORT peer-group
 neighbor TRANSPORT remote-as external
 neighbor TRANSPORT capability extended-nexthop
 neighbor 10.0.0.1 peer-group SERVICE
 neighbor 10.0.0.2 peer-group SERVICE
 neighbor Ethernet0 interface peer-group TRANSPORT
 neighbor Ethernet4 interface peer-group TRANSPORT
 address-family ipv4 unicast
  redistribute connected
  neighbor TRANSPORT activate
  neighbor TRANSPORT allowas-in
 exit-address-family
 address-family l2vpn evpn
  neighbor SERVICE activate
  advertise-all-vni
 exit-address-family
exit

動作確認

<通信>

csr1000v-0
HOST#ping vrf 101-2 192.168.2.102 source 192.168.2.101 repeat 3
Type escape sequence to abort.
Sending 3, 100-byte ICMP Echos to 192.168.2.102, timeout is 2 seconds:
Packet sent with a source address of 192.168.2.101
.!!
Success rate is 66 percent (2/3), round-trip min/avg/max = 5/5/6 ms
HOST#
HOST#!#ping:ok

<パケットキャプチャ>

  • csr1000v-0 ~ sonicvsvm-7

  • sonicvsvm-5 ~ sonicvsvm-7

【補足など】

IPv4 FIBの状態

準備過程で、BGPで交換されたIPv4経路はインポートされるが、その経路に対するIPv4通信は失敗する現象が発生していました。この時、show ip routeでは表示されるエントリが、ip route showでは表示されない状態となっていました。
この現象は、一度再起動することで解消されました。

before reloading
admin@R101:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

B>r 10.0.0.1/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
  r                    via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
B>r 10.0.0.2/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
  r                    via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
B>r 10.0.0.11/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:05:53
B>r 10.0.0.12/32 [20/0] via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:05:53
B>r 10.0.0.13/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
  r                     via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
B>r 10.0.0.14/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
  r                     via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
C>* 10.0.0.101/32 is directly connected, Loopback0, 10:02:07
B>r 10.0.0.102/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
  r                      via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19

admin@R101:~$
admin@R101:~$ ip route show
240.127.1.0/24 dev docker0 proto kernel scope link src 240.127.1.1 linkdown
admin@R101:~$
admin@R101:~$ ping 10.0.0.102 -I 10.0.0.101 -c 3
PING 10.0.0.102 (10.0.0.102) from 10.0.0.101 : 56(84) bytes of data.

--- 10.0.0.102 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2027ms

admin@R101:~$
after reloading
admin@R101:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

B>* 10.0.0.1/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:30
  *                    via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:30
B>* 10.0.0.2/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:30
  *                    via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:30
B>* 10.0.0.11/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:30
B>* 10.0.0.12/32 [20/0] via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:30
B>* 10.0.0.13/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:25
  *                     via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:25
B>* 10.0.0.14/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:17
  *                     via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:17
C>* 10.0.0.101/32 is directly connected, Loopback0, 00:06:34
B>* 10.0.0.102/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:25
  *                      via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:25

admin@R101:~$
admin@R101:~$ ip route show
10.0.0.1 nhid 226 proto bgp src 10.0.0.101 metric 20
        nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
        nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
10.0.0.2 nhid 226 proto bgp src 10.0.0.101 metric 20
        nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
        nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
10.0.0.11 nhid 227 via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 proto bgp src 10.0.0.101 metric 20
10.0.0.12 nhid 228 via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 proto bgp src 10.0.0.101 metric 20
10.0.0.13 nhid 226 proto bgp src 10.0.0.101 metric 20
        nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
        nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
10.0.0.14 nhid 226 proto bgp src 10.0.0.101 metric 20
        nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
        nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
10.0.0.102 nhid 226 proto bgp src 10.0.0.101 metric 20
        nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
        nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
240.127.1.0/24 dev docker0 proto kernel scope link src 240.127.1.1 linkdown
admin@R101:~$
admin@R101:~$ ping 10.0.0.102 -I 10.0.0.101 -c 3
PING 10.0.0.102 (10.0.0.102) from 10.0.0.101 : 56(84) bytes of data.
64 bytes from 10.0.0.102: icmp_seq=1 ttl=61 time=3.81 ms
64 bytes from 10.0.0.102: icmp_seq=2 ttl=61 time=3.29 ms
64 bytes from 10.0.0.102: icmp_seq=3 ttl=61 time=3.05 ms

--- 10.0.0.102 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 3.045/3.381/3.806/0.316 ms
admin@R101:~$

bridge tableの状態

L2通信の成功後、BGPで交換されたEVPN Route Type 2の経路はインポートされるが、そのエントリはshow mac等の出力には表示されない現象が発生していました。
今回の環境では、表示上の問題か、もしくはユニキャスト通信もBUM転送で疎通している可能性があります。この点に関しては、例えばNVEの追加によって切り分けを進めることができそうです。

sonicvsvm-6
admin@R101:~$ #ping:ok
admin@R101:~$
admin@R101:~$ show mac
  No.    Vlan  MacAddress         Port       Type
-----  ------  -----------------  ---------  -------
    1       2  00:00:00:00:01:01  Ethernet8  Dynamic
Total number of entries 1
admin@R101:~$
admin@R101:~$ vtysh

Hello, this is FRRouting (version 8.2.2).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

R101#
R101# show bgp l2vpn evpn
BGP table version is 9, local router ID is 10.0.0.101
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 10.0.0.101:1
*> [2]:[0]:[48]:[00:00:00:00:01:01]
                    10.255.0.101                       32768 i
                    ET:8 RT:65001:5002
*> [3]:[0]:[32]:[10.255.0.101]
                    10.255.0.101                       32768 i
                    ET:8 RT:65001:5002
Route Distinguisher: 10.0.0.102:1
* i[2]:[0]:[48]:[00:00:00:00:01:02]
                    10.255.0.102             0    100      0 i
                    RT:65001:5002 ET:8
*>i                 10.255.0.102             0    100      0 i
                    RT:65001:5002 ET:8
* i[3]:[0]:[32]:[10.255.0.102]
                    10.255.0.102             0    100      0 i
                    RT:65001:5002 ET:8
*>i                 10.255.0.102             0    100      0 i
                    RT:65001:5002 ET:8

Displayed 4 out of 6 total prefixes
R101#

Discussion