Microsoft ExpressRoute, as much as it can be complicated and hard work to implement, I have done the hard work and turned the complication into something that is understandable.
To begin with, I’ll point out a couple of significant changes to ExpressRoute this year:
- Bye bye Public peering. Beginning April 1, 2018, public peering is no longer available to implement. All Microsoft public addresses/services (across Office 365 & Azure) are available now on Microsoft peering. Move a public peering to Microsoft peering. Existing Azure public peering configurations will continue to be supported, though Microsoft recommend moving to Microsoft peering with route filters.
- ExpressRoute for Azure Active Directory on public peering and Microsoft peering for Azure will no longer be supported by default. Software as a service offerings like Azure AD are designed to work by going directly through the Internet, without requiring private connections like ExpressRoute. Because of this, on August 1, 2018, we will stop supporting ExpressRoute for Azure AD services that use public peering or Azure communities in Microsoft peering. You may notice affected services’ Azure AD traffic gradually shifting from ExpressRoute to the Internet. Please note this change does not apply to customers who use ExpressRoute for Office 365 and the Other Office 365 Online services (12076:5100) community.
On with the blog…. If you split up ExpressRoute, customer on one side and Microsoft on the other side, this blog has two parts in which it focuses on, the Microsoft side and the Customer side…..
Microsoft side focus
- The C-Tag is different per peer type (Private/Microsoft). There are two C-Tags, one for Microsoft peering and one for Private peering. The C-Tag is used to identify the peer/routing domain. This tag is defined by customer and input in Azure Portal.
- The S-Tag has a one to one relationship with the ExpressRoute circuit, this applies to each circuit, an S-Tag for each circuit. This is defined by Microsoft and customer is not aware of it.
- For Microsoft peering, as there’s no vNet gateway, it doesn’t need isolation, as all IP addresses are public & unique, there is no need for VRF. The difference being, with private peering, there’s a chance of an overlap of IP address ranges hence the need for VRF.
- VRF, works on Layer 2.5 at the MSEE level. VRF does isolation within routing, like a namespace in Linux. The Exchange Provider is not aware of VRF, because The Exchange Provider works on Layer 2, this is the maximum layer it can go to.
- Traffic to Azure | The Exchange Provider adds the S-Tag, then the MSEE removes both the S-Tag & C-Tag from the packet.
- Traffic from Azure | The MSEE adds both the S-Tag & C-Tag to the packet.
- The Exchange Provider equipment is a highly available layer 2 switch stack.
- Cross Connects are layer 1 physical connections between the Exchange Provider & the MSEE.
Customer side focus
- The BGP peers are setup independently as a first step prior to any NAT’ing & advertising of address ranges.
- Unlike with Public Peering, Microsoft Peering requires an additional public subnet to be available so it can be used for NAT’ing and advertising. This subnet cannot be the same as the subnet used for peering.
- Public IP addresses advertised to Microsoft over ExpressRoute must not be advertised to the Internet. This may break connectivity to other Microsoft services. However, Public IP addresses used by servers in your network that communicate with Office 365 endpoints within Microsoft may be advertised over ExpressRoute.
- With Microsoft peering, traffic destined to Microsoft cloud services must be SNATed to valid public IPv4 addresses before they enter the Microsoft network.
So there you go, hope this helps someone.
Also, a blog post wouldn’t be complete without a cool white-boarding video on ExpressRoute.
As for routing and optimisation.
- From Azure: use AS PATH prepending – if you continue to advertise both of the prefixes on both ExpressRoute circuits
- From the Customer side: Microsoft use BGP Communities so you can use BGP’s Local Preference to influence routing
- Between virtual networks: Solution: assign a high weight to local connection
More details on this here….