Microsoft ExpressRoute, as much as it can be complicated and hard work to implement, I have done the hard work and turned the complication into something that is understandable.

To begin with, I’ll point out a couple of significant changes to ExpressRoute this year:

• Bye bye Public peering. Beginning April 1, 2018, public peering is no longer available to implement. All Microsoft public addresses/services (across Office 365 & Azure) are available now on Microsoft peering. Move a public peering to Microsoft peering. Existing Azure public peering configurations will continue to be supported, though Microsoft recommend moving to Microsoft peering with route filters.
• ExpressRoute for Azure Active Directory on public peering and Microsoft peering for Azure will no longer be supported by default. Software as a service offerings like Azure AD are designed to work by going directly through the Internet, without requiring private connections like ExpressRoute. Because of this, on August 1, 2018, we will stop supporting ExpressRoute for Azure AD services that use public peering or Azure communities in Microsoft peering. You may notice affected services’ Azure AD traffic gradually shifting from ExpressRoute to the Internet. Please note this change does not apply to customers who use ExpressRoute for Office 365 and the Other Office 365 OnExpressRoute TSP,CSA poster v2line services (12076:5100) community.

On with the blog…. If you split up ExpressRoute, customer on one side and Microsoft on the other side, this blog has two parts in which it focuses on, the Microsoft side and the Customer side…..

Microsoft side focus

• The C-Tag is different per peer type (Private/Microsoft). There are two C-Tags, one for Microsoft peering and one for Private peering. The C-Tag is used to identify the peer/routing domain. This tag is defined by customer and input in Azure Portal.
• The S-Tag has a one to one relationship with the ExpressRoute circuit, this applies to each circuit, an S-Tag for each circuit. This is defined by Microsoft and customer is not aware of it.
• For Microsoft peering, as there’s no vNet gateway, it doesn’t need isolation, as all IP addresses are public & unique, there is no need for VRF. The difference being, with private peering, there’s a chance of an overlap of IP address ranges hence the need for VRF.
• VRF, works on Layer 2.5 at the MSEE level. VRF does isolation within routing, like a namespace in Linux. The Exchange Provider is not aware of VRF, because The Exchange Provider works on Layer 2, this is the maximum layer it can go to.
• Traffic to Azure | The Exchange Provider adds the S-Tag, then the MSEE removes both the S-Tag & C-Tag from the packet.
• Traffic from Azure | The MSEE adds both the S-Tag & C-Tag to the packet.
• The Exchange Provider equipment is a highly available layer 2 switch stack.
• Cross Connects are layer 1 physical connections between the Exchange Provider & the MSEE.

Customer side focus

• The BGP peers are setup independently as a first step prior to any NAT’ing & advertising of address ranges.
• Unlike with Public Peering, Microsoft Peering requires an additional public subnet to be available so it can be used for NAT’ing and advertising. This subnet cannot be the same as the subnet used for peering.
• Public IP addresses advertised to Microsoft over ExpressRoute must not be advertised to the Internet. This may break connectivity to other Microsoft services. However, Public IP addresses used by servers in your network that communicate with Office 365 endpoints within Microsoft may be advertised over ExpressRoute.
• With Microsoft peering, traffic destined to Microsoft cloud services must be SNATed to valid public IPv4 addresses before they enter the Microsoft network.

So there you go, hope this helps someone.

Sharing the same ExpressRoute circuit

A question I get asked a lot is “can I share the same ExpressRoute circuit across multiple subscriptions and multiple subscriptions across tenants?” The short answer is Yes.

It’s all to do with the Azure entities in the Azure portal – the ExpressRoute Circuit, the Connection and the Virtual Network Gateway. In fact, it’s the Connection entity that glues the ExpressRoute Circuit and the Virtual Network Gateway together. The glue, requires a Circuit authorisation key in order for other Connections to utilise it.

Below is what it looks like:

Popcorn time!

Also, a blog post wouldn’t be complete without a cool white-boarding video on ExpressRoute.

Part 1:

Part 2:

Part 3:

As for routing and optimisation.

• From Azure: use AS PATH prepending – if you continue to advertise both of the prefixes on both ExpressRoute circuits
• From the Customer side: Microsoft use BGP Communities so you can use BGP’s Local Preference to influence routing
• Between virtual networks: Solution: assign a high weight to local connection

More details on this here….

Megaport

Megaport is one of the providers for ExpressRoute. A couple things Megaport have shared with me:

• Megaport’s method of connectivity for Office 365 is for customers to connect via Microsoft peering. Customers need to first seek approval for the Office 365 prefixes to be advertised back to them from Microsoft – http://aka.ms/errequest
• Imagine if a customer may not have public addresses currently assigned to their organisation and would like to have Megaport perform the IP delegation and NAT. Megaport can do this via their new Megaport Cloud Router/MCR product. Video of how it works here thanks to Jason Bordujenko – https://megaport.wistia.com/medias/5qpir7qdja.

Megaport Cloud Router/MCR product

This diagram representing a single or dual customer port model using a single MCR, this allows Megaport to run multiple peering types. On the right hand side shows both Microsoft peering & Private Peering connectivity – traversing a single MCR. This gets mixed down on the customer side to a single 802.1q/dot1Q combined peering at Layer 3. This brings together all the C-Tags at this point.

It is optional to connect from a secondary customer port.

What Megaport is able to do with the MCR product, particularly in Sydney is split the Azure primary & secondary circuits across two separate deployments, physically diverse across the Megaport infrastructure in different datacenters within the Sydney region to wherever the customer port/s are located bringing them into two separate physical locations or two separate ports in the same datacenter.

Routing/Security on the Microsoft side

Firstly, to explain the ASNs:

• 12076 is the AS number that the Azure ExpressRoute service uses, for all peering types.
• 8075 is the AS number that Microsoft uses for its internet peering (not just for Azure cloud services).

I get the question about security a lot, as an example, for a service like Dynamics 365, while this is a public service advertised on Microsoft’s ASN8075, it can be also advertised over ExpressRoute Microsoft peering using route filters using ASN12076. After you setup Microsoft peering with an ExpressRoute circuit, nothing is advertised back from Microsoft by default, hence the need for route filters. Route Filters use BGP community values, for a list of the BGP community values and the services they map to, see BGP communities. So, using the same example, the BGP community value for Dynamics 365 is 12076:5040.

As for routing and advertisements, the diagram below pretty much sums it up:

Great References:

• Routing with ExpressRoute for Office 365 – https://docs.microsoft.com/en-us/office365/enterprise/routing-with-expressroute
• Asymmetric routing with multiple network paths – https://docs.microsoft.com/en-us/azure/expressroute/expressroute-asymmetric-routing
• Optimize ExpressRoute Routing – https://docs.microsoft.com/en-us/azure/expressroute/expressroute-optimize-routing
  • BGP communities
  • AS PATH prepending
  • Connection weights
• Designing for disaster recovery with ExpressRoute private peering – https://docs.microsoft.com/en-us/azure/expressroute/designing-for-disaster-recovery-with-expressroute-privatepeering
  • You can influence Azure to prefer one ExpressRoute circuit over another one using one of the following techniques (listed in the order of effectiveness):
    • advertising more specific route over the preferred ExpressRoute circuit compared to other ExpressRoute circuit(s)
    • configuring higher Connection Weight on the connection that links the virtual network to the preferred ExpressRoute circuit
    • advertising the routes over less preferred ExpressRoute circuit with longer AS Path (AS Path prepend)

Also note: To be able to attach route filters with Office 365 services on them, you must have authorisation to consume Office 365 services through ExpressRoute. If you are not authorised to consume Office 365 services through ExpressRoute, the operation to attach route filters fails. For more information about the authorisation process, see Azure ExpressRoute for Office 365. Connectivity to Dynamics 365 services does NOT require any prior authorisation. Also, the ExpressRoute Premium add-on mandatory for Microsoft peering.