We continue our discussion of the public reference architecture on the Dish 5G buildout on AWS with a look at the physical network design and datacenter layout.
Cloud Infrastructure Architecture
DISH 5G netowrk’s architecture utilizes Amazon Virtual Private Cloud (Amazon VPC) to represent NDCs/RDCs or BEDCs (xDCs). Amazon VPC enables DISH to launch CNF resources on a virtual network. This virtual network is intended to closely resemble an on-premises network, but also contains all the resources needed for Data Center functions. The VPCs hosting each of the xDCs are fully interconnected utilizing AWS global network and AWS Transit Gateway. AWS Transit Gateway is used in AWS Regions to provide connectivity between VPCs deployed in the NDCs, RDCs, and BEDCs with scalability and resilience (Fig2).
AWS Direct Connect provides connectivity from RAN DUs (on-prem) to AWS Local Zones where cell sites are homed. Cell sites are mapped to a particular AWS Local Zone based on proximity to meet 5G RAN mid-haul latency expected between DU and CU.
5G Core Network Connectivity
In the AWS network, each Region hosts one NDC and three RDCs. NDC functions communicate to each other through the Transit Gateway, where each VPC has an attachment to the specific regional Transit Gateway. EC2 and native AWS networking is referred to as “Underlay Network” in this network architecture. Provisioning of Transit Gateway and required attachments are automated using CI/CD pipelines with AWS APIs. Transit Gateway routing tables are utilized to maintain isolation of traffic between functions.
Some of the 5G core network functions require support for advanced routing capabilities inside VPC and across VPCs (Ex. UPF, SMF and ePDG). These functions reply to routing protocols such as BGP for route exchange and fast failover (both stateful and stateless). To support these requirements, virtual routers (vRTRs) are deployed on EC2 to provide connectivity within and across VPCs, as well as back to the on-prem network.
Traffic from vRTRs is encapsulated using Generic Routing Encapsulation (GRE) tunnels, creating an “Overlay Network.’ This leverages the Underlay network for end-point reachability. The Overlay network uses Intermediate Systems to Intermediate Systems (IS-IS) routing protocol in conjunction with Segment Routing Multi-Protocol Label Switching (SR-MPLS) to distribute routing information and establish network reachability between the vRTRs. Multi-Protocol Border Gateway Protocol (MP-BGP) over GRE is used to provide reachability from on-prem to AWS Overlay network and reachability between different regions in AWS. The combined solution provides DISH the ability to honor requirements such as traffic isolation and efficiently route traffic between on-prem, AWS and 3rd parties (e.g., voice aggregators, regulatory entities etc.).
AWS Direct Connect for DISH’s RAN Mid-Haul
AWS Direct Connect is leveraged to provide connectivity between DISH’s RAN network and the AWS Cloud. Each Local Zone is connected over 2*100G Direct Connect links for redundancy. Direct Connect in combination with Local Zone provides a sub 10 msec Midhaul connectivity between DISH’s on-prem RAN and BEDC. End-to-end SR-MPLS provides connectivity from cell sites to Local Zone and AWS region via Overlay Network using the vRTRs. Through this DISH has the ability to extend multiple Virtual Routing and Forwarding (VRF)from RAN to the AWS Cloud.
AWS Local Zone and Internet Peering
Internet access is provided by AWS within the Local Zone. This “hot potato” routing approach is the most efficient way of handling traffic, rather than backhauling traffic to the region, a centralized location or incurring the cost of maintaining a dedicated internet circuit. It improves subscriber experience and provides low latency internet. This architecture also reduces the failure domain by distributing internet among multiple Local Zones.
Network Resiliency
In telco-grade networks, resiliency is at the heart of design. It’s vital to maintain the targeted service-level agreements (SLAs), comply with regulatory requirements and support seamless failover of services. While redundancy and resiliency are addressed at various layers of the 5G stack, we will focus here on transport availability in failure scenarios. High availability and geo-redundancy are NF dependent, while some NFs are required to maintain state.
- NDC
-
- High Availability: High availability is achieved by deploying two redundant NFs in two separate availability zones within a single VPC. Failover within an AZ can be recovered within the region without the need to route traffic to other regions. The in-region networking uses the underlay and overlay constructs, which enable on-prem traffic to seamlessly flow to the standby NF in the secondary AZ if the active NF becomes unavailable.
-
- Geo-Redundancy: Geo-Redundancy is achieved by deploying two redundant NFs in two separate availability zones in more than one region. This is achieved by interconnecting all VPCs via inter-region Transit Gateway and leveraging vRTR for overlay networking. The overlay network is built as a full-mesh enabling service continuity using the NFs deployed across NDCs in other regions during outage scenarios (e.g., Markets, B-EDCs, RDCs, in us-east-2 can continue to function using the NDC in us-east-1).
- RDC
-
- High availability and geo-redundancy are achieved by NFs failover between VPCs (multiple Availability zones) within one region. These RDCs are interconnected via Transit Gateway with the vRTR-based overlay network. This provides on-premise and B-EDC reachability to the NFs deployed in each RDC with route policies in place to ensure traffic only flows to the backup RDCs, if the primary RDC becomes unreachable.
- PEDC
-
- DISH’s RAN network is connected, through PEDC, to two different direct connect locations for reachability into the region and local zone This allows for DU traffic to be rerouted from an active BEDC to backup BEDC in the event a local zone fails (Fig3).
The next blog will discuss key architecture lessons by means of which DISH are able to address latency-sensitive application requirements. By utilizing the right mix of AWS resources in the regions and in Local Zones, DISH was able to address latency-sensitive requirements of 5G applications by deployment in AWS Local Zones while maintaining central deployment of the rest of 5G core applications in AWS Regions.
Image by torstensimon from Pixabay