We continue our discussion of the public reference architecture on the Dish 5G buildout on AWS with a detailed look at the deployment automation of the implementation.
Deployment Automation: As a Code
For network automation as well as scalability, the AWS and DISH architecture teams selected the infrastructure as code (IaC) to enable automation. It can be tempting to create resources manually in the short term, but using infrastructure as code:
- Enables full auditing capabilities of infrastructure deployment and changes.
- Provides the ability to deploy a network infrastructure rapidly and at scale.
- Simplifies operational complexity by using code and templates as well as reduces the risk of misconfiguration.
All infrastructure components such as VPCs and subnets to TGWs are deployed using AWS Cloud Development Kit (AWS CDK) and AWS CloudFormation templates. Both AWS CDK and Cloud Formation use parameterization and embedded code (through Lambda) to allow for automation of various environment deployments without the need to hardcode dynamic configuration information within the template.
AWS has pioneered the development of new CI/CD tools to help a broad spectrum of industries develop and roll out software changes rapidly while maintaining systems stability and security. These tools include a set of DevOps (software development and operations) services, such as AWS CodeStar, AWS CodeCommit, AWS CodePipeline, AWS CodeBuild, and AWS CodeDeploy. Moreover, AWS has also been evangelizing the idea of IaC using AWS CDK, AWS CloudFormation, and API-based third-party tools (e.g., Terraform). Using these tools, NF deployment processes can be stored in AWS as a source code and maintain the same source code in the CI/CD pipeline for continuous delivery.
The DISH and AWS teams worked with independent software vendors (ISVs) to deploy CNFs following cloud-native principles, with full CI/CD, observability and configuration through cloud-native tools, like Helm and ConfigMaps. This approach, combined with the resiliency components discussed earlier, maximizes 5G service availability in outage scenarios at the networking domain or when NFs are impacted by failures.
The CI/CD process, developed in partnership with DISH and ISVs, includes the following steps:
- Network setup – AWS CDK and CloudFormation initiate the creation of the network prerequisites
- Networking stack ( VPC, Subnets, NAT gateway, route table, and internet gateway)
- Infrastructure deployment – AWS CDK and CloudFormation initiate the creation of the following resource stacks:
- Compute stack (EKS cluster creation, EKS worker nodes, Lambda)
- Storage stack ( S3 buckets, EBS volumes, and EFS)
- Monitoring stack (Amazon CloudWatch, Amazon OpenSearch Service)
- Security stack ( IAM roles, IAM policies, EC2 security groups, VPC NACLs)
- CNF deployment – In this stage, CNF is deployed onto EKS clusters using Kubectl and Helm chart tools. This stage also deploys any specific applications or tools needed by the CNFs to work efficiently (e.g., Prometheus, Fluentd). CNFs can be either deployed via Lambda functions or AWS CodeBuild, which can be part of the AWS CodePipeline stages.
- Continuous updates and deployment – These are a sequence of steps that will be carried out iteratively to deploy changes \ as part of container and configuration changes resulting in upgrades. Similar to the CNF deployment case, this is automated using AWS services with the trigger from AWS CodeCommit, Amazon Elastic Container Registry or third-party source systems such as GitLab Webhook.
Figure 4: AWS CI/CD Pipeline Flow
The CI/CD pipeline is built using AWS CodePipeline and utilizes a continuous delivery service that models, visualizes and automates the steps required to release software. By defining stages in a pipeline, you can retrieve code from a source code repository, build that source code into a releasable artifact, test the artifact and deploy it to production. Only code that successfully passes through all stages will be deployed. In addition, other requirements can be added to the pipeline, such as manual approvals, to help ensure that only approved changes are deployed to production.
Figure 5: DISH CI/CD Pipeline Architecture
By leveraging infrastructure as code, DISH can automate the creation (and decommissioning) of environments. This increases the pace of innovation, reduces human error, and ensures compliance with DISH’s security postures. The automated CI/CD pipeline checkpoints and constant monitoring tools like Amazon GuardDuty (threat protection) and Amazon Macie (sensitive data identification and protection) are included in the infrastructure as a code for security checks.
CI/CD Security
Security is a critical element in the network design and is introduced to various parts of the solution. Below is a list of security steps that DISH and AWS CI/CD processes would take into account while deploying an application.
- Source: The ECR repository allocated to the ISV is configured with “Scan on Push” flag enabled, so that any uploads of Docker images will be immediately subjected to a security scan. Any known common vulnerabilities and exposures (CVE) are flagged with notifications. Apart from ECR, the charts put into AWS CodeCommit repository, the ISVs will be requested to encrypt any passwords used using AWS Secrets Manager rather than in plain text.
- Artifacts integrity: The artifacts used across the pipeline are encrypted, both at rest (using AWS managed keys) and in transit (using SSL/TLS).
- IAM users and roles: User or resource permissions provided are based on the principle of minimum permission. Cross-IAM role trust relationship is configured and used for operating across resources in different services, for example, AWS CodeBuild needing permission to run commands on an EKS cluster.
- Audit: The auditing capability of AWS CloudTrail service is used to maintain an audit trail in the DISH environments. It tracks each and every API call across services and user operations and will allow an evaluation of any past events.
- Image vulnerability scanning: CNF images that are uploaded to Amazon ECR are automatically scanned for security vulnerabilities. A report of the scan findings is available in the AWS console and can also be retrieved via API. The findings can also be sent to the accountable parties for corrective action, including the replacement of the CNF image.
Security checks are kicked off at various stages of the pipeline to ensure the newly uploaded image is secure and complies with the desired compliance checks, and a notification can be sent to DISH for approval. The container registry would scan for any open CVE vulnerabilities. The configuration is checked against leaking out any sensitive information (known PII pattern), the test stage triggers compliance check rules (for example, unexpected open TCP/UDP ports, DOS vulnerabilities), and eventually verifies backward and forward compatibility for graceful upgrade and rollback safety. Apart from the application, it is critical to provision pipeline security by ensuring encrypted transfer of artifacts across stages, whether at rest or in transit.
Beginning of 5G
In this post, we presented a cloud-native architecture for deploying 5G networks on AWS Cloud. By utilizing the AWS Cloud, DISH was able to architect, design, build and deliver a complete cloud-native 5G network with full operations and network deployment automation, including the adoption of the Open Radio Access Network (O-RAN) and other open community projects. This platform allows DISH to create innovative new services and react faster to customer demands. It also mitigates heavy lifting activities that put additional workloads on the architecture, engineering, and operations teams.
- Deploying 5G at Cloud Scale: By utilizing the AWS Cloud platform, DISH was able to architect, design, build, and deliver the first 5G data and voice call using the 5G core platform deployed in the public cloud in a short period of time.
- Tech Stack Innovation: By utilizing a common cloud architecture across core and edge, the DISH team was able to promote the adoption of CNFs and cloud-native architecture across a diverse set of 5G partners and software providers.
- Edge Optimization: By utilizing the right mix of AWS resources in the regions and in Local Zones, DISH was able to address latency-sensitive requirements of 5G applications by deployment in AWS Local Zones while maintaining the central deployment of the rest of 5G core applications in AWS Regions.
DISH is an active member of O-RAN Alliance as well as other open community projects. To learn more about DISH’s digital innovation opportunities, visit https://www.dishwireless.com/
Image by Horacio Lozada from Pixabay