iTranslated by AI

The content below is an AI-generated translation. This is an experimental feature, and may contain errors. View original article
📖

re:Invent 2025: Meta's Insights on Implementing AWS Security Controls at Scale (Identity, Network, and Resource)

に公開
2025/12/10

Introduction

By transcribing various overseas lectures into Japanese articles, we aim to make hidden, high-quality information more accessible. This project focuses on the following presentation!

For information on re:Invent 2025 transcription articles, please check this Spreadsheet.

📖re:Invent 2025: AWS re:Invent 2025-Meta's insights on implementing AWS security controls at scale (SEC308)

This video provides a detailed explanation of how Meta, with 3.5 billion daily active users, scales security on the AWS cloud. AWS Solutions Architects Robin Rodriguez and Syed Shareef, along with Meta Senior Engineering Manager Peter Nieuwenhuizen, explain security principles spanning three domains: identity, network, and resource. Meta's case study introduces short-lived credential issuance via Auth-D, IPv6-only network configuration, traffic inspection leveraging AWS Network Firewall, data governance via Privacy-Aware Infrastructure, and secure construction methods for AI clusters. Practical approaches proven in hyper-scale environments are shared, such as federated policy management using Service Control Policies (SCP) and Resource Control Policies (RCP), automation through infrastructure as code, and data perimeter implementation.

https://www.youtube.com/watch?v=5rTuf6bsYlM
Please note that this is an automatically generated article that maintains the content of the original lecture to the maximum extent possible. Please be aware that there may be typos, omissions, or incorrect information.

Main Content

Thumbnail 0

Protecting the Digital Experience of 3.5 Billion People: Meta's Cloud Security Challenge

That's wonderful. Thank you all for joining. I would like you to imagine something with me for a moment. Imagine you are responsible for ensuring the security of the digital experiences of 3.5 billion people. This means nearly half of the world's population logs in daily, shares memories, connects with loved ones, and builds communities across 150 countries. Each country has its own regulatory requirements, compliance standards, and security expectations. Now, here is the important part: this is not a hypothetical scenario. This is a typical Wednesday afternoon at Meta.

My name is Robin Rodriguez, and I am an AWS Solutions Architect. I am very happy to be with you all at re:Invent today. Joining me on stage today are my wonderful colleagues: Peter Nieuwenhuizen, Senior Engineering Manager at Meta, and Syed Shareef, Security Solutions Architect at AWS. Syed and I have had the privilege of working with Meta's teams to provide a secure experience that supports billions of users. And the scale of the challenges these teams are tackling is truly staggering.

Thumbnail 90

Today's session is about what we call "Cloud Fortification," but it's actually much more than that. It's about how one of the world's largest technology companies has rebuilt security for the cloud era. It's about hard-learned lessons, innovative solutions discovered, and practical wisdom that can benefit organizations of all sizes. Whether you are managing thousands of workloads on AWS or just beginning your cloud journey, the principles we share today can transform your thinking about security.

In this hour, we will discuss this AWS Cloud Security Playbook. What are the key principles for strengthening identity, network, and resources on AWS? We will discuss the unique challenges Meta has faced with hyper-scale infrastructure and how it has evolved into a hybrid cloud. Peter will share how Meta has implemented security controls across the domains of identity, network, and resources, as well as real-world lessons. We will also delve deep into how they protect large-scale AI clusters and the next steps in this security evolution. I would like to invite Syed to the stage to explain the AWS Cloud Security Playbook.

Thumbnail 210

Security at Scale: Key Characteristics and Anti-fragile Design Philosophy

Thank you, Robin. Let's start with the AWS Cloud Security Playbook. What does it look like? As an AWS security specialist for over six years, I have had the unique privilege of working with our largest customers. Based on our collective understanding of security at scale, we have identified several key characteristics that apply to all customers when implementing cloud security.

Security at scale, especially for hyper-scale companies like Meta, is defined by the fusion of ephemeral workloads, distributed systems, and automation. When trying to ensure security at that scale, this is an absolute necessity. W. Edwards Deming, the father of quality management, once said that you cannot achieve quality by inspection; quality must be built into the process. Similarly, security at scale must be built into the process—from the design phase, integrated into development, embedded in deployment, and operated at runtime.

When we say security is federated, we mean that security is no longer a gatekeeper or someone who simply says "yes" or "no." Rather, it is woven into the fabric of cloud deployment. The goal shifts from preventing change to ensuring that change is made within established safety parameters. Resilience in cloud security is not just the ability to prevent bad days. We all understand that bad days happen. What matters is how well you can withstand them and recover from them.

Our goal is to be anti-fragile. If you have read Nicholas Taleb's book, you will understand this concept. So, what does it look like? Instead of having a firewall that breaks and fails open when hit by a DDoS attack, you have automated rules that add malicious IPs to a Web Application Firewall and create honey-pots, allowing you to actually withstand the attack and recover from it.

Thumbnail 390

When operating at scale, what we see very often is a virtuous cycle created by companies learning from past events. We all agree that bad days happen, and it might be on a Thursday morning. However, this is only possible if there is an iterative process where teams can create controls for tomorrow's protection from yesterday's events. This truly relies on a blameless culture, focusing on the technology and timeline of the attack rather than individuals or human error, thereby turning every incident into a valuable learning asset for the entire team.

Strengthening the Identity Domain: A Hierarchical Approach to Federation, Policy Management, and Service Authorization

Now, as we move forward in talking about security at scale, let's look at some of the key domains. One of the first, and most important, is the identity domain. There are a few key points I want to emphasize here. Regarding Identity Providers and Single Sign-on mechanisms, the first is to use Identity Provider and Single Sign-on mechanisms that allow human users to access AWS applications or accounts within AWS. I want to reiterate that when using Identity Center, you can use it to access AWS applications as well as AWS accounts. You can choose to use either feature, or both.

Thumbnail 450

Now, let's look at common policies when trying to implement security at scale—specifically in a federated and automated way. When federating, we block bad things for the entire enterprise at the organization level, but then we leverage the organizational structure to apply policies at the organizational unit or account level, aligning them with the security requirements of the workloads running there. For example, you can apply Service Control Policies at the organization level to ensure no one in the organization can modify CloudTrail. Or, you can apply Resource Control Policies to prevent, for instance, specific types of encryption used in S3 buckets, such as those used by customers during ransomware attacks.

Thumbnail 530

An important point to emphasize here is that there are also management policies, one of which is declarative policy. You can use these to basically prevent non-compliant actions. For example, you could block public internet access to an entire VPC and create helpful error messages for end-users. Now, when using AWS at scale, you probably aren't enabling all services in your production accounts. You are likely using only the services you want. So, how do you effectively manage service authorization at scale while maintaining security and enabling developer innovation? Developers want to try the latest services. Many of you here are probably interested in trying out the new security agents or other amazing products announced just yesterday.

So what we do is implement a hierarchical approach, creating sandbox environments where developers can freely experiment in environments isolated from other workloads. A common pattern is to set a 30-day lifespan for sandbox accounts, after which they are deleted, but at the same time, ensure that those sandbox environments have some baseline security controls incorporated, such as allowing specific types of data, keeping data internal, controls on who can or cannot access it, and usage metrics collected in those sandbox environments. Because when someone experiments and finds that something works, they might want to move it to the next stage. At that point, rather than creating an entire workload somewhere else, they might have to make the choice to migrate that account to a main workload or production account.

So, how do we use metrics to facilitate that migration? The overall idea is to replace manual controls with automated guardrails so that we can scale safely.

Thumbnail 630

Thumbnail 670

Network Domain Security: Zero Trust Access and Traffic Inspection Patterns

Now, let's talk about some of the features of the network domain. One of the most important is zero trust access, right? This is about ensuring that actors, whether they are humans, machines, or devices, can access the resources they should be using in a way that doesn't rely on proximity. Instead, we leverage identity and other security features to perform authorization. AWS provides these types of zero trust mechanisms across the entire stack, from the edge to the cloud API. You can use a shared authorization context and consistent authorization evaluation throughout, and since you can collect telemetry across all phases, you can monitor access extensively.

Thumbnail 690

Now, naturally, you need to perform inspection as traffic passes through. Among customers operating at scale, we see three common patterns quite frequently. The first is the outbound pattern, right? Keeping safe things inside and not letting them out. In other words, how do you prevent connections from resources to untrusted connections or destinations? In contrast to this, potential risks that can be mitigated using, for example, a network firewall include the exfiltration of sensitive data and command-and-control type activities.

Thumbnail 720

Thumbnail 740

The counterpart to this is the ingress pattern, where traffic is sent from untrusted sources to internal resources. Similarly, you can use the same mechanisms here to mitigate threats such as SQL injection or the exploitation of vulnerabilities. And when operating at scale, you will have different trust boundaries across your organization, right? We talked about sandboxes and non-sandbox environments, but similarly, you will also perform East-West inspection. This is done between VPCs within the same region or between VPCs across different regions, and you will use the same mechanisms to inspect that traffic. You can see the details in the later slides.

Thumbnail 770

Thumbnail 780

Thumbnail 790

Thumbnail 800

Thumbnail 810

Another point I want to emphasize is that in large-scale operations, some customers are using identity-aware service authorization. I talked about how humans use this, but this is an example of how services use it. We have a service we provide called VPC Lattice. With VPC Lattice, you can create a lattice network for an application running in one VPC to connect to another application running in a different VPC. At this point, these applications do not have access rights, but by using VPC Lattice to create service definitions, target groups, action policies, and rules, we can enable this access. However, under the hood, we are leveraging other building blocks like AWS IAM to enable this connection.

Thumbnail 820

Data Governance and Resource Protection: KMS Key Management and Data Gravity

Next, let's talk about data governance, especially regarding resources. The foundation for this is understanding your data landscape. Security here starts with data classification and protection measures tailored to the sensitivity of the data. You cannot protect everything in the same way. You must choose to apply security resources to your most valuable assets. Early in my security career, a mentor told me, "Don't put a steel lock on a plywood door," and I have never forgotten that.

Important decisions here include the regions where data can reside for compliance reasons, encryption requirements, and backup requirements. Now, one of the key things that comes up, and something customers operating at scale practice very well, is AWS KMS key management. When operating at scale, you will have many KMS keys. AWS KMS keys are protected by FIPS 140-3 Level 3 hardware security modules. I've used a lot of jargon, but what I mean is that this gives you many options. It provides many options in terms of where and how you create keys, as well as where the key material comes from.

One best practice we often see is that a central security team wants to keep keys under their own management. However, a challenge often faced with this pattern is that the central management account where you hold those keys becomes subject to all KMS-level quotas.

Instead, you should move keys closer. Data has gravity. Similarly, encryption keys are drawn to it. However, this does not mean the central team loses control. By using the SCPs and RCPs I mentioned earlier, you can get the same governance and ensure that these keys are properly protected. But now, the keys are close to the data itself, and in particular, quotas regarding key usage come from the account where the key is being used. Similarly, when using KMS keys with bring-your-own-key-material, KMS allows it, but what you must consider is the performance and operational overhead you take on by doing so. I understand there are always very good compliance or regulatory reasons for doing it, but it is important to understand where they apply and to use them appropriately, rather than applying them uniformly across the organization.

Thumbnail 970

Infrastructure as code validation and building trusted security boundaries with immutable infrastructure

Next, I want to talk about how organizations implement defense in depth using infrastructure as code validation. When you automate, you end up defining all your infrastructure as code. Hopefully, you are not clicking in the console. What you need there is for infrastructure validation to happen in multiple layers—from the development phase in your IDE and CI/CD pipeline, all the way to deployment. So, what does that look like? Let's take a look at an example.

Thumbnail 1000

Thumbnail 1010

Thumbnail 1020

Terraform core is calling a Terraform template created using the AWSCC provider. This calls the Cloud Control API, where, if you have encountered CloudFormation hooks, this is basically a way to intercept these in the target environment and check for compliance with rules applied in that target environment. You can then take those CloudFormation hooks and evaluate them. In this scenario, for example, if you are currently creating an S3 bucket, you create a rule in this target environment—not across the entire organization scanning all templates, but in the target environment—where you say that actually very specific settings are required. For example, if you are doing a federal-type project and require dual-layer encryption, you can create a rule that says all S3 buckets here require dual-layer encryption, and the hook will use that to evaluate it. And based on pass or fail, you can provide helpful messages back to the developer.

Thumbnail 1050

Immutable infrastructure means treating infrastructure as code that is never changed after deployment. Basically, no in-place updates, no infrastructure drift, and changes trigger new deployments. This eliminates unauthorized changes or "click ops," ensures a consistent security posture, provides a clear audit trail, and enables rapid security response. The idea is that when doing large-scale cloud security, immutability transforms infrastructure from a potential vulnerability into a trusted security boundary.

Thumbnail 1100

Thumbnail 1110

Thumbnail 1130

Data perimeters: Multi-layered defense with Identity, Network, and Resource boundaries

Now, let's talk about data perimeters. This combines many of the elements we've discussed and is something we frequently see large customers implement to achieve their security goals. So, what is this? Before you build solid boundaries like the ones shown on the slide, you need to define what they mean for your organization. What are your approved data patterns, and what do you trust? What you own, what AWS owns on your behalf, or what trusted third parties own for you? In other words, in this case, you have trusted identities, resources, and networks.

Thumbnail 1140

Now, let's look at the primary objectives of data perimeters. As you can see on the slide, these are the main objectives regarding who can access what and from where, defined by different perimeters: identity, network, and resource. The identity perimeter primarily mitigates unauthorized access by identities outside the organization. The resource perimeter mitigates unintended data movement to resources outside the organization, and the network perimeter mitigates, for example, the use of credentials outside the corporate environment.

Thumbnail 1170

Thumbnail 1180

Thumbnail 1190

Thumbnail 1200

Let's take a quick look at how these work together. Accessing a trusted resource from a trusted identity—no problem, everything goes through. But what about untrusted identities? For example, I work for a company, but I try to bring and use my credentials from my personal AWS environment. What happens? The identity perimeter blocks it. Now, what if I try to access those resources from home using corporate credentials? The network perimeter blocks it.

Thumbnail 1210

Thumbnail 1230

Thumbnail 1240

And if I try to access a trusted resource using personal credentials, I am blocked again by the identity perimeter. Now, what about a data exfiltration scenario where I am working in a corporate environment and try to access my personal S3 bucket? How do these perimeters function? If I try to access non-corporate resources using corporate credentials, the resource perimeter blocks it. Similarly, if I am trying to use personal credentials but am coming from outside the network and trying to access these resources, the network perimeter blocks it. In this table, you can see where the different data perimeters apply, the appropriate policies to use, and the key IAM features used to achieve the purpose of that perimeter.

I hope this is helpful. I realize it was a lot of information, but I wanted to make sure you understood some of the key characteristics of large-scale enterprise security. At this point, I would like to invite Peter from Meta to talk about how they have built cloud security at scale. Thank you very much.

Thumbnail 1310

Meta's Hyperscale Infrastructure: Global Systems Supporting 3.5 Billion Users and the AI Challenge

Thank you, Syed. Today, I want to talk about the challenges of scale in the reality of Meta's cloud security. You might have thought about building for scale, but what does that really mean? It means being able to handle the traffic when everyone wants to connect to the service at the same time, like on New Year's Eve or Christmas Eve. Today, I'm going to talk about building at meta-scale. And as you can imagine, operating at this scale comes with many challenges. So, let's start by digging into some of the interesting and unexpected things that happen with planet-scale social media.

Thumbnail 1330

Before we dive deep into Meta's cloud infrastructure, I'd like to take a moment to share Meta's mission. It is to build the future of human connection and the technology that makes it possible. We have been a social technology company from the very beginning, and our technology stack reflects that in many ways. Compared to much of the industry, we are contrarian. This is because the problems of social connection technology we are trying to solve are unique. Our innovation provides people with new ways to connect, and we are committed to keeping everyone safe and having a positive impact on the world.

Thumbnail 1360

With 3.5 billion daily active users, the world relies on Meta for connection and communication. We have a truly global user base, and we cannot build with certainty about who is connected to whom. We are a tech company by birth, and we maintain an incredibly strong bottom-up engineering culture. It goes without saying. We love leading the development of emerging technologies like mixed reality and open-source AI. Today, I am happy to share some of the exciting work we are doing in expanding our infrastructure to AWS.

Thumbnail 1400

Supporting 3.5 billion daily active users, many of whom are generating content, requires hyperscale infrastructure. Social media workloads typically exhibit higher transaction rates than most large internet sites and systems. For me, that is expected. In social media, the cost per transaction—like "liking" a post or watching a video—does not require the movement of physical goods or the exchange of money. In other words, the real-world transaction cost is very low. We are not limited by the number of widgets in an e-commerce warehouse. We are not limited by planes, seats, or airport capacity. This means we see transaction rates that are orders of magnitude higher than most other global distributed computing platforms. The CAP theorem is definitely at play here, and eventual consistency is something we think about a lot.

Thumbnail 1460

Because there are no constraints of the physical world, our global user base loves to connect in unexpected ways. Our systems support diverse security, data, and privacy compliance requirements. We have invested heavily in developing privacy-aware infrastructure. It is built based on the context of each request and enforces privacy policies in real-time. This allows teams to programmatically define who, what, when, and where regarding when their services should be called and where data should flow. Meta addresses these unique requirements by building a globally consistent graph database. Not many workloads require this type of infrastructure. If you look up the TAO system, you can find research papers on how we did this.

Thumbnail 1510

Thumbnail 1530

Meta is building innovative experiences that are at the forefront of reshaping how we interact in our communities. Building systems to support experiences like mixed reality and open-source foundation models requires new approaches to scalability and the secure handling of data. We are working to solve these challenges at the infrastructure level.

Meta's strong engineering culture means we remain focused on building highly scalable hyperscale infrastructure. We are already a hyperscaler, and we continue to invest in our own data centers, backbone cables, and edge connectivity. It is because of these investments that we were able to provide the Threads app to 100 million users in its first week. Without this type of infrastructure, that would have been impossible.

Thumbnail 1590

However, for AI, that alone is not enough, so we are implementing a large-scale hybrid cloud. We are terminating Direct Connect fibers within our own facilities, and our security infrastructure is already in place. This gives us the confidence to venture further into the cloud. Many of the security premises we already have in place help us work while keeping our infrastructure secure across multiple clouds.

Meta has been at the forefront of AI application research for many years. It started with content ranking for News Feed and Instagram Reels. Afterward, we doubled down on building GPU infrastructure. 2022 was the year of large language models, and Meta entered this race with the Llama system. Subsequent Llama versions were trained on our own infrastructure, and as you have seen in the media, Mark launched the Super Intelligence Labs initiative.

We are currently building the best-in-class Prometheus 1-gigawatt and Hyperion 5-gigawatt superclusters. Despite all these efforts, we still have AI research workloads that require more capacity, featuring job sizes suitable for cloud GPU capacity. These jobs do not require the hyperscale contiguous clusters we are building in our own data centers. In this talk, we will delve deeper into what makes Meta's cloud deployments unique and how we are configuring AWS architecture to scale safely.

Thumbnail 1670

The Role of the Cloud Foundation Team: Identity Management and Short-Lived Credential Issuance via Auth-D

So, let me give you some background on Meta. Most of Meta's software engineering teams are developing systems that run on existing hyperscale container environments. These are suitable for small-scale web requests generated from our family of apps. Meta also runs large-scale compute workloads on the cloud.

The Cloud Foundation team is composed of Meta's cloud experts, responsible for low-level infrastructure such as transit accounts, Direct Connect links, and the construction of systems that enable the efficient implementation of cloud security standards. We promote cloud usage across the business and are building an infrastructure fabric layer that provides identity, authentication, public key infrastructure, metrics, monitoring, and many other support services. Our team is accelerating the efficient and compliant use of cloud resources.

Thumbnail 1740

We provide building blocks such as pre-packaged AMIs, Kubernetes images, and build pipelines. This allows software functional teams to automatically support privacy-aware infrastructure structures and move quickly while maintaining security and compliance. As we expand beyond our own data centers, the network edge no longer provides a strong data boundary, and we are increasingly relying on sophisticated identity systems to grant least-privileged access to users and service roles.

We have a centralized IAM system integrated with all internal systems. We use role-based access control to grant only the minimum set of functions required to execute workloads and management tasks. We use machine attestation as an input to ensure host integrity and generate short-lived, role-specific credentials.

Short-lived credentials reduce the blast radius of lost or stale credentials, reducing the need for complex revocation processing or untimely updates. We stream CloudTrail and other AWS logs into Meta's production infrastructure and scan for anomalies in real-time.

Thumbnail 1820

With strong identity and public key infrastructure, we have the confidence to extend our network to the cloud and support these workloads on-demand. Auth-D is our credential issuance service. It abstracts different credential and token acquisition mechanisms across different clouds. When a request comes from a Metaprod user or system, Auth-D receives a service call containing the cloud account ID, the role, and several associated identities.

Thumbnail 1840

Thumbnail 1860

In the authentication check step, we verify that the requested resource exists within the cloud account and check if the request is permitted by the policies stored in our internal IAM system. If the check fails, we return an authorization error. If the request is approved, we obtain credentials from the cloud service provider. This is the step that translates between our internal security structures and the different types of credentials required in the cloud.

Thumbnail 1870

In the case of AWS, this can take the form of STS tokens, Kubernetes tokens, or SSH certificates. In many cases, we generate a set of credentials used only for that specific request. This provides full end-to-end traceability for each user session or job creation, which is very important for maintaining data lineage. I will explain data controls and share an example of how our privacy-aware architecture maintains context when processing data.

Thumbnail 1910

If assembling the credential package fails, we return a service error to the user. We have standardized error categories—permission, service, or cloud errors—so we can correctly handle exceptions and route them to the right service team as needed. Once a user obtains the correct credential package, they can assume temporary roles with the necessary permissions to access cloud resources. The user then invokes assume role via SAML, initiates an SSH session, or executes a job.

Thumbnail 1960

Cloud Foundation maintains building blocks to refresh credentials as needed. This setup works well for us because we have highly technical teams using the cloud, and this extra layer of complexity pays off by saving time on maintaining static credentials or performing on-demand rotation. Meta has a fairly unique IPv6-only infrastructure, and we spend what we save on NAT or IP address space exhaustion calculations on a BGP infrastructure that optimizes routing tables and ensures they are not too large for routers to support.

IPv6 Network Architecture and Thorough Implementation of Infrastructure as Code

We still maintain significant network control and do our best to enforce a policy of one workload per cloud account. This automatically ensures VPC isolation between workloads, and Cloud Foundation maintains transit VPCs and common service VPCs in separate accounts. We pay a lot of attention to the strict control of internet ingress and egress, which is very helpful in maintaining data lineage for our privacy-aware infrastructure.

Thumbnail 2030

We allow East-West connectivity between VPCs quite freely, and by not restricting these connections too severely, we are able to enforce the one-account-per-workload rule. As you will see, even with IPv6 everywhere, the subnet complexity still remains significant. Meta uses IPv6 everywhere. It starts with multiple public /32 ranges.

There are a vast number of addresses, and I won't try to comprehend the number of zeros today, but there are more IP addresses than grains of sand on Earth. There is no need to worry about depletion. With so many addresses, we can assign a /62 per host, and furthermore, we assign one IP per Tupperware task. This is equivalent to our Kubernetes task.

In other words, anything running on the Meta infrastructure can be routed to any device or other task. And IP address conflicts or NAT within the network no longer exist. The tradeoff here is a massive BGP infrastructure that manages routing tables and ensures they do not exceed the capabilities of physical routers. This becomes an even bigger problem for some cloud providers that limit the number of static routes that can be used per account or per VPC.

However, overall, being able to extend a Layer 3 network to cloud VPCs is truly amazing. Then, we can have network engineering manage Direct Connect links and decide whether to route traffic to cloud infrastructure via Direct Connect, VPN, or the internet.

Thumbnail 2110

Before we get to the interesting stuff, let me explain a few more prerequisites regarding infrastructure. Cloud Foundation provides a managed experience of sorts to the cloud-using teams within Meta. We worked with the security team to set fairly strict rules on how to use the cloud. Some of these rules are for protecting digital assets, and others come from hard-won lessons we learned while scaling the number of people managing specific accounts.

We only allow cloud users to view accounts from the console. We quite strictly enforce the use of Infrastructure as Code through Terraform modules and the use of integrations we have built into Meta's main CI/CD system. Terraform code can go through the same peer review process as all other code that goes into our monorepo. We are moving toward true continuous deployment of Terraform code, with some of the larger systems running Terraform apply every two hours.

We use service gating for production accounts to ensure that internal support is in place before customers can freely start using new or complex AWS services. If teams just need to experiment, we provide short-lived sandbox accounts, which are automatically isolated at the two-week mark. We also provide Meta Instance. These are cleanly packaged AMIs loaded with all the software required to connect to Meta services, making things like identity, logging, metrics, and service registration available immediately at no cost.

Thumbnail 2210

Robust encryption controls are a key element in building trust when migrating workloads to the cloud. I won't go into detail here, but it is important to know that these exist within the infrastructure and are crucial for enabling the rapid construction of cloud capacity. Our internal public key infrastructure includes multiple root CAs and cloud-dedicated CAs, allowing us to track where certificates are being used and prevent them from being passed across infrastructure boundaries.

We use EC2 instance attestation to generate X.509 machine certificates, ensuring we can trust the hardware and operating system environment. By regenerating these certificates periodically, we can determine if anything has changed in the hardware or firmware. We protect all connections with mutual TLS, ensuring that both ends of a connection are verified and helping to mitigate man-in-the-middle attacks. Direct Connect links are encrypted at Layer 2 using MACsec.

Thumbnail 2270

Building Secure AI Clusters: Integrated Design of GPU, Storage, and Network Firewalls

Having covered the basic building blocks, let's dive into a case study on building secure AI clusters on AWS. AI clusters present some unique challenges due to GPU shortages and the massive data transfers required to run these large clusters. They also don't require the same level of resiliency and availability that would be needed when providing online production traffic. In a sense, it's an all-or-nothing situation. If a GPU goes offline, does it mean anything if the data store it is paired with is available? I don't think so.

Due to physical space and power constraints, we cannot specify which Availability Zones our GPU hosts are placed in. So, we set up infrastructure for rapid deployment of AI clusters and do not allow AI training jobs to span multiple clusters. Researchers also have unique needs. They often work through interactive sessions on the same hosts as the physical hardware. They also frequently pack Slurm jobs across large blocks of GPU capacity.

They collaborate directly with other researchers and frequently exchange code via GitHub. Sometimes they need to pull datasets from the public internet, so in short, they need easy access to the internet while preventing the leakage of weights, biases, or finished models. Now, let's explain in more detail how we build this secure infrastructure.

Thumbnail 2370

Let's start this diagram from Meta's production data centers. I've reduced them to two here for simplicity, also because the slides aren't big enough. You'll hear about this theme quite often as we go into detail. These are themselves very large. You can think of them as being analogous to an AWS region with multiple physical data centers inside two purple boxes. We terminate our own fiber in the data centers, and there are no colocation facilities on the left side of this diagram.

And we have shiny new GPU hosts delivered by Amazon as Reserved Instances in specific Availability Zones. The location of the GPUs and accompanying storage hosts determines the Availability Zones where we build the rest of the infrastructure. Once the hosts are placed, automation tools set up the control plane for the EKS HPC cluster and validate the GPU pods and login pods used by AI researchers.

Thumbnail 2440

Thumbnail 2460

Thumbnail 2470

The GPU hosts are not directly connected to the internet. Instead, we pair this VPC with a Management VPC. This hosts components involved in internet connectivity, such as routers, firewalls, Route 53 private hosted zones, bastions, and proxies, enabling access to other parts of the infrastructure and GPU cluster management services. You may have seen this architecture in previous re:Inforce sessions, but what's new today is the Storage VPC.

GPUs might be the brains of AI, but storage is the heart and circulatory system. Modern GPUs require very high bandwidth and low-latency data access to maintain high utilization. This is by no means easy in a generic public cloud where we don't have access to the specialized hardware, storage, and networking technologies we can use in our own data centers. This is especially important when running a cluster of 10,000+ GPUs with several petabytes of data distributed across smaller files.

We have faced many challenges regarding file system metadata performance, such as simply listing the contents of a directory. By creating an independent Storage VPC and placing a dedicated team of engineers in this area, we were able to conduct various types of tests, benchmarks, and tuning, including S3, S3 Express One Zone, various FSx for Lustre and OpenZFS offerings, and running our own Hammerspace instances. We are conducting all kinds of experiments in this area, including on bare metal instances.

Thumbnail 2540

Thumbnail 2550

This is where VPC peering comes into play, connecting the GPU VPC and the Storage VPC. We also store checkpoints, weights, and biases in specialized storage systems running in the Storage VPC. To keep everything secure, we block unauthorized access within the Storage VPC using S3 bucket policies.

Thumbnail 2570

Next, we add Direct Connect as a layer. This is the connection from Meta's production data centers to the Storage VPC. Important processing happens here. With LAG, DIF, and multiple lines, highly redundant bandwidth is secured, mitigating the impact of individual fiber cuts, devices, or rack outages. Through these links, we are transferring massive amounts of training data into and out of the cloud environment. Since GPUs are like hungry beasts, a lot of work has gone into optimizing the data storage layer and connectivity to keep this part efficient.

Thumbnail 2620

Thumbnail 2640

The final piece of the high-level view is DNS. Please remember that we use IPv6 everywhere. Therefore, the Storage VPC has a Route 53 private hosted zone, providing DNS resolution for everything accessible via these Direct Connect links. With this setup, we can decide, for example, whether to run research or login traffic via the internet or via Direct Connect.

AI research workloads often require some level of internet connectivity. Inbound traffic usually occurs in the form of researcher SSH logins, service account connections, and engineers connecting to review and optimize cluster performance. To support this, we have a sophisticated setup using AWS Network Firewall, Network Load Balancer, and NAT gateways.

On the internet ingress side, we perform traffic inspection to prevent intrusions or unauthorized connections. By using AWS Network Firewall, we can cover all types of application traffic in one place. Even if traffic is encrypted, we can apply controls beyond the network and transport layers. We use Server Name Indication extensions with these Network Load Balancers to enforce TLS versioning and fingerprinting. We also use AWS-managed threat signature support, allowing us to detect threats and block attack patterns of known vulnerabilities without having to write and maintain individual rules.

Thumbnail 2720

Our internet gateway setup is configured to allow only ingress traffic to the AWS Network Firewall within the ingress subnet, and access is restricted via routing tables. We use standard industry tools to write Suricata rules for IP filtering, and we have hundreds of entries for Meta's CIDR blocks. Because we have so much control over where to assign IPv6 CIDRs, we can reasonably whitelist inbound traffic based on source IP, and as a result, the connection can reach the GPU VPC.

Thumbnail 2750

On the egress side, Meta again deploys AWS Network Firewall endpoints in their own subnets, with access restricted via routing tables. We filter HTTP and HTTPS traffic based on domain names and configure domain filtering using a whitelist approach. Security groups are configured very loosely, generally either none, Meta's CIDR blocks, or completely open. VPC routing is configured to allow egress traffic only via the egress subnet, and domain whitelist registration is done in AWS Network Firewall.

Thumbnail 2790

We have a self-service system that allows researchers to assign domains to their clusters. This is available according to the needs of researchers, partners, or vendors. Egress traffic follows the path of the pink arrows. Egress inspection provides a vector for protecting against unauthorized outbound traffic and detecting malware communication patterns.

To summarize the internet connectivity portion, using AWS Network Firewall allows us to scale in ways that security groups cannot. In particular, it solves the problem of CIDR rule limits. Because Network Firewall has the capacity to support 1 million CIDRs, we can configure Meta's IP space in fine detail. Even if the CIDR blocks swell from about 30 to 200 or 2,000, it can handle it. Standard Suricata rules integrate well with our internal firewall management tools. Domain filtering allows us to use domain whitelists and blacklists beyond IP rules. AWS has a fantastic whitepaper on centralized ingress and egress filtering approaches, and we have used this to scale up to our current solution.

Thumbnail 2870

Privacy Aware Architecture and Data Lineage: Understanding Automated Privacy Policy Enforcement with the Banana Data Example

Moving beyond infrastructure to the application layer, we integrate with Meta's data privacy controls. This is a much more difficult area because policy decisions require understanding the data type, meaning, and request context before deciding whether to allow or block a request. For example, depending on which of Meta's family apps is being used for the request, different privacy policies may apply. Meta's Privacy Aware Architecture is a sophisticated system that automatically applies privacy policies as applications process data. While I will only touch on a few aspects of PAI today, you can find more details on the Meta Engineering blog.

PAI includes a central integrated metadata store where table schemas and data understanding signal annotations are saved. The signal collection system is responsible for identifying the semantic type and shape of data contained in columns and verifying that it matches the schema and intended purpose. These annotations are used to enforce policies and establish lineage, allowing us to understand the entire data lifecycle and reflect it in policy decisions.

Thumbnail 2960

Let's briefly explain the classic banana data example from our engineering blog. Suppose Alice is a user in the Meta system and is storing data about a bunch of bananas. She might own this bunch of bananas she purchased at a store. These diagrams show how that data is represented in the context of several different data warehouses. On the top left, there is a table context where data about individual bananas is stored, including color, country, flavor, etc. You can think of this as the online or base representation of the data.

When processing this into a data warehouse, you might be interested in a column representation where parts of Alice's banana data are included in a sundae instance. Alternatively, you might consider a row-based representation where different fruits have type, ripeness, and price. What about a cell-based representation where Alice's banana data is used to make juice with size and freshness? These diagrams depict how Alice's banana data propagates throughout the data warehouse, and the key point here is that data lineage is difficult. And, just like the actual fruit, it's not always possible to turn the sundae back into the banana.

Thumbnail 3050

Next, suppose we want to model user interactions where Alice's phone interacts with smoothie, fruit basket, and bread applications. In this case, we have set a policy that Alice's banana data can be used for smoothie and fruit basket use cases, but not for the bread application. To complete this analogy, perhaps there is a policy that blocks usage here because we don't want to heat the bananas needed when baking bread. For more details on how our privacy-aware infrastructure works, please check out our security and privacy blog posts. There, we explain in more detail how Meta enforces what we call purpose limitation through our large-scale privacy-aware infrastructure.

Thumbnail 3110

To wrap this up, I want to land this talk on our threat detection and incident response systems and close the loop. Closing the loop with these systems gives us confidence that our cloud AI environment is well-protected. We stream VPC Flow Logs into internal systems and run real-time analysis to detect anomalies. We monitor what's happening in S3 buckets by streaming S3 data events and utilize a wide range of third-party monitoring applications, including Wiz and GuardDuty, for threat detection. Automation outperforms human response times. By instrumenting everything and using a central account to host our detection systems, we gain the ability to automate remediation and isolation when issues occur.

Thumbnail 3180

Five Principles of Cloud Security: Automation, Proactive Design, Federation, Resiliency, and a Culture of Learning

Let's go back to Robin to wrap up. Thank you, Peter. Yes, we are entering the summary, and I appreciate everyone taking the time for this journey today. Looking back, I hope you remember that we started this journey by imagining what it takes to protect those experiences for our 3.5 billion daily active users. Peter gave us fantastic insights into Meta's real-world implementation. We could actually see that security at scale is not just about doing more of the same controls. It's about fundamentally rethinking how security works when operating at a planetary scale.

Returning to the principles we discussed earlier, the ability to automate manual processes is essential because manual processes inevitably fail at scale. Automation is not an option; it's the only way to ensure consistent security enforcement across a vast infrastructure. I recall Peter talking about their cloud deployment, automated credential provisioning through Active Directory, and how it enables both security and speed for developers. Proactive security by design—ensuring that security is built from the start, rather than being an afterthought in the organization. Their AI approach and policy validation throughout the CI/CD pipeline ensure that these controls are built in from the beginning.

Federate, and through the utilization of a multi-account strategy and Service Control Policies, provide teams with the freedom to innovate within organizational boundaries, enabling speed while managing risk by establishing guardrails. The final two points: be able to rely on resiliency, understand failure modes, and design systems that can detect and recover from those events. At scale, the question is not whether something will fail, but when it will fail.

Therefore, Meta's approach to network segmentation and threat detection allows them to respond quickly to unexpected events. Encompassing all of this is a culture of learning. Utilizing those real-world experiences where every event, every incident, becomes an opportunity to learn and strengthen our security posture.

Thumbnail 3330

What I want everyone to take away today is the incredible scale Meta operates at, which Peter shared. At the beginning, I called it just a Wednesday afternoon, but we all have our own Wednesday afternoons. Regardless of the scale, the fundamentals of effective cloud security remain consistent: know your important assets, implement appropriate controls, and build security into the process from the start. I encourage you to ask yourself which of these principles you can apply, where you can use automation to reduce risk, and how you can build a culture of learning around security.

Start with what you can do today. Whether that's implementing your first Service Control Policy, adopting infrastructure as code, or adding automation to an existing security workflow. What I learned from these engagements with Meta is that scaling security is not just about protecting more resources. It's about enabling the business to grow with confidence, knowing that your security posture is growing along with it.

Thumbnail 3420

Thank you for joining us here today. Peter, thank you for sharing Meta's amazing efforts. And Syed, thank you for explaining the details of the AWS security services that support large-scale deployments. Please fill out the session survey in the mobile app. Your feedback helps us continue to provide valuable content. Please enjoy the rest of re:Invent. And remember, security at scale starts with the right principles. Thank you.


This article was created automatically using Amazon Bedrock, maintaining information from the original video as much as possible.

Discussion