iTranslated by AI
re:Invent 2025: AWS WAF Best Practices and HSBC's Large-Scale Deployment Case Study
Introduction
By transcribing various overseas lectures into Japanese articles, we aim to make hidden, high-quality information more accessible. This project focuses on the following presentation!
For transcript articles of re:Invent 2025, please check the information summarized in this Spreadsheet. Please take a look.
📖 re:Invent 2025: AWS re:Invent 2025 - A day in the life of an AWS WAF administrator (NET317)
This video explains practical best practices for WAF operations through a day in the life of an AWS WAF administrator. It introduces WAF placement on CloudFront, origin cloaking using VPC origins and Origin Access Control (OAC), and procedures for building initial policies using protection packs. Dan Avidan from HSBC shares real-world deployment cases processing billions of requests across 35 markets, discussing the time required for tuning managed rules, handling DDoS attacks, and scaling log solutions. It emphasizes traffic management using labels, configuring Layer 7 DDoS protection, countermeasures against AI agents via bot control, and the importance of treating WAF configurations as Infrastructure as Code.
Note: This article is automatically generated while maintaining the content of the existing lecture as much as possible. Please be aware that there may be typos or incorrect information.
Main Content
Introduction: Following the Day of a WAF Administrator
Hello, and welcome to re:Invent 2025. I hope you enjoyed the first day and will enjoy the rest of the week. My name is Tzoori Tamam, and I am a Principal Solutions Architect Specialist with AWS Education Services. Joining me is Dan Avidan from HSBC, who will introduce himself later in the presentation. Today, we will cover a level 300 talk about a typical—or perhaps not so typical—day for a WAF administrator.
It will be a bit hectic and technically dense. Since this is a level 300 talk, we assume you already have basic knowledge, so we won't delve into what rules or actions are. We will dive a bit into labels and new features, but we assume a baseline technical level from the start. I apologize for the pace. I naturally speak fast, and there is a lot to cover in 60 minutes, so it will be even faster. You can watch this later on YouTube at 0.75x speed, which might be easier to understand, but for now, we will move quickly.
We will structure this talk as a story. I will embody the WAF administrator and take you through my day. Afterwards, we will hear a real-world story from Dan at HSBC. He will explain their journey through adopting AWS WAF. Let's start with waking up. It's morning, a new day, and a new web application. On my way to work this morning, I received a notification that there is a new application out.
Morning Start: Protection Planning for a New Application
My company likely misunderstood the hype around bots and AI, so we built a real bot-robot sales website and mobile app. Now, as a WAF administrator, I have a new application that I need to protect. First, this is the architecture. It is built to send typical API calls from web users and mobile applications, as well as from unknown automated clients, including good and bad bots. It is there, and everyone can use it. Traffic first hits CloudFront, our CDN service, as the ingress point, and from there, CloudFront routes traffic to multiple backend and frontend components.
An Application Load Balancer acts as the load balancer for web components. API Gateway routes API calls to different Lambda workers and Lambda integrations. Since there is static content from a single-page application, HTML, CSS, and JavaScript files are all hosted in an S3 bucket, and CloudFront handles all of that. This is my headache. After drinking my morning coffee, I now need to decide how to protect this newly launched application. There are multiple questions we need to ask ourselves when deciding how to protect a specific application.
We need to ask ourselves: where will we protect it? We saw that there are multiple components. Where will we put the WAF? Will we place it at the edge of CloudFront? Will we put separate WAFs on the API Gateway and the ALB? Where should it go? It really depends on the situation. It depends on who manages it and how much time we have. If we want to take our time and really tune policies for different components of the application, it makes sense to have different Web Application Firewall policies on top of the ALB and API Gateway.
However, we don't have enough time to manage all these applications separately. More importantly, when an attack comes to the edge of the network, we want to block it there. So, we will probably enable it on CloudFront. Next, we need to ask, when? When do we need to protect this application? If we had time, we could plan ahead, start in staging, test, and tune. And when ready to launch to production, we could start turning on different rules. But I got this call this morning. And the answer to "when" is now. The application is already out. I have to protect it.
In this case, we usually have to make some compromises. Since we don't have enough time in the world, there is usually some urgency when deploying a WAF. At least initially, we have to make some compromises. Some we enable in blocking mode immediately, while others we enable in count mode so we can tune as traffic starts flowing through the application. But again, we won't know until we look. We have to turn it on and see what happens.
Another question is, who manages this Web Application Firewall policy? Is it me, the security person? Is it the application team? Each application component has its own WAF policy, which means each of these people needs to manage their own WAF. This really changes the complexity we can impose on them. If they have to manage something they don't really know well, they will usually do the simple things. On the other hand, if the application people manage the Web Application Firewall policy, they really know when they introduce new changes to the application and what those changes are, so they can tune the Web Application Firewall policy accordingly.
So, again, we need to balance. We need to decide by understanding what goes where and who manages what. And finally, what. The WAF policy is a complex thing. It can do many different things. Obviously, we could turn on all the different toggles. That would probably be a mess. Probably things that weren't intended will get blocked, and we will need to roll back and do something else. So, "what" refers to what we implement on day one, what we plan to implement on day 100, and how we plan to roll it out. So, what goes into my policy? I don't have time for everything today, so this talk will focus on the two questions: where and what.
CloudFront WAF Placement and Origin Protection Implementation
So, talking about where to place it, one of the things we can do—and is usually the best option—is to place the policy on top of CloudFront. Why? First, especially when talking about volumetric attacks, we want to block traffic at the very edge. We don't want it to reach the origin, inside the VPC. There are a few reasons we don't want that, but remember that CloudFront has over 750 points of presence, and each of them has a capacity of hundreds of gigabits per second. This is a very well-built mechanism designed to withstand traffic peaks, so blocking bad traffic at that point makes a lot of sense.
On the other hand, you might also want to consider building smaller, more granular Web Application Firewall policies to place on top of different application components, letting the application teams handle them. In some designs we see, we perform coarse-grained control at the very front of CloudFront, and more fine-grained control with smaller WAF policies running on top of ALB, API Gateway, Lambda, or other underlying services. At the very least, we want to see WAF running on CloudFront. This is the best place to start blocking incoming attacks.
Once you place CloudFront on top of your application, there is one thing you must consider: how to ensure no one bypasses our WAF. To do that, you need to utilize a mechanism known as origin cloaking or origin masquerading. Basically, you want to ensure that only traffic originating from the WAF can reach the origin, whether it is an S3 bucket, an API Gateway, or an ALB.
When protecting the most common use case where you have a VPC-based application and need to ensure only traffic coming from CloudFront and WAF can reach it, the best response is to enable VPC origins. This is a kind of new feature we've had for a few months, so I don't know how many people are already using it and know about it. However, this is the best way to protect your origin. Basically, by enabling VPC origins, you turn your load balancers into internal resources. They no longer have public IP addresses. They become inaccessible from the internet. The only thing that can connect to these load balancers is the specific CloudFront distribution you've assigned to it. Not even the entire CloudFront; it's a very specific CloudFront distribution, and this is what we call VPC origin. This is the best scenario.
If you have an S3 bucket, the best way to protect it without making the bucket public is to enable Origin Access Control, or OAC. Basically, this means that when CloudFront communicates with that bucket, it uses an IAM policy. In practice, it assumes a role and gains access to the bucket just like any other policy you would place on the bucket. The S3 bucket is assigned a CloudFront role and is permitted to fetch objects from the bucket.
With this solution, you can mask and protect your resources. Basically, you ensure traffic only flows through CloudFront, and it cannot be accessed in any other way. If your application lacks one of these components—for example, an application running on-premises or an application running in another cloud—I will explain other methods later. Still, there are ways to ensure no one can access the origin without going through CloudFront.
Basically, you want to restrict access at the IP level. You can use managed prefix lists and unmanaged prefix lists. There is a JSON file available on the web and a managed prefix list available in every VPC. You can use these in your security groups to ensure that only CloudFront's incoming IP addresses are permitted. However, this alone is not enough. This is because any CloudFront distribution could access your application.
Therefore, you usually want to add a secret header to your CloudFront distribution settings. Something you rotate from time to time would be ideal. This secret header is added to every request going from CloudFront to the origin. And you need to enforce it on the origin side. It might be an ALB rule or an NGINX rule. Someone needs to check the incoming header and ensure that only requests with that secret header are allowed to pass.
Building WAF Policies: Leveraging the CloudFront Console and Protection Packs
This is a supplementary explanation for maintaining a stable, best-practice application architecture. Now, let's move on. The way to start building AWS WAF policies was one of the biggest topics in conversations with customers in the past. I have been dealing with customers regarding WAF for many years—for a longer time than I'd like to admit, actually. However, it has become really easy recently.
One of the simple ways to enable a WAF policy on CloudFront is to go to the CloudFront console. I don't know how many of you have seen this, but this is relatively new. It was released about two weeks ago. You simply choose a flat-rate pricing plan that includes WAF, and the WAF components are already included.
If you zoom in a bit, you can see various plans starting from scratch. It includes CloudFront even for very simple configurations, includes DNS, includes a simple WAF configuration, and continues on to various pricing plans. By choosing one of these plans, you essentially get a pre-built policy. This is usually "good enough." "Usually good enough" is a big word, but it's much better than nothing, and it's a very good start to begin adjusting rules to match your policy.
You can start from a safe and manageable place and progress from there. I will talk a little later about the balance between manageability and security, but let's zoom in on the business offering in the middle and see what it includes. Let me hijack the discussion a bit for marketing purposes. It includes quite a lot. It includes DDoS protection, storage for S3, and DNS requests. It is a really great package, and you can have multiple of these within different distributions.
For example, let's say you have 50 distributions. You can attach one of these packages to each. So, if you are dealing with an application you know nothing about, or if you yourself don't have enough knowledge to start with a robust web configuration, starting from the CloudFront console would be a very good way to go.
When you start or enable WAF configuration in CloudFront, there are very simple questions you can answer by checking boxes. Do you need SQL protection? Do you need rate-limiting rules? Do you want to run only in count mode or monitor mode? These are very simple toggles. You don't even need to know about WAF, and the application owners you delegate to don't really need to know anything about WAF either. It is good enough to get started.
Certainly, and perhaps the best way, is to build on top of this and add more controls as you go. This is all in the AWS CloudFront console, and it is really easy to start building WAF on top of that. The experience is a little different if you start from the WAF console. This is a change we committed a few months ago, so I don't know how many of you are familiar with it.
Basically, when creating a new Web ACL—or what we now call a "protection pack"—you are asked simple questions about your application. What kind of application is it? Content delivery? API? A combination of those? Retail? You just check the boxes, and you can start building your application configuration from there.
Basically, you or the application owner will answer questions about the application that you should know. From there, you can choose from these pre-prepared protection packs. These include sufficiently good security policies. They contain more features than the policies deployed in CloudFront. If you are already using the WAF console, you probably have some knowledge, and you will be able to access more advanced features. So, there are three options here as well. The basic one has rate limiting and IP reputation services enabled, and bot control is in count mode. You can also proceed to the essential pack. This provides more features. However, more is not necessarily always better. We obviously want to aim for a very robust and feature-rich policy that protects against various types of threats that occur on the internet. However, that can bring you operational costs.
Usually, when you start with a WAF that is too strong from the beginning—I have seen this happen many times with all WAFs—if you come on too strong at the first sign of trouble, false positives and blocked requests occur. You won't be able to handle them, and eventually, you will just turn it off. The hurdle you have to jump over to re-enable it after failing once is very high. Therefore, I recommend starting low and building on what you already have. Once you establish trust in how the mechanisms work, know where the logs are, and know how to tune things, you can start building from that foundation.
After setting up your application policy through WAF, there are several other things you can set, such as rate limit thresholds. It can be whatever suits your application. Here, we started with 1,000 requests per 5 minutes per IP, but you can tune it down to as low as 10 requests if your application requires it, or up to as high as 2 billion requests if your application requires it. So, we start with good standard numbers, but you can tune them. And although this part is very simple, it is very important to enable it. It's logging. You just check the box and say you want logging. The same applies to the CloudFront console. Enabling web logging is very easy, and it sends logging toward CloudWatch logs.
Returning to the fixed-price plans we announced a few weeks ago, logs are included in that pricing plan, so it is no longer a concern. Since CloudWatch logs, which is the easiest way to manage WAF logs, is included in the pricing plan, you can just go ahead and check the box and enable it. Even in the traditional pay-as-you-go model without using the pricing plan, we have lowered the price of CloudWatch logs using WAF. Currently, we provide 500 megabytes of free logs for every 1 million WAF requests sent, so it really helps in cost reduction when not using the pricing plan.
In summary, you now understand what we are aiming for with the Essential Pack and what you can get. You get IP-based blocking, geolocation-based blocking, and very good core HTTP compliance protection rules. You get known malicious inputs like Log4J—that is, known vulnerabilities. All with a simple click. Furthermore, you also get the new Layer 7 DDoS rules in count mode, so they are already enabled. They are placed in the correct place at the top of the policy. All that remains is for you to monitor what they do and enable them when you are ready.
Coffee Time: Monitoring Dashboards and Logs
It's time to have a coffee. I arrived in the morning, heard the news about the new application, and built a WAF policy that met my needs with just a few clicks. Now it is time to look at the logs. This is what makes me feel good—looking at the dashboard all day. I really love it. Imagine this is me. It's the best thing I could do using AI. I am a WAF person, not an AI person. AWS WAF has these built-in dashboards that are ready to use. Again, setting up dashboards in WAF used to be difficult. I would say it was relatively difficult. You had to know what queries were needed to build CloudWatch dashboard widgets.
Since last June, we released this new UI. There are very useful dashboard widgets where you can check where the traffic is coming from, which rules are being hit, what kind of bot traffic is being seen, and what kind of DDoS traffic is being seen. You can drill down into different insights such as top IP addresses, top URIs, top user agents, and top fingerprints. All of these useful features are enabled by default. Most importantly, you can navigate logs directly from the console. This means you don't need to run tedious CloudWatch log queries. Everything is built-in.
You can click on an IP address in the top IP list, filter by that IP, and start seeing what these IPs are doing in your application. Each of these rows is basically clickable. You can see full request details, IP addresses, reputation, query string parameters, and labels that we have accumulated for that specific request. Everything is actionable, so you can take something from there, go back to the filter, and ask for more requests by this URL or ask for more requests by this fingerprint. If you need more comprehensive or more advanced queries, you can also use old-school CloudWatch Log Insights queries.
Naturally, this applies if you are using CloudWatch Logs, but it makes sense because it's cheap and included in the pricing plan. If you are using other destinations for logs, you can use similar mechanisms to build your own dashboards and run Athena queries on top of S3 as you did before. That part hasn't changed.
Lunch Break Crisis: Handling False Positives and Leveraging Labels
Looking at the logs calms me down. I also had my coffee. It's lunchtime, and just as I sat down to eat, I heard shouting from Marketing upstairs. Something about 403 complaints from customers, or maybe I misheard. So I start looking at the dashboard and I see these 403 errors. I see multiple blocks coming from different IP addresses, and all of them are hitting the cross-site scripting body rule. I have a feeling that this is a false positive that I need to address.
So I start digging into the logs in more detail. I see different IP addresses, and looking at these IP addresses, they are basically harmless. They are doing good things. They are logging in, using shopping bots, or posting on forums. But at the same time, they are triggering one rule on a specific webpage, where it is detecting cross-site scripting attempts in the body. I think this is a false positive that needs to be addressed.
So, the first rule when dealing with false positives is to stay calm. Do not turn off logging or roll back changes. Try to understand exactly what the pattern triggering the false positive is. Once you know that, it could be a specific URI, it could be a specific parameter, or it could be anything in the request. By looking at and interacting with the dashboard, you can really identify it and start building a rule based on it.
Now, I have also regained my composure. First, I turn off the rule that triggered the false positive. In our case, it was a cross-site scripting attempt in the request body, so I either turn it off or switch it to count. This way, users will no longer be blocked, but on the other hand, I am not enforcing cross-site scripting in the body, which is not good.
So, I build another rule immediately after the rule that triggered the false positive. That rule is structured to still block cross-site scripting attacks based on this label—I will talk about labels later. It still mitigates cross-site scripting attacks, but not if the URI is the slash-missing-robots-form. Because that is what I saw in the logs. By doing so, I basically returned to enforcing cross-site scripting in the body but excluded that specific URI from that rule. All other attacks against that URI are still valid, so this is a very good, fine-grained way to mitigate false positives.
Then, I start tuning from there. Tuning helps avoid chaos. Tune your policies frequently. Tune them early and do not wait until something happens. Check the logs to ensure you are doing the right thing and that enough traffic is being blocked.
And verify that enough traffic is getting through. If you make changes, make them incrementally. Do not add five different rules on the same day. With WAF, more is not necessarily better. Add rules incrementally and monitor their effects. Manage traffic with labels. There is a small starter using labels. I don't know how many of you know about labels, but labels are your best friend when it comes to WAF.
Labels are metadata. We add them to every request that passes through WAF. You can add them with managed rules. By default, every managed rule adds different labels. For example, these are labels that come out of the core rule set. This is just a part of it, there are more. You can add labels as part of an action in custom rules. You can use them as a match on a rule or as a scope-down condition. Think about the case where you don't want to inspect certain traffic from a specific IP address. You can create a custom rule that adds a label to all traffic from that IP address, and then use that label to scope down another rule so that it doesn't inspect that specific traffic.
Labels help with investigation. Because they appear in logs. They also appear in CloudWatch metrics, and you can use them if you have too much noise, high costs, or want to filter out specific rules or specific traffic patterns from logs for other reasons. Labels are a very useful tool. They are like a multi-tool for WAF, and you can really use them for anything. Always add labels first, then think about what to do with them. Labels are your best friend.
Afternoon Coffee Break: Practicing Cost Optimization
Finally, don't worry, I am not late. Lastly, since I am from Israel, I usually have coffee at 2:00 PM. I don't know how it is in the US, but we take a coffee break at 2:00 PM. Usually, it's strong black coffee. I use that time to optimize costs. So, I have adjusted the policy. I know I am blocking the bad stuff. I know I am passing the good stuff. It is already deployed, but now I am making sure I am not making mistakes regarding costs.
To manage costs, first, you need to ensure that the right things are being blocked in the right place in WAF. Ensure that the Layer 7 DDoS rule is first. Because every time you block traffic, it does not incur costs. Blocked requests are free with Layer 7 DDoS. It's free even without a pricing plan. Ensure that this rule runs first, and try to narrow the scope for other page rules like bot control so that they only get the traffic you think should be inspected. Also, check your logs to ensure logs are not being retained for a long period. Set specific log retention periods based on your business needs. Whether it's 14 days or 7 years, just stick to it and do not retain logs forever.
Use labels again to filter out noisy logs and switch to the flat-rate pricing plan. The flat-rate pricing plan is truly predictable, with no extra charges. You know what you get every month. Regarding logs, CloudWatch Logs are the easiest to set up and usually the cheapest, so they are sufficient, especially if your traffic rate is low. If your traffic rate is high, you might want to consider other ways to send WAF logs. You might want to consider Kinesis Data Firehose. This is outside the pricing plan and is likely the cheapest way for very high volumes of traffic. You can also use it to send traffic to different places, such as S3 or external third-party vendors like SaaS offerings.
Finally, you can send WAF logs directly to an S3 bucket using an S3 bucket. Behind the scenes, it uses CloudWatch Logs, so you may incur unexpected vending costs at large volumes. However, this is the easiest to set up. If possible, use CloudWatch Logs. If not, use Kinesis Firehose. If you need S3, use Kinesis Firehose to send traffic or logs to S3. Finally, if you are using a pricing plan, it is really easy to check if you are about to violate the contract you signed. You can see how fast you are hitting the quota based on what has already happened that month. For example, how many of the 500 million requests you have already sent. The goal is the same for data transfer. However, as long as it is not a consistent violation where you are violating it every month, it is fine even if you are about to violate it.
If you violate the quota in any given month, there are no extra charges. If this happens frequently and you are violating the quota, you will eventually experience throttling. However, there is a very clear dashboard in the UI where you can track where you are heading and whether that plan is the right one for you. This concludes my coffee break. I will pass it on to Dan.
HSBC Case Study: Background of WAF Adoption in a Global Bank
Hello everyone. I hope you are enjoying re:Invent. I am Dan Avidad from HSBC Retail Bank, and I belong to the IWPB International Wealth and Premier Banking team. Today, I will talk about our initiatives. First, regarding who we are and our position within the organization, we are digital enablers. We provide common frameworks and platforms, such as mobile authentication and authorization, business process orchestration, and of course, the edge routing and protection platform that I will talk about today. The services we provide allow various value streams to focus on their specific business verticals without worrying about things like DDoS protection.
HSBC is one of the world's largest banks and operates in many markets. The majority of them can adopt AWS, but some cannot. For example, in highly regulated markets with data residency requirements like China, we cannot use AWS, and in markets like Egypt, we cannot use any cloud providers at all. We need to operate across multiple cloud providers. We operate in highly regulated environments, and any downtime must be reported to regulators. Therefore, protecting our brand and defending against DDoS attacks is of the utmost importance.
There is an interesting anecdote. For instance, in markets where we own white-label brands, such as Marks and Spencer Money in the UK, or competing brands in the Asia-Pacific region, we must maintain separation between them to comply with antitrust laws. Examples include Hang Seng Bank and the Hong Kong retail bank. We provide not only banking services but also insurance products. Ultimately, this means that our site traffic patterns are very diverse, and naturally, we need to account for this across various systems.
We have a regulatory obligation to enable our customers to perform transactions. If people cannot access their money, trust in the financial industry is naturally damaged. Regulators take this very seriously. Peak times often coincide with periods of geopolitical tension or natural disasters, which also pose challenges. Regarding our team, we operate in many markets with multiple pre-production and production AWS accounts. We support many value streams, and each value stream has multiple applications. If you do the math, it ends up being a very large attack surface, so our protection truly needs to be world-class.
We handle hundreds of terabytes worth of traffic and process billions of requests every month. We have hundreds of CloudFront distributions and multiple WebACLs, and again, we support multiple cloud providers. Unfortunately, the reality is that we are under attack every single day. It is not something that happens rarely; literally, some form of attack is occurring 24 hours a day. I will not go into detail on this slide, but while these banking industry challenges are not unique to us, it is interesting that regulators are becoming increasingly interested in them. This is not as much the case in the US, but it is certainly true in Europe, driven by the European Central Bank, and in the Asia-Pacific region. This is now becoming a mandatory regulatory requirement, so it is very interesting to see regulators mandating the use of these protections.
HSBC's Journey: Implementing WAF in a Serverless Architecture
Regarding when our journey began, interestingly, it was at re:Invent in 2018, and it coincided with the need to create a new cloud platform.
Personally, I was truly inspired by all things serverless, so I had a personal mission to do it completely serverless. And we actually achieved that. Of course, the migration to the cloud is taking longer than expected. We are still operating in a hybrid cloud environment, which requires special considerations. The intent was specifically to shift security to the left, abstract our origins from third-party service providers and our origins from sensitive security tokens, and not have to worry about edge protection. This is what we provide, doing everything completely serverless through compute at the edge.
This is the architecture, and it is not particularly exotic. Shield provides Layer 4 mitigation and passes requests to CloudFront. We inspect the requests, and if they are not blocked, they are passed to Lambda. As you can see, there are no EC2 reverse proxies. Everything is done serverless. This predates CloudFront Functions. Ideally, we would migrate some of these to CloudFront Functions now. Another important point is that we are not just protecting AWS origins. We have created a generic, abstracted platform that provides protection for various origins, such as on-premises or other cloud providers, for example, GCP-based origins.
HSBC's Challenges: Evolving Threats and Organizational Barriers
There are several challenges. One is that the threat landscape is constantly evolving. When we started, DDoS wasn't really much of a problem. If it happened, AWS Shield would mitigate Layer 4 type attacks, and we could actually claim credit without doing much at all. AWS Shield would kick in, mitigate the attack, and that was it. Then there are Layer 7 attacks. Malicious attempts at domain name slash GTC password, which might succeed if you're lucky. Then there are volumetric attacks, which the industry addressed with rate-limiting-based tools. And of course, there are slow and low attacks. And naturally, the current concern is agentic. For example, if I have a legitimate agent that I really want to transfer money on my behalf, how do you distinguish between a legitimate agent and a malicious one?
The reality is that, unfortunately, executives don't really understand these things. Even if you're not a Tier 1 bank, it's the type of thing your management has probably said to you, like, make sure you get paged every time we're under attack. No, the team absolutely cannot sleep. We are constantly under attack. For example, I want to be paged when the origin is struggling, when the origin is in distress.
There is a misconception that once you invest in WAF, you're done. For example, if you have money invested this year, you won't need to invest next year. No, the reality isn't like that; you need to continue to invest continuously. You need to continuously enhance the WAF to protect against new threats. Especially because of the serverless nature of our platform, if there's an attack that breaches our defenses, we get told, for example, it's because you're doing it serverless that problems happen. Why are you using Lambda? It wouldn't happen if you were using EC2.
The important point is that no matter how much computing resource you have in the world, you cannot mitigate a really massive DDoS attack. It's not a question of the type of computing you use. The important thing is to have really strong protection measures to prevent attacks. It's not the type of computing you use; that's the point that matters. Another reality is that engineers and application teams don't really understand this. Unfortunately, this is not something they deal with daily. They are well-versed in API Gateway, Kubernetes, and EC2, but not in security. On top of that, they are naturally under pressure to deliver new business features. WAF is probably not one of them.
On the other hand, the cybersecurity team understands security well, but not so much about applications and how to protect them. That's where our team comes in, trying to bridge the gap between the cybersecurity team and the application team. We try to step in between them and bridge the gap. From an application perspective, if you are using AWS Firewall Manager organizational policies, you should talk about, for example, OWASP 10 rules, but then you need to consider how to integrate application-specific rules or AWS account-specific rules into those policies. Whether you do it at build time or at runtime, there are things to consider there.
AWS WAF, unfortunately, does not have deployment stages like, for example, API Gateway, so you really need to think about how to roll out new rules. For example, let's say you've done a really good job of consolidating CloudFront into one and consolidating WAF backends into one.
But now, if that one WAF does something it shouldn't have, suddenly the entire organization goes down. So you have to think about things like whether to use blue-green web buckets to toggle, or whether to just create more web appliances than necessary to artificially minimize the blast radius of the organization.
It's actually counterintuitive, but detecting actual attacks can be complex. For example, let's say the business ran a really successful campaign, but they didn't notify you, and there are many, many distributed requests coming in. Is that real user traffic or is it an attack? So what you have to think about is, do you really know you're under attack, or is it genuine traffic?
If you're using AWS managed WAF, naturally, you need to think about how to tune it. For example, let's say there is a rule in AWS WAF that limits a specific size. What happens if suddenly one of your applications needs to support a value higher than that threshold, for example, for document uploads? So you really want to tune it for that specific journey, for that specific application, and minimize the impact. And this tuning takes time.
I've shown a concrete example of Amazon managed rule tuning here. It took a significant amount of time, including AWS support. I won't explain it line by line, but you can see the types of changes that were needed, from how to get AWS WAF tokens from application changes to changes by the service team on the AWS side. Thanks to the service team. Thank you. It was really great agility, and AWS rolled out extensions across the state within a sprint. So that's truly the agility we should aim for. And ultimately, it was worth the time. Because we were able to achieve a level of false positives that the business could accept.
Don't be afraid. Ultimately, it was truly worth it. Because this allowed us to create a cookie-cutter report that we could successfully roll out to the other 35 markets after that initial tuning period in those two markets.
HSBC's Lessons: Technical Challenges and Future Outlook
There are several challenges on the WAF side. So really, as Suri said, you need to think about how to protect your on-premises origins. Previously, there was this limitation with VPC origins for AWS origins, but that's gone now, so I'm excited that we can roll it out more widely. VPC origins no longer need to be in the same AWS account as the CloudFront distribution.
You need to remember that if you are adding CloudFront distributions regularly, they aren't automatically registered for Shield Advanced auto-mitigation. When you add CloudFront distributions, you must ensure they are registered to be protected. Actually, I recently learned that WAF can add custom labels, and while CloudFront Functions and Lambda Edge don't have triggers, WAF can actually inject headers. This allows for some very interesting things within CloudFront Functions or Lambda Edge, which is quite convenient.
We learned the hard way that our logging solution didn't scale during a DDoS attack. This is something you want to think about. You might think you have a functional logging solution, but you want to ensure it can scale up during a DDoS attack. This is very important. Otherwise, you won't know how to analyze the attack and mitigate it, for example, by adding IP-based rules. We have CloudWatch Logs Insights, but we noticed that with our traffic, we could have queries that run for quite a long time. It's not the cheapest service, so we are considering using AWS OpenSearch, for example.
Lessons learned at the end. It's like saying sell high and buy low, but what I want to say is to protect against tomorrow's attacks, not yesterday's. Have you added new infrastructure? Have you added new applications? Are there new WAF features you really want to add to your application? WAF comes with an API. If you aren't familiar with the WAF automation framework, you can mine logs to find interesting events. For example, why not automatically block IPs that are introducing 404s, or IPs with a large number of 500 errors? For example, for a period of 24 hours.
Know who your TAMs (Technical Account Managers) and solution architects are. We get tremendous value from them. They can register you for AWS WAF betas and various previews. Continue investing in Security and Compliance (SAC) and the WAF team. This isn't something you set and forget for WAF. Continuous investment is required. Of course, the AWS SRT team is always there, but you really want to be able to analyze logs independently and mine logs yourself. For example, you want to add JSON for fingerprint blocking rules yourself. Automation is absolutely critical. You don't want to be there at 2 AM trying to figure out how to revert a rule, so automation is absolutely critical.
As for our next steps, it's finally to leverage the investment we've made in WAF so far. To consolidate thousands and thousands of domains and endpoints into top-level domains, so we can finally benefit from the level of protection I've described. Thank you, Dan. I know HSBC is a huge bank. Many of the challenges they encountered might not be your challenges, but it's a really good practice to see that even at their scale, it's possible to basically use WAF as a layer to protect every application they have (thousands and thousands of applications).
Twilight Reflections: Evolution of WAF and Best Practices for Continuous Improvement
Now, as the sun goes down, I can't help but get philosophical, looking to the future and thinking about what else is possible. As Dan said, you need to look ahead when it comes to AWS WAF. Don't be passive. You need to be proactive and respond as fast as, or faster than, the attackers. Testing new rules, adding new rules, configuring settings, adjusting and hardening security—I'll talk about three different things in the next few minutes. First, I'll talk about bot control, then Layer 7 protection, and finally, the general use of new rules, new custom rules, and new managed rules.
Regarding Bot Control, my company completely misunderstood what bots actually mean. With Bot Control, you can use the managed tool we provide to first identify general bots, both good and bad. Specifically, allowing search engines and SEO engines is important, and this can be achieved with general Bot Control. Or you can block noisy health checkers coming from the web just because they only self-identify. This is something easily done with general Bot Control. Detecting and finding more advanced bots, known as targeted bots, which are bots built specifically to harm your application—this is where you typically enable targeted Bot Control. This is the next level of Bot Control, where you can track anomalous behavior. You can look not just at specific requests, but throughout the entire session duration to see which sessions are generating more traffic or are drastically different from normal user sessions, and challenge them with CAPTCHAs or JavaScript challenges, or simply block them if they are acting too suspiciously.
Now, in the age of AI and AI bots, much of the traffic you receive—and when I say much, sometimes over 50 percent of the traffic you receive—might be generated by AI agents. Some are harmless, others are not, and all of them are very noisy; you need to ensure that only the good ones enter your application. Make sure to leverage Bot Control and upgrade to version 4, the latest version of Bot Control. With this, you can use Web Bot Off to ensure that only authorized and verified agents enter your application. All other agents likely need to register and be verified for this mechanism. This isn't a standard yet, but it's a mechanism used by most web providers and agent registries to ensure only good bots pass through.
Regarding Layer 7 DDoS, these are new rules, so if you are not aware of them or not familiar with them, get familiar. There is a QR code at the end, so you can scan it and learn more. By enabling the essentials pack, which is included in the business pricing plan, you can start using it. It was created with override checks enabled, so if you look at the dashboard—there is a specific dashboard for DDoS—and confirm that there are no false positives, and there won't be, this is a very safe rule.
You can check this box to ensure it only functions during a DDoS attack. It's very quiet and does nothing during normal times, but only when an actual event occurs, it starts targeting attackers by specific IP addresses, countries, AS numbers, user agents, or fingerprints. Only these will receive this challenge or be blocked if they don't take action. When evolving your WAF, be sure to use this very important and very powerful tool.
When adding new rules, be sure to track traffic and traffic changes. It's possible that you go viral and traffic increases, or you go viral in a bad way and attackers try to harm you. Be sure to track all traffic, not just the blocked traffic, but the allowed traffic as well. Are different rules hitting? Are there any rules that suddenly stopped hitting? You might be missing something. Attackers might have evolved and you can no longer find them. Be sure to track traffic constantly.
On the other hand, strike the right balance between secure policies and manageable policies. The most secure computer is one that is locked in a vault and disconnected from the world, and that cannot be used. Strike the right balance for you. I don't know how much time you have to invest in WAF, but it is an investment of time. Add the rules that provide the most value. It doesn't have to be a perfect policy. That's enough. The best policy is one that you can manage. Otherwise, you'll turn it off and do nothing, which is much worse.
Strike the right balance, and tighten the right rules. Make sure the rate-based rules you introduce track not just IP addresses, but also fingerprints, methods, and labels. Leverage what WAF has to offer to the fullest. This is a continuous effort. As attackers become more sophisticated, tighten your rules more and more.
Another good practice is to maintain a staging environment. Dan touched on this a bit, track changes, and roll them out incrementally. When deploying to production, try to predict what will happen. Build your label logic in a way that allows you to test in a staging environment. If you have the capacity for that, that's great. In some scenarios, you can add the same rule to the same policy multiple times in count mode. You can add multiple rules, including managed rules, and see how production traffic behaved in count mode instead of block mode.
If you don't have the capacity to run a full staging environment, at least do this. Add multiple versions of the same rule to the policy and set different actions. Order them correctly and see what happens if you switch the rules to block mode. Again, labels are your friend.
Finally, don't forget that AWS WAF settings are code. If you look at the WAF console, there is a JSON button here. Your entire WAF configuration is just a JSON document. Use this. Use this power. Use the APIs WAF provides to automate and leverage Infrastructure as Code tools. Version control becomes better, change management becomes better. Automation during new deployments becomes possible. If you have the resources—and AI now writes most automation—it's really simple and can save you a lot of trouble. On day one, everything is improvised. By day 100, you should probably be doing better.
That's it for today. The application is fine for now. Tomorrow is a new day. Naturally, on my way home, I'm thinking about the lessons learned from this busy day. We talked about architecture and origin cloaking. We talked about having a sufficient initial policy. Build it however you want. Manually, with pricing plans, with WAF configuration protection packs, or whatever, but do something on day one. Even a policy with a single rule in count mode with logging is far better than having no policy. At least you get visibility.
Do one thing you can manage, and grow from there. However, make sure to follow up. And follow up quickly and repeatedly. The web is a bad place. I'm not here to scare you. I'm just here to show you that WAF is a continuous effort.
I've done too much WAF by now. With the new pricing plans, it's really predictable and simple. A lot of value is included at a fixed price. Ensure your WAF configuration is evolving. Since attackers evolve, introduce new rules, upgrade rule versions, keep testing managed rules, keep looking at logs, and keep watching for changes in traffic and rule hits.
Which rules are hitting and which aren't hitting are equally important. If possible, automate and test changes. This is an iterative, repeatable, predictable, and easily rollbackable process. WAF configuration is code. Treat it like code.
Well, I promised you three QR codes. Don't scan them yet. Take a picture instead. One is a blog about the new flat-rate pricing plan. It's new, so you can learn about it. I didn't cover it in depth, but it's very useful. These other two QR codes are about Layer 7 DDoS mitigation using new managed rules. One blog is the launch blog in the middle, and the one on the right is a more detailed explanation of how to configure it and how to tune it if necessary.
Thank you very much for your time. I know it's a busy time and a long day. Some of you might still have jet lag, and others might have it even tougher. Thank you for your time. If you don't mind, giving feedback in the app would really help. Have a great rest of the week. Thank you, everyone.
- This article was automatically created using Amazon Bedrock, while maintaining the information from the original video as much as possible.





















































Discussion