iTranslated by AI

The content below is an AI-generated translation. This is an experimental feature, and may contain errors. View original article
📖

re:Invent 2025: How Slack Achieved 90% Cost Reduction and Improved User Satisfaction with Amazon Bedrock and Nova

に公開
2025/12/09

Introduction

By transcribing various overseas lectures into Japanese articles, we aim to make hidden, high-quality information more accessible. This project takes on a new presentation!

For transcriptions of re:Invent 2025 articles, please check this Spreadsheet for aggregated information.

📖 re:Invent 2025: AWS re:Invent 2025 - Delighting Slack users safely and quickly with Amazon Nova and Bedrock (AIM384)

This video introduces Slack's practical efforts in delivering Slack AI to millions of users by leveraging Amazon Bedrock and Nova. By migrating from SageMaker to Bedrock, they enabled the operation of over 15 LLMs in production, achieving a cost reduction of over 90% (more than $20 million) despite expanding their customer base. While maintaining FedRAMP Moderate compliance, they significantly improved utilization efficiency by shifting from Provisioned Throughput to On Demand and leveraging cross-region inference. They also developed a proprietary experimentation framework using LLM-as-a-judge and Amazon Bedrock Guardrails to build a mechanism for objectively measuring quality. For their search query understanding feature, switching to Nova Lite reduced P50 latency by 46% and costs by 70% while maintaining quality. Ultimately, they report a 90% reduction in cost per monthly active user and a 15-30% improvement in user satisfaction while scaling operations fivefold.

https://www.youtube.com/watch?v=6vBNKmSc614
Note: This article was automatically generated while maintaining the content of the original lecture as much as possible. Please be aware that there may be typos or incorrect information.

Main Content

Thumbnail 0

The Challenge of Bringing Generative AI Applications to Production: A Case Study of Slack and Amazon Bedrock

Hello, everyone, hello. I hope you are enjoying re:Invent 2025 this week. Thank you for coming today. Before we actually start, I would like to ask for a quick show of hands. How many of you have already built a generative AI application and are actually helping your users, trying to make their lives more effective or more productive? Go ahead, please raise your hands, don't be shy. That's a pretty good number. I think this is truly a testament to how wonderfully the industry has transformed over the past few years.

Think about it; it all started with initial research and development, which led to the first set of frontier models. They showed us, and the whole world, the potential and technological possibilities of users interacting with systems in their own natural language and responding creatively. Because of that, many tools have been born today. Not only do they search for information when you cannot answer questions yourself, but they also generate new rich media content based on provided descriptions or captions. Furthermore, when you want to share a new concept as a prototype with people, or when you want to fill an application gap or make small fixes, they can generate entirely new code. The industry has evolved that rapidly in the last few years.

Thumbnail 140

However, I think all of you who said you deployed something have also likely seen all the challenges associated with taking such applications to production-level quality. These are the challenges to bring it to a level of quality that you can feel comfortable providing to users over time. So, my name is Gene Ting. I am a Principal Solutions Architect at Amazon Web Services. And today, I am joined by Austin Bell and Shaurya Kethireddy from Slack, and they will talk about their journey in overcoming those challenges and delighting millions of Slack users around the world using Amazon Bedrock and Nova.

Thumbnail 150

Thumbnail 160

Thumbnail 170

Basic Principles of Designing Production-Ready Generative AI Applications and Leveraging Amazon Bedrock

Let's get started. To make generative AI applications production-ready, at a high level, we follow steps similar to what we have thought about for more traditional applications in the past. You need to think carefully about who you are building for and what their personas are. That defines user requirements and the functionality you want to design for the application. Once those decisions are made, you need to think about reliability, operational excellence, and cost management, especially as the application grows over time. You always want to protect your application from malicious damage and ensure that user sessions and their data can be protected.

Thumbnail 190

Thumbnail 200

Thumbnail 220

Thumbnail 230

However, a new and interesting nuance for generative AI applications is how to ensure the system can actually respond to users in a safe way and how to ensure the system only processes the types of workflows or requests you have designed. Therefore, there are several aspects to consider when defining user requirements and designing generative AI applications. Sometimes you need to think about whether to build features through static LLM workflows. Sometimes you will want to see how you can apply more dynamic, agent-like workflows. But what these have in common is that they all come with different levels of complexity based on the task you are trying to achieve.

You might want to respond instantly or as close to real-time as possible, such as answering search phrases and returning rich summaries of search results. On the other hand, you might want to take large amounts of conversation threads or text and summarize them on a daily basis. These are types of workflows that are less sensitive to latency. But the important thing is that, just like with traditional applications, you always need to be aware of and conscious of the latency sensitivity for the user based on the type of use case under consideration at that time.

Thumbnail 290

This is where Amazon Bedrock comes in. We talked about different levels of complexity for different tasks, and within Amazon Bedrock, there are actually many models you can choose from to accomplish those tasks.

Having these choices and making those selections helps you identify the best fit for the type of use case you are trying to achieve, and also helps you manage costs over time. Similarly, when deciding between real-time or asynchronous responses, Bedrock gives you choices. You can choose Provisioned Throughput, priority tiers, or the standard tier for answering the majority of questions. If you don't need to answer questions in a hurry, you can also defer some inference to batch inference or the flex tier. Making these choices also helps with cost management.

Thumbnail 350

Thumbnail 370

As applications mature and improve over time, you will want to have a strategy for how to upgrade and replace those models, whether that is upgrading to a new version of the same model or upgrading to a different model that you think can accomplish the same task. You also need a way to iteratively optimize model responses over time. This is where Amazon Bedrock prompt management and prompt tuning come in. By using "LLM as a judge" and having a cataloged set of prompts, you can actually apply them to different models and compare them over time. Using prompt tuning and the prompt optimizer, you can get suggestions on how to essentially send different prompts and contexts to verify results.

Thumbnail 400

Thumbnail 410

Thumbnail 420

Monitoring Strategy for Large-Scale Operations, Security Design, and Ensuring Safety with Bedrock Guardrails

After all those fundamental application decisions are made, you want to think about how you operate and essentially run at scale. It starts with the decision of which region to choose to host your application, based on user requirements or regulatory needs. This defines where you start your interaction with Bedrock. You want to establish a monitoring strategy immediately. From there, you will monitor "time to first token" for requests. You can also use Amazon CloudWatch metrics to actually see end-to-end latency, error frequency, and throttling frequency. For example, using CloudWatch logs or your own logging system, you want to see how agents are responding or what the LLM is doing on the backend.

Thumbnail 450

Thumbnail 470

You want to be able to tune for performance. Therefore, tuning for performance always involves some kind of caching strategy. For example, by using explicit prompt caching on Bedrock, you can save time and costs when converting common, frequently used inputs into input tokens. And I talked about how Bedrock has the model flexibility to help with cost management. This is also very useful in terms of resiliency. In other words, you need to establish a deeper model strategy at all times so that if for some reason a model starts acting up or if there are availability concerns regarding that model, your application can be covered.

Thumbnail 500

Thumbnail 520

As your application really grows over time, you can use other regions that have the capacity hosting those models. With Bedrock's cross-region inference, you can instruct your request to be sent to another region hosting that model, whether it's on the same continent or on the other side of the globe. Security by design is always an essential part of making an application production-ready. Starting with the basic blueprint I talked about earlier, the first thing that will put you at ease is that Bedrock is designed such that model providers cannot access the environment where they actually host the models. And the converse is also true; by design, it does not share the inputs and outputs that are fed into and generated by the models you use on Bedrock.

Thumbnail 560

Thumbnail 570

Thumbnail 580

Thumbnail 590

Apply IAM policies to ensure that only specific parts of your organization and environment, and specific users, can actually access the models you have already begun utilizing on Bedrock. Use service control policies to confirm that only specific models approved by your organization are actually available within your infrastructure. There are also additional controls via VPC endpoints, requiring that you traverse specific paths to reach Amazon Bedrock through your infrastructure. And remember when I talked about using cross-region inference to reach capacity wherever it is available on Bedrock? You can also restrict the behavior of that cross-region inference to only specific geographical locations you choose, based on specific regulatory requirements or data sovereignty needs.

Thumbnail 630

So, let's return to this nuance regarding ensuring your application responds in a secure manner and only processes what you have specifically designed it to handle. We have been talking about basic request-response workflows. Looking at the user input side, you can apply Amazon Bedrock Guardrails to filter for specific keywords. You can basically reject dangerous content. You can actually choose other types of custom topics you don't want your system to process. And if any of these policies are triggered, you can have Bedrock respond with a custom moderated response.

Thumbnail 680

At the other end of the spectrum, after the initial input has been scrutinized and fed into the model, you can have Bedrock Guardrails apply the same type of policies again, adding additional automated inference checks for accuracy before sending the response back to the user. Now that I've talked about the basic blueprint for moving from an idea to production, let's look at how one of our largest customers, Slack, applies some of these principles and rigor to effectively, securely, and efficiently provide Slack AI. Austin, please go ahead.

Thumbnail 720

Slack AI Overview: Efforts to Integrate Generative AI Across the Product

Great, Gene, thank you for the introduction. And thank you all for joining us. I'm going to talk to you about how we at Slack provide Slack AI features effectively, securely, and efficiently. Specifically, our goal is to weave generative AI throughout the entire Slack product. It's quite easy to reach 80% of generative AI features. But what truly becomes a differentiator, setting us apart from others, is that remaining 20%—that is, increasing cost efficiency or achieving higher quality. In this presentation, I will talk about our journey in how we at Slack managed to achieve scale and meet our quality requirements.

Thumbnail 750

Specifically, I'll talk about three different areas. First, I'll explain how we developed an infrastructure layer that allowed us to meet our scaling needs while ensuring high security and meeting all compliance requirements. Next, I'll talk about how we developed an internal experimentation framework that enabled us to objectively measure the output quality of our generative AI and gave us the confidence that we were actually improving quality. Finally, I'll talk about how these two sections combine to allow us to more seamlessly integrate generative AI throughout our product.

Thumbnail 790

Thumbnail 820

But first, a little bit about us. My name is Austin Bell, and I'm the Director of Machine Learning, Search, and AI at Slack. And I've come here with a colleague. Hello everyone, my name is Shaurya, and I'm an engineer working on the infrastructure and platform team at Slack. Now, to set the stage—I'll assume you have some level of knowledge about what Slack is—but for those who don't know about our Slack AI offering, we support over a dozen generative AI features and tools that span a wide spectrum of complexity and use cases.

For example, we provide AI summary features across various product surface areas. We have an internal QA system where you can ask questions related to data stored within Slack and find information. We also have a feature that creates a daily digest, allowing you to get up to speed on everything going on very quickly. These are just a few examples, and we are regularly continuing to add even more features. Now, for the first topic, I'll hand it over to my colleague Shaurya. He will explain how we built the infrastructure layer at Slack.

Departure from SageMaker: Initial Infrastructure Challenges and Scaling Limits

Austin, thank you for the introduction. Hello everyone, I'll talk about how we are scaling Slack AI for the millions of daily active users we have. When we generally talk about generative AI, we usually speak about speed—that is, how fast tokens are generated, how quickly the product moves, and how fast the market is moving. However, here at Slack, we also add another element: trust. We ensure that the services we use have the necessary reliability to protect our customers' data and that we live up to the trust placed in us by our customers so that they can work within our platform with peace of mind.

Thumbnail 910

One of the phrases we usually cherish is that the true test of scalable infrastructure is not just how fast it grows, but how well it protects what is important as it grows. As you can imagine, many people save all their work content within Slack. Whatever features we add, we want to ensure that users can continue their work on Slack without us adding friction.

Thumbnail 940

Thumbnail 990

Before we dive into the technical aspects of how we are scaling Slack AI, I'd like to provide some background on the product promise for our features. The first pillar is trust. When we started scaling out Slack AI, the question we received most often from customers was whether their data was being used to train generative AI models. The answer is no. We do not train generative models on our customers' data. We do not log our customers' data. We also allow Slack workspace administrators to opt out of features. This means that administrators can opt in or opt out of specific features. Additionally, the data retention period is zero, and data sent to LLM providers is not shared. As Gene mentioned, inputs and outputs are not shared.

The second pillar is security. Because we operate in the FedRAMP Moderate compliance space, we verify that the services we use meet the strict standards set by federal guidelines. We also ensure that the services we use remain within our trust boundaries, as a lot of data is moving. We also have technical access controls in place. For example, if there is a message in a private channel, we want to ensure that people not participating in that channel cannot access it. We ensure the security of messages and shared answers.

Thumbnail 1030

The third pillar is reliability. Having trust and security is not enough. The feature must also be highly available. This allows our customers to actually use it in their daily processes. Furthermore, contextual relevance is also necessary. That is, we want to ensure that when customers send a request, they get an answer relevant to what they are actually looking for. And transparency is also required. That is, when they get an answer, we want to include citations whenever possible, so that customers can trace back to the original messages that the LLM may have used to generate that specific answer.

Thumbnail 1080

Having all these pillars might be very easy if you have 10 users, but let's look at how Slack is currently operating. Every week, we process over 1 billion to 5 billion messages, 100 million to 500 million files, and 1 billion to 5 billion searches. As you can see, Slack operates at massive scale. You shouldn't just connect AI features to a public API and hope for the best. You need to ensure that the services you use are properly tested behind the scenes, and that they can seamlessly handle millions and billions of requests while maintaining your security and trust posture.

Thumbnail 1120

Now that I've explained the product context and Slack's scale, let's start with our journey in the past, from mid-2023 to mid-2024, on how we provided Slack AI. Back then, we had limited LLM choices. It was costly, less flexible, and we were operating in a provisioned throughput environment. From there, we have now evolved into a future that realizes high flexibility, high utilization efficiency, and improved reliability.

Thumbnail 1160

When we first started in mid-2023, we looked for services provided by Amazon that were within the FedRAMP Moderate compliance scope and adhered to our security and trust posture, so we started with SageMaker. As background, Slack primarily operates within the US East One region, and we have a VPC that includes our Slack instances. From there, requests pass through VPC endpoints. Slack AI requests can essentially pass through AWS's internal network, and those requests are sent to SageMaker endpoints. This is basically a wrapper for the model. When we first started, we only had one model, and this model was provided via provisioned throughput. Also, inside the VPC...

you might see a small box called a concurrency checker. This is what we use to maintain the load. During peak load, we were able to reduce lower-priority requests. For example, internally we have three priorities: the highest priority for the most latency-sensitive ones, the medium priority for those that can be completed within a 5-10 minute SLA, and the third-tier priority for things like batch jobs that run at night. We had high concurrency, and when the cluster was almost reaching peak utilization, instead of scaling up—which at the time could take over an hour—we could very quickly reduce the load.

Thumbnail 1250

This worked well during the early days of Slack AI while we were still scaling up, but we noticed that Slack AI was also being added to our customer base exponentially. So, we noticed some issues. The first was peaky traffic. Two different types of requests were coming in: time-sensitive ones and batch workloads. These are fairly predictable in daily scenarios, but the throughput consistency was different. Also, it was difficult to obtain GPUs at that time. Because of the GPU shortage, it was taking weeks to get the GPUs needed to scale up for the new customer base added to Slack AI. Due to the GPU shortage, we couldn't easily scale up or down. We had to maintain GPU instances with on-demand capacity reservations so that we could keep them during off-peak times.

The impact of these two issues was that we were over-provisioned for most of the day. This is because we had to keep the GPUs running. Because of this, we were scaling our infrastructure to support peak traffic, but this wasn't ideal because it essentially caused cost inefficiencies. We were paying for instances and GPU time that we were not actively using to serve customer requests. Because of this fixed cost, we couldn't diversify our LLMs. We had one, or at most two, which also slowed us down when experimenting with new models or adding new features.

These two issues meant that we were over-provisioned for the majority of the day, as we had to keep those GPUs running. We were scaling our infrastructure to support peak traffic, but this wasn't ideal because it led to cost inefficiencies; we were paying for instances and GPU time that weren't being actively used to serve customer requests. This fixed cost also prevented us from diversifying our LLMs. We only had one, or at most two, which slowed us down when experimenting with new models or adding new features.

Transitioning to Amazon Bedrock: The Journey to FedRAMP Compliance and Over 90% Cost Reduction

With these issues in mind, we had a vision of where we wanted to go. We wanted to find a new service that was managed but also offered model diversity. That's when we found Bedrock. In mid-2024, Bedrock became FedRAMP Moderate compliant, allowing us to seriously consider the service. We could now process all these requests within our trust boundary, and because Bedrock promises that inputs and outputs are not shared with providers, they cannot use them to train their models. Bedrock has an entire collection of state-of-the-art models, which allowed our product engineers to add more features. The speed at which models are added to the Bedrock model registry is also fast, with these models becoming available within the Bedrock platform within a day of their public release.

Thumbnail 1410

Thumbnail 1450

After deciding on Bedrock as our migration destination, we followed several migration steps to ensure customer trust and a highly reliable migration without downtime. The first aspect was understanding the Bedrock infrastructure, as the two differences were provisioned throughput and on-demand. In the SageMaker world, we were using the provisioned throughput infrastructure type, so to facilitate the migration, we maintained the same in Bedrock and decided to handle the transition to on-demand in the next phase. The second part involved conducting internal load testing and performing some compute calculations. This meant we were working to scientifically figure out what the equivalent of Bedrock Model Units (BMUs) was for the SageMaker compute we already had. This allowed us to obtain equivalent compute without repeating guesses and checks.

Thumbnail 1470

Here is one example of our load testing. We were able to run Claude Instant and Claude Haiku on both SageMaker and Bedrock. As you can see, for Claude Instant, it was a 1:1 mapping. That is, when we ran SageMaker on P4d instances, it was equivalent to one Bedrock Model Unit. When we ran Claude Haiku instances on P5 instances, two P5 instances were equivalent to one Bedrock Model Unit. By using these ratios, we were able to acquire equivalent compute, and we collaborated with the Amazon Bedrock service team to have it provisioned to our account.

Thumbnail 1510

Once that compute was provisioned, we were able to start running shadow traffic. This meant that every time a request was sent to SageMaker, we also sent a duplicate request to Bedrock. This allowed us to understand the internals of the service. It also helped us build monitoring dashboards. We were able to validate latency, time-to-first-token, and other metrics we had set up.

Thumbnail 1540

Thumbnail 1560

After shadow traffic ran at 100% for two weeks, we started the full cutover process. This was also done in phases, progressing from 1% to 5%, 10%, and finally 100%. This basically meant that instead of shadow mode, we started serving responses sent from the Bedrock service instead of the SageMaker service. Moving to Bedrock helped us reduce costs and allowed us to experiment more, but we also noticed some gaps we needed to work on to make the service more efficient.

The first challenge was that we couldn't scale Provisioned Throughput. It remained a static compute cluster, which kept us facing issues regarding cost efficiency. On the platform side, we realized it would be more useful to enable the addition of backup models. When regressions occurred in Bedrock models or we noticed some features weren't performing as expected, adding backup models allowed us to essentially reroute without making code changes when incidents occurred.

We also added an emergency stop feature for specific features and models. This works in conjunction with backup models. When the emergency stop is turned on, the entire feature could potentially be turned off, or if a model is turned off on our end, all requests are rerouted to the backup model. By using Amazon Bedrock, we were also able to use the various tools models offer, such as tools, prompt caching, guardrails, and other model-specific capabilities. We then extended our internal AI platform to give developers access to these.

Thumbnail 1650

While transitioning away from Provisioned Throughput, we realized we couldn't scale, so we decided to move into the On Demand world at this point. This is a different infrastructure type, not bare instances, but based on quotas. This is done based on tokens per minute, input tokens, and requests per minute. Also, because we had a lot of metadata on our end during the Provisioned Throughput era, we were able to essentially calculate the requests and tokens per minute from that metadata and pass those compute requests to Amazon Bedrock to have those quotas delivered to our account.

Similar to the SageMaker world, we had Slack apps in the US East One region. The VPC holds the Slack instances. Requests are sent to VPC endpoints, which perform all internal routing and send them to the Bedrock service. Then, based on the model ID we added to the request, it points to a specific model. One difference from the SageMaker architecture diagram is that instead of a concurrency checker, we had a requests-per-minute and tokens-per-minute checker on our end. This allowed us to essentially control different features and prevent any specific feature from monopolizing the entire Bedrock cluster.

Thumbnail 1740

Another benefit gained from Bedrock On Demand is that we can now use US cross-region inference profiles. Since we are a FedRAMP Moderate shop, we need to stay within US boundaries, and Bedrock provides that capability. By also deploying in US West Two, we now have two regions to choose from, allowing us to get compute delivered at a faster pace.

Thumbnail 1770

Through all of this, we achieved many results by the end of the migration. When we first started, there was only one LLM to choose from, but by the end of the migration and in our current state, we are experimenting with and providing over 15 LLMs in production. Reliability also improved. With increased LLM flexibility, we can now fall back to specific models, and we can quickly switch models during incidents or experiment to find the best model in terms of quality and cost. Furthermore, the biggest numerical achievement was utilization efficiency.

Throughout the migration, Slack AI usage was increasing exponentially as customer onboarding progressed. However, despite the growth in the customer base, we achieved a cost reduction of over 90%, amounting to a reduction of over $20 million. In summary, we started with the phrase that it's not about how fast we can scale infrastructure, but how well we can protect data while scaling. By utilizing services provided by Amazon, we were able to work with our internal cloud team, platform team, and the AWS side to deliver all AI features at scale.

Thumbnail 1870

Thumbnail 1890

Measuring Generative AI Output Quality: Developing an Internal Experimentation Framework and Evaluation System

Thank you. Now, I will pass the baton to Austin to talk about how we deliver with high quality. Thank you, Shaurya. Let's shift gears a bit and talk about how we developed an internal experimentation framework to more objectively measure the quality of generative AI output, and how we are actually working to improve this quality. Now, this is a quote from a Slack engineer, and you might have been in a situation like this yourselves: you demo a generative AI feature, it looks great, and you're really excited. The time comes to put it into production, but you need to improve some edge cases for quality. You change the prompts, you change the pipeline, and you try to release it. But a few days later, you start to notice that quality is actually regressing in certain areas. This becomes a cycle, a whack-a-mole situation. You try to fix one problem and another appears, and after a few weeks, you haven't actually seen any quality improvement.

Thumbnail 1930

So, why does this happen? It's because evaluating generated output is difficult. We are in the midst of a paradigm shift from classical machine learning. In classical machine learning, we could leverage engagement metrics in an online setting to evaluate quality, or utilize things like precision and recall in an offline setting. Generative AI output is far more subjective. What I consider good output might be vastly different from what you consider good output. It might also depend on the surface area of the product where you actually want to display this generative AI output. The question then becomes: how do you actually measure it in a way that meets your goals and allows for continuous improvement?

Thumbnail 1970

Now, a key belief we have at Slack is that you can only improve what you have the ability to measure. So, we started on a journey to define what we want to measure. We defined it with two pillars: quality, meaning the answer actually provides what is requested and is accurate; and safety, meaning we are fostering a desirable and appropriate environment at Slack, and protecting data as safely as possible. We further broke each of these down.

Thumbnail 2010

First, regarding objective measurement. We consider this more deterministic output, sort of like table stakes. Being able to render correctly; parsing JSON or XML output correctly. Are the IDs formatted correctly? Are links formatted correctly to navigate Slack appropriately? If these are missing, users will notice, and it doesn't really matter what the generative AI content is. Because users cannot overlook these issues. On a more difficult side, there is subjective measurement. This is things like factual accuracy: based on grounded context, is it actually telling the truth? Answer relevance: is it answering the question the user actually asked? Attribution accuracy, which is a big deal for Slack: are we attributing correct content to the correct user or correct file?

Thumbnail 2070

On the safety side, we also divided this into two separate categories: harm and security. Harm refers to things we measure regarding harmfulness. Are we capturing bias? Are we ensuring a good environment and that the LLM is responding in a way that aligns with our values? Second is security: are we protecting against prompt injection attacks as much as possible? Are we preventing unintended or intentional search poisoning that we don't want to happen?

Thumbnail 2110

Now, that's step one, defining what we want to measure. The next part is generating evaluators that give us the ability to actually measure this. This was a journey from the start. We wanted to achieve a few things. We wanted to ensure product engineers had the ability to very quickly manually review output across a small dataset based on prompt changes or pipeline changes. We also wanted to introduce various automated programmatic metrics that could capture objective measurement in the quality pillar in an automated way. This ensures that we aren't seeing regressions in formatting or rendering capabilities, which we consider table stakes at Slack.

Thumbnail 2160

This is adding complexity to our journey, and this is where Slack stands today. We are starting to tackle more complex quality definitions. We are leveraging LLM-based quality metrics to be able to measure factual accuracy and answer relevance. We are leveraging guardrails to capture safety and harm issues. The overall goal here is that by combining these automated programmatic metrics, LLM judges, and guardrails, we can start evaluating generative AI quality on far larger datasets that are more representative of the production environment. This gives us the ability to run far larger experiments and tests.

Now, where do we go from here? The goal is to start developing CI/CD for generative AI. By giving ourselves the ability to define verified output and automate a set of tests, we can catch regressions and improve quality in a faster and more efficient way. Let me pause for a moment and explain a bit more about what I just mentioned. At Slack, we are running dozens of different task-specific LLM judges. We work with each product team developing AI features at Slack to create a rubric for what the definition of quality actually is for that specific feature. This changes for every feature. By giving engineers the ability to simply define a rubric without actually having to write code to deploy these, they can roll them out very quickly.

Thumbnail 2270

We also leverage Amazon Bedrock Guardrails. This gives us the ability to very easily measure toxicity, harmfulness, and prompt injection in both the input and output of the LLM. Here, I have highlighted two steps. First, we defined what we want to measure, and second, we defined how to measure it. However, what really boosts engineer productivity is developing workflows that allow them to leverage these features within their daily development experience. For those of you with a background in machine learning, this might sound somewhat familiar.

First, we conduct a series of offline experiments. We start with what we call a "golden set." These are validated outputs from internal data that allow users to apply prompt changes to the LLM and run manual reviews on a small dataset of 10 to 20. Once confident in that, we can move to running experiments on what we call "validation sets." These are far larger and more representative of production data. Thus, by using a combination of these automated programmatic metrics and quality metrics, we can capture large-scale regressions and confirm that we are actually making the intended quality improvements.

Thumbnail 2360

What is important here is that we want to provide the fastest feedback loop possible at each step. This gives engineers the ability to fail as quickly as possible, go back to the drawing board, and release features more rapidly. The goal here is to eventually automate this as much as possible, but leveraging a "human in the loop" is key. Since we aren't perfect in the development of these evaluators, giving engineers the ability to very quickly confirm the impact of changes and LLM responses across data allows us to move significantly faster.

Thumbnail 2390

Now, after verifying many of these in an offline environment, we move to online evaluation. Here we start running A/B tests and also integrate all the evaluators mentioned earlier into the A/B tests. This allows us to actually measure using both these quality metrics and user feedback metrics.

Before we actually decide to roll out to production, we make quality changes and gain confidence that we aren't causing regressions in specific areas before too many users actually see them.

Thumbnail 2430

Going back a bit, let me explain some of what I just talked about in more detail. We typically handle three different types of datasets. Golden sets are manually scrutinized, very small, and around 10 to 50 samples. This allows engineers to manually review the output. Validation sets are typically in the 100 to 1,000 sample range. Here, automated quality evaluation and programmatic metrics really show their strength. As you can imagine, reviewing 1,000 samples, for example, is nearly impossible for a human. And for A/B testing, we typically run experiments on 1% to 25% of Slack AI queries for that specific feature.

Thumbnail 2480

These are great, but are they actually valuable? I have selected three examples here showing a few different areas we are working on at Slack. In terms of prompt engineering, we recently changed how we serialize content sent to the LLM, and as a result, factual accuracy and user attribution accuracy improved by 5% and 6%, respectively. When a new version of a model or a new LLM is released, we frequently conduct model upgrades. Our recent upgrade increased user satisfaction by 11% and improved key quality metrics by 3% to 5%. In fact, there have been times we decided not to roll out an upgrade because the new version caused a regression, so we now run this flow in a more automated way for every LLM upgrade we perform. Beyond quality, cost management is a key area. We sometimes want to reduce costs or improve latency while maintaining quality. In a recent change, we realized a 60% cost reduction by migrating to an equivalent quality LLM.

Thumbnail 2560

Applying to Search Query Understanding: Achieving a 46% Latency Reduction and 70% Cost Reduction with Nova Lite

I have talked about two different things separately so far. First, how we built our infrastructure layer, which gave us the ability to leverage LLMs more efficiently, choose between different LLMs, and switch between them seamlessly. Second, how we developed this internal experimentation framework that allows us to actually measure AI output quality and have confidence that the changes we want to implement are indeed being implemented. In this section, I will explain how we leverage both of these to integrate generative AI more seamlessly across our application, maintaining cost efficiency while sustaining the quality and scale users expect.

Thumbnail 2610

Thumbnail 2630

Thumbnail 2640

To do that, we will look at a few use cases, but before that, I want to emphasize that generative AI has a spectrum of complexity. On the low-complexity side, there are things that were probably done with traditional machine learning models in the past: classification, conversion from unstructured to structured data, data parsing, and so on. Moving to medium complexity, we have summarization, very basic image generation, and content editing features. And on the high-complexity side, things the media loves to cover: agentic workflows, video generation, tool use, and such big things. Now, at Slack, we have numerous generative AI applications, and they span the entire spectrum from low to high complexity. We don't want to use state-of-the-art frontier LLMs for every use case, but answering the question "what is exactly the right LLM to achieve our goals without sacrificing quality for the user?" is a difficult problem.

Thumbnail 2680

Thumbnail 2690

Now, let's look at a specific use case that tackles this low-complexity area, which we call "search query understanding" at Slack. But before that, let's look at the search area of Shaurya's scale slide again. Now, we perform a massive number of searches at Slack. The majority of them are simply finding people or channels. A very small portion of those are very complex searches, like asking questions of data or performing searches across large amounts of messages. Even if it's a small portion of this number, if they aren't executed efficiently, it could lead to high costs.

Thumbnail 2730

So, what is search query understanding? Let's say a user comes into Slack and adds a search term like: "Can you find the FY25 sales deck that John Doe recently sent me?" Now, you might not want to send this directly to the search cluster. It might not be helpful, but there is a lot of information hidden within it, which can make the search more targeted and increase the likelihood of finding the files or messages you are actually looking for. So, we run it through a large language model pipeline so we can generate JSON that includes filters and other potentially relevant information.

Let's break this down. We know the query we actually want to search for is not the entire text. It's probably just "FY25 sales PowerPoint deck." This is just one query, but often we execute multiple queries in parallel, so there might be multiple versions of this search term. They used the word "recently," so we can narrow this down to a specific time range like the last 3 months, indicating that it increases the chance of finding what they are looking for. They said they are looking for a "sales deck," so it is highly likely that this includes file types like presentations. So, can we limit it only to threads or conversations containing presentations? And they mentioned that "John Doe" sent it to you, so let's look only at conversations or threads he is involved in.

Thumbnail 2820

Thumbnail 2830

Thumbnail 2840

Thumbnail 2850

So, what does this actually look like in the search pipeline? A user sends a search. We have a machine learning model that can very quickly determine whether this is an information search or a navigation search. In the case of a navigation search, it goes through a very specific search and machine learning ranking pipeline. In the case of an information search, it goes through the query understanding pipeline to try and target the actual search more accurately, and then proceeds to the remaining search retrieval machine learning, and finally, the LLM response.

Thumbnail 2860

Now, where do we start from here? We had a problem. The existing LLM we were using in this case was working. Quality goals were met, but it exceeded our search latency budget and was very high-cost considering the scale we needed to operate at. So, what we wanted to do in this specific use case was to switch to or leverage prompt engineering that could reduce both latency and cost while maintaining quality. Now, the benefit was that we had both these infrastructure layer changes and this experiment framework. So, we wanted to see how we could leverage both of these to choose an LLM suitable for this specific problem. At the same time, while having confidence that we are meeting all three of these goals.

Thumbnail 2920

Now, let me explain how this actually looks in our internal experiment framework. This shows a simple summary of what it looks like when running offline experiments. In this case, we compare Nova Lite with the original LLM we were using to determine whether this actually leads to the improvements we want. Of particular importance are latency and key quality metrics. This is a rolled-up summary. In reality, there are dozens of different evaluation indices, which are aggregated into these summaries. This is being run on a very small sample set, but as you can see, we saw a significant reduction in latency and improvement in quality in this small sample. Through this offline test, we were able to gain confidence that we were actually achieving the improvements we wanted.

Thumbnail 2990

So, we moved to online evaluation. We started running A/B tests, and this is part of the picture when we tackled the key metrics we were worried about. Specifically, we wanted to check latency and quality metrics from the perspective of automated quality metrics and user satisfaction. As a result, latency was significantly improved, and there were no significant changes in user satisfaction and both automated quality metrics.

Thumbnail 3000

So, what does this mean? Regarding query understanding, we finally switched to Nova Lite. As a result, after a series of prompt changes, we were able to switch without any quality degradation visible to the user, while simultaneously reducing P50 latency by 46% and reducing the service provision cost for this specific feature by 70%.

Thumbnail 3030

Results of an Integrated Approach: Achieving Both a 90% Reduction in Cost Per Monthly Active User and Quality Improvement

So, what were we able to achieve here? We have these two independent sections.

Thumbnail 3060

Thumbnail 3080

With these infrastructure improvements, we gained the ability to choose appropriate LLMs for the future and operate the infrastructure at a much more efficient utilization rate. We also have quality and evaluation systems, which give us the ability to more objectively benchmark and evaluate generative AI output quality. And this allows us to choose the right LLM for the job, maximize the time actually spent on prompt engineering while minimizing the engineering time spent on this, operate the infrastructure much more efficiently, and communicate the improvements we are making to other teams and leadership with the confidence that those improvements are real.

Thumbnail 3110

We are doing this across dozens of different features. What were the results of all these changes over the past year? We were able to reduce the cost of providing Slack AI per monthly active user by 90%. At the same time, we scaled our operational size by about 5x. While achieving all of this, we were able to increase user satisfaction and user feedback across key Slack AI features by 15% to 30%.

Fantastic. Thank you for watching today. We are happy to answer any questions you may have.


  • This article is automatically generated using Amazon Bedrock, maintaining the information from the original video as much as possible.

Discussion