iTranslated by AI

The content below is an AI-generated translation. This is an experimental feature, and may contain errors. View original article
📖

re:Invent 2025: Multimodal Capabilities and Performance Evaluation of Amazon Nova 2 Omni

に公開
2025/12/08

Introduction

By transcribing various overseas lectures into Japanese articles, we aim to make hidden, high-quality information more accessible. In this project, we feature the following presentation!

For the transcript articles of re:Invent 2025, please check the information summarized in this Spreadsheet.

📖 re:Invent 2025: AWS re:Invent 2025 - [NEW LAUNCH] Amazon Nova 2 Omni: A new frontier in multimodal AI (AIM3324)

This video introduces the latest features of multimodal AI, focusing on Amazon Nova 2 Omni. Nova 2 Omni is the industry's first hybrid reasoning model capable of understanding text, images, video, and audio in an integrated manner, and generating both text and images. It supports a context window of up to 1 million tokens and is compatible with over 200 languages. It demonstrates high performance comparable to Gemini 2.5 Flash and GPT-4o Mini in document understanding, speech transcription, and video analysis, earning 2nd place overall on the MMAU leaderboard. Its image generation capability has been significantly improved compared to Nova Canvas, particularly excelling in Japanese text rendering and complex editing operations. A case study from Dentsu Digital reports that it achieved a correlation value of 0.88 in video creative prediction accuracy using Nova Omni, and successfully developed 7 applications in just 7 days.

https://www.youtube.com/watch?v=-nMiqOgQbHc
Please note that this article was automatically generated while maintaining the content of the existing lecture as much as possible; therefore, there is a possibility of typos or incorrect information.

Main Content

Thumbnail 0

Opening of the Amazon Nova 2 Omni Session and the Nova Family Overview

Hello everyone, and thank you for joining us this afternoon. Today's session is about Amazon Nova 2 Omni, the new frontier of multimodal AI. Joining me today are two co-presenters: Ashwin Swaminathan, Director of Applied Science who is involved in the development of Amazon Nova models, and Mr. Yamamoto, Chief AI Officer at Dentsu Digital.

Thumbnail 40

Our agenda for today starts with an introduction to the entire Amazon Nova family of models. Then, we will touch upon the Amazon Nova 2 family announced by Matt Garman in yesterday's keynote. We will dive deep into Amazon Nova, specifically the Omni model, which is optimized for multimodal workflows—multimodal understanding and multimodal generation. We will also provide several examples and demos to give you ideas of what you can build for your business. Furthermore, we will compare the performance of these models with the best models in the industry. Finally, Mr. Yamamoto will talk about how Dentsu Digital is transforming their business and supporting customers using Nova Omni.

Thumbnail 90

We launched the Amazon Nova foundation model family at last year's re:Invent. In fact, I was right here at this venue. We launched a series of models. The primary Nova models were launched in two categories: one we call the Nova understanding models, which take text, images, and video as input to extract metadata, summarize content, or generate text through question answering. These are available in Micro, Lite, and Pro versions. We subsequently launched Premier, the largest model with the best performance in this category. We also launched Nova Canvas, an image generation model, and Nova Reel, a video generation model. Later, we launched a speech-to-speech model for real-time conversational AI applications such as customer support calls; this is Amazon Nova Sonic. Very recently, just last month, we launched the industry's first native multimodal embedding model. This is perfect for semantic search and agentic RAG applications, especially when dealing with large amounts of unstructured data such as documents, images, videos, audio, and text. We are pioneers in this field, providing the industry's first model that delivers the highest performance with much lower latency and cost characteristics, all integrated together.

Thumbnail 190

Thumbnail 200

Thumbnail 210

The Four New Models of the Amazon Nova 2 Family: Features of Lite, Pro, Omni, and Sonic

Nova is already being used by tens of thousands of enterprises and startups, some of which are listed here. Now, let’s touch upon what was launched yesterday. We are introducing four models in the Amazon Nova 2 family. First is Nova 2 Lite, a fast and cost-effective reasoning model for everyday workloads. It is our first reasoning model and a hybrid reasoning model; this means developers can control whether to enable reasoning and what level of reasoning to apply to specific tasks. For example, if a task involves document understanding and you want some level of thought or reasoning to verify if extracted values are correct, you might want to use a certain degree of reasoning. For everyday tasks, you can disable that reasoning, and the model will provide results directly. This uses fewer tokens and generates results faster.

Thumbnail 260

We also launched Nova 2 Pro in preview. It is similar to the Nova 1 family but at a higher tier, providing higher performance with different latency and cost characteristics. This is the most intelligent reasoning model for highly complex tasks. Think of coding tasks, complex agents, or multi-agent scenarios; you would want to use Nova 2 Pro in those cases.

Thumbnail 290

And the third model we will talk about today, also launched in preview, is Nova 2 Omni. This is an integrated model for multimodal reasoning and image generation.

We believe that future models will be fully multimodal. It’s the same way humans interact, right? Through spoken language, visual information, and written text. Every provider will head in that direction. This is the first model available on Bedrock capable of understanding all modalities, including audio and voice. Simultaneously, it can generate high-quality images within the same model. It is also the industry’s first reasoning model that can reason across any modality and generate images in a single model.

Thumbnail 350

Thumbnail 370

As the second generation of Amazon Nova Sonic, we have Nova 2 Sonic. Compared to the first generation, performance is improved, more languages are supported, and conversations feel more natural. In this presentation, we will dive deeper into the Omni model. This is a brief overview of the capabilities of all four of these models.

Thumbnail 380

All four of these models support a large context window of up to 1 million input tokens. They support over 200 languages for input text. Since Pro and Omni can also understand audio, they can process audio and voice in up to 10 languages. One more thing to mention about Sonic is that it now supports more voices naturally. You can select different male or female voices to match your workflow across multiple languages.

Key Features of Nova 2 Omni: Multimodal Understanding, Reasoning, and Integrated Image Generation

As I mentioned, to summarize, Omni can accept any modality as input and generate text and images. It feels more natural because, as I stated earlier, it mirrors how humans interact. By having the model understand all content in the same space before generating text or images, you get higher quality output than a combination of multiple task-specific models. Additionally, you don't need to worry about creating complex pipelines with multiple models and systems; since you can integrate them within a single model, you can reduce construction and maintenance costs and shorten time-to-market.

Thumbnail 470

These are the main characteristics offered by Amazon Nova 2 Omni. As mentioned, this is a hybrid reasoning model where developers can control the level of reasoning or choose whether to enable it at all. The model excels at following instructions, tool invocation, and simple text-based natural language tasks, such as NLP tasks, sentiment analysis, and classification. It is state-of-the-art in multimodal recognition, and the model is highly optimized for all multimodal tasks: input document understanding, input image understanding, video understanding, and audio understanding.

Cross-modal reasoning is a new feature and a point where this model truly excels compared to the best models currently on the market. We will show you some of the figures we have already published in our technical report. Regarding audio understanding, as mentioned, Nova 2 Pro and Omni are the first models on Bedrock to support audio understanding. This means it can transcribe audio, perform multi-speaker diarization—understanding which speaker is talking at which point—support multiple languages, and handle various types of tasks that we will explain in more detail later.

This model supports much higher quality image generation and natural language-based image editing. Supported by the text LLM, it can generate much higher quality text rendered within images. Especially long text in images has always been a challenge, and we faced that in Canvas as well. In this model, we are attempting to fix all the remaining gaps we had in the previous generation of image generation models, and this model should be very compelling for image generation.

Thumbnail 610

Thumbnail 680

Let’s look at the performance. This chart is based on Artificial Analysis, conducted by a third party, not us, and shows their comprehensive Artificial Analysis Index. It measures performance across more than 10 benchmarks and shows how top-tier models function. The number 58 is an aggregated value from many different benchmarks across various categories, such as instruction following, tool invocation, and agentic coding. As you can see, in this tier that competes with models like Gemini 2.5 Flash or GPT-4o Mini, this model is very competitive and ranks very high on the leaderboard. Those on the left are a higher tier. They are Gemini 2.5 Pro, which is similar to the Nova 2 Pro category, so we are not comparing them there. These are figures related to language understanding—knowledge, reasoning, instruction following—and tool invocation, which are important for testing how good the model is and also for tool invocation for agentic applications. These are key elements, and as you can see, Nova 2 Omni is very competitive in all of these figures as well.

Thumbnail 710

Achieving High-Precision OCR and Key Information Extraction in Document Understanding

Now, let's look at some use cases. Particularly in multimodal use cases, this model has been significantly improved from the foundation models of the Nova 1 family, and it is also very competitive compared to other models. The main use case we hear from customers is document understanding. Almost every company has documents, and some are of a very complex nature from a layout perspective. There are handwritten documents, documents with a mix of handwritten and typed languages, and so on; document understanding is a very complex and difficult problem. What this model does is provide much higher accuracy compared to previous generations of models. I will show you figures regarding measurements on public benchmarks to demonstrate where we excel.

Thumbnail 800

Some of the key points we tried to fix with this are OCR—character recognition—and the extraction of important information. If you have PDF reports or receipts, it's about how accurately the model can extract all the information from them. In some cases, you can use built-in tools to perform verification and reveal inconsistencies in the content described. This is an example of OCR; the image on the left can be thought of as a combination of very non-traditional types of documents. There is text, there are images in various areas, and this is one of the basic use cases every company has. In other words, how accurately can we extract information from here, perform OCR, and create this text? You can also instruct the model to generate output in structured formats like JSON or XML, which you can then use when calling various tools in subsequent processing. As you can see on the right, it extracts all information in a structured way. It is not shown as JSON here, but that is something the model can execute accurately.

Thumbnail 860

Thumbnail 900

This is also a document with a very complex layout on the left, but the model is able to extract information from here very accurately. As I mentioned earlier, you can prompt the model to look for inconsistencies in specific areas and perform calculations. That is, the model uses built-in tools, such as a code interpreter capable of simple mathematical calculations, to tell you whether the information presented in the document is correct and where the inconsistencies are.

Thumbnail 910

New Features in Audio Understanding: Transcription, Speaker Diarization, and High Ratings on the MMAU Leaderboard

Now, let's take a look at audio understanding, a new feature of this model. We have AWS Transcribe as one of our solutions for ASR and speech recognition-type use cases. With this model, you can transcribe audio, summarize what was said in the audio file, perform question answering, and execute all of these while calling tools. As I mentioned earlier, it supports speaker diarization for up to three speakers, allowing you to separate which speaker is talking at which point in the audio. This is a very high-performance model.

On the MMAU leaderboard—the Massive Multitask Audio Understanding and Reasoning leaderboard—we are ranked second overall, far ahead of the models provided by Google and OpenAI in this field. This measures accuracy for speech, soundscapes (i.e., non-speech audio), and music. In other words, the model can understand various types of audio input.

Thumbnail 990

Thumbnail 1000

Thumbnail 1020

Let's see how it works. In this case, we have a short snippet from the Q3 earnings call conducted by CEO Andy Jassy. When we provided this input, or in the various types of tasks we asked of the model, we tested to see how the model would behave. In this case, it's just an internal tool where we uploaded this file and started with a very simple transcription. In that instance, we checked to see if the model missed any words. We checked to see if the model mistakenly quoted words that were not spoken in the input audio. It's hard to read here, but we performed the analysis and testing ourselves. The results were very accurate.

Thumbnail 1040

Thumbnail 1060

Then, you can ask follow-up questions such as "What are the main points?" It then provides the main items contained in this audio file in bullet points. You can also ask further follow-up questions like "Summarize the main achievements mentioned by Andy." In other words, you can analyze all kinds of audio files and execute tasks like this with much higher accuracy. This is also one of the summaries. We will share the results when we talk about the summary of multimodal recognition later.

Deepening Image and Video Understanding: Object Detection, Temporal Understanding, and Cross-Modal Reasoning Benchmark Results

Now, to talk about image and video understanding, I would like to hand it over to my co-presenter, Ashwin. Thank you, Rohit. As Rohit mentioned, I will talk a bit about the work we have done as part of the Nova 2 Omni model. I will focus on image and video understanding use cases and also explain how Omni can solve the image generation aspect.

Thumbnail 1120

Looking at the broad categories of image and video understanding use cases, they can be roughly divided into three specific areas. The first is recognition and object detection. This is when you have a scene and want to understand what objects are in that scene and create bounding boxes. The second is question and answering. This is when you want to ask questions about an image and grasp what is happening within it. And the third is temporal understanding. This is for when you transition from images to video and want to capture the temporal aspects of a scene so you can ask questions about a specific video. For example, it is about identifying when a specific event occurred within a video or when something appeared—things generally required in video understanding tasks.

Thumbnail 1170

Let's look at some examples. This is an example of a scene. It is a complex scene. It contains many elements. When you ask a simple question like, "Detect the number of plants, cushions, tables, and televisions in the scene," some things immediately jump out at you as a human looking at the image. For instance, the large potted plant right next to the sofa on the right. And as you look around, you can find multiple smaller plants scattered throughout the scene.

Thumbnail 1200

Let's see what the model generated. The model detected not only the large potted plant behind the scene but also all the smaller plants throughout the space.

This includes the plants on the shelves, the plant on the center table, and the plant next to the television. It also detected every individual cushion and extracted bounding boxes for those cushions. Furthermore, it did the same for the television specified in the prompt.

The second point I want you to focus on is the accuracy of the bounding box positions and how tightly they fit around the objects. These are aspects we optimized when building Amazon Nova 2 Omni. We wanted to ensure that by generating very high-quality output while simultaneously creating very accurate bounding boxes, we could enable various types of applications, such as automation, real-time decision-making, robotics, and others. We wanted to optimize the model to work really well in these scenarios.

Thumbnail 1260

Now, let's look at recognition and question answering. When you look at this scene, you can ask a few questions. How many teams are on the screen? What is this event? What are they playing? Did the runner pass the baton? When you pass an image like this to the Nova Omni model, it determines that this is a relay race featuring three teams: Great Britain, Switzerland, and Belgium. It can also detect the position of the baton, identify who passed it, who hasn't passed it yet, and who finished first in the race.

Thumbnail 1320

This understanding of the entire scene becomes very powerful in supporting various types of applications. This is what we have heard from customers like you regarding how we can support such complex understanding use cases. By doing this, you can reduce the portion of automation handled by humans and improve the overall power of AI for your workloads.

Thumbnail 1340

The third example is about temporal and multimodal attention. If you want to support long-duration event detection to infer between frames and identify specific events in a scene, let's look at an example. The prompt here is to enable identification of all segments in the video for this specific prompt. The prompt is "man standing on a boat," and we are asking the model to generate results in a highly structured output format. I will pause it briefly so you can check the accurate timestamps as we look through this.

Thumbnail 1360

Thumbnail 1370

Thumbnail 1380

Thumbnail 1390

As we look through this, various types of landscapes and scenes are displayed, but between 12 and 14 seconds, you can see a man standing on a boat. As we progress through the video, the man is now walking; he is no longer standing next to the boat, so although the person is still visible, it is not counted as the model's output. The model accurately understands and infers the type of prompt the customer is looking for and provides output in a form that can solve complex tasks.

Thumbnail 1400

Thumbnail 1410

In this case, "man standing next to a boat" includes not only people just walking or standing, but also the relationships between objects and how these objects combine to answer the question the customer is asking. These are the types of complex use cases that the Amazon Nova 2 Omni model can support.

Thumbnail 1420

Regarding performance, as Rohit mentioned, we have evaluated the model against a wide range of benchmarks. We have published a very detailed technical report on the Amazon Science website, so I encourage everyone to visit that site and look at all the benchmarks included in the technical report. Here, I am only pointing out some of the benchmarks we have verified.

In terms of video understanding, we have conducted extensive benchmarking across a wide range of use cases, including the Video MME benchmark. What is interesting about the Video MME benchmark is that it includes not only general video understanding use cases but also cross-modal understanding. You can see how the video performs when video and audio are input together, and what performance improvements result from that. Through the combination of video and audio and cross-modal understanding, you can see that our performance is far superior to other models in the same category.

For document understanding, there is the OCR Bench, an OCR-based benchmark. Here too, the Nova Omni model exceeds the state of the art and provides very good results in a wide range of use cases for this specific benchmark. Regarding audio understanding, as Rohit mentioned, we are ranked second on the leaderboard in popular benchmarks like MMAU.

Overall, you can see that it has excellent performance across a wide range of tasks. Another thing we did while building the Omni model was learn from all the feedback we received from you regarding the Nova 1 model. We received a lot of feedback regarding cases where the model worked well and cases where it didn't work well while you were testing the video understanding use cases.

Thumbnail 1570

In cooperation with CMU, we created a new benchmark called Mavericks. This benchmark is also public. Additionally, based on all the feedback we received from you over the past year, we evaluated the model based on actual customer use cases. We are also positioned in the top class in terms of performance on this Mavericks benchmark. Moreover, this benchmark helps evaluate how video combined with voice or video combined with audio can work together to provide very high-quality results for real-world customer use cases. This is part of our benchmarking efforts, and I would like everyone to check it out.

High-Quality Image Generation and 9 Types of Editing Operations: From Text Rendering to Complex Editing

Now, let's talk a little bit about image generation. What is unique about Omni is that it doesn't just understand all kinds of content like text, images, videos, audio, and speech; it can also generate images. This allows you to realize complex use cases that previously required multiple models. The Nova Omni model can generate high-quality, realistic images of people, text rendering, and spatial understanding. Since it has an excellent understanding backbone for content and images, it can use that backbone to generate higher-quality images and perform higher-quality editing tasks. You can see that in these examples.

First, these are examples of images created with the Nova 2 Omni model. This is the text-to-image part. When you provide a text prompt, it generates various types of images. You can see some of those images in this slide. As Rohit mentioned earlier, we also did a lot of optimization for visual text rendering. We want to ensure that the model generates the correct text output. This is a really difficult problem for most image generation models that exist today. This has always been a challenge, especially for long text within images, and we faced that challenge with Canvas as well. We are trying to fix all the remaining gaps we had with previous generation image generation models, and this model should be very compelling in image generation.

Thumbnail 1660

As for how we evaluate the model, we look at win rates. As for how we calculate win rates, based on all the feedback we received from you, our customers, regarding the Nova Canvas model last year, we curated a dataset of prompts and conducted blind AB tests compared to other competing models on the market. This is a blind AB test. The annotators did not know which image corresponded to Nova Omni and which corresponded to a competitor, and they looked at each image and provided an evaluation of win, tie, or loss. We then aggregated the results across all annotations. The accurate process and procedure for human evaluation are also documented in our technical report.

Next, we calculated the win rate. The win rate is basically the win percentage plus half of the ties. In other words, we are looking at how well our model performs, and this essentially provides a quantification that anything above 50 indicates that our model is on par with or better than other competing models. As you can see in the figure, Nova Omni performs significantly better than Nova Canvas. This means that compared to the previous generation model we launched last year, it is a step-function improvement in terms of performance. It also significantly outperforms models like Flux Kontext Max and Flux Kontext Pro and is on par with GPT-image 1 and Gemini-family models.

Thumbnail 1740

Thumbnail 1760

These are more detailed results looking at specific categories. The specific areas we focused on were improving the rendering of people and the rendering of visual text and scenes. This is also an area we improved when transitioning from Nova Canvas to Nova Omni as part of the image generation task. Here is a complex example. The prompt is an image of Paris with a crowd of pedestrians all looking at the camera. Here, we are not just trying to generate a group of people, and this is a really difficult thing for an image generation model. That is because it involves generating a lot of small faces throughout the scene. Furthermore, the prompt asks for the people to be looking at the camera.

Thumbnail 1800

Here too, the Nova Omni model understands the user's intent and is able to generate an image that accurately represents what the user is looking for in the prompt. If you look at the scene with many people, you can see that about 90% of them are actually looking at the camera. This is something that most other competing models cannot do today.

As I mentioned earlier, the advantage of having a model that integrates understanding and generation is that the model's understanding component can learn from the generation side, and the model's generation component can learn from the understanding side. This is useful for tasks like editing. The Nova 2 Omni model can support nine different types of editing operations, such as adding new objects, changing objects, extracting information about specific objects, replacing, removing, changing backgrounds, and style transfer. Now, let's look at some examples so you can understand the types of operations we support.

Thumbnail 1840

These can all be done with just text prompts. By simply providing an input image and a text prompt, the model understands the user's intent and can assist with various types of editing tasks. Here is an example of an add operation. It is when you want to add a wooden bench facing the right side of the image. The model generated the bench, but it also understood the style and layout of the scene. When creating the bench, it created it in the exact style of the image so that it harmonizes well with the scene and scene layout.

Thumbnail 1880

Thumbnail 1900

Here is another example of when you want to add a door number to the image on the left. The model understands the intent and generates an image with the number in the correct position on the door. This is an example of a change. In this case, the prompt is asking to change the color of the bicycle to red. The model understands the position of the bicycle, segments that area, and can convert only the relevant part to red.

Thumbnail 1920

Thumbnail 1930

This is another example of changing the frosting color to a different color. Here too, the model understands your intent and can make the change. This is an example of changing a scene, where a snowy background is changed to a more sandy background. When extracting information, for example, suppose there is an image of people wearing dresses on Amazon.com. You want to extract and display that dress and express this information very accurately so that the user can make a purchasing decision.

Thumbnail 1950

This is also an area where the model excels. When replacing an object, what is interesting here is that the user's prompt specifically focuses on not changing the hairstyle. When changing the hat, you want to avoid changing the hairstyle as well and not influence the overall representation of the scene. The model understands that intent and can generate an image that touches only the hat without touching the hair, so it generates those images and updates only the hat.

Thumbnail 1990

This is an example of removal, deleting a polar bear from the scene and deleting a mobile phone from the user's hand. This is also an example of a complex editing operation, and there are multiple prompts. Not only is it deleting the mobile phone from the user's hand, but it is also changing the color of the sweater from pink to blue. It understands both aspects and generates an image that follows the described prompt.

Thumbnail 2020

This is an example of background change, style transfer, and motion change. This is about how to represent a scene, for example, how the same person looks with different types of expressions, and creating expressions that look realistic without changing the person's appearance, attributes, or clothing. This is also an important aspect.

Thumbnail 2040

These are hybrid changes, such as not only removing a hat but also changing the color of the dress that person is wearing. These are the types of complex scenes that we have evaluated and tested while building the Nova 2 Omni model. By having the ability to understand tasks, we can also improve the overall quality of the generation side.

Thumbnail 2070

We evaluated our model against image editing benchmarks. This graph shows the performance of the model. As I explained earlier, in image editing tasks, we are better than the Flux model and perform on par with GPT-4o, GPT-4.1, and Gemini models. Let's look at examples here. There are various types of editing operations that you can do with the Nova model.

Thumbnail 2100

I want to convert a bright Scandinavian style to white. I will add muted minimalistic art.

Thumbnail 2110

Thumbnail 2120

Thumbnail 2130

Thumbnail 2140

Thumbnail 2150

Thumbnail 2160

In other words, you can use it to rearrange and create your own environment. If you are a real estate manager, you can expand to similar use cases for similar scenarios and workloads. And if you want to create advertisements, if you have objects, you can create captions and place objects in various environments to create various types of advertising and marketing assets for your workload.

Currently, as part of our beta program, we see many customers using the model. Monks is one such company. One of the advantages of the Omni model is that, given the power and capability of understanding and generation, you don't need to have multiple models to solve the same task. Because the model itself can understand and express the task and also assist in the generation of content, it becomes more powerful. Therefore, companies don't need to have very powerful and complex workflows, and they are realizing that many workflows can be simplified by the power of the Omni model.

Thumbnail 2220

Dentsu Digital's Nova Omni Case Study: Predicting and Generating Video Creative

Now, I will hand it over to Yamamoto-san. He will talk a little bit about how Dentsu is using the model, its use cases, and workloads. Yes, thank you. Hello, hello, thank you very much. Thank you for the introduction, and thank you for this wonderful opportunity. Let me introduce myself. My name is Satoru Yamamoto, and I serve as Chief AI Officer at Dentsu Digital, and I am also scheduled to serve as Deputy Chief AI Officer at Dentsu Japan.

Thumbnail 2230

Thumbnail 2260

I would like to express my gratitude. I am truly grateful for all the support from AWS. I have had many opportunities to present at many AWS-related events such as re:Invent, Summit, and Cannes Lions. So, I am a big fan of AWS. Today, I would like to tell you all how powerful AWS AI is, especially Amazon Nova Omni. But before we get into the main topic, let me briefly introduce our company, Dentsu.

Dentsu is the largest advertising agency in Asia, and we are currently operating globally in over 100 countries, with more than 70,000 employees. Ah, I have to say one thing. Let me say thank you. It was happy news for me. Thank you very much. In yesterday's keynote session, CEO Matt Garman mentioned Dentsu as a marketing giant. No, no, no, we are not such a tall giant. We are just number one in Asia. Matt Garman, thank you very much. It feels like this.

Thumbnail 2300

Thumbnail 2320

By the way, by the way, we have a solution called Mugen AI. This is a kind of digital marketing solution. We are making great use of AWS AI services here. Today, I would like to introduce how we are utilizing Amazon AWS AI, especially Nova Omni, within this solution. So, let's get into the main agenda. There are three topics: Creative, Planning Operations, and Next-Gen Experience. This agenda follows the evolution of generative AI such as multimodal AI, agent AI, and physical AI. Let's get into the first topic, Creation.

Thumbnail 2340

In Creation, we have a solution called Me AI A. It has three functions: creative generation, performance prediction, and improvement suggestions. Thankfully, it has already been introduced to over 200 companies, and we have succeeded in achieving an average improvement of over 150%. I want to emphasize here that we are utilizing Amazon Nova to make this solution conversational. This is to ensure that employees can use it as much as possible. However, there is still a challenge. That is video creative, video creative.

For example, we couldn't get accurate video predictions, but I can say that using Amazon Nova Omni can change this situation. So, let me explain. In the conventional approach, what we were doing was converting the original video into more simple and fragmented information like keyframes. And we also needed to convert keyframes into text summaries, and we also needed to obtain information from audio. Thus, everything was separated. But this is truly different from how humans watch videos.

As a result, the accuracy was not that high. On the other hand, Amazon Nova 2 Omni is a truly authentic multimodal AI, so it can understand the video itself just like us. As a result, we can obtain very high accuracy in prediction. Also, with the conventional method, what was required was a very vast amount of data. But because Nova Omni has already learned vast knowledge in the background, such an amount of data is not required. You only need to prepare a small amount of high-quality data.

Thumbnail 2450

Thumbnail 2460

Thumbnail 2470

So, let me show you a demo here. All you need to do is upload the video like this. And select an advertising campaign to prepare training data for the prediction model. The model will tell you what kind of video creative will have higher performance, like this. And finally, you can get the predicted value of the uploaded video. What I want to emphasize here is that the correlation value between the predicted value and the actual delivery value is very high, like 0.88. So, we can truly perform predictions accurately. By the way, I forgot to mention, this example is from ANA. The largest airline in Japan, right? I also used that plane to come here, so thank you very much for providing the example.

Thumbnail 2510

Thumbnail 2550

By the way, you can utilize Nova Omni not only for prediction but also for generation. Because Nova Omni is truly multimodal. So, we prepare a brain that can understand the tendency of which videos have high performance, and use that brain to generate a storyboard. Because it is multimodal, it can also draw pictures. So, let's check the demo like this. Just ask the same AI, "Come on, let's make a storyboard." First, you can get a text storyboard. You can convert this storyboard into a visual storyboard. And based on that storyboard, you can generate a moving video very easily like this.

Thumbnail 2560

And here I would like to emphasize two things. This is related to Omni's image generation capability. The first is that since Omni's understanding is truly high, it is very easy to handle or edit images with natural language. The second is the quality itself. It is competitive against major models. And what truly surprised me is that it can generate Japanese characters truly accurately and beautifully. For us, it is almost the first time. It is truly high quality.

Thumbnail 2600

Thumbnail 2610

Thumbnail 2620

Thumbnail 2630

Yes, not only can you generate storyboards and videos from scratch, but you can also reuse existing videos. Because Omni understands which parts are important. So let's look at the demo again. I am a little tired, so please listen to this instead of me. Today, we will demonstrate how AI accelerates content production. First, we upload the raw video file. Nova Omni instantly analyzes the footage and generates detailed metadata for all scenes. Next, it acts as a creative director, creating a viral strategy that includes specific hooks and scripts. Finally, it leverages AI's deep understanding of interviews and visuals to efficiently create the final video edit. From raw footage to viral-ready, it is streamlined by intelligent analysis.

Thumbnail 2650

Thumbnail 2660

From Planning and Operations to Next-Generation Experiences: Developing 7 Applications in 7 Days

Thus, by using Nova Omni, you can handle complex data like video very easily. So, let's move on to the next topic. Planning and Operations. We have a solution called Mugen AI Agent Canvas. This is a so-called citizen development solution where non-engineers can easily build agents. And we have already succeeded in developing solutions for marketers such as strategic planning, journey planning, and media planning, but by utilizing Nova Omni for agents, we can go even further. I think this part is interesting.

Well, what we did was make Omni imitate humans. The video on the left is where a human is actually operating the UI of Amazon's automation UI.

Thumbnail 2710

Thumbnail 2720

We are using an automation UI called Amazon Ads Operation Automation using Xnurta, and we input this video into Nova Omni. Nova Omni understands the workflow and writes a prompt for Nova Act. Nova Act is an AI that can handle browser operations. As you can see, without any human hand, this process can be fully automated.

Thumbnail 2730

These two pages are not directly related to Omni, but they are very good examples, so let me introduce them. So far, I have introduced AI solutions for Japan such as creative prediction, generation, and planning and operations. What we are going to do from now is utilize the Bedrock Agent's multi-agent collaboration function to integrate all solutions into one AI agent. So, let's look at the demo.

Thumbnail 2760

Thumbnail 2770

Thumbnail 2780

What we will do from now, sorry, it is written in Japanese, but I will speak in English. So, let's make a new product, sparkling water. How about "Shining Sparkling Water"? So, let's talk to the AI persona and ask the persona what they think. And if the taste itself is not that important, if it looks good, it will be fine. So, let's summarize the interview. And based on the interview, we can consider the name, key message, and key visual.

Thumbnail 2790

Thumbnail 2800

Thumbnail 2810

Thumbnail 2820

And we will summarize this as a value proposition. Based on that proposition, we can prepare a customer journey, and for the acquisition part, we will need things like digital banner generation. In other words, what I mean is that we are already applying Amazon Nova AI lead to our solutions, and furthermore, we can proceed using Amazon Nova Omni, and we can combine those with Bedrock Agent. Well, explaining this is a kind of advanced way to utilize AI for marketing agents.

Thumbnail 2830

Thumbnail 2850

Thumbnail 2860

However, for our final topic, let's talk about next-generation experiences. It is more conversational, interactive, real, and physical. The first example is using chat. With this solution, you can utilize client-specific data to prepare a chat bot for clients very easily. Of course, this is AWS-based and utilizes the Amazon Nova series. Like this, this is an example of golf course booking. But the problem is that this is just a text-to-text conversation. It is quite normal, but by utilizing Nova Omni, you can provide a richer experience like this.

Thumbnail 2890

Thumbnail 2900

He is my friend, and he is talking to AI by voice. I have analyzed your video. Please tap the microphone and talk. Yeah, initially she is speaking in English, and then he is inputting information in Japanese. It is very interesting. He is just speaking in English, and then he is speaking in Japanese, but the conversation is not interrupted. And here you can also upload videos. Since the AI can understand our videos, it gives advice on how to improve a golf swing like this. This is a kind of new generation experience related to the real world.

Thumbnail 2920

And there are more examples. This is like an in-store experience. What is happening in the video on the left is a video from a customer's perspective in the store, this customer is looking at perfume, and after waiting a bit, maybe she, this customer is talking to the clerk like this. And what we are doing is extracting the persona of this customer. For example, judging from this video, this customer might be a woman. And it looks like a gorgeous and style-conscious office worker. Like this, we can extract a persona.

Thumbnail 2980

And we can also use this persona in the digital world. I think this is a quite interesting example. What we did was use this persona in Amazon Nova Act. In other words, based on the behavior in the store, we can infer how this customer will behave in an e-commerce shop. And such technology that connects real stores and e-commerce stores is very important for the era when VR and AR glasses are widely spread.

Thumbnail 3010

Thumbnail 3030

Well, with this, the presentation is almost over. Today, I introduced some applications based on Amazon Nova 2 Omni, but I am aware that this is not a marketing session. This is a technology session. So, I will tell you very shocking news. Actually, I was informed last Tuesday of last month that there would be a presentation today, and the preparation period was only 7 days. But since I introduced 7 applications, it is 1 application per day. It is quite rapid development. Actually, a woman named Suzuki-san who is sitting here developed these solutions all by herself.

Therefore, compared to before, the development environment has changed quite a bit. Previously, one solution took at least 3 months, but now it is only 1 day. So now, we have become able to focus only on our own business. And I think this is the right way to work with AI. This is almost my final comment. Thanks to AWS AI, we have been significantly freed from concerns regarding development. And as marketers, we can focus on moving people's hearts, creating new value, and changing the world. Thank you very much. I am finished.

Discussion