iTranslated by AI
re:Invent 2025: Building and Scaling a Global Market Data Platform with dbt at Nasdaq
Introduction
By transcribing various overseas lectures into Japanese articles, we aim to make hidden, high-quality information more accessible. This is the presentation we are featuring in this project, which operates under that concept!
For transcription articles of re:Invent 2025, please check the information summarized in this Spreadsheet.
📖re:Invent 2025: AWS re:Invent 2025 - Scaling Data Transformation w/ dbt & Nasdaq: Unlocking Ent Power Global Mkts
In this video, Michael Weiss, Product Strategist for the Nasdaq Eqlipse Intelligence Platform, explains the construction of a global market data infrastructure utilizing dbt. Nasdaq is composed of three pillars: Market Services, Capital Access Platforms, and Financial Technology, with the Intelligence Platform covering the four domains of data management, analytics, reporting, and billing. The reasons for adopting dbt Enterprise include multi-tenancy, scalability through dbt mesh, the provision of context via the semantic layer, and visibility into governance and lineage. Nasdaq defines its mapped data layer as a contractual layer, providing standard analytical models while also allowing for custom model extensions by clients. Ultimately, the goal is to establish a globally shared ontology and semantics, integration with AI agents, and the realization of a data model marketplace. A key feature is the construction of an environment where non-technical users can also participate in data modeling by providing a no-code experience.
Note: This article has been automatically generated while maintaining the content of the existing lecture to the greatest extent possible. Please be aware that there may be typos, omissions, or incorrect information.
Main Content
Introduction to Nasdaq's Three Pillars and the Financial Technology Division
Thank you, everyone. My name is Michael Weiss. I serve as a Product Strategist for the Nasdaq Eqlipse Intelligence Platform. Today, I would like to talk a little bit about how we have utilized dbt to empower our markets, and how we have empowered markets on a global level. Let’s start with a brief agenda. I hope it won't be too long. Happy hour is coming up soon, and I'm sure you're all looking forward to that more than listening to me. But first, let me give you a brief introduction to what Nasdaq is, so you know what we are talking about.
When talking about Nasdaq, you can think of it primarily in terms of three pillars. First, there is the Market Services pillar. This is the part many people know as Nasdaq. These are the markets that Nasdaq owns and operates in North America and Europe. This includes the largest trading market in the United States by volume. Second is Capital Access Platforms. This is the division primarily responsible for listing businesses, and we currently have approximately 3,500 companies listed. A significant part of this area is index services. You may have heard of the QQQ, which makes up a large part of our portfolio. And the final pillar is the Financial Technology division. This is a part of Nasdaq that I suspect many people don't think about. Nasdaq serves approximately 4,000 customers globally to support their various solution needs in financial markets.
I would like to delve a little deeper into Financial Technology so you can understand the breadth of services Nasdaq can provide to its clients. We cover everything to ensure market compliance, from financial crime management with Verafin products to regulatory technology with AxiomSL for regulatory reporting and surveillance. Finally, there is Capital Markets Technology. This includes both Calypso and Nasdaq Eqlipse, covering the full trading lifecycle for both banks and markets.
Overview and Architecture of the Eqlipse Intelligence Platform
Now, let me spend just one more minute talking about what Nasdaq Eqlipse is and the products that make up that domain. Afterward, I will explain the product I am talking about today and the role dbt plays in it. There are four major technical domains in Eqlipse. First is the trading system. This is what markets deploy to execute trades. Next is clearing, which supports our customers' clearing needs. And finally, at the end of that main trading lifecycle, there is custody. The domain I am talking about today is our new product, the Intelligence Platform. The Intelligence Platform is responsible for gathering all this information and enabling customers to leverage it for new use cases, new products, and new services to sell to their own customers.
Now, let's dive a bit deeper into the Eqlipse Intelligence Platform to help you understand what it is. When thinking about the Eqlipse Intelligence Platform, there are really four major domains from a platform perspective. The foundation of everything is data management. This includes everything from how we acquire data from our own products and other third parties, how we store that information, how we ensure data governance, and how we maintain an open ecosystem so that various different use cases and services—from other data transformation services to AI and agentic use cases—can communicate with that data.
Next, the main pillar we consider is analytics—the insights we need to get from the data to run the market and the business. There are several things to consider here, and this is the area where dbt primarily falls. But in reality, what we are trying to serve on the analytics side is the business. We are trying to provide access to as much information as possible in the most intuitive way possible. Traditionally, this was done with dashboards and reporting, but clearly, in today's world, we are also considering more agentic use cases and digital worker type use cases.
The third pillar is about reporting. Despite your best efforts, sending CSV files and PDFs remains a large part of what we have to do. So, we are providing a service that allows customers to automate that, reducing friction for their own customers by providing self-service portals and reporting capabilities. Technology or engineers do not actually need to be involved. And the last one, which might seem like it doesn't fit into the other three categories, is our managed billing solution. This focuses on how customers need to charge their clients for transaction-related or non-transaction-related fees.
When I talk about the Eqlipse Intelligence Platform, I won't call this an architecture diagram, but it will give you an idea of how the components are arranged. Returning to Capital Markets Technology, the first thing we are trying to do is integrate natively with other Nasdaq products and services. When a customer purchases the Intelligence Platform, we want them to not have to think about how to get data from other Nasdaq services. We will handle that. That is everything on the left. In addition to proprietary data from Nasdaq, we are allowing customers to bring their own data into the platform, and we allow them to fuse it and place it alongside the data we are already managing.
In the gray box, there are four major things. The ingestion engine is actually running there, and this is critical for integration with all other products and services. Regarding the cloud data lake, you all probably understand what a data lake is, but this is the primary retention and storage of information. We provide various query and access endpoints that you can use to integrate with your own services if you want to bring your own applications into the ecosystem. And finally, there is the context-aware understanding and knowledge about the data, which is the data catalog service.
And the green part on the far right can be thought of as the business products and applications that sit on top of everything else. When I mentioned analytics, that is the Insights HQ product. This focuses on more traditional dashboard use cases, the ability to create, publish, and share dashboards, and can be used both internally and externally. The Report HQ product focuses on reporting, and Revenue HQ is the billing component I mentioned. And as I said, these can be exposed to internal customers who buy Intelligence products from us, or we can provide access to our customers' clients to reduce friction points and ensure that customers get the information they need to make decisions.
Strategic Use of dbt: Integrated Approach and Reasons for Choosing dbt Enterprise
Now, let's get into the meat of it. The role of dbt in this platform, in this ecosystem, because dbt plays a very important role in what we do. Before getting into what dbt does, I think it is important to explain some of the goals we had in leveraging dbt. And in many respects, these are the same goals we had back in 2021 when everything was focused solely on Nasdaq use cases. We understood that we needed to integrate with existing products and services to make that information usable for internal customers. We need to provide better business models and analytical models to end customers, so they can actually analyze and understand what is happening in their business.
In addition to that, we wanted to provide the ability for internal users, and now external users, to extend and participate in model creation. Often, those data models and traditional ETL processes were locked behind engineering resources. We wanted to lower the barrier to entry so that more teams could collaborate and participate in this process. And dbt has truly enabled that internally, and now it is enabling us to do that externally. And the last part is to provide an access point where we can enforce standards and contracts, both for users and for AI. In today's world, data without context is a bit limiting. When you spin up an agent, you want to know that it is interacting with the data in the right way. And that is really a key driver of what we are trying to achieve with dbt.
So why are we doing this? What is our strategic vision, and why did we decide to build the Intelligence platform, and why dbt? It really boils down to a few target outcomes we are looking to achieve. We believe that by removing the burden of managing the data platform and building the baseline data models, we can provide faster innovation to our customers because we are doing it for ourselves. We let them focus on their real differentiators, while we take on more of the heavy lifting.
Another thing to think about is that we want to start thinking of the global financial fabric as something more unified. What we are starting to do is build a standard understanding and semantics for that information. Whether we are operating in the US, Europe, Asia, or Latin America, we can all speak the same structure, the same ontology, and the same semantics. And that would be a major unlock for a more global-based economy. Obviously, now we can all share and provide information to one another. And that also leads to globally-sourced model definitions. This is essentially a massive outsourcing effort. It is the ability for customers not just to use our models, but to contribute to those models and share them more seamlessly with partners and users.
So, from a dbt integration and product perspective, how are we actually thinking about this architecturally? I apologize, this is high-level, but it gives you an idea of how we have laid this out. The way we deploy the Intelligence platform is actually what we call a hybrid tenant approach. Many people probably don't know what that means, but what we have here is service separation and isolation based on shared components (which are all multi-tenant) and single-tenant components (where the customer's data resides). And there are a few reasons why we do that; local laws are the biggest part of it, but of course, there are security and other fun things as well.
We are taking a similar approach with dbt. There is a main dbt account where Nasdaq does most of the dbt development, testing models, and publishing them. And for customers interested in collaborating on dbt models, we provide a dedicated dbt account, pre-configured with various things they don't need to worry about. From how we manage source control in GitLab, to CI/CD processes that test test coverage and coding standards,
and even how to rely on Nasdaq's data models within dbt to extend them and add their own information. So, from the perspective of dbt project structure, what would this look like? It is interesting how we thought about this. Everything in the blue section on the left is what Nasdaq provides out-of-the-box. Obviously, first and foremost, we are always going to integrate with our own products and services, and I touched on that.
However, you might also notice that we have the Nasdaq Mapped Data Layer in the contract section. You might wonder why that is there. There are a few reasons why we think about it. One is that it is the lowest level of data we want to define in business terms. That is, we are thinking about defining transactional data in a more business-friendly way. But second, we want to provide the flexibility and capability for non-Nasdaq entities and services to plug into the ecosystem as well. Having that contract layer allows other tools and services in the same domain as the products we already serve to immediately plug in and map their own data.
Once we enter the contract layer, we start talking about Nasdaq standard analytical models and insight models. These are provided automatically to all customers within the domains they are interested in. However, that is not always enough. It probably covers 80% of what people need. But we understand that customers have things they want to do on their own. That is where the functionality for customers to provide their own information to the platform, create their own contracts, create their own models, or extend our models becomes really important. Because we don't want to be a bottleneck for customers to extract true value and functionality from their data if we can't cover every use case.
The last interesting part of this is the semantic layer. I won't say that 'semantic layer' is the perfect term. Because I don't think the term 'semantic' describes what we are trying to achieve. When we think about 'semantic', what we really mean is 'context'. What we are trying to do is provide a layer of contextual knowledge about the data—whether for traditional BI use cases or agentic use cases—so that they can understand the data they are interacting with and know that they are using it in the right way.
In addition to that, we want to be able to enforce certain criteria on the data they get. For example, if there is a calculation and you only want them to get the average, just going to a traditional data store doesn't give you a way to enforce it. They can do whatever they want. With a semantic layer, you can enforce these types of functionalities, ensuring data is utilized only in the most logical ways.
So why did we choose dbt and use dbt Enterprise? Everything I just talked about could probably be done with open-source dbt. It is a fantastic open-source project, and I think anyone who has used it would agree. So, why did we go with dbt Enterprise? First of all, it provided the easiest path to multi-tenancy compared to building it ourselves. Second, dbt Mesh—the ability to depend on projects within the dbt Enterprise ecosystem—is absolutely critical for providing extensibility to our customers.
The semantic layer—the ability to expose those rules and enforce them from access points—is also becoming more important to our overall offering. Governance, lineage, and visibility are essential. We want to ensure the right users have access to the right things, understand how data is derived, and be able to see what is happening. Finally, dbt Enterprise has many great tools for continuous delivery and enforcing standards.
One last thing I want to mention about dbt is the general development experience. If you haven't used it, dbt Enterprise has a fantastic interface for building dbt models. Whether you are a more technical user who wants to dive into the SQL side, or you use something like the Canvas dbt Enterprise now has for more drag-and-drop capability, it allows various users—not just inside Nasdaq but also our customers—to participate in managing data contracts and the data lifecycle.
Roadmap for Global Expansion and Setting Success Metrics
So, how are we thinking about rolling this out? There is a lot to think about. When we talk about deploying something at a global scale, there are many moving parts. One is, how do we make sure we are doing it the right way? Are we doing it in a secure manner, and also, how are we benchmarking ourselves to ensure we are moving in the right direction?
The way we thought about this was, first, to focus on the areas we know best. That is defining a common ontology and semantics for the data we know. Because if we can build these contracts and these data models correctly, we can move to the next phase. That is enabling our clients to provide that extensibility. Once we reach the extensibility side, then I want to talk about how we expose that information to AI and agentic use cases. How do we ensure that digital workers are interacting with that data and doing the right thing without going off the rails?
The final phase, which is one of the goals we are trying to reach, is a data model marketplace. When we talk about data and data monetization, we always talk about the data itself. We never talk about the data models that could potentially be sold. If we all have a baseline common definition and understanding, customers now have the opportunity to create their own structures, re-package them, and sell them to other consumers on the Intelligence Platform at a global scale. This is a new monetization method that was impossible before without something like dbt.
To wrap this up, what are the value drivers we are thinking about from a client perspective? What are we trying to measure against? As we think about this over the next 12 to 18 months, how will we know we are being successful? First of all, are our customers easily accessing the data? Do they understand that data? Are they making better decisions because of that data? This is obviously the simplest thing we can think about, because this has been universal for 20 years at this point.
Secondly, have we made it easier to engage with and use agentic use cases? Is the work we are doing enabling customers to move faster with AI? Thirdly, have we enabled new customer data access services? Are we providing them with the ability to interact with data in new and interesting ways they couldn't before?
Also, we want to measure how much of our models are actually reusable. We don't want to be in the business of building custom, one-off models. We want to build models that we know will benefit every customer, so we want to make sure we are measuring usability across a broad range here.
And one more thing I want to point out is the no-code experience. Are we enabling non-technical people to truly engage with data in ways they couldn't before? On the data modeling side, with things like Canvas and dbt Enterprise, and likewise on the digital work and agent side. Is the data being built specifically for their use cases? Are they getting the right answers? Because if not, none of this matters. We need them to be able to trust what they are doing.
So, we are three minutes early, so you can all go get a beer or a cocktail. If you have any questions, feel free to ask. But thank you all for coming today. And I hope you enjoy the rest of your time at re:Invent.
- This article is automatically generated using Amazon Bedrock, while keeping the contents of the original video as faithful as possible.
Discussion