iTranslated by AI
re:Invent 2024: New York Life's AWS-based Lakehouse and Generative AI Use Case
Introduction
By transcribing various overseas lectures into Japanese articles, we aim to make hidden, high-quality information more accessible. This is the presentation we are featuring in this project!
📖 AWS re:Invent 2024 - New York Life: Data platform modernization to generative AI innovation (FSI324)
This video introduces the data infrastructure and AI utilization initiatives at New York Life, the largest mutual life insurance company in the United States. Scott Symank, Lead Architect at the company, explains the transition from a legacy Hadoop data lake to an AWS-based Lakehouse architecture, while Boris Simanovic shares their journey of building an MLOps platform and Gen AI use cases. In particular, an LLM-based letter generation system introduced to streamline the work of Claims Managers successfully reduced a task that used to take several hours to about 15 seconds. It is also worth noting that they have conducted AWS's "Experience-based acceleration" more than 12 times, enhancing innovation capabilities across the entire organization.
Note: This article is automatically generated from the video. There may be typos or incorrect information, so please refer to the original video for accurate details.
Click on an image to jump to the corresponding scene in the video.
For other transcribed articles related to re:Invent 2024, please check the information summarized in this Spreadsheet.
Main Content
Overview of New York Life's Data Strategy and AI Platform
We are joined by Scott Symank, Lead Architect at New York Life, and Boris Simanovic, who oversees MS at the company. Scott will begin by introducing New York Life, what the company aims to achieve, and its data strategy. Afterward, he will explain how they transformed their data platform from legacy systems. Then, Boris will talk about the AI and Gen AI platform they built, and the actual impact it has brought to Claims Managers. He will further explain the partnership between New York Life and AWS, and how they were able to accelerate innovation through it. Finally, we will summarize the lessons learned throughout this journey, including what worked and what didn't.
Scott Symank on New York Life's Data Transformation Journey
Hello. I am Scott Symank, Domain Architect for the AI and Data organization at New York Life. After spending over 25 years in the pharmaceutical industry, I joined New York Life about seven and a half years ago. To briefly introduce New York Life, we are the largest mutual life insurance company in the United States. This means we are responsible to our policyholders, not Wall Street. As mentioned earlier, we are a company with a 179-year history, having thrived through both economic booms and busts. We actively leverage technology, data, and AI to provide superior experiences for both our agents and policyholders.
About a year ago, we hired a Chief Data and Analytics Officer and launched an AI and Data organization. We then embarked on a new data strategy and mission. Our strategy is to activate the power of New York Life's data through enterprise AI and data products to generate meaningful business impact. This strategy consists of four pillars. The first pillar is to scale the data infrastructure and technology for building enterprise data products. The second pillar is to improve data intelligence through AI and analytics. The third pillar focuses on transforming into a culture where decisions are driven by data. And the final pillar is to provide proactive and personalized experiences for both agents and policyholders.
Since this breakout session is about modernization efforts, let's look at where we started. We had an on-premises Hadoop data lake environment with data flowing in from about 100 different sources. We set levels for data standardization and organization, and built a data warehouse using IBM BigIntegrate (essentially DataStage for Hadoop). We placed Tableau on top of that to realize business analytics and dashboards. As this technology aged, we faced many challenges. The first was that support became very limited due to the end of vendor support for Hadoop 3.6.4. Furthermore, with the merger of Hortonworks and Cloudera, they shifted entirely to a CDP approach, abandoning the legacy Horton stack, which turned our limited support into a deprecated platform.
At New York Life, we were pursuing a new compute strategy to become a cloud-focused, cloud-first company. We recognized that the legacy platform had many features that needed improvement. It lacked ACID capabilities to improve data pipeline performance. As more data flowed into the lake, requirements emerged from new personas who wanted to access the data to gain better insights. We decided to build a Lakehouse architecture using S3 as the data lake and Redshift as the data warehouse. The cloud migration was carried out by combining the redesign and re-engineering of many components, along with a lift-and-shift of some.
I will explain how New York Life operates on AWS. First, the foundation is the Enterprise Cloud Platform team. If you liken it to a wedding cake, they are the bottom layer, serving as the foundation for New York Life's AWS operations. They are responsible for account creation, Service Control Policies, and setting Transit Gateway rules. They also manage the process of making approved services available. Just because AWS releases a new service and makes it generally available, it doesn't automatically mean it's available at New York Life. We need to ensure that proper security policies and guardrails are in place to use that service. There is close coordination with the Cloud Enterprise team, and we focus only on services that are in demand. In our efforts, we had to get usage approval for various services such as Glue, Lambda, and Redshift.
The second important team in our efforts is the SRE (Site Reliability Engineer). These are reliability engineers placed in departments or organizations who function as Platform Engineers. They manage the entire infrastructure, are responsible for Terraform coding, and handle cross-account access and IAM principal development. They operate at the departmental level, rather than at the enterprise or project level. The third element is the organization's engineering team, responsible for building data pipelines, data ingestion, and ETL or ELT to build domain models in the warehouse. The final layer consists of consumers and personas who want to utilize the data that is increasing day by day. This includes analytics engineers, data scientists, MLOps specialists, and those who want to perform exploratory data analysis.
At New York Life, we have implemented a micro-account strategy in line with the AWS Well-Architected Framework. This approach establishes boundaries that isolate business applications and require explicit permission to access different applications due to differing compliance and security requirements. This is a proven, secure approach that ensures the right people have access to the right services. We have developed a set of architectural best practices and principles. The first is to run everything through a HIPAA ingest account. All data pipeline activities store data in the HIPAA account. Since we handle PII (Personally Identifiable Information) and PHI (Protected Health Information) in life insurance applications, we maintain strict security regarding data handling.
Access to the HIPAA account is not granted to processes or people; all data must be pushed out from the HIPAA account. Different roles have different permissions. The data ingestion portion has appropriate access to the landing bucket, while processes that read and process the data have only read permissions for that bucket, with no write permissions to other buckets. This prevents multiple roles from having write access to the same bucket. All plaintext data must be written to the landing bucket. We are in the process of implementing AWS Lake Formation to manage access rights to the data lake. When users want to access or query the data lake, they use Redshift external tables, as done through Redshift. We aim to use serverless functionality whenever possible and cost-effective, but the main Redshift cluster for ETL and ELT tasks is running most of the time, so we use provisioned clusters. For consuming clusters that scale up and down, we typically use a serverless approach. Cross-account access is implemented through trust policies, so in the event of a breach, instead of changing IAM access, we can simply disconnect the trust policy to prevent cross-account access.
Finally, de-tokenization capabilities are managed by the LDP group. We use a tool called Protegrity to obfuscate PII data within the data lake. This is reversible, but requires appropriate permissions managed through LDA. This flows into Redshift and Redshift user-defined functions capable of handling that specific functionality.
Building and Operating an MLOps Platform with Boris Simanovic
When explaining pipeline work, I will first give an overview of each account and then go into details. Basically, I will cover all accounts except the ML Ops account. Boris will explain the architecture and strategy for ML OPS in his presentation. The first part focuses on what we call the "Ingestion/Data Pipeline." Specifically, I will discuss the Ingestion account, Data Lake account, and Data Governance account.
As you can see from the architecture, data flows into the Data Lake from various sources. The diagram only shows SFTP and DMS, but in reality, we have message-based systems for AWS Full Replication, AWS DMS CDC, and various other data patterns entering the Landing bucket. Instead of building individual ingestion solutions, we built a Glue framework with patterns to ingest data using an Amazon Aurora metadata repository. When using Database Migration Services for full ingestion, loading the database with appropriate metadata automatically generates Glue jobs. We do not build point-to-point ingestion; whether it is file ingestion or message-based, everything ingests data into the Landing bucket through the framework.
Data in the Landing bucket is in the exact same form as the source system, with no transformation at all. We have built one pipeline process, so regardless of how data reaches the Landing bucket, the next steps are identical depending on the contents of the metadata repository. In case change points cannot be sent, we perform our own delta detection. Since AWS charges based on compute usage, continuously calculating data without changes not only degrades pipeline performance but also increases costs. Therefore, we try to identify deltas as early as possible in this process. After that, we carry out a series of standardization steps, such as standardizing date formats and standardizing addresses using Address Doctor, and here we use Protegrity for tokenization of PII fields. This data is written to the Raw bucket within the Ingestion account.
The Raw bucket stores only standardized and tokenized change data. We then apply these changes to the Data Lake. The Data Lake is in a separate account and is the first layer where end users can utilize the data. The Data Lake retains all Type 2 history, so for example, if someone has been a policyholder for 30-40 years and has moved 4-5 times, their address and the duration of each are recorded. We have converted about 4,000 tables within the Lake. When we first went live in the cloud, everything was Parquet, but analysis revealed that 1,200 to 1,400 tables would benefit from moving to Iceberg. We converted about 500 tables to Iceberg, achieving a 25 to 40% performance improvement. Migration to Iceberg is about one-third complete, with 700 tables remaining. We found that converting small tables has little effect, so we are focusing on major performance gains for updates and writes. We are handling Glue Data Cataloging with Lake Formation here.
You can see the Data Governance account at the bottom. When we initially migrated from Hadoop to the cloud, it was 100% structured data and sources. Since then, we have moved into Generative AI, dealing with unstructured data. We were able to use the same architecture to ingest, process, and move data to the Lake. The difference was that we needed to add Data Governance to tag documents for use with appropriate models. Just because a document is in the Lake doesn't mean it's suitable for a specific model's purpose, so document owners tag them to indicate whether they are suitable for use with different Generative AI models.
Now, let's explain the consumption side: the Transform Account, IMC Account, Warehouse Account, and Tableau. Most of the explanation so far has been about re-architecture and re-engineering, but the Transform Account represents a lift-and-shift. That is, we are running DataStage in the cloud and performing ETL work to generate the Redshift data warehouse. We are currently migrating this to IMC Informatica, which has both ETL and ELT capabilities. The Platform SRE team can scale DataStage horizontally and shut it down to reduce costs. Since it is not a tool optimized for the cloud, it requires detailed engineering work, but we wanted users to be able to use Redshift early without waiting for the full migration of ETL work. If you come back here in four months, the Transform Account should be decommissioned.
We also have a series of Redshift clusters. There is one large provisioned cluster where all ETL and ELT to calculate our domain models run. Many of the other clusters are Serverless, which can read from each other via data sharing. When data scientists need to access production data to build Model-ready Datasets, they have data shares on their clusters; this isolates compute resources, so their calculations and model executions do not compete with our ETL or ELT jobs. People performing exploratory data analysis are in their own clusters, so they can use the exact same production data while isolating compute resources.
We also see Producer 2.0, an Amazon Aurora database that functions as an Operational Data Store. It contains a lot of agent information and is updated in near real-time. We are aiming for zero-ETL with post-Aurora, which will be announced this week, so this is placed in the Warehouse Account. We also want to perform analysis of this data through Redshift. Our goal is to make Producer 2.0 data available in the Redshift cluster so that all data can be queried in one place. Although we only show Tableau as a consumer account, many users now connect to our environment through Redshift. We had to define external tables for each of those clusters, but AWS recently released an update allowing external tables to be part of the data sharing process.
How did our data enable Generative AI? We wanted to use the same data lake structure to centralize high-quality, reliable data for data scientists. We want to ensure that data scientists and their Generative AI applications do not have to search for data or hunt through different sources. We want to make that data easily accessible to them. We also have the capability to ingest unstructured data. For early ingestion use cases, we built our own frameworks to handle various data sources, such as Salesforce, Adobe, and SharePoint platforms. We built a generic framework that can be used to ingest that data. This improves data access and scalability by leveraging AWS and the scalability of the data lake.
Data governance and compliance are critical. For the Data Governance account mentioned earlier, the main role is tagging to ensure the right data is used for the right models. Because we have tracking and data provenance, we have lineage from the source system through the lake to BI reports, just like with BI reports. Now, for ML and GenAI projects, we can track what data is used in which environment, from source to GenAI model.
Gen AI-Powered Innovation in Claims Management
Now, I will hand it over to Boris to tell you more about the world of MLOps. Thank you, Scott. Hello everyone, and thank you for your time today. My name is Boris Simonovic. At New York Life, I lead the MLOps team, which includes everything from data utilization and model building to ML engineering for production environments, as well as AI solutions and the platforms required for them. Before diving into the platform details, I would like to talk about the principles we use when building our platform. Please remember that we started from scratch because we did not have a platform like this before.
One of the architectural principles was scalability. If you have experience in software development, you know this is a standard SDLC element. We needed a provider that could scale as we desired. Thus, Kubernetes was a clear choice, and we use EKS. We also needed automation. A full CI/CD pipeline from development to production is essential. In MLOps, deployment to production means everything we build and automate. The reason is to ensure reproducibility. We want to be able to reproduce everything that runs and everything that is built from start to finish. Everything is version-controlled in GitHub and progresses from Docker to ECR.
We are a regulated organization and spend a lot of time on governance. One reason we want our pipelines to be fully reproducible and automated is so we can demonstrate to regulators how we built a particular model and how we reached a particular solution. For that, we also need monitoring. Many of these concepts are standard DevOps principles that have existed for decades, but AI and GenAI solutions have unique requirements. Another principle was modularity. We wanted to modularize everything—infrastructure, applications, models, and LLMs—so we could scale them as needed.
Security is an area we are accustomed to, including standard security scans for both code and Docker containers. Also, as Scott mentioned, we prioritize data protection. We are concerned about LLM protection, IP protection, and where data goes when we send it to an LLM. Last but not least, a key point we continue to work on is democratization. We are a centralized team serving the entire New York Life organization, with decentralized data science teams. We want everyone in the company to be able to use all the data science tools and platforms that host our models, AI, and GenAI solutions.
Let's start with a very high-level development architecture. On the left is the Data Lake account managed by the data team. This is where the datasets Scott mentioned exist, and where all data—structured and unstructured—comes together. For exploratory data analysis, we leverage SageMaker Studio—the notebooks that data scientists use for experimentation.
Our ML engineers use slightly different tools to deploy models into production. All experiments are stored in GitHub. In GitHub, we created templates for ML engineers to wrap all experimental code, containerize it with Docker, and push it to the ML pipeline. This pipeline is deployed to our server, EKS. Looking back, one point I forgot to add to this diagram is that SageMaker can also access Bedrock. For data scientist experiments, they have full access to modeling datasets and Bedrock. That concludes the explanation of the development environment.
So, how do we operationalize this? Everything is version-controlled in GitHub. Much of it follows standard SDLC principles and pipelines. We adopt a modular architecture, where specific technology teams are responsible for the UI. The UI team builds the UI, AI solutions, and GenAI solutions. Incidentally, these are all treated in the same way—all are containerized and have REST APIs. Whether hosting a model on an internal Kubernetes server or using a Foundation Model, architecturally, all solutions are wrapped in exactly the same way.
For GenAI, we needed a few additional components. We needed an OpenSearch database (vector database), and we used an open-source serverless version. I touched on monitoring earlier; there is a difference from normal software where you only need application monitoring. While we do full infrastructure monitoring, with models, you also need application performance monitoring. Furthermore, model metrics are required. For governance reasons and internal operations, we need to save all data that the model inputs and outputs. This allows data scientists to reproduce and review the performance of any model and improve the product in the next version.
To that end, we used DynamoDB to record everything. This is because, honestly, we didn't fully grasp everything we were doing at first. We had just started, were learning, and are still learning today. This is all a journey. I have been doing AI and ML for over 10 years, and I learn new things every day. In DynamoDB, we saved every prompt, completion result, start and end time, token length, and any other information we could capture. We didn't know what would become important. The data scientists who review this information later had access to Bedrock. From a scaling perspective, Bedrock enabled scaling in many ways. This was a truly good choice. We could access many models, launch our own models (i.e., open-source models), or use Foundation Models.
Why do we do all of this? Scott talked about the data journey, and I am talking about the platform—all of this is to develop solutions for our customers. When we say customers, we refer to internal customers, business partners, and ultimately, our actual customers and policyholders. Over a year ago, business partners brought us a problem. They talked about the insurance claims management experience, where they had to generate custom letters based on highly complex medical records. Because everything was being generated manually, there were many issues. The process was non-existent. This was an extremely complex problem that previous AI couldn't solve. All the tools we built were for this purpose. The problem was that it was very manual. Since these letters were all created by people, there were legal risks. Every person writes differently. It is not unique like mathematics; if you recall English class, there were various graded essays, and you couldn't always understand why that result occurred. So, we aimed for standardization.
We wanted to standardize how these letters were generated. Since letter generation took a very long time and the content or reasons for the letters were sometimes unclear, we wanted to improve customer experience. We also wanted to ensure that all generated content was fact-based, created according to specific standards, and that legal risks were reduced. If a claims manager writes incorrect language in an email when denying a claim, and that could lead to misunderstandings, legal risks might arise.
In the traditional Claims Manager workflow, to make a decision regarding a specific medical claim, they had to review documents such as nurse case manager notes, claim notes, and other necessary memos. After reviewing all these documents, they would decide whether to approve or deny, and then begin the arduous task of organizing all the data logically and creating the letter. This part took up the majority of the day and was the task everyone disliked the most. Once the draft of the letter was completed, it typically underwent a review, often by a supervisor, before being sent to the customer.
We considered ways to improve this process. The good news was that we already had a 'Human in the Loop.' We are a risk-averse organization with a 179-year history, and since we want to avoid being on the front page of the New York Times, the review process was important. The difference was that by using LLMs to generate letters, tasks that took hours were completed in seconds. I wish I could show you the expression on the face of one of our Claims Managers when they saw that a task that used to take half a day was generated in about 15 seconds in their first experiment.
By adopting LLMs, we were able to build consistency. The model is very consistent and can generate data at specific reading levels, making everything easier to read. Some of these letters were very complex, and we wanted to avoid situations where you would need a lawyer to understand them. Through this process, by using prompts to create letters, we improved the work experience for Claims Managers. Data scientists performed all of this prompt work, which was the main task as I explained when showing the SageMaker architecture earlier. Of course, we also do pipeline work such as extraction and chunking.
Everything was automated, and the time required was reduced from hours to minutes, improving efficiency. Risks were also reduced through prompts. Asking an LLM—it is interesting to use the word 'ask' because we are accustomed to giving instructions to models, but we speak to these models like assistants—one thing was to maintain a specific tone. We wanted to ensure that the model used appropriate expressions, especially when delivering negative results to customers. We wanted to improve customer experience, and that was achieved through fact-based generation. Large Language Models are good at comprehension-level tasks, and we ensured they were built at specific levels with appropriate tones. There were dozens of instructions where I summarized at a very high level, and these improved customer experience in several ways.
We are now able to generate letters faster, more quickly, more concisely, and with the other characteristics I mentioned earlier. Looking back, building this application itself wasn't very difficult, but because it was our first attempt, it was a major learning opportunity. At the time, it seemed insurmountable. It wasn't a standard algorithm problem of just classifying or extracting specific words—it wasn't a problem like named entity recognition—it was a challenge of generating letters. Therefore, we needed to think about how to decompose the problem.
We started with design collaboration. We worked with the data team, business partners, and AWS Management to plan a roadmap that included optimizing the data platform including unstructured data across the organization. Everything Scott talked about also included the construction of platforms and tools, which required management buy-in at the CEO level. Fortunately, the CEO at the time had a clear policy to 'become one of the best gene-related companies.' This is what Jimmy will talk about next, but for me, if I had to describe this entire success in one word, it would be Experience-based acceleration.
At first, we didn't know how to approach the problem. However, that is exactly what helped us decompose the problem and test it in various ways. AWS sent experts to help with all the testing and ultimately helped us provide an excellent solution. Experience-based acceleration is a mechanism we use at AWS to accelerate development. From the experiences we have seen, the main reason projects are delayed or fail is not due to technical issues, but human working styles. It is mainly organizational issues—not communicating with each other, silos existing within the organization, and not collaborating sufficiently.
This is a way to ensure that customers can innovate and collaborate in the way we do at AWS. It is a transformative approach where business stakeholders, developers, and decision-makers are brought into one room to actually build. The key is to build and to have the appropriate decision-makers in the room so that when obstacles arise, they can make decisions quickly. This creates organizational capability for innovation, facilitates development acceleration, and strengthens the capabilities of builders by receiving appropriate guidance in the same room as AWS experts.
This is a 3-6 week project, with actual construction performed in 3 days. The first part is building collaboration with management. This is because management buy-in is necessary for the team to work on this project. We define success criteria and identify what to do in the acceleration phase. We define multiple workstreams and proceed with tasks in parallel to accelerate development. Then, we carry out the actual 3-day effort. Since the pandemic, we have also conducted this remotely; it works remotely, in hybrid, or in person. Everyone participates in the construction in the same room or via Zoom meeting. There are decision-makers, and a method for escalation is provided if a decision is needed, and we build in those 3 days.
At New York Life, we have found this to be a very effective mechanism. We have conducted at least 12 or more EBs. At first, there is resistance because it takes effort, but once they see the results, how much delivery is accelerated, how it promotes innovation, and how it raises organizational capability, it became a daily effort for New York Life. It is now fully established. I have heard that the Cloud team conducted an EB in the production environment on the weekend. For them, it has become a daily activity. The important thing is that this is an investment from AWS, and they send experts free of charge to support innovation.
Lessons Learned from the Data and AI Journey and Future Outlook
We have learned numerous lessons through this data and AI journey. I want you all to take away what you should and should not do. What we learned is that you should always design with scale in mind. You need to build iteratively while keeping a North Star (a clear goal) in mind. You cannot do everything at once. It is important to build in small units, fail early, and pivot as necessary. You should prioritize architectural reviews. We learned these lessons during the process of data transformation, but when implementing those small parts of the architecture, always perform an architectural review and ask AWS to review the architecture to ensure you are following AWS best practices and using appropriate tools.
Foundation Models have now become commoditized. Everyone in the world can access the same Foundation Models. The key is data. That is the differentiator from other organizations. Always build a robust data foundation and bring the right model for that data. Don't align data to the model; align the model to the data. Again, failing early is very important. This is how we work at Amazon. Fail early, pivot, and make changes. Make decisions with a sense of speed, fail early, and try to make changes if necessary.
As I touched on earlier, large companies have many silos. I have worked with large companies for over 20 years and have seen those silos. Communication between departments is lacking, and conversations only occur through email, Teams, or Zoom. You need to break this down. At Amazon, we operate with small teams where business stakeholders and technical stakeholders work together. We call these 'Two-pizza teams.' It is important to create small teams that produce concrete results. And finally, be unwavering in your vision. Again, keep the North Star in mind, be unwavering on that, but be flexible on how to achieve it, and be sure to adjust course as necessary.
I want to close with a quote from one of the innovators who represents our era. We should not fear change. Change is always present. We are very lucky to live in an era where such great technological innovation is happening, especially regarding AI. Let's embrace it, absorb it, and innovate quickly. Embrace that change, create something new, and make magic happen. That concludes my explanation. Be sure to perform reviews.
- This article was created entirely automatically using Amazon Bedrock.
- Given concerns about internet pollution caused by generative AI articles, this article aims to maintain the value of the original content itself by converting session videos into text and images with almost no change in the amount of information, while hoping to increase accessibility and Googleability in multiple languages.
Discussion