[AWS] Understanding Big Data Analytics Platforms: Redshift, Athena, and Glue Explained with a Food Factory Analogy

*This page includes promotions.

Introduction: Freshness is Key for Data

"Big Data" sounds cool, but essentially, it's just "masses of ingredients" (logs, sales data, etc.).
These cannot be consumed as they are.

1. Transport (Collection)
2. Wash and cut (Processing)
3. Store (Storage)
4. Taste (Analysis/Visualization)

This sequence of steps is called a "data pipeline."
In this article, we will organize AWS analytics services by comparing them to a "food processing factory line."

1. The Conveyor Belt: Amazon Kinesis

It processes "streaming data" that flows in continuously.

• Amazon Kinesis Data Streams:
  • Role: Streams data in real-time.
  • Analogy: "The never-stopping conveyor belt."
  • A place to "receive data for now" that occurs every second, such as IoT sensor data or web click logs.
• Amazon Kinesis Data Firehose:
  • Role: Collects and saves data to S3.
  • Analogy: "The automated packing robot."
  • Takes data from the conveyor belt, boxes it up (e.g., every 60 seconds), and throws it into a warehouse (S3). It is extremely convenient because you can use it without writing code.

2. The Prep Work Robot: AWS Glue

Data stored in the warehouse (S3) is messy—some is JSON, some is CSV. We use this to clean it up.

• AWS Glue:
  • Role: Extract, Transform, Load (ETL) of data.
  • Analogy: "The fully automatic peeling & cutting machine."
  • Automates mundane tasks like "reading CSV files, deleting unnecessary columns, and converting them to Parquet format (compression)." It is serverless, so no management is required.

3. The Giant Refrigerator and Kitchen: Redshift & Athena

• Amazon Athena:
  • Role: Run SQL queries directly on data in S3.
  • Analogy: "Snacking in the warehouse (instant)."
  • Features: No servers required. Without any prior preparation, you can instantly search files in S3 by asking, "What are the current sales?" It is easy to use, but since it scans every time, it is not suitable for massive amounts of data.
• Amazon Redshift:
  • Role: Store and analyze data in a specialized format (DWH).
  • Analogy: "The organized, giant commercial kitchen."
  • Features: Data is ingested into Redshift once. In exchange, you can aggregate even petabyte-scale data at blazing speeds. This is the "main analysis platform" for enterprises.
• Amazon EMR:
  • Note: A "rental lab" that uses Hadoop or Spark. Intended for experts who want to perform more advanced distributed processing, such as machine learning preprocessing.

4. The Menu: Amazon OpenSearch Service

A service specialized in "searching" rather than "analysis."

• Amazon OpenSearch Service (formerly Elasticsearch):
  • Role: Full-text search engine.
  • Analogy: "A super detailed menu and customer feedback box."
  • Use case: Creating a "product search bar" for an e-commerce site or ingesting massive amounts of system logs to instantly search for the word "error" (log analysis).

5. Plating: Amazon QuickSight

Listing numbers will only get you scolded by the CEO. Finally, we turn them into charts.

• Amazon QuickSight:
  • Role: BI tool (Business Intelligence).
  • Analogy: "The floor server (dashboard)."
  • The AWS version of Tableau or Power BI. It displays the results from Redshift or Athena as clean pie charts or bar charts.

Summary: The Gold Standard for Analysis Patterns

Beginners should first memorize this "Royal Road" route:

1. Save to S3 using Kinesis Firehose.
2. Clean and organize with Glue.
3. Analyze quickly with Athena (or perform heavy-duty analysis with Redshift).
4. Visualize with QuickSight.

As long as you can visualize this flow, Big Data won't be scary anymore!

Afterword

If you can write SQL, "analysis" is fun. While AWS analytics services are powerful, you ultimately use "SQL" at the end. As an engineer, if you can "build the infrastructure and also query the data content with SQL," your market value will skyrocket.

I learned how to build a "serverless analysis platform" using Athena by getting hands-on with this book.
"AWS-based Data Lake"
https://amzn.to/4jzQ5Pg