Data analytics
Kinesis, Firehose, Glue, Athena, EMR, Redshift, Lake Formation, OpenSearch: collecting, processing and querying big data.
- DomainHigh-performing architectures
- DomainCost-optimized architectures
- 12 diagrams
- 11 scenarios
- 12 services
5-minute essentials
- Kinesis Data Streams: a real-time data stream where you write your own consumers; data is retained for 24 hours (extendable up to 365 days) and can be replayed.Question says "multiple applications reading the same stream", "replay old data", "keep ordering by key" → Kinesis Data Streams.
- Amazon Data Firehose: loads streaming data into S3, Redshift, OpenSearch, Splunk… in near real time, with no shards to manage.Firehose batches data by buffer (size or time) and then writes; it can convert to Parquet and call Lambda to transform. "Dump logs into S3 with the least operations" → Firehose.
- A shard in provisioned mode: writes up to 1 MB/s or 1,000 records/s, reads 2 MB/s. On-demand mode scales automatically.ProvisionedThroughputExceeded error: add shards, switch to on-demand, or choose a partition key with a more even distribution (avoid hot shards).
- Amazon MSK is managed Apache Kafka.Question says "already using Kafka", "need the Kafka API", "move a Kafka cluster to AWS" → MSK, not Kinesis.
- Managed Service for Apache Flink does stateful stream processing: filtering, time-window aggregation, anomaly detection directly on Kinesis or Kafka."Compute a 5-minute average over the data stream" → Flink.
- Athena: runs SQL directly on data in S3, serverless, billed by the amount of data scanned.To cut cost and speed up: store in columnar format (Parquet / ORC), compress, and partition (e.g. by date) so each query scans less data.
- AWS Glue: Data Catalog (shared metadata store), crawlers that infer schemas, and serverless ETL running Spark.Athena, Redshift Spectrum and EMR all read tables from the Glue Data Catalog. "Automatically discover the schema of new data in S3" → Glue crawler.
- Redshift is a columnar data warehouse for analytics (OLAP) and BI on large structured data.Complex joins, many concurrent analysts, periodic reports → Redshift. Uneven load, no desire to manage a cluster → Redshift Serverless.
- Redshift Spectrum queries data in S3 without loading it into Redshift.Hot data lives in Redshift, historical data stays cheap in S3, and they can still be joined in one SQL statement.
- Amazon EMR runs Hadoop / Spark / Hive / Presto clusters that you configure.Need a specific big data framework or cluster customization → EMR. Task nodes do not hold HDFS data, so they can run on Spot to cut cost.
- Lake Formation builds a data lake on S3 with centralized permissions down to the column and row level."Team A sees only some columns of a table" when querying via Athena / Redshift Spectrum / EMR → Lake Formation, not a bucket policy.
- OpenSearch Service: full-text search, log analytics and dashboards (OpenSearch Dashboards)."Keyword search in a product catalog" or "near real-time log analytics with dashboards" → OpenSearch.
- Amazon Quick (formerly QuickSight) is a BI tool for building dashboards from Athena, Redshift, S3, RDS…Question asks for "a dashboard for business users, serverless" → Quick / QuickSight.
Diagrams
Real-time stream: Kinesis Data Streams with multiple consumers
Step by step
- Sources write records with a partition key; the same key goes to the same shard and keeps ordering.
- Three consumers read the same stream independently; one that fails can re-read from its old position within the retention period.
- Lambda writes results to DynamoDB; Firehose buffers and stores the raw data in S3.
Firehose: load logs into S3 as Parquet, then query with Athena
Step by step
- The application sends logs to Firehose.
- Firehose calls Lambda to clean the data and uses the schema in the Glue Data Catalog to convert to Parquet.
- When the buffer size or time is reached, it writes a file to S3.
- Athena queries with SQL, reading only the columns it needs.
The partition key decides the shard: avoid hot shards
Step by step
- A key with many values (e.g. user ID) spreads the load evenly across shards.
- A single fixed key sends everything to one shard: beyond 1 MB/s or 1,000 records/s it gets ProvisionedThroughputExceeded.
- Fix: a more dispersed partition key, add shards, or switch to on-demand.
Data lake on S3: Glue Data Catalog and Lake Formation
Step by step
- A Glue job cleans raw data and writes partitioned Parquet.
- The crawler scans S3 and creates / updates tables and partitions in the Data Catalog.
- Athena, Redshift Spectrum, EMR read the schema from the Data Catalog; Lake Formation decides who sees which tables / columns / rows.
Athena scans only the partitions in WHERE
Step by step
- The query filters on the partition column dt.
- Athena reads only the dt=2026-10-16 folder: less data scanned, less cost.
- The other partitions are skipped entirely.
Redshift + Spectrum: hot data in the cluster, old data in S3
Step by step
- The dashboard sends one SQL statement joining a Redshift table with the external table.
- The external table part is processed in parallel by Spectrum directly on S3, with the schema from the Data Catalog; data does not need to be loaded into the cluster.
EMR cluster: On-Demand core nodes, Spot task nodes
Step by step
- The primary node assigns work to core and task nodes.
- Main data lives in S3 (EMRFS): the cluster can be shut down when the job ends without losing data.
Near real-time log analytics with OpenSearch
Step by step
- Firehose batches logs and loads them into OpenSearch.
- Full-text search and dashboards in OpenSearch Dashboards.
- Firehose backs up records (all, or only failed records) to S3.
AWS Glue: crawler, Data Catalog, ETL job
Step by step
- The crawler recognizes the format and schema, groups year=/month= style folders into partitions, and writes tables to the Data Catalog.
- The ETL job cleans, converts to Parquet and partitions: later queries scan less data.
- Athena and Redshift Spectrum read the same tables in the Data Catalog, with no data copying.
Three ways to run Amazon EMR
Step by step
- EMR on EC2: choose instance types, HDFS, Spot task nodes, install extra applications; you operate the cluster.
- EMR on EKS: run Spark on an EKS cluster shared with other applications.
- EMR Serverless: submit a job and it runs; workers scale up and down with the job.
Batch pipeline: S3 → Glue → Redshift → dashboard
Step by step
- The schedule triggers a Glue job every night: clean raw data, write Parquet.
- Redshift loads the cleaned data with COPY.
- The dashboard queries Redshift; business users view it without writing SQL.
Choosing a data analytics service
Step by step
- Application logs must be loaded into S3 every few minutes with no consumer code: Firehose.
- Occasionally query old logs in S3 with SQL: Athena.
Quick comparisons
Kinesis Data Streams, Firehose, MSK
| Kinesis Data Streams | Amazon Data Firehose | Amazon MSK | |
|---|---|---|---|
| Latency | Real time (milliseconds) | Near real time (buffered) | Real time |
| Who reads the data | Consumers you write (Lambda, KCL, Flink…) | Firehose writes to the destination itself | Kafka consumers |
| Replay old data | Yes, within the retention period (24 hours – 365 days) | No | Yes, per Kafka's retention configuration |
| Capacity management | Shards (provisioned) or on-demand | Scales automatically | Brokers / serverless mode |
| Best for | Many consumers, custom processing | Loading data into S3 / Redshift / OpenSearch | Systems already using Kafka |
Choosing a query and analytics tool
| Athena | Redshift | EMR | OpenSearch | |
|---|---|---|---|---|
| Type | Serverless SQL on S3 | Data warehouse | Hadoop / Spark cluster | Search and log analytics |
| Data lives in | S3 | Redshift tables (+ S3 via Spectrum) | S3 (EMRFS) or HDFS | OpenSearch indexes |
| Billing | Per data scanned | Per node or Redshift Serverless | Cluster EC2 + EMR fee | Per node / serverless |
| Use when | Ad hoc queries, no infrastructure wanted | BI, large joins, many concurrent users | Custom big data jobs, multiple frameworks | Full-text search, logs, dashboards |
Making Athena cheaper and faster
| Technique | Why |
|---|---|
| Columnar format Parquet / ORC | Reads only the columns the query needs |
| Compression (Snappy, GZIP…) | Fewer bytes to scan |
| Partitioning (e.g. year/month/day) | A WHERE on the partition scans only that folder |
| Merge small files into larger ones | Fewer S3 requests, more efficient reads |
| Workgroup data scanned limit | Blocks queries that scan too much |
Glue, Lake Formation, EMR: who does what
| Service | Role in the data lake |
|---|---|
| Glue Data Catalog | Shared catalog of tables / schemas |
| Glue crawler | Scans S3, automatically creates / updates tables in the Data Catalog |
| Glue job | Serverless ETL (Spark, Python) |
| Lake Formation | Centralized permissions by database / table / column / row |
| EMR | Custom big data cluster when Glue is not enough |
Exam scenarios
1Clickstream data must be processed independently by 3 different applications, and data from 7 days ago must be replayable after a processing error.
Show answer
Right: Kinesis Data Streams, extend retention to 7 days, each application is a consumer
Multiple consumers read the same stream, and can replay by position within the retention period.
Wrong: Amazon Data Firehose
Firehose writes straight to the destination, has no custom consumers and cannot replay.
Wrong: A single SQS queue
Messages are deleted once read; multiple applications reading would need multiple queues, and there is no replay.
2You need to put application logs into S3 as Parquet for Athena to query, with the least operations.
Show answer
Right: Amazon Data Firehose with Parquet format conversion, destination S3
Firehose buffers, converts the format and writes to S3 by itself; no servers.
Wrong: Kinesis Data Streams + a self-written consumer converting to Parquet
It works but you must write and operate the consumer.
Wrong: EC2 running a script that batches logs hourly
You manage the machine yourself, with poor fault tolerance.
3Occasionally the operations team needs to run SQL on ALB logs stored in S3; they do not want to build infrastructure.
Show answer
Right: Athena with a table pointing at the log folder in S3
Serverless, pay per data scanned, no data loading.
Wrong: Build a Redshift cluster and load the logs
Pays for an always-running cluster for an occasional need.
Wrong: An EMR cluster
You must create and manage a cluster.
4The Athena bill has risen sharply; the data is large CSV files and queries often filter by date.
Show answer
Right: Convert the data to compressed Parquet and partition by date
Athena bills by data scanned: columnar format + compression + partitioning sharply reduce the amount scanned.
Wrong: Switch to Redshift to save money
Does not fix the root problem of scanning too much data, and adds cluster cost.
5Hundreds of analysts run BI reports with many joins over tens of TB of structured sales data.
Show answer
Right: Amazon Redshift
A columnar data warehouse that parallelizes queries, built for OLAP / BI.
Wrong: RDS for PostgreSQL
An OLTP database, not suited for TB-scale analytics with many concurrent users.
Wrong: DynamoDB
A NoSQL key-value store with no join support.
6The most recent 3 months of data are in Redshift and 5 years of older data are in S3; occasionally a report must combine both without paying for Redshift storage.
Show answer
Right: Redshift Spectrum: an external table pointing at S3, joined with tables in Redshift
Queries S3 data without loading it.
Wrong: Load all 5 years into Redshift
Wastes capacity and cost on rarely used data.
7You need to run existing Spark, Hive and HBase jobs, customize versions and cluster configuration, and reduce compute cost.
Show answer
Right: Amazon EMR, data in S3, task nodes on Spot
EMR lets you pick frameworks and configuration; task nodes hold no HDFS data, so Spot is safe.
Wrong: AWS Glue
Glue runs serverless Spark / Python ETL, does not run Hive / HBase, and offers little cluster customization.
Wrong: Athena
It is only SQL on S3.
8Data lake on S3: the marketing team may see only some columns of the customer table when querying with Athena.
Show answer
Right: Lake Formation granting column-level permissions to the marketing group
Centralized permissions, applied to Athena, Redshift Spectrum, EMR.
Wrong: A bucket policy restricting by prefix
A bucket policy controls only down to the object level, not columns.
Wrong: Create a copy of the data with columns removed
Duplicates data and is hard to keep in sync.
9New data lands in S3 daily with a schema that may change; the table must always be up to date for querying.
Show answer
Right: A Glue crawler on a schedule, updating the Glue Data Catalog
The crawler infers new schemas and partitions automatically.
Wrong: Edit the table DDL by hand each time
Easy to forget, easy to get wrong.
10An e-commerce website needs a product search box by keyword, with suggestions and ranked results.
Show answer
Right: Amazon OpenSearch Service
Full-text search with relevance ranking.
Wrong: Athena
Batch queries on S3, not for low-latency interactive search.
Wrong: DynamoDB scan
No full-text search; scans are slow and costly.
11An application writing to Kinesis Data Streams gets ProvisionedThroughputExceeded even though total traffic is below the combined capacity of the shards.
Show answer
Right: Choose a partition key with many values for even distribution, or use on-demand mode
A "hot" partition key sends everything to one shard even though other shards sit idle.
Wrong: Switch to SQS
Loses stream characteristics (ordering, multiple consumers, replay).
Quick picks
Where does your data need to go, and how is it processed?
| When you need | Use |
|---|---|
| Real-time stream, many consumers, replayable | Kinesis Data Streams · see diagram |
| Load a stream into S3 / Redshift / OpenSearch, no consumer code | Amazon Data Firehose · see diagram |
| Already using Apache Kafka | Amazon MSK |
| Time-window stream analytics | Managed Service for Apache Flink · see diagram |
| Ad hoc SQL on S3, no infrastructure | Athena (+ Parquet, partitions) · see diagram |
| Schema catalog, serverless ETL | AWS Glue (Data Catalog, crawler, job) · see diagram |
| BI / OLAP on large structured data | Amazon Redshift (or Redshift Serverless) · see diagram |
| Join Redshift data with older data in S3 | Redshift Spectrum · see diagram |
| Custom Hadoop / Spark / Hive | Amazon EMR (Spot task nodes) · see diagram |
| Data lake permissions down to column / row | Lake Formation · see diagram |
| Full-text search, log analytics with dashboards | OpenSearch Service · see diagram |
| BI dashboards for business users | Amazon Quick (QuickSight) · see diagram |
Common pitfalls
- Choosing Firehose when the question needs multiple custom consumers or replay: that is Kinesis Data Streams.
- Treating Firehose as real time: data is batched by buffer, so it is near real time.
- Choosing Kinesis when the question says the system already uses Kafka: that is MSK.
- Using a partition key with few values: one shard is overloaded while others are idle.
- Letting Athena scan uncompressed, unpartitioned CSV, then complaining about the bill.
- Building Redshift for a few occasional queries on S3: Athena is cheaper and simpler.
- Loading all historical data into Redshift when Spectrum can read it directly from S3.
- Running EMR core nodes on Spot: core nodes hold HDFS data, so losing a node loses data; Spot suits task nodes.
- Using a bucket policy for column-level permissions: it cannot be done, you need Lake Formation.
- Using Athena or DynamoDB for a product search box: full-text search is OpenSearch's job.
- Confusing RDS (OLTP) with Redshift (OLAP) when the question describes analytical reports over a large volume of data.
Services in this module
Kinesis Data Streamsreal-time data streams
Data Firehoseload streams into data stores
MSKmanaged Apache Kafka
Managed Flinkstream processing with Apache Flink
Glueserverless ETL and data catalog
AthenaSQL directly on S3
EMRSpark/Hadoop big data clusters
Redshiftanalytics data warehouse- Redshift ServerlessRedshift without managing clusters
Lake Formationdata lake permission governance
OpenSearch Servicesearch and log analytics
Quick (QuickSight)AI workspace for business
Numbers that change (checked against AWS docs)
- Kinesis Data Streams (provisioned): each shard writes up to 1 MB/s or 1,000 records/s, reads up to 2 MB/s. checked 2026-10-16 ↗
- Kinesis Data Streams retains data for a minimum of 24 hours, extendable up to 8,760 hours (365 days). checked 2026-10-16 ↗
- Kinesis Data Streams on-demand: a new stream defaults to 4 MB/s write and 8 MB/s read, scaling automatically. checked 2026-10-16 ↗
- Firehose to S3: buffer 1–128 MiB (default 5) and 0–900 seconds (default 300); it writes whichever condition is reached first. checked 2026-10-16 ↗
- Athena bills by data scanned; the pricing page uses an example of 5 USD per TB. Compression and columnar formats (Parquet) reduce the amount scanned. checked 2026-10-16 ↗
- Athena: when WHERE includes a partition column, only that partition's data is scanned. checked 2026-10-16 ↗
- Redshift Spectrum queries data in S3 without loading it into Redshift tables; multiple clusters can query the same dataset. checked 2026-10-16 ↗
- A Glue crawler infers the schema and writes tables (including folder-based partitions) to the Data Catalog; ETL jobs use these tables as sources and targets. checked 2026-10-19 ↗
- EMR Serverless is an EMR deployment option with no cluster management and automatically scaling workers; pre-initialized capacity lets jobs start in seconds. checked 2026-10-19 ↗
- EMR on EKS runs open-source big data frameworks on an Amazon EKS cluster, shared with other container applications. checked 2026-10-19 ↗