AWS Glue
Serverless ETL and data catalog
A serverless data integration service: crawlers automatically read schemas into the Glue Data Catalog, and ETL jobs run Spark/Python to clean and transform data.
- Cost$$ Medium
- Difficulty●●●●●
- Costs money when idle?No
- Serverless
- In examsSAA · DVA
Think of it asLike a librarian who both catalogs the books and tidies up the shelves.
When to use it
✓ Use it when
ETL into a data lake/warehouse and needing a shared metadata catalog for Athena, Redshift, EMR.
✗ Avoid it when
Need full control to customize Spark clusters → EMR; real-time streams → Managed Service for Apache Flink.
Common uses
- Crawlers create tables for S3 data
- Convert CSV to Parquet
- Shared catalog for Athena
What does it cost?
- Pricing model: ETL jobs are charged per DPU-hour (billed per second); the Data Catalog is charged by objects stored and requests beyond the free tier.
Setting too many workers or running crawlers too often drives up DPU-hours.
When the question says serverless ETL that discovers schemas and builds a data catalog for Athena or Redshift Spectrum, choose AWS Glue; for a deeply customized Spark/Hadoop cluster, EMR; for just delivering a stream into S3 without complex transformation, Data Firehose.
Frequently asked questions
How is Glue different from EMR?
Glue is serverless: you define an ETL job, AWS provisions resources and bills by run time, and you manage no cluster. EMR gives you Hadoop/Spark clusters and other frameworks with full control and tuning, at the cost of more management. For regular ETL choose Glue; for special frameworks or deep control choose EMR.
What are the Glue Data Catalog and crawlers for?
The Data Catalog is a central metadata store (table names, columns, types, S3 locations). A crawler scans source data, infers the schema, and creates or updates tables in the catalog. Athena, Redshift Spectrum and EMR all use this catalog, so you define a schema once for many tools.
Will rerunning a Glue job process the same data twice?
Job bookmarks remember what has been processed, so the next run picks up only new data rather than reading everything again. Glue also supports streaming ETL reading from Kinesis or Kafka. Billing is based on processing capacity (DPUs) multiplied by run time, so small, tight jobs are cheaper.
Related services
SAA-C03 study guide
Glue is covered in these study modules: