From ten hours
to a few minutes.
I led a rebuild of core ETL pipelines, replacing complex, scheduled AWS Glue jobs with simpler, event-driven Lambda transforms.
The result: faster processing, lower costs, and integrations that could ship in weeks.
lower processing costs
pipeline processing time
to deploy new integrations, down from 10 months
Inside the rebuild
The challenge
Long-running jobs and overlapping schedules left data unprocessed. Unnecessary steps increased cost and made changes difficult.
The decision
Use event-driven, serverless transforms for relatively simple processing. Make schema changes and new integrations configuration-driven.
Why it mattered
The system became simpler to operate and easier to extend. Processing that took up to ten hours finished in a couple of minutes.