Data engineering
NYC Taxi Data Pipeline
A data pipeline that takes NYC taxi datasets and processes them with Python and Spark on AWS. The project was mostly about working with data too big for one machine and running distributed jobs in the cloud.
- Python
- Spark
- AWS
What it does
- Distributed processing
- Large-scale datasets
- Cloud execution
Where it stands
I’m rebuilding this one right now. I’ll add the source code and a live demo once it’s in good shape.
The illustration above is drawn in code to show how the project works. It isn’t a screenshot of the actual app.
How it’s built
01
Input — NYC taxi datasets
02
Processing — distributed Spark workloads
03
Runtime — AWS cloud environment
Have questions about this project?
Get in touch