Things I've learned building data pipelines in production — written down so I don't forget them, and shared in case they help you too.
A complete walkthrough of how I set up this very portfolio — private S3 bucket, CloudFront OAI, and everything managed with Terraform from scratch.
Notes from building Change Data Capture into a production data platform — the architectural decisions, the gotchas, and how we used Great Expectations to catch bad data at the gate.
The Medallion architecture is well-documented in theory. In practice, making it work cleanly across multiple ETL jobs for different clients required building Euclidean — here's what drove those decisions.
How I use Great Expectations to validate data at ingestion time, where it fits in the pipeline, and the test suite patterns that actually survive production pressure.
Metrics, alerting, and lineage — the three things you need to sleep well knowing your pipelines are healthy. Here's the exact stack I run and why each tool earns its place.
Practical patterns for managing EMR clusters, Redshift, S3 lifecycle policies, and IAM in Terraform — lessons from doing it the wrong way first and refactoring everything.