I'm a Data Engineer working with Spark, Apache Iceberg, and dbt on AWS. I maintain a SaaS production Lakehouse platform with dozens of Glue jobs and write about real-world engineering problems on Medium.
Real-world production engineering challenges and Lakehouse insights
To build reliable Text-to-SQL agents, your semantic layer needs a Schema Routing Engine . Here is how it works, what we learned testing it in real benchmarks, and the best practices for implementin...
A deep dive into the disk architectures of RocksDB/Flink and PostgreSQL, how memory access patterns shape I/O trade-offs, and when to use each. As software engineers or Data Engineers, we often tre...
Read a real routing config, run it against a live agent, then deliberately revert one function to its old, buggy version and watch a real misrouting bug come back on your own machine. My previous p...
Data Science exercises, ML algorithms & C programming notes
A series for data engineers who can run PySpark jobs but want to understand what happens underneath: MapReduce, the DAG, shuffle, joins, skew, caching, the Spark UI, Iceberg, and streaming.
How much context does an LLM actually need to answer SQL questions correctly? A series built from a real dbt project and database: zero context, column descriptions, business logic, routing, and access control.
A running log of real problems from day-to-day work: incidents, dead ends, things that broke in production, and the small realizations that changed how I approach the job.
Most engineers learn streaming as batch, but faster. Kleppmann flips that model in Designing Data-Intensive Applications: batch is the special case, streaming is the default. This pill sets up the whole series.