I'm a Data Engineer working with Spark, Apache Iceberg, and dbt on AWS. I maintain a SaaS production Lakehouse platform with dozens of Glue jobs and write about real-world engineering problems on Medium.
Real-world production engineering challenges and Lakehouse insights
A weekly merge job kept getting slower, week after week. Every signal pointed at the usual suspect. It wasn’t. Here’s how we told the difference, and what was actually eating the extra time. A few ...
Building a Semantic Layer from Zero (Part 2): Can Column Descriptions Alone Stop an LLM From Hallucinating SQL? Same 6 questions, same database — but this time the model gets the schema. 6/6 correc...
6 questions, 12 attempts, 1 silent fabrication that would have gone straight into a report This is part 1 of a 6-post series that answers one question: How much context does a model actually need t...
Data Science exercises, ML algorithms & C programming notes
A series for data engineers who can run PySpark jobs but want to understand what happens underneath: MapReduce, the DAG, shuffle, joins, skew, caching, the Spark UI, Iceberg, and streaming.
How much context does an LLM actually need to answer SQL questions correctly? A series built from a real dbt project and database: zero context, column descriptions, business logic, routing, and access control.
A running log of real problems from day-to-day work: incidents, dead ends, things that broke in production, and the small realizations that changed how I approach the job.
A message tells one consumer what to do. An event tells everyone that something happened. Mixing the two up is how a streaming pipeline turns into a distributed RPC system.