TopicPartition

postgres can be your data lake (with pg_lake)

This is an engineering conversation around pg_lake - a new OSS Postgres extension that lets you query and manage Iceberg tables directly form Postgres. Marco Slot, who has EXTENSIVE experience, shares with us various engineering internals, like:• how pg_lake makes analytics (literally) 100x faster• why Postgres is architecturally terrible at analytical queries (and how vectorized execution fixes this)• how (and why) pg_lake intercepts query plans and delegates parts of the query tree to DuckDB• Marco's hard-won experience through a decade+ career in Postgres• versatility as the real moat of Postgres• the practical differences in engineering b/w OLTP and OLAP• and a lot more--------------------------------------------------------------------*TIMELINE*0:02 What is pg_lake?2:23 Postgres' 100x slower problem and columnar storage experiments they had to make Postgres fast for analytics6:00 practical examples and internals16:20 perf internals - vectorized execution & CPU Optimization23:00 pg_lake architecture (why DuckDB isn't embedded) and the connection-per-process issue29:16 how pg_lake intercepts the query plan tree and delegates parts to DuckDB41:09 Iceberg catalogs48:24 postgres to iceberg ingestion patterns (and pg_incremental)53:40 Marco's (long) career: early AWS, Citus, Microsoft, Crunchy Data & Snowflake1:04:20 Marco's observations around the merging between OLTP and OLAP (and the subtle dev differences there)1:15:30 reverse ETL1:33:08 Iceberg as the TCP/IP for tables1:35:00 Marco's thoughts on the "Just Use Postgres" fever-----------------------------------------*MARCO*You can find Marco on:- LinkedIn: https://www.linkedin.com/in/marcoslot/- X: https://x.com/marcoslot- GitHub: https://github.com/marcoslot ----------------------------------------- *pg_lake* You can find the project on GitHub:- https://github.com/snowflake-labs/pg_lake -----------------------------------------*TRANSCRIPT*Feed this into your favorite AI for summarization, or to prompt it specific questions:https://gist.githubusercontent.com/stanislavkozlovski/65c037a8963e49d8121b25003ec94715/raw/4f51f5dcd562b42e8d511b8bc58f0fff6ad5302e/foo.md(or just send Gemini this video link and ask it)------------------------------------------*OTHER PLATFORMS*Watch on YouTube here:https://youtu.be/Jd0DcX2fO_kApple Podcasts:https://podcasts.apple.com/us/podcast/topicpartition/id1814926834General RSS:https://anchor.fm/s/104fd76e0/podcast/rss-----------------------------------------If you found anything useful from this episode, please consider supporting our growth (so we can continue delivering valuable content). You can do this by simply sending it to a friend. It takes 2 seconds to do, and recording/producing this takes us 8hrs+

why IBM bought Confluent

An insightful, casual conversation I had with a past Confluent colleague of mine - Justin Manchester.Justin was employee number 47 in Confluent, where he worked as many things (as one does in startups) but most lately a Technical Support Engineer diving into customers' problems using Kafka. In this episode, he shares his thoughts about the IBM acquisition through a lens that I haven't seen explored yet - Mainframes and Data Gravity (lock in). ---------------------------------------TIMELINE:0:00 - Mainframe Opening3:24 - Justin's Background10:49 - Data Gravity (why mainframe migration fails)14:37 - Joining Confluent as Employee #4718:58 - Kafka as a forward cache (RBC bank example)28:47 - Why is IBM interested in Confluent?45:09 - How IBM may monetize & pay back its acquisition (bundling)1:03:53 - AI Washing and the ZIRP Hangover in the tech space1:18:27 - Confluent's failed bets1:26:04 - What happens to Confluent post-IBM: Flink, WarpStream, etc1:34:23 - Why every data company is chasing the next 10x (to their detriment)---------------------------------------TRANSCRIPT:Feed this to our chatbot of choice to get an interactive summary:https://gist.github.com/stanislavkozlovski/3cd41999066e3316819885f20f7963ae---------------------------------------WHERE TO FIND JUSTIN:LinkedIn: https://www.linkedin.com/in/justin-manchester-48400858/

why Just Use Postgres /w Denis Magda

**JUST USE POSTGRES**Denis is the author of the brand new book "Just Use Postgres!". Consistent with the name, it talks about how most people do NOT need a collection of specialty databases nor systems. Instead, it focuses how you can fulfil the majority of these use cases with a plain old Postgres (and a bit of extensions).Over time, PostgreSQL has grown its network effect to become the most powerful general-purpose database in the world, namely due to its rich extension support and active community.In this super fun conversation, we talk through:• what MySQL got wrong w.r.t community and why Postgres succeeded• the recent (commercial) explosion of Postgres development in the world (Supabase, Neon, Crunchy, ClickHouse)• when Postgres instead of Kafka for messaging• the significance of the meme “just use postgres”• Postgres for data analytics-----------------------------------------**TIMELINE**00:00 - intro & Denis' history04:26 - The Just Use Postgres movement08:36 - The Small Data trend and its iImplications15:49 - The timely shift from distributed systems to simplicity17:15 - Updating your priors re: PG capabilities20:28 - MySQL's OSS community split & our experiences with such things39:48 - Postgres for analytics55:49 - PG in the lake house59:27 - today's AI marketing slop01:02:19 - How you'd build a brand new app on Postgres01:05:24 - Speedrun through PG extensions 01:11:56 - PG as a Message Queue-----------------------------------------**TRANSCRIPT**Feed this into your favorite AI for summarization, or to prompt it specific questions:https://gist.github.com/stanislavkozlovski/a8a38f9557486086bae2f3b81a1ad835-----------------------------------------**BUY THE BOOK**Listeners to this show have a special discount of ~40%!Enter code "2minstrMagda" at checkout to make use of it.Use it before it expires on March 1st 2026:✅ https://www.manning.com/books/just-use-postgres-----------------------------------------**DENIS**You can find Denis on:- LinkedIn: https://www.linkedin.com/in/dmagda/- X/Twitter: https://x.com/denismagda

tansu - pluggable stateless kafka (written in rust)

Tansu is an open-source (Apache-license), stateless Kafka broker that supports pluggable storage engines (PostgreSQL, SQLite, S3 and in-memory). Similar to WarpStream, it is a modern diskless Kafka implementation which does not explicitly store data on disk but rather outsources the durability and replication to another storage backend. Tansu is the only implementation (that I'm aware of) which supports pluggable storage backends, including SQL databases like Postgres and SQLite. Critically, unlike the other alternatives, it does not depend on a coordinator service. The combination of these two unlock a truly stateless Kafka which allows for some interesting use cases, like: * instant auto-scalability * incredibly flexible deployment models (per VPC, AZ, region, etc.) * embedded Kafka * dead-easy CI/CD integration and more.It is also the only truly open source diskless Kafka implementation. Tansu is made for developers. It is fully Kafka API compatible, but makes certain ergonomic decisions that help it boast better UX than regular Kafka.

The Ins and Outs of KIP-1150: Diskless Topics in Apache Kafka

An interview with Greg Harris and Ivan Yurchenko, part of the authors of the new KIP-1150 that introduces diskless topics to Apache Kafka. Diskless topics are a special kind of topic that write directly to an object store. This outsources the data replication and durability guarantees to the store (e.g., S3) and avoids the costly inter-zone data transfer fees that clouds charge you for. The result is a 80%+ reduced Kafka bill and a simpler architecture where brokers are stateless (or have less state) – making them significantly easier to maintain operationally. In this podcast, we dive deep into the technical details and tradeoffs in this complex KIP. ---------------------------------------- ⏱️ TIMELINE * 04:00 – Why Diskless? * 09:17 – Architecture Walkthrough: How Does It Work * 18:50 – The Batch Coordinator * 22:00 – The Different Batch Coordinator Implementations * 28:00 – The Inkless Fork (And Why) * 39:40 – Topic-Based Batch Coordinator * 45:40 – Bottlenecks in the Batch Coordinator? * 47:48 – The Read Path * 54:40 – Caching * 01:03:00 – Shared Log Segment Merging (AKA Object Compaction) * 01:19:06 – Latency * 01:26:30 – Practical Real-World Latency Requirements * 01:39:00 – The Power of Open Source * 01:47:30 – Cost Breakdown * 01:58:00 – Ops and Engineering Time Savings * 02:06:30 – Client Bootstrapping, Leaderless Partitions, One Broker Connection per Client * 02:11:20 – Multi-Tenant Kafka Future * 02:14:05 – Iceberg, Parquet, and First-Class Schema Support * 02:23:10 – S3 Express ---------------------------------------- If you found this episode interesting, please give it a like to signal to the algorithm that it's good. It takes 2 seconds to do, and it took us 5 hours to produce. ---------------------------------------- 💬 HOW TO GET INVOLVED We recommend participating in the Apache Kafka mailing list. Beyond that, you can find Greg and Ivan here: * Ivan Yurchenko – X: @ivan0yu [https://twitter.com/ivan0yu] * Greg Harris – X: @gharris1727 [https://twitter.com/gharris1727] ---------------------------------------- ⚠️ DISCLAIMERS All views expressed in this podcast are personal opinions of the participants. They do not represent the views or positions of Aiven, the employer of Ivan and Greg, or any other entity. Apache®, Apache Kafka®, Kafka, and the Kafka logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.