Under the Hood
I’m aren’t reinventing the wheel here; we’re just putting the best wheels on the same chassis. The project automates the setup of a stack that includes:
- PostgreSQL: The database we all know and love, equipped with the
pg_lakeextension. - DuckDB: Acts as the engine to process those Parquet files at lightning speed.
- Object Storage: Your choice of MinIO (the default) or SeaweedFS if you’re feeling adventurous.
The magic happens via the pg_lake extension (shout out to Snowflake Labs). It creates foreign tables in Postgres that map to your Parquet files. When you run a query, Postgres hands off the heavy lifting to DuckDB, which pulls the data from MinIO/SeaweedFS, crunches it, and hands the results back. You get the familiar interface of Postgres with the analytical power of a column-store database.
Why SeaweedFS?
By default, the stack uses MinIO because it’s the standard for S3 compatibility. But I’ve also added support for SeaweedFS. Why? Because I personally prefer it and am a bit of pony, but also because at the edge—where resources are tight and simplicity is king—SeaweedFS is often a lighter, more efficient alternative for distributing files. You can switch to it just by setting MODE=seaweed-fs.
The “So What?”
This isn’t just for playing around (though it is fun). It’s about democratizing the “Lakehouse” concept. You can use this to:
- Develop Locally: Test your analytical queries on your MacBook before paying for cloud compute.
- Edge Deployment: Run a powerful analytical node on a decent VPS or even a localized server without the cloud overhead.
- CI/CD: Spin up a disposable Data Lake for your integration tests.
It’s open-source, it’s available now, and it’s ready for you to break it.
Check it out on GitHub: TheMadBotterINC/PG-Lake-Make
As always, issues and pull requests are welcome. Let’s make our data stacks a little less “cloud native” and a little more “cloud smart.”
