01
Real-Time Crypto Streaming Pipeline
Python · Redpanda · PostgreSQL · dbt · Airflow · Metabase · Docker · Oracle Cloud
A six-stage streaming system pulling live crypto prices from CoinGecko through Redpanda, persisting them in PostgreSQL, transforming with dbt, orchestrating with Airflow, and surfacing them in a Metabase dashboard. Eight containers, end-to-end. Deployed to Oracle Cloud's free-tier ARM VM with CI/CD shipping changes in ninety seconds.
Architecture / 6 stages · 8 containers
End-to-end · dockerized
01
Source
CoinGecko REST API · 10 crypto assets ·
60s polling cadence
02
Ingest
Redpanda event broker · Python producer publishes to
crypto-prices · consumer commits safely with poison-pill handling
03
Store
PostgreSQL raw warehouse · indexed for read · volume-persisted across container restarts
04
Transform
dbt three-layer models ·
staging → intermediate → mart · LAG windows, moving averages, daily summaries
05
Orchestrate
Apache Airflow · 5-minute schedule · freshness gates · automatic retries · task-level observability
06
Analyze
Metabase six-panel live dashboard · price trends, deltas, top movers, volumes — refreshed in near real-time
14,400/day
Throughput
< 5 min
Freshness
~ 90 s
Deploy
8 svc
Containers