All Projects
Data Engineering
GCP E-Commerce Data Platform
This platform solves a real-world problem: comparing product prices and metadata across multiple e-commerce marketplaces in near real-time. It scrapes product listings from Amazon, Noon, and Jumia, normalizes the data into a unified schema, streams it into a data warehouse, and transforms it into analytics-ready tables for price comparison, trend analysis, and product intelligence.
Gallery
Screenshots & diagrams
Architecture
How it's built
Client Request (FastAPI)BigQuery Cache CheckScraper Engine (Amazon / Noon / Jumia)Pub/SubApache Beam / DataflowBigQuery + GCS Data Lakedbt Transform (stagingmart)
Sole data engineer — designed architecture, built scraper engine, streaming pipeline, API service, and dbt transforms.
Highlights
Key achievements
- 01Multi-site concurrent scraping with anti-bot resilience: proxy rotation, browser fingerprint impersonation, CAPTCHA detection
- 02Dual-sink Dataflow pipeline writing to BigQuery (structured) and GCS (raw JSON data lake) simultaneously
- 03Cache-first API design — checks BigQuery for recent results before triggering new scrapes, reducing latency and cost
- 04Incremental dbt mart with merge-based deduplication ensuring one row per product URL with the latest data
- 05Fully containerized deployment: API on Cloud Run, dbt as Cloud Run Job, streaming on Dataflow
Stack
Technologies used
PythonPythonGCPGCPBigQueryBigQuerydbtdbtAirflowAirflowDockerDocker