All Projects
Data Engineering
EGX Big Data Pipeline
A production-grade, end-to-end big data pipeline for the Egyptian Exchange (EGX) stock market. The pipeline follows a Lambda Architecture pattern with both batch and streaming ingestion paths that converge in a unified HDFS data lake. It computes 10+ technical indicators (SMA, EMA, RSI, MACD, Bollinger Bands) and serves analytics through Hive tables connected to Power BI via DirectQuery.
Gallery
Screenshots & diagrams
Architecture
How it's built
yfinance APIKafka (real-time ticks) + Batch IngestionHDFS Data LakeSpark ETL (technical indicators)Hive TablesPower BI DirectQuery
Sole data engineer — designed Lambda architecture, built ingestion, Spark ETL, orchestration, and serving layers.
Highlights
Key achievements
- 01Lambda Architecture with both batch and streaming paths converging in HDFS
- 02Kafka-based real-time tick ingestion with Spark Structured Streaming consumer
- 0310+ technical indicators computed via PySpark: SMA, EMA, RSI, MACD, Bollinger Bands, ATR, OBV
- 04Airflow DAGs orchestrating daily batch ETL, indicator computation, and Hive table refresh
- 05Fully containerized with Docker Compose: Hadoop, Spark, Kafka, Hive, Airflow — all services in containers
Stack
Technologies used
KafkaKafkaSparkSparkAirflowAirflowDockerDockerPythonPythonSQLSQL