All Projects
Data Engineering
Real-Time Fraud Detection Pipeline
A production-grade, end-to-end real-time fraud detection pipeline built on modern data engineering principles. The system ingests banking transactions via Change Data Capture (CDC), enriches them with customer and account context from Redis, engineers 30+ fraud features, scores each transaction with a machine learning model in real time, and lands everything into a Snowflake data warehouse modeled as a Kimball star schema for analytics.
Gallery
Screenshots & diagrams
Architecture
How it's built
MSSQL (CDC)DebeziumKafkaFlink (enrichment + ML scoring)S3 (Parquet)Airflow (COPY INTO)Snowflakedbt (stagingdimsfacts)
Sole data engineer — designed CDC architecture, built Flink pipeline, trained ML model, created dbt warehouse.
Highlights
Key achievements
- 01Change Data Capture via Debezium capturing every INSERT, UPDATE, DELETE at the row level
- 02Real-time Flink pipeline engineering 30+ fraud features and scoring with ML model
- 03Redis-first dimension lookup with MSSQL fallback for sub-millisecond enrichment
- 04Dual-mode ML training: unsupervised (Isolation Forest) and supervised (XGBoost) with automated feedback loop
- 05Kimball star schema in Snowflake with dbt transformations: staging → dimensions → facts
Stack
Technologies used
KafkaKafkaPythonPythonSnowflakeSnowflakedbtdbtAirflowAirflowDockerDocker