data-pipeline/README.md

data-pipeline

ETL pipeline for Nexus analytics platform. Python + Apache Airflow.

Setup

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
airflow db init
airflow dags list

Architecture

S3 (raw events) → ingest → transform → load → Redshift (analytics)

Dags run nightly at 01:00 UTC. See config/pipeline.yml for full schedule config.