No description

Find a file

Nina Seidel d657c8375a Test PII anonymisation		2026-03-24 10:15:00 +00:00
config	Move pipeline to 01:00 UTC — 03:00 was conflicting with Redshift maintenance	2026-03-10 08:10:00 +00:00
dags	Add Airflow DAG skeleton for events ETL	2026-02-19 11:30:00 +00:00
src	Add PII anonymisation step — legal requirement from data protection review	2026-03-24 09:30:00 +00:00
tests	Test PII anonymisation	2026-03-24 10:15:00 +00:00
README.md	Initial data pipeline scaffold	2026-01-08 09:00:00 +00:00
requirements.txt	Initial data pipeline scaffold	2026-01-08 09:00:00 +00:00

data-pipeline

ETL pipeline for Nexus analytics platform. Python + Apache Airflow.

Setup

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
airflow db init
airflow dags list

S3 (raw events) → ingest → transform → load → Redshift (analytics)

Dags run nightly at 01:00 UTC. See config/pipeline.yml for full schedule config.