18 lines
413 B
Markdown
18 lines
413 B
Markdown
# data-pipeline
|
|
|
|
ETL pipeline for Nexus analytics platform. Python + Apache Airflow.
|
|
|
|
## Setup
|
|
|
|
python -m venv .venv && source .venv/bin/activate
|
|
pip install -r requirements.txt
|
|
airflow db init
|
|
airflow dags list
|
|
|
|
## Architecture
|
|
|
|
```
|
|
S3 (raw events) → ingest → transform → load → Redshift (analytics)
|
|
```
|
|
|
|
Dags run nightly at 01:00 UTC. See `config/pipeline.yml` for full schedule config.
|