nexus/data-pipeline
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Reduce batch size — 50k was OOMing the Redshift loader, 10k works fine, don't ask why

Browse source
This commit is contained in:
Markus Fischer 2026-04-21 08:00:00 +00:00
parent 4c29bcc54c
commit 0b31344c4c
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
config/pipeline.yml
View file

@ -7,5 +7,5 @@ pipeline:
host: redshift.nexus.local host: redshift.nexus.local
database: analytics database: analytics
schema: public schema: public
batch_size: 50000 batch_size: 10000
max_retries: 3 max_retries: 3