How I Fixed a Data Ingestion Bottleneck With Date Partitioning
Nie udało się dodać produktów
Nie udało się dodać tytułu do koszyka.
Nie udało się dodać tytułu do schowka.
„Usuń ze schowka“ nie powiodło się.
„Obserwuj podcast“ nie powiodło się
„Przestań obserwować podcast“ nie powiodło się
-
Czyta:
-
Autor:
This story was originally published on HackerNoon at: https://hackernoon.com/how-i-fixed-a-data-ingestion-bottleneck-with-date-partitioning.
I fixed a recurring data ingestion bottleneck by working with a vendor to introduce date-partitioned exports and reduce unnecessary file discovery.
Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #data-engineering, #distributed-systems, #data-ingestion-pipelines, #data-partitioning, #object-storage, #incremental-data-ingestion, #idempotent-data-processing, #data-pipeline-reliability, and more.
This story was written by: @rahuln. Learn more about this writer by checking @rahuln's about page, and for more stories, please visit hackernoon.com.
Our vendor-data ingestion jobs were repeatedly enumerating a growing history of export files. I worked with the vendor's engineers on date-partitioned delivery and prepared our consumer in parallel. This case study explains how partitioning bounds file discovery, why event-date partitions need late-arrival and replay policies, and how to coordinate the producer-consumer cutover.