Filter, reshape, and redact logs before export with Transformers
Curated from Cloudflare Blog
If you manage high-volume log pipelines, you likely spend significant engineering cycles building custom ingestion layers to sanitize data before it hits expensive storage. This article details a shift in that paradigm, moving transformation logic upstream to the source. By leveraging SQL-based filtering and redaction directly within the log stream, you reduce the bandwidth and storage costs associated with shipping raw, unprocessed data. The underlying use of Apache DataFusion and Arrow suggests a focus on performance, which is critical when dealing with real-time telemetry. For practitioners, the immediate value is the elimination of fragile, middleware-heavy ETL jobs. The concrete takeaway: audit your current log export destinations for redundant fields or sensitive PII, then prototype a pre-export SQL transformation to measure the immediate reduction in egress volume and storage spend.
Logpush now supports Transformers, allowing teams to filter, redact, and reshape log streams using SQL before export. Powered by Apache DataFusion and Apache Arrow, Transformers apply those changes directly to log data before it reaches the destination.
— Cloudflare Blog