Setting the file. One moment.
Chapter 04 · Cloudflare Deploy
Subchapter 4.147
references/pipelines/README.mdMarkdown3 KBView on GitHub
ETL streaming platform for ingesting, transforming, and loading data into R2 with SQL transformations.
Pipelines provides:
Status: Open beta (Workers Paid plan)
Pricing: No charge beyond standard R2 storage/operations
Data Sources → Streams → Pipelines (SQL) → Sinks → R2
↑ ↓ ↓
HTTP/Workers Transform Iceberg/Parquet| Component | Purpose | Key Feature |
|---|---|---|
| Streams | Event ingestion | Structured (validated) or unstructured |
| Pipelines | Transform with SQL | Immutable after creation |
| Sinks | Write to R2 | Exactly-once delivery |
# Interactive setup (recommended)
npx wrangler pipelines setupMinimal Worker example:
interface Env {
STREAM: Pipeline;
}
export default {
async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
const event = { user_id: "123", event_type: "purchase", amount: 29.99 };
// Fire-and-forget pattern
ctx.waitUntil(env.STREAM.send([event]));
return new Response('OK');
}
} satisfies ExportedHandler<Env>;Need SQL queries on data?
→ R2 Data Catalog (Iceberg)
✅ ACID transactions, time-travel, schema evolution
❌ More setup complexity (namespace, table, catalog token)
Just file storage/archival?
→ R2 Storage (Parquet)
✅ Simple, direct file access
❌ No built-in SQL queries
Using external tools (Spark/Athena)?
→ R2 Storage (Parquet with partitioning)
✅ Standard format, partition pruning for performance
❌ Must manage schema compatibility yourselfNew to Pipelines? Start here:
Task-based routing: