> ## Documentation Index > Fetch the complete documentation index at: https://docs.folksbase.joselito.dev/llms.txt > Use this file to discover all available pages before exploring further. # Streaming Architecture > How folksbase handles large CSV files without running out of memory. CSV files can be large — tens of thousands of rows, hundreds of megabytes. Loading an entire file into memory would crash the server. folksbase uses streaming throughout the CSV import and export pipelines to keep memory usage constant regardless of file size. ## Why Streaming? The naive approach to handling file uploads is simple: read the entire file into a buffer, parse it, process it. This works for small files but fails catastrophically for large ones: * A 200 MB CSV file would consume 200 MB+ of server memory just to hold the raw content * Parsing it into objects doubles or triples that memory usage * Multiple concurrent uploads could exhaust the server's memory entirely Streaming solves this by processing data in small chunks as it arrives. At any point, only a small portion of the file is in memory. ## CSV Import Flow The import pipeline has two phases: upload (synchronous, in the route handler) and processing (asynchronous, in an Inngest background job). ```mermaid theme={"dark"} sequenceDiagram participant Client participant API as Route Handler participant Blob as Vercel Blob participant Redis participant Job as Inngest Job participant DB as Postgres Client->>API: POST /api/imports (CSV file) Note over API: Stream.tee() splits into two branches par Upload branch API->>Blob: Stream file to blob storage and Count branch API->>API: Count rows via streaming parser end Note over API: Read first 64KB for preview API->>API: Parse preview (headers + 5 rows) API->>Client: Return preview + AI suggestions Client->>API: POST /api/imports/:id/confirm API->>Job: Trigger Inngest event Note over Job: download-and-chunk step Job->>Blob: Stream CSV from blob Job->>Redis: Store 500-row chunks (1h TTL) Note over Job: process-batch-N steps (one per chunk) loop For each chunk Job->>Redis: Read chunk Job->>DB: Upsert contacts (50-row sub-batches) Job->>Redis: Delete processed chunk end Job->>DB: Mark import as completed ``` ### Phase 1: Upload (Route Handler) When a CSV file is uploaded, the route handler does several things simultaneously without buffering the entire file: 1. **Stream splitting with `tee()`** — The file's `ReadableStream` is split into two branches using `.tee()`. One branch streams to Vercel Blob for storage, the other counts rows via a streaming CSV parser. 2. **First 64KB preview** — A custom `teeWithHead()` function reads the first 64KB from the upload stream without consuming the rest. This small chunk is enough to extract headers and 5 preview rows for the column mapping UI. 3. **Byte-level size enforcement** — A `limitStream()` transform wraps the upload stream with a byte counter. If the stream exceeds 200 MB (even if the `Content-Length` header was spoofed), the upload is aborted mid-stream. 4. **Binary content detection** — The first 8KB is scanned for null bytes to detect binary files masquerading as CSVs. At no point is the full file held in memory. The preview uses 64KB, the row counter processes one record at a time, and the blob upload streams directly from the client to storage. ### Phase 2: Processing (Inngest Job) Once the user confirms the column mapping, an Inngest background job processes the file: 1. **`download-and-chunk`** — Streams the CSV from Vercel Blob through a `csv-parse` parser. As rows are parsed, they accumulate in a buffer. Every 500 rows, the buffer is flushed to Redis as a JSON chunk with a 1-hour TTL. Memory usage stays constant at \~500 rows regardless of file size. 2. **`process-batch-N`** (one step per chunk) — Each chunk is processed independently as a separate Inngest step. This means: * If batch 3 fails, only batch 3 retries — not the entire import * Each batch reads its chunk from Redis, upserts contacts in 50-row sub-batches, and deletes the chunk when done * Progress is reported after each sub-batch for real-time UI updates 3. **`finalize`** — Marks the import as completed, invalidates caches, generates an AI summary, and sends notification emails. ### Why Chunked Steps? The previous approach used a single monolithic step that streamed, parsed, and inserted all rows. If it failed at row 9,000 of 10,000, the entire import would retry from the beginning. The chunked approach isolates failures. Each 500-row chunk is an independent Inngest step with its own retry budget. A transient database error on one chunk doesn't affect the others. ## CSV Export Flow Exports use a similar streaming approach, but in reverse — reading from the database and writing to blob storage. ```mermaid theme={"dark"} sequenceDiagram participant Job as Inngest Job participant DB as Postgres participant CSV as csv-stringify participant Blob as Vercel Blob Note over Job: resolve-export-metadata step Job->>DB: Get custom field keys + row count Note over Job: stream-export step loop Cursor-based batches of 1000 Job->>DB: Fetch next 1000 contacts DB->>CSV: Pipe contacts to csv-stringify end CSV->>Blob: Stream CSV content to blob Job->>DB: Mark export as completed ``` ### How It Works 1. **Metadata resolution** — A dedicated step discovers all custom field keys (via `SELECT DISTINCT jsonb_object_keys`) and counts the total rows upfront. This way the streaming step already knows the full column list and row count without scanning the data twice. 2. **Streaming CSV generation** — The `createCsvStream()` function creates a `ReadableStream` that: * Fetches contacts in cursor-based batches of 1,000 * Pipes each batch through `csv-stringify`'s streaming API * Never holds more than one batch in memory 3. **Direct blob upload** — The CSV stream is piped directly to Vercel Blob. There's no intermediate file or buffer. ### Why Cursor-Based Batching? OFFSET-based pagination (`LIMIT 1000 OFFSET 5000`) gets slower as the offset grows — Postgres has to scan and discard all the skipped rows. Cursor-based pagination (`WHERE id > last_seen_id ORDER BY id LIMIT 1000`) is consistently fast because it uses the primary key index. For an export of 100,000 contacts, cursor-based batching keeps every batch equally fast. OFFSET-based would make the last batches significantly slower than the first. ## Key Design Decisions ### Why Vercel Blob for storage? Vercel Blob supports streaming uploads and downloads natively. The upload stream from the client can be piped directly to blob storage without buffering. For downloads, the blob stream can be piped directly to the HTTP response. ### Why Redis for chunk storage? Inngest steps are isolated — they can't share in-memory state. Redis provides fast, temporary storage for the parsed CSV chunks between the download step and the processing steps. The 1-hour TTL ensures chunks are cleaned up even if the job fails. ### Why 500-row chunks and 50-row sub-batches? * **500 rows per chunk** balances Redis storage (each chunk is a few hundred KB of JSON) against the number of Inngest steps (a 10,000-row import creates 20 steps, not 10,000). * **50 rows per sub-batch** balances database insert performance against progress reporting granularity. Users see the progress bar update every 50 rows. ## What's Next? The full import flow from the user's perspective. The layered pattern that organizes this streaming code.