Bulk scans
resource.scan(options, consume) reads every row that matches a request, in batches. Use it for exports and background jobs. Normal pagination selects a new page each time; on PostgreSQL a scan runs the selection once and fetches from a cursor.
Read the batches
const exportedRows = await ordersResource.scan(
{
request,
context: { orgId: "acme" },
batchSize: 1000,
count: "exact",
load: "detail",
},
async (batches) => {
let processed = 0;
for await (const { rows, totalRows } of batches) {
processed += rows.length;
console.log({ processed, totalRows });
}
return processed;
},
);
The callback receives an async iterable of { rows, totalRows? }. Whatever it returns is what scan returns. Rows are typed like query's, with the default load profile, the load you pass, or a view.
| Option | Default | Notes |
|---|---|---|
request | none | Filters, sorting and search. Pagination and facets are ignored. |
batchSize | 1000 | A positive integer, set by your server. Independent of the public page-size limit. |
count | "none" | "exact" adds totalRows to every batch, but may process the whole selection first. |
signal | none | An AbortSignal, checked between database operations. |
db | the engine's | An existing repeatable-read or serializable transaction. |
- The scope and validation run once.
- A unique id is added to the sort, so the order is stable.
- Await your writer inside the loop. The engine doesn't fetch ahead or keep rows in memory.
- An empty selection yields no batches, and scans don't compute summaries.
Snapshot and connection
PostgreSQL scans run DECLARE … NO SCROLL CURSOR, FETCH and CLOSE in a repeatable-read, read-only transaction. Rows and their relations come from one snapshot, so concurrent writes don't shift rows between batches.
That connection and snapshot stay open for the whole callback, so:
- bound the number of concurrent exports;
- don't upload files or wait on other services inside the callback;
- expect sorting and exact counts to use temporary storage in the database.
A scan can't resume from a checkpoint.
The cursor is closed when the callback returns, on break, on errors and on cancellation. Cancellation doesn't interrupt a statement that is already running; use statement timeouts for that.
To share a snapshot with other work, pass your own transaction as db. The engine doesn't commit or roll it back. A read-committed transaction is rejected.
import { drizzle } from "drizzle-orm/node-postgres";
const db = drizzle({ connection: { database: "app" }, relations });
await db.transaction(
async (transaction) => {
await ordersResource.scan(
{ request, db: transaction, context: { orgId: "acme" } },
async (batches) => {
for await (const { rows } of batches) {
console.log(rows.length);
}
},
);
},
{ isolationLevel: "repeatable read", accessMode: "read only" },
);
Other databases and custom steps
The cursor path needs:
- an asynchronous PostgreSQL driver with interactive transactions (not HTTP-only drivers);
- a unique, non-null root id, and unique relation keys for the sort columns. A sort through an ambiguous relation is rejected rather than exporting duplicates.
Other databases, and resources with a custom strategy.query or strategy.ids, scan by bounded pagination instead. The scope resolves once, facets are skipped and an exact count runs on the first page only, but there is no single snapshot. A custom strategy.rows works with both paths.