Skip to content

SDK ​

dirsql ships as a library for Python (PyPI: dirsql), TypeScript (npm: dirsql), and Rust (crates.io: dirsql). All three are thin bindings over one shared Rust core, so behavior is identical; only the calling conventions differ.

Install ​

bash
uv add dirsql
bash
pnpm add dirsql
bash
cargo add dirsql

Import ​

python
from dirsql import DirSQL, Table, RowEvent
typescript
import { DirSQL, Table } from "dirsql";
import type { TableDef, RowEvent, ExtensionSpec, DirSQLOptions } from "dirsql";
rust
use dirsql::{DirSQL, AsyncDirSQL, Table, Row, RowEvent, Value, Extension, DirSqlError};

DirSQL ​

Constructor ​

python
DirSQL(
    root: str | None = None,
    *,
    tables: list[Table] | None = None,
    ignore: list[str] | None = None,
    no_ignore: bool = False,
    config: str | list[str] | None = None,
    persist: bool = False,
    persist_path: str | None = None,
    extensions: list[dict] | None = None,  # [{ "path": str, "entrypoint"?: str }]
)
typescript
new DirSQL(configPath: string)
// or
new DirSQL({
    root?: string,
    tables?: TableDef[],
    ignore?: string[],
    noIgnore?: boolean,
    config?: string | string[],
    persist?: boolean,
    persistPath?: string,
    extensions?: ExtensionSpec[],  // [{ path: string, entrypoint?: string }]
})
rust
DirSQL::builder()
    .root(root)                     // optional
    .tables(tables)                 // optional; append one with .table(t)
    .ignore(patterns)               // optional
    .no_ignore(true)                // optional; default false
    .config(config_toml_path)       // optional; repeatable — call again to
                                    //   merge another config in call order
    .persist(cache_path)            // optional; Some(path), or None for the
                                    //   platform cache dir
    .extensions(extensions)         // optional; append one with .extension(e)
    .poll_interval(duration)        // optional; watch-loop cadence, default 200ms
    .build()                        // -> Result<DirSQL>  (synchronous scan)
// Shortcuts: DirSQL::new(root, tables), DirSQL::with_ignore(root, tables, ignore),
//            DirSQL::from_config_path(path)  // reads an explicit config file

Creates a SQLite index over a directory. The index root is the explicit root when given (a relative root resolves against the cwd once), else the process cwd — the config file's location never sets the root. A config's [[table]] globs anchor at the config file's own directory instead (see glob anchor); root governs programmatic tables and path-tables. The Python and TypeScript SDKs read the cwd when DirSQL is constructed, though the scan runs later, so a chdir after construction does not move the root.

Constructing with neither a config nor programmatic tables defines no named tables — the same as the CLI with no -c. Query the filesystem with a path-table (SELECT * FROM './'); a SELECT ... FROM files in that state fails with a hint pointing at that form. There is no implicit <root>/.dirsql.toml discovery: to read a config on disk, pass its path via config (Rust: .config(path) / DirSQL::from_config_path(path)). The root-joining DirSQL::from_config(root) shortcut was removed in #603 — use DirSQL::from_config_path(root.join(".dirsql.toml")).

Parameters:

  • root — Directory to index. When omitted, the index roots at the process cwd (even when config is supplied).
  • tables — Programmatic Table definitions.
  • ignore — Glob patterns matched against root-relative paths under the glob rule (* one level, ** any depth); matched files are skipped entirely (scan and watch).
  • no_ignore / noIgnore — Scan files a .gitignore would hide. Path-tables respect .gitignore files by default; the built-in .git skip and the ignore parameter's patterns still apply. The CLI's --no-ignore.
  • config — Path to a .dirsql.toml. Its [[table]] entries are appended after any programmatic tables; its ignore patterns and [[dirsql.extension]] entries are appended likewise. The config file does not set the index root: with no explicit root, the index roots at the process cwd. Its [[table]] globs anchor at the config file's directory (or at the literal prefix of an absolute / ~/ glob), not at root; its ignore applies to its own tables only. Multiple configs merge in order on every SDK — Python config accepts a str or a list[str], the Rust builder's .config() is repeatable (call it once per file), and TypeScript config accepts a string or a string[]; the configs merge in call order ([[table]] / ignore / [[dirsql.extension]] accumulate; a duplicate table name across configs errors), mirroring the CLI's repeatable -c/--config.
  • persist — Keep the SQLite index on disk between runs (default off: ephemeral, rebuilt every startup). The cache lives in the platform cache directory by default (outside <root>); on restart, only files whose stat changed are re-parsed. (The CLI exposes the same switch as the --persist [PATH] flag; it is not a config key.)
  • persist_path / persistPath — Override the cache location. Ignored when persistence is off. Constructor values are used as given. In Rust these two parameters collapse into a single builder method: .persist(None) enables persistence at the default location, .persist(Some(path)) enables it at path.
  • extensions — SQLite extensions to load at startup, before any table DDL (enable → load → disable, so SQL load_extension() is never exposed). Each entry pairs a shared-library path with an optional entrypoint init symbol. In Python and TypeScript, path may be a bare package name, resolved from the installed package (importlib / node_modules) — for both the programmatic list and a config file's [[dirsql.extension]] entries. The Rust SDK is file-path-only. Programmatic entries load first, then the config's. A relative programmatic path is used verbatim (resolved by the OS against the process working directory); config-file paths resolve against the config file's parent.

Construction is asynchronous in Python and TypeScript. The constructor returns immediately and the initial scan runs in the background; query and the other methods transparently await readiness. In Rust, .build() scans synchronously; use .build_async() → AsyncDirSQL for the non-blocking equivalent.

The scan is the only walk of the tree. A file watcher over the named tables starts before it and runs for the life of the instance; the changes it sees during the scan are applied once, before the build finishes, and later changes update the index as they arrive. See The file watcher for the difference between platforms.

Rust builder extras
  • Table appears in the builder via .table(t) (append) or .tables(v) (replace).
  • .poll_interval(Duration) sets the channel-based watch() loop's poll cadence (default 200 ms): lower values give tighter event latency at higher idle CPU.
  • .suppress_config_extensions(true) skips loading the config file's own [[dirsql.extension]] entries. Launcher plumbing (used with pre-resolved .extensions(...)); not needed in application code.
  • Extension { path: PathBuf, entrypoint: Option<String> } is the extension spec type.

ready ​

python
await db.ready() -> None
typescript
await db.ready  // awaitable property
rust
// AsyncDirSQL only; DirSQL::build() is already complete when it returns.
db.ready().await -> Result<()>

Waits for the initial scan to complete, re-raising any construction error. On Linux, a tree with more directories than inotify's watch limit fails construction with the OS error ENOSPC and the message inotify watch limit reached: Python raises OSError with errno.ENOSPC, TypeScript rejects with an Error whose code is "ENOSPC", and Rust returns DirSqlError::Io whose os_error() is errno 28. To raise the limit, see The watch limit. Safe to call multiple times. In Python and TypeScript every other method awaits readiness internally, so an explicit ready is only needed to observe construction errors before issuing a query.

query ​

python
await db.query(sql: str, params: Sequence | Mapping | None = None) -> list[dict]
typescript
await db.query(sql: string, params?: unknown[] | Record<string, unknown> | null) -> Record<string, unknown>[]
// Runs on the libuv threadpool; the JS event loop stays responsive.
rust
db.query(sql: &str) -> Result<Vec<Row>>   // Row = HashMap<String, Value>
db.query_with(sql: &str, params: &Params) -> Result<Vec<Row>>

Executes a SQL query and returns rows keyed by column name. If a result repeats a column name, later occurrences receive :N suffixes (for example, a and a:1); suffixes avoid names already present in the result.

Bind parameters. params reaches SQLite's own parameter binding; the values never enter the SQL text, so text holding quotes needs no escaping. A sequence (a TypeScript array) fills ? placeholders in order; a mapping (a TypeScript plain object) fills :name, @name and $name placeholders, keyed by the name without its prefix. It works for path-tables and declared tables alike:

python
await db.query(
    "SELECT path FROM './*.json' WHERE content ->> 'advice' = :advice",
    {"advice": text},
)
await db.query("SELECT * FROM notes WHERE author = ? AND year = ?", ("ann", 2024))
typescript
await db.query(
  "SELECT path FROM './*.json' WHERE content ->> 'advice' = :advice",
  { advice: text },
);
await db.query("SELECT * FROM notes WHERE author = ? AND year = ?", ["ann", 2024]);
// queryOrdered takes params the same way.
rust
use dirsql::{Params, Value};
use std::collections::HashMap;

db.query_with(
    "SELECT path FROM './*.json' WHERE content ->> 'advice' = :advice",
    &Params::Named(HashMap::from([("advice".into(), Value::Text(text))])),
)?;
db.query_with(
    "SELECT * FROM notes WHERE author = ?",
    &Params::Positional(vec![Value::Text("ann".into())]),
)?;
// query_ordered_with and query_each_with bind the same way;
// AsyncDirSQL::query_with is the async twin.
  • Values bind as SQLite's native types. Python accepts None, int (bool binds as 0/1), float, str, bytes and bytearray; anything else raises TypeError rather than being stringified, and an int outside the signed 64-bit range raises OverflowError. TypeScript accepts null, number (an integral value binds as an INTEGER, any other as a REAL, so 1.0 binds as 1), bigint, boolean (0/1), string, and Buffer / Uint8Array; anything else, undefined included, rejects with a TypeError, and a bigint outside the signed 64-bit range with a RangeError. Rust binds a Value.

  • The params must fit the placeholders exactly, or the query fails with an error naming the problem: a wrong count for a sequence, a placeholder name the mapping lacks, a mapping key no placeholder uses, or a sequence for named placeholders (and a mapping for ?). Python raises a RuntimeError for these, and a TypeError when params is neither a sequence nor a mapping (a str counts as neither). TypeScript rejects with an Error for these, and a TypeError when params is neither an array nor a plain object. Rust returns DbError::Params.

  • Without params, a statement with placeholders fails the same way: a placeholder is never left unbound.

  • Waits for the batch being applied. The watcher's changes are applied in the background, a batch at a time, and foreign keys are checked once per batch. A query that arrives while a batch is being applied waits for it and is answered as soon as it ends; otherwise it is answered at once. A batch is every change that was waiting when applying started, and it is never split, so a query reads whole batches only. on_file runs once per change: with a 2 ms callback, a batch of 500 changes holds a query for about a second. Under a writer that never pauses, a slow callback lets one batch grow to thousands of changes, and a query waits as long as that batch takes. The measurements, the Linux limit of 16,384 queued changes, and what keeps the wait short are in How long a query waits.

  • Can miss a file you just wrote. A query does not wait for changes still queued or not yet reported by the operating system, so a write followed at once by a query can return the old rows: see A query can miss a file you just wrote. On Linux changes are applied in the order the kernel reports them, so a writer creating a parent file and then its child does not fail a query; macOS and Windows give no such guarantee. While a batch a foreign key rejected waits to be retried, the query raises its violation instead, unless a file's on_file is failing, whose error is raised first: see Foreign keys. A file is read when it is closed after writing or renamed into place; a query issued the moment a write returns may read the previous rows, so wait for the change on watch first. In TypeScript onFile runs on the JavaScript thread, so a change reaches the index when that thread is next free, and a busy thread lengthens the batch and the wait; the SQL runs on the threadpool. In Python on_file needs the GIL, so a busy Python thread does the same.

  • Read-only. Each statement is classified by SQLite itself (sqlite3_stmt_readonly); anything classified as a write — INSERT, UPDATE, DELETE, DROP, CREATE, ALTER, VACUUM, … — is rejected before producing rows. Rust surfaces this as DirSqlError::WriteForbidden; Python raises a RuntimeError and TypeScript rejects with an Error carrying a "read-only" message. This is the query-layer half of dirsql's broader guarantee that it never modifies your files — see Read-only by design.

  • No session state. BEGIN, COMMIT, ROLLBACK, SAVEPOINT, RELEASE, and any PRAGMA that sets a value are rejected as not authorized, so a statement returns the same rows whatever ran before it. Reading pragmas (PRAGMA table_info(t), PRAGMA cache_size) still work.

  • One statement. A query runs exactly one statement. Any SQL after it (SELECT 1; SELECT 2, or a typo after the ;) is rejected rather than dropped; a trailing ;, whitespace, or a comment is fine.

  • Internal tracking columns (_dirsql_file_path, _dirsql_row_index) are excluded from SELECT * results; name them explicitly to see them.

  • Column order. Python row dicts are keyed in the order the SELECT list names the columns. Rust and TypeScript rows do not carry that order; use queryOrdered.

  • SQLite values map back to language types:

SQLitePythonTypeScriptRust
TEXTstrstringValue::Text
INTEGERintnumberValue::Integer
REALfloatnumberValue::Real
BLOBbytesBufferValue::Blob
NULLNonenullValue::Null

An INTEGER outside JavaScript's safe integer range (magnitude greater than Number.MAX_SAFE_INTEGER, 2^53 − 1) cannot be represented as a JS number without precision loss, so the TypeScript SDK throws rather than return a rounded value. Python's int is unbounded, so it always round-trips.

queryOrdered ​

typescript
await db.queryOrdered(sql: string, params?: unknown[] | Record<string, unknown> | null) -> { columns: string[]; rows: Record<string, unknown>[] }
rust
db.query_ordered(sql: &str) -> Result<QueryResult>   // QueryResult { columns, rows }
db.query_ordered_with(sql: &str, params: &Params) -> Result<QueryResult>

Runs query and also returns the column names in the order the SELECT list names them. Use it to render columns in the order the caller asked for: a row's keys do not carry that order. In TypeScript, object keys that look like integers always enumerate first.

watch ​

python
async for event in db.watch():   # AsyncIterator[RowEvent]
    ...
typescript
for await (const event of db.watch()) {   // AsyncIterable<RowEvent>
    ...
}
rust
use futures::StreamExt;
let mut stream = db.watch()?;   // WatchStream: impl Stream<Item = RowEvent>
while let Some(event) = stream.next().await { ... }

Returns a stream of RowEvents reflecting filesystem changes under the root. The stream never terminates on its own; stop consuming it to stop.

The watcher is live once both watch() and ready have returned, in either order (a Rust DirSQL is ready when build() returns); a file written after that point produces an event and reaches the index. A stream reports the changes applied after it attaches, once each; a change applied before reaches no stream, and a write made while the initial scan is still running reaches the index but may be missed by the stream.

An instance with no named tables cannot be watched: path-tables emit no watch events, so watch() (and the polling primitives) raise an error naming that cause. Define a table first.

Python/TypeScript: each watch() call returns an independent stream, and every stream on one instance receives every event observed after it was created, so several consumers (one per client, say) can share one instance.

Rust: watch() may be called once per instance and is mutually exclusive with the polling API below — mixing them returns an error.

Low-level watch primitives ​

typescript
await db.startWatcher()               // optional, idempotent; pollEvents attaches on its own
await db.pollEvents(timeoutMs) -> RowEvent[]
rust
db.poll_events(timeout: Duration) -> Result<Vec<RowEvent>>

The first pollEvents / poll_events call attaches; each call returns the row events applied since the previous one, blocking up to the timeout for the first and returning an empty batch if none arrives. Safe to call in a loop. TypeScript runs the poll on the libuv threadpool. Python does not expose public polling primitives — use watch().

AsyncDirSQL (Rust) ​

Rust's non-blocking wrapper: the constructor returns immediately while the scan runs on a background thread. Python and TypeScript need no equivalent — their DirSQL is already async-by-default.

rust
let db = DirSQL::builder().root("./data").tables(tables).build_async()?; // -> AsyncDirSQL
db.ready().await?;                        // required before use
let rows = db.query("SELECT ...").await?; // spawn_blocking under the hood
let stream = db.watch()?;                 // same WatchStream as DirSQL
let sync: DirSQL = db.sync()?;            // unwrap the inner sync handle

Shortcuts mirror DirSQL: AsyncDirSQL::new, with_ignore, from_config_path. Unlike the Python/TypeScript SDKs, methods called before ready().await completes return a "not ready" error rather than waiting — an intentional, language-idiomatic difference.

Table ​

python
Table(*, name: str, ddl: str, glob: str, on_file: Callable[[str], list[dict]], strict: bool = False)
typescript
new Table({ name, ddl, glob, onFile, strict? })
// or a plain object — TableDef and Table are interchangeable:
{ name: string, ddl: string, glob: string, onFile: (path: string) => Record<string, unknown>[], strict?: boolean }
rust
Table::new(name, ddl, glob, on_file)      // on_file: Fn(&str) -> Vec<Row>, infallible
Table::try_new(name, ddl, glob, on_file)  // on_file: Fn(&str) -> Result<Vec<Row>, _>
Table::strict(name, ddl, glob, on_file)   // Table::new with strict = true

Maps files to table rows.

  • name — The table's SQL name, declared rather than derived from ddl. Must be unique across all tables. Building fails if the ddl creates no table by this name (checked against SQLite's catalog, before any file is indexed).
  • ddl — A SQLite CREATE TABLE statement, run verbatim.
  • glob — Glob pattern matched against root-relative paths. Every table whose glob matches a file receives that file's rows — a file can populate multiple tables. A {name} segment is rewritten to * (it matches one path segment but captures nothing).
  • on_file (on_file / onFile) — Callback receiving the matched file's full path (the root joined with the file's relative path — absolute when root is absolute) and returning the rows that file contributes. Required. A row's columns are exactly what the callback returns, narrowed to the DDL — dirsql injects nothing (see Columns). A callback that wants the path, stat metadata, or file body computes it from the path it receives; dirsql never reads file contents itself. Return an empty list to skip a file. A callback that raises, or returns a row the table rejects (a strict key mismatch, NOT NULL, UNIQUE, CHECK), fails ready and every query with an error naming the table, the file and the original error; no file is skipped. Have writers write to a temporary file in the same directory and rename it into place, so a reader sees only the old content or the new.
  • strict — Default off: extra row keys are dropped and missing declared columns become NULL. When on, any extra or missing key is an error (see strict).

Readable attributes: ddl, glob (all SDKs), plus strict.

RowEvent ​

Emitted by watch and the polling primitives; one event per row-level change caused by a filesystem event.

FieldPythonRustTypeScript
Actionaction: strenum variant (Insert / Update / Delete / Error)action: string
Tabletable: str | Nonetable: String (Option<String> on Error)table: string | null
New/current rowrow: dict | Nonerow / new_row on the variantrow?: Record | null
Previous rowold_row: dict | Noneold_row on UpdateoldRow?: Record | null
Error messageerror: str | Noneerror: String on Errorerror?: string | null
File pathfile_path: str | Nonefile_path (String; PathBuf on Error)filePath?: string | null

Actions: insert (new row; no previous), update (row changed in place; carries both old and new), delete (row removed), error (an extraction or watch failure — carries error, never row/old_row; table is None/null when the failure isn't tied to one table). file_path is relative to the root. Errors never terminate the stream.

Value types ​

An on_file callback (and the SQLite columns it feeds) accepts:

SQLite columnPython valueTypeScript valueRust Value
TEXTstrstringValue::Text(String)
INTEGERint, boolnumber, boolean, bigintValue::Integer(i64)
REALfloatnumberValue::Real(f64)
BLOBbytesBuffer / Uint8ArrayValue::Blob(Vec<u8>)
NULLNonenullValue::Null

Integers are stored as a signed 64-bit Value::Integer. A value that does not fit that range is a hard error, not a lossy conversion: Python raises OverflowError, and a TypeScript bigint outside the i64 range throws. A TypeScript bigint within i64 range maps to INTEGER. Only a real bytes/bytearray (Python) or Buffer/Uint8Array (TypeScript) maps to BLOB — a list/array of integers does not.

In Python, any other type (Decimal, Fraction, a numpy scalar other than numpy.float64, dict, list, memoryview, ...) raises TypeError naming the column and the type. It is never coerced to REAL or stored as its repr; convert it first (int(x), float(x), str(x), json.dumps(x)).

In TypeScript, undefined stores NULL; any other type (a plain object, Array, Date, Map, a function, a symbol, a typed array other than Uint8Array, ...) fails the build with an error naming the column and the type. It is never stored as its String() form; convert it first (JSON.stringify(x), d.toISOString()).

Progress on construction ​

Constructing a DirSQL walks the tree and ingests every matched file, which on a large corpus is the slowest thing your program does. The core reports both phases on stderr while they run, then erases the live line and leaves one summary of what each cost:

dirsql: indexed 41231 files in 3m12s

A query that calls a worker-backed function is reported the same way, as a running count of round trips (dirsql: running 9204 worker calls) — the shape dirsql-plugin-embeddings produces when it embeds a corpus one row at a time.

This is terminal-only by default, and only for a phase that runs longer than half a second — a program whose stderr is a pipe, a file, or a log collector sees nothing, whatever the scan costs. Set DIRSQL_PROGRESS=never to guarantee silence even on a terminal, or DIRSQL_PROGRESS=always to report regardless. The full table is in the CLI reference; the setting lives in the shared core, so it behaves identically from all three SDKs.

Released under the MIT License.