SDK
dirsql ships as a library for Python (PyPI: dirsql), TypeScript (npm: dirsql), and Rust (crates.io: dirsql). All three are thin bindings over one shared Rust core, so behavior is identical; only the calling conventions differ.
Install
uv add dirsqlpnpm add dirsqlcargo add dirsqlImport
from dirsql import DirSQL, Table, RowEventimport { DirSQL, Table } from "dirsql";
import type { TableDef, RowEvent, ExtensionSpec, DirSQLOptions } from "dirsql";use dirsql::{DirSQL, AsyncDirSQL, Table, Row, RowEvent, Value, Extension, DirSqlError};DirSQL
Constructor
DirSQL(
root: str | None = None,
*,
tables: list[Table] | None = None,
ignore: list[str] | None = None,
no_ignore: bool = False,
config: str | list[str] | None = None,
persist: bool = False,
persist_path: str | None = None,
extensions: list[dict] | None = None, # [{ "path": str, "entrypoint"?: str }]
)new DirSQL(configPath: string)
// or
new DirSQL({
root?: string,
tables?: TableDef[],
ignore?: string[],
noIgnore?: boolean,
config?: string | string[],
persist?: boolean,
persistPath?: string,
extensions?: ExtensionSpec[], // [{ path: string, entrypoint?: string }]
})DirSQL::builder()
.root(root) // optional
.tables(tables) // optional; append one with .table(t)
.ignore(patterns) // optional
.no_ignore(true) // optional; default false
.config(config_toml_path) // optional; repeatable — call again to
// merge another config in call order
.persist(cache_path) // optional; Some(path), or None for the
// platform cache dir
.extensions(extensions) // optional; append one with .extension(e)
.poll_interval(duration) // optional; watch-loop cadence, default 200ms
.build() // -> Result<DirSQL> (synchronous scan)
// Shortcuts: DirSQL::new(root, tables), DirSQL::with_ignore(root, tables, ignore),
// DirSQL::from_config_path(path) // reads an explicit config fileCreates a SQLite index over a directory. The index root is the explicit root when given (a relative root resolves against the cwd once), else the process cwd — the config file's location never sets the root. A config's [[table]] globs anchor at the config file's own directory instead (see glob anchor); root governs programmatic tables and path-tables. The Python and TypeScript SDKs read the cwd when DirSQL is constructed, though the scan runs later, so a chdir after construction does not move the root.
Constructing with neither a config nor programmatic tables defines no named tables — the same as the CLI with no -c. Query the filesystem with a path-table (SELECT * FROM './'); a SELECT ... FROM files in that state fails with a hint pointing at that form. There is no implicit <root>/.dirsql.toml discovery: to read a config on disk, pass its path via config (Rust: .config(path) / DirSQL::from_config_path(path)). The root-joining DirSQL::from_config(root) shortcut was removed in #603 — use DirSQL::from_config_path(root.join(".dirsql.toml")).
Parameters:
root— Directory to index. When omitted, the index roots at the process cwd (even whenconfigis supplied).tables— ProgrammaticTabledefinitions.ignore— Glob patterns matched against root-relative paths under the glob rule (*one level,**any depth); matched files are skipped entirely (scan and watch).no_ignore/noIgnore— Scan files a.gitignorewould hide. Path-tables respect.gitignorefiles by default; the built-in.gitskip and theignoreparameter's patterns still apply. The CLI's--no-ignore.config— Path to a.dirsql.toml. Its[[table]]entries are appended after any programmatictables; itsignorepatterns and[[dirsql.extension]]entries are appended likewise. The config file does not set the index root: with no explicitroot, the index roots at the process cwd. Its[[table]]globs anchor at the config file's directory (or at the literal prefix of an absolute /~/glob), not atroot; itsignoreapplies to its own tables only. Multiple configs merge in order on every SDK — Pythonconfigaccepts astror alist[str], the Rust builder's.config()is repeatable (call it once per file), and TypeScriptconfigaccepts astringor astring[]; the configs merge in call order ([[table]]/ignore/[[dirsql.extension]]accumulate; a duplicate table name across configs errors), mirroring the CLI's repeatable-c/--config.persist— Keep the SQLite index on disk between runs (default off: ephemeral, rebuilt every startup). The cache lives in the platform cache directory by default (outside<root>); on restart, only files whose stat changed are re-parsed. (The CLI exposes the same switch as the--persist [PATH]flag; it is not a config key.)persist_path/persistPath— Override the cache location. Ignored when persistence is off. Constructor values are used as given. In Rust these two parameters collapse into a single builder method:.persist(None)enables persistence at the default location,.persist(Some(path))enables it atpath.extensions— SQLite extensions to load at startup, before any table DDL (enable → load → disable, so SQLload_extension()is never exposed). Each entry pairs a shared-librarypathwith an optionalentrypointinit symbol. In Python and TypeScript,pathmay be a bare package name, resolved from the installed package (importlib/node_modules) — for both the programmatic list and aconfigfile's[[dirsql.extension]]entries. The Rust SDK is file-path-only. Programmatic entries load first, then the config's. A relative programmatic path is used verbatim (resolved by the OS against the process working directory); config-file paths resolve against the config file's parent.
Construction is asynchronous in Python and TypeScript. The constructor returns immediately and the initial scan runs in the background; query and the other methods transparently await readiness. In Rust, .build() scans synchronously; use .build_async() → AsyncDirSQL for the non-blocking equivalent.
The scan is the only walk of the tree. A file watcher over the named tables starts before it and runs for the life of the instance; the changes it sees during the scan are applied once, before the build finishes, and later changes update the index as they arrive. See The file watcher for the difference between platforms.
Rust builder extras
Tableappears in the builder via.table(t)(append) or.tables(v)(replace)..poll_interval(Duration)sets the channel-basedwatch()loop's poll cadence (default 200 ms): lower values give tighter event latency at higher idle CPU..suppress_config_extensions(true)skips loading the config file's own[[dirsql.extension]]entries. Launcher plumbing (used with pre-resolved.extensions(...)); not needed in application code.Extension { path: PathBuf, entrypoint: Option<String> }is the extension spec type.
ready
await db.ready() -> Noneawait db.ready // awaitable property// AsyncDirSQL only; DirSQL::build() is already complete when it returns.
db.ready().await -> Result<()>Waits for the initial scan to complete, re-raising any construction error. On Linux, a tree with more directories than inotify's watch limit fails construction with the OS error ENOSPC and the message inotify watch limit reached: Python raises OSError with errno.ENOSPC, TypeScript rejects with an Error whose code is "ENOSPC", and Rust returns DirSqlError::Io whose os_error() is errno 28. To raise the limit, see The watch limit. Safe to call multiple times. In Python and TypeScript every other method awaits readiness internally, so an explicit ready is only needed to observe construction errors before issuing a query.
query
await db.query(sql: str, params: Sequence | Mapping | None = None) -> list[dict]await db.query(sql: string, params?: unknown[] | Record<string, unknown> | null) -> Record<string, unknown>[]
// Runs on the libuv threadpool; the JS event loop stays responsive.db.query(sql: &str) -> Result<Vec<Row>> // Row = HashMap<String, Value>
db.query_with(sql: &str, params: &Params) -> Result<Vec<Row>>Executes a SQL query and returns rows keyed by column name. If a result repeats a column name, later occurrences receive :N suffixes (for example, a and a:1); suffixes avoid names already present in the result.
Bind parameters. params reaches SQLite's own parameter binding; the values never enter the SQL text, so text holding quotes needs no escaping. A sequence (a TypeScript array) fills ? placeholders in order; a mapping (a TypeScript plain object) fills :name, @name and $name placeholders, keyed by the name without its prefix. It works for path-tables and declared tables alike:
await db.query(
"SELECT path FROM './*.json' WHERE content ->> 'advice' = :advice",
{"advice": text},
)
await db.query("SELECT * FROM notes WHERE author = ? AND year = ?", ("ann", 2024))await db.query(
"SELECT path FROM './*.json' WHERE content ->> 'advice' = :advice",
{ advice: text },
);
await db.query("SELECT * FROM notes WHERE author = ? AND year = ?", ["ann", 2024]);
// queryOrdered takes params the same way.use dirsql::{Params, Value};
use std::collections::HashMap;
db.query_with(
"SELECT path FROM './*.json' WHERE content ->> 'advice' = :advice",
&Params::Named(HashMap::from([("advice".into(), Value::Text(text))])),
)?;
db.query_with(
"SELECT * FROM notes WHERE author = ?",
&Params::Positional(vec![Value::Text("ann".into())]),
)?;
// query_ordered_with and query_each_with bind the same way;
// AsyncDirSQL::query_with is the async twin.Values bind as SQLite's native types. Python accepts
None,int(boolbinds as0/1),float,str,bytesandbytearray; anything else raisesTypeErrorrather than being stringified, and anintoutside the signed 64-bit range raisesOverflowError. TypeScript acceptsnull,number(an integral value binds as an INTEGER, any other as a REAL, so1.0binds as1),bigint,boolean(0/1),string, andBuffer/Uint8Array; anything else,undefinedincluded, rejects with aTypeError, and abigintoutside the signed 64-bit range with aRangeError. Rust binds aValue.The params must fit the placeholders exactly, or the query fails with an error naming the problem: a wrong count for a sequence, a placeholder name the mapping lacks, a mapping key no placeholder uses, or a sequence for named placeholders (and a mapping for
?). Python raises aRuntimeErrorfor these, and aTypeErrorwhenparamsis neither a sequence nor a mapping (astrcounts as neither). TypeScript rejects with anErrorfor these, and aTypeErrorwhenparamsis neither an array nor a plain object. Rust returnsDbError::Params.Without
params, a statement with placeholders fails the same way: a placeholder is never left unbound.Waits for the batch being applied. The watcher's changes are applied in the background, a batch at a time, and foreign keys are checked once per batch. A query that arrives while a batch is being applied waits for it and is answered as soon as it ends; otherwise it is answered at once. A batch is every change that was waiting when applying started, and it is never split, so a query reads whole batches only.
on_fileruns once per change: with a 2 ms callback, a batch of 500 changes holds a query for about a second. Under a writer that never pauses, a slow callback lets one batch grow to thousands of changes, and a query waits as long as that batch takes. The measurements, the Linux limit of 16,384 queued changes, and what keeps the wait short are in How long a query waits.Can miss a file you just wrote. A query does not wait for changes still queued or not yet reported by the operating system, so a write followed at once by a query can return the old rows: see A query can miss a file you just wrote. On Linux changes are applied in the order the kernel reports them, so a writer creating a parent file and then its child does not fail a query; macOS and Windows give no such guarantee. While a batch a foreign key rejected waits to be retried, the query raises its violation instead, unless a file's
on_fileis failing, whose error is raised first: see Foreign keys. A file is read when it is closed after writing or renamed into place; a query issued the moment a write returns may read the previous rows, so wait for the change onwatchfirst. In TypeScriptonFileruns on the JavaScript thread, so a change reaches the index when that thread is next free, and a busy thread lengthens the batch and the wait; the SQL runs on the threadpool. In Pythonon_fileneeds the GIL, so a busy Python thread does the same.Read-only. Each statement is classified by SQLite itself (
sqlite3_stmt_readonly); anything classified as a write —INSERT,UPDATE,DELETE,DROP,CREATE,ALTER,VACUUM, … — is rejected before producing rows. Rust surfaces this asDirSqlError::WriteForbidden; Python raises aRuntimeErrorand TypeScript rejects with anErrorcarrying a "read-only" message. This is the query-layer half of dirsql's broader guarantee that it never modifies your files — see Read-only by design.No session state.
BEGIN,COMMIT,ROLLBACK,SAVEPOINT,RELEASE, and anyPRAGMAthat sets a value are rejected as not authorized, so a statement returns the same rows whatever ran before it. Reading pragmas (PRAGMA table_info(t),PRAGMA cache_size) still work.One statement. A query runs exactly one statement. Any SQL after it (
SELECT 1; SELECT 2, or a typo after the;) is rejected rather than dropped; a trailing;, whitespace, or a comment is fine.Internal tracking columns (
_dirsql_file_path,_dirsql_row_index) are excluded fromSELECT *results; name them explicitly to see them.Column order. Python row dicts are keyed in the order the SELECT list names the columns. Rust and TypeScript rows do not carry that order; use
queryOrdered.SQLite values map back to language types:
| SQLite | Python | TypeScript | Rust |
|---|---|---|---|
| TEXT | str | string | Value::Text |
| INTEGER | int | number | Value::Integer |
| REAL | float | number | Value::Real |
| BLOB | bytes | Buffer | Value::Blob |
| NULL | None | null | Value::Null |
An INTEGER outside JavaScript's safe integer range (magnitude greater than Number.MAX_SAFE_INTEGER, 2^53 − 1) cannot be represented as a JS number without precision loss, so the TypeScript SDK throws rather than return a rounded value. Python's int is unbounded, so it always round-trips.
queryOrdered
await db.queryOrdered(sql: string, params?: unknown[] | Record<string, unknown> | null) -> { columns: string[]; rows: Record<string, unknown>[] }db.query_ordered(sql: &str) -> Result<QueryResult> // QueryResult { columns, rows }
db.query_ordered_with(sql: &str, params: &Params) -> Result<QueryResult>Runs query and also returns the column names in the order the SELECT list names them. Use it to render columns in the order the caller asked for: a row's keys do not carry that order. In TypeScript, object keys that look like integers always enumerate first.
watch
async for event in db.watch(): # AsyncIterator[RowEvent]
...for await (const event of db.watch()) { // AsyncIterable<RowEvent>
...
}use futures::StreamExt;
let mut stream = db.watch()?; // WatchStream: impl Stream<Item = RowEvent>
while let Some(event) = stream.next().await { ... }Returns a stream of RowEvents reflecting filesystem changes under the root. The stream never terminates on its own; stop consuming it to stop.
The watcher is live once both watch() and ready have returned, in either order (a Rust DirSQL is ready when build() returns); a file written after that point produces an event and reaches the index. A stream reports the changes applied after it attaches, once each; a change applied before reaches no stream, and a write made while the initial scan is still running reaches the index but may be missed by the stream.
An instance with no named tables cannot be watched: path-tables emit no watch events, so watch() (and the polling primitives) raise an error naming that cause. Define a table first.
Python/TypeScript: each watch() call returns an independent stream, and every stream on one instance receives every event observed after it was created, so several consumers (one per client, say) can share one instance.
Rust: watch() may be called once per instance and is mutually exclusive with the polling API below — mixing them returns an error.
Low-level watch primitives
await db.startWatcher() // optional, idempotent; pollEvents attaches on its own
await db.pollEvents(timeoutMs) -> RowEvent[]db.poll_events(timeout: Duration) -> Result<Vec<RowEvent>>The first pollEvents / poll_events call attaches; each call returns the row events applied since the previous one, blocking up to the timeout for the first and returning an empty batch if none arrives. Safe to call in a loop. TypeScript runs the poll on the libuv threadpool. Python does not expose public polling primitives — use watch().
AsyncDirSQL (Rust)
Rust's non-blocking wrapper: the constructor returns immediately while the scan runs on a background thread. Python and TypeScript need no equivalent — their DirSQL is already async-by-default.
let db = DirSQL::builder().root("./data").tables(tables).build_async()?; // -> AsyncDirSQL
db.ready().await?; // required before use
let rows = db.query("SELECT ...").await?; // spawn_blocking under the hood
let stream = db.watch()?; // same WatchStream as DirSQL
let sync: DirSQL = db.sync()?; // unwrap the inner sync handleShortcuts mirror DirSQL: AsyncDirSQL::new, with_ignore, from_config_path. Unlike the Python/TypeScript SDKs, methods called before ready().await completes return a "not ready" error rather than waiting — an intentional, language-idiomatic difference.
Table
Table(*, name: str, ddl: str, glob: str, on_file: Callable[[str], list[dict]], strict: bool = False)new Table({ name, ddl, glob, onFile, strict? })
// or a plain object — TableDef and Table are interchangeable:
{ name: string, ddl: string, glob: string, onFile: (path: string) => Record<string, unknown>[], strict?: boolean }Table::new(name, ddl, glob, on_file) // on_file: Fn(&str) -> Vec<Row>, infallible
Table::try_new(name, ddl, glob, on_file) // on_file: Fn(&str) -> Result<Vec<Row>, _>
Table::strict(name, ddl, glob, on_file) // Table::new with strict = trueMaps files to table rows.
name— The table's SQL name, declared rather than derived fromddl. Must be unique across all tables. Building fails if theddlcreates no table by this name (checked against SQLite's catalog, before any file is indexed).ddl— A SQLiteCREATE TABLEstatement, run verbatim.glob— Glob pattern matched against root-relative paths. Every table whose glob matches a file receives that file's rows — a file can populate multiple tables. A{name}segment is rewritten to*(it matches one path segment but captures nothing).on_file(on_file/onFile) — Callback receiving the matched file's full path (the root joined with the file's relative path — absolute whenrootis absolute) and returning the rows that file contributes. Required. A row's columns are exactly what the callback returns, narrowed to the DDL —dirsqlinjects nothing (see Columns). A callback that wants the path, stat metadata, or file body computes it from the path it receives;dirsqlnever reads file contents itself. Return an empty list to skip a file. A callback that raises, or returns a row the table rejects (astrictkey mismatch,NOT NULL,UNIQUE,CHECK), failsreadyand every query with an error naming the table, the file and the original error; no file is skipped. Have writers write to a temporary file in the same directory and rename it into place, so a reader sees only the old content or the new.strict— Default off: extra row keys are dropped and missing declared columns becomeNULL. When on, any extra or missing key is an error (seestrict).
Readable attributes: ddl, glob (all SDKs), plus strict.
RowEvent
Emitted by watch and the polling primitives; one event per row-level change caused by a filesystem event.
| Field | Python | Rust | TypeScript |
|---|---|---|---|
| Action | action: str | enum variant (Insert / Update / Delete / Error) | action: string |
| Table | table: str | None | table: String (Option<String> on Error) | table: string | null |
| New/current row | row: dict | None | row / new_row on the variant | row?: Record | null |
| Previous row | old_row: dict | None | old_row on Update | oldRow?: Record | null |
| Error message | error: str | None | error: String on Error | error?: string | null |
| File path | file_path: str | None | file_path (String; PathBuf on Error) | filePath?: string | null |
Actions: insert (new row; no previous), update (row changed in place; carries both old and new), delete (row removed), error (an extraction or watch failure — carries error, never row/old_row; table is None/null when the failure isn't tied to one table). file_path is relative to the root. Errors never terminate the stream.
Value types
An on_file callback (and the SQLite columns it feeds) accepts:
| SQLite column | Python value | TypeScript value | Rust Value |
|---|---|---|---|
| TEXT | str | string | Value::Text(String) |
| INTEGER | int, bool | number, boolean, bigint | Value::Integer(i64) |
| REAL | float | number | Value::Real(f64) |
| BLOB | bytes | Buffer / Uint8Array | Value::Blob(Vec<u8>) |
| NULL | None | null | Value::Null |
Integers are stored as a signed 64-bit Value::Integer. A value that does not fit that range is a hard error, not a lossy conversion: Python raises OverflowError, and a TypeScript bigint outside the i64 range throws. A TypeScript bigint within i64 range maps to INTEGER. Only a real bytes/bytearray (Python) or Buffer/Uint8Array (TypeScript) maps to BLOB — a list/array of integers does not.
In Python, any other type (Decimal, Fraction, a numpy scalar other than numpy.float64, dict, list, memoryview, ...) raises TypeError naming the column and the type. It is never coerced to REAL or stored as its repr; convert it first (int(x), float(x), str(x), json.dumps(x)).
In TypeScript, undefined stores NULL; any other type (a plain object, Array, Date, Map, a function, a symbol, a typed array other than Uint8Array, ...) fails the build with an error naming the column and the type. It is never stored as its String() form; convert it first (JSON.stringify(x), d.toISOString()).
Progress on construction
Constructing a DirSQL walks the tree and ingests every matched file, which on a large corpus is the slowest thing your program does. The core reports both phases on stderr while they run, then erases the live line and leaves one summary of what each cost:
dirsql: indexed 41231 files in 3m12sA query that calls a worker-backed function is reported the same way, as a running count of round trips (dirsql: running 9204 worker calls) — the shape dirsql-plugin-embeddings produces when it embeds a corpus one row at a time.
This is terminal-only by default, and only for a phase that runs longer than half a second — a program whose stderr is a pipe, a file, or a log collector sees nothing, whatever the scan costs. Set DIRSQL_PROGRESS=never to guarantee silence even on a terminal, or DIRSQL_PROGRESS=always to report regardless. The full table is in the CLI reference; the setting lives in the shared core, so it behaves identically from all three SDKs.