db/fact_disk_storage.ts

Filesystem CAS for fact bytes stored outside Postgres — the disk half of PgFactStore, threaded over the injectable runtime/*Deps rather than raw node:fs, so it runs unchanged under Node, Deno, and a mock runtime.

Large facts (over the embedded threshold) live on disk at the canonical sharded layout <facts_dir>/<shard>/<rest> — <shard> is the first 2 hex chars of the blake3 digest, <rest> the remaining 62 — with the fact row carrying external_url = file:<shard>/<rest> (disk-root-relative). The layout is single-sourced by fact_disk_path in db/file_fact_url.ts, so the write path here and the URL minted into the row can't drift. The TS twin of the Rust fuz_fact disk CAS.

Writes land through <facts_dir>/.tmp/<rand>.tmp, are fsynced, then renamed into the content-addressed final path. The rename is atomic on POSIX (a *concurrent reader* observing the path sees either the full content or nothing), but atomicity is not durability — the fsync before the rename is what guards against a *host crash* leaving a torn/zero file at a published CAS path, because the serving path streams the hash-named file without re-hashing it (server/serve_fact_route.ts). This twins the Rust fuz_fact §fsync posture: data-sync before the rename; the parent-dir fsync stays deliberately waived (a lost dirent is regenerable under content addressing). If the final path already exists the temp is dropped instead of renamed over — idempotent dedup (same hash → byte-identical content), mirroring the Rust commit path. .tmp/ is a sibling of <shard>/ under the same facts_dir so rename is always same-filesystem (no EXDEV).

view source

Declarations
#

9 declarations

FACT_TMP_DIRNAME
#

db/fact_disk_storage.ts view source

".tmp" import {FACT_TMP_DIRNAME} from '@fuzdev/fuz_app/db/fact_disk_storage.js';

Subdirectory under facts_dir for in-flight atomic temp files.

FACT_TMP_ORPHAN_MAX_AGE_MS
#

db/fact_disk_storage.ts view source

number import {FACT_TMP_ORPHAN_MAX_AGE_MS} from '@fuzdev/fuz_app/db/fact_disk_storage.js';

Default age (1 hour) past which a .tmp/* file is considered orphaned.

FactDiskStorageDeps
#

db/fact_disk_storage.ts view source

FactDiskStorageDeps

type Pick<FsReadDeps, "stat" | "readdir" | "read_file"> & Pick<FsWriteDeps, "mkdir" | "rename" | "write_file" | "fsync"> & Pick<FsStreamDeps, "write_file_stream"> & Pick<FsRemoveDeps, "remove">

import type {FactDiskStorageDeps} from '@fuzdev/fuz_app/db/fact_disk_storage.js';

Filesystem capabilities the disk CAS needs, drawn from runtime/deps.ts. A full RuntimeDeps (Node or Deno) satisfies this; each function below picks the narrow subset it actually uses.

stat

Get file/directory stats, or null if path doesn't exist.

type (path: string): Promise<StatResult | null>

path

type string
returns Promise<StatResult | null>

readdir

List directory entries (names, not full paths). Throws if the directory does not exist.

type (path: string): Promise<string[]>

path

type string
returns Promise<string[]>

read_file

Read a file as bytes. Throws if the file does not exist.

type (path: string): Promise<Uint8Array<ArrayBufferLike>>

path

type string
returns Promise<Uint8Array<ArrayBufferLike>>

mkdir

Create a directory. mode applies at creation (no-op where modes don't apply).

type (path: string, options?: { recursive?: boolean | undefined; mode?: number | undefined; } | undefined): Promise<void>

path

type string

options?

type { recursive?: boolean | undefined; mode?: number | undefined; }
optional
returns Promise<void>

rename

Rename (move) a file.

type (old_path: string, new_path: string): Promise<void>

old_path

type string

new_path

type string
returns Promise<void>

write_file

Write bytes to a file.

type (path: string, data: Uint8Array<ArrayBufferLike>): Promise<void>

path

type string

data

type Uint8Array<ArrayBufferLike>
returns Promise<void>

fsync

Flush a file's data to stable storage (fsync). Call on a temp file after writing it and *before* rename-ing it into place when the renamed path is later served without re-verification — otherwise a host crash after the rename can surface a torn/zero file as authentic content. The fact disk CAS (db/fact_disk_storage.ts) is the one such path; it twins the Rust fuz_fact §fsync posture (data-sync before rename; the parent-dir fsync stays deliberately waived — a lost dirent is regenerable under content addressing). Real runtimes open the path, fsync, and close; create_mock_runtime no-ops (it models no durability).

type (path: string): Promise<void>

path

type string
returns Promise<void>

write_file_stream

Write a ReadableStream of bytes to a file, consuming it with backpressure (peak memory is one chunk). Creates or truncates path. Throws on any I/O error; a partially-written file may remain, so callers needing atomicity write to a temp path then rename.

type (path: string, data: ReadableStream<Uint8Array<ArrayBufferLike>>): Promise<void>

path

type string

data

type ReadableStream<Uint8Array<ArrayBufferLike>>
returns Promise<void>

remove

Remove a file or directory.

type (path: string, options?: { recursive?: boolean | undefined; } | undefined): Promise<void>

path

type string

options?

type { recursive?: boolean | undefined; }
optional
returns Promise<void>

read_fact_bytes_from_disk
#

db/fact_disk_storage.ts view source

(deps: Pick<FactDiskStorageDeps, "read_file">, facts_dir: string, url: string): Promise<Uint8Array<ArrayBufferLike>> import {read_fact_bytes_from_disk} from '@fuzdev/fuz_app/db/fact_disk_storage.js';

Read a disk-backed fact's bytes from the <facts_dir>/<shard>/<rest> layout the writers above produce, addressed by the row's file:<shard>/<rest> external_url. The only read of external bytes PgFactStore.get performs. Does NOT verify hash content — PgFactStore.get calls fact_hash_verify(hash, bytes) on the result and returns null on mismatch.

Defense at the read seam is the FILE_FACT_URL_PATTERN regex (via parse_file_fact_url) — .. segments, foreign schemes, and non-hex chars fail before any disk access.

deps

type Pick<FactDiskStorageDeps, "read_file">

facts_dir

type string

url

type string

returns

Promise<Uint8Array<ArrayBufferLike>>

throws

  • Error - when `url` is not a `file:<shard>/<rest>` URL, or when the read fails

stream_fact_to_disk
#

db/fact_disk_storage.ts view source

(deps: Pick<FactDiskStorageDeps, "stat" | "mkdir" | "rename" | "fsync" | "write_file_stream" | "remove">, facts_dir: string | undefined, source: ReadableStream<...>, max_bytes: number, embedded_threshold: number): Promise<...> import {stream_fact_to_disk} from '@fuzdev/fuz_app/db/fact_disk_storage.js';

Stream source to storage with bounded memory: hash BLAKE3 + SHA-256 incrementally in one pass, buffer in memory until the bytes cross embedded_threshold, then spill the buffer + remaining chunks through a temp file and atomically land it in the disk CAS. Peak heap is O(chunk + embedded_threshold), never O(artifact), so a multi-GB upload never buffers in RAM.

  • Embedded vs disk. A body <= embedded_threshold stays in memory and is returned as {kind: 'embedded'} for the PG bytes column. Above it (with a facts_dir), the buffer + remaining chunks spill to <facts_dir>/.tmp/…, then rename into <facts_dir>/<shard>/<rest> once the hash is known — {kind: 'disk'}. A body over the threshold with facts_dir === undefined throws FactPayloadTooLargeError (matches PgFactStore.put).
  • Cap enforcement. Aborts with FactPayloadTooLargeError the moment the running byte count passes max_bytes — the mid-stream backstop for a chunked or mis-declared Content-Length.
  • Disk-full. An ENOSPC from the temp-file write surfaces as FactStorageFullError.

deps

type Pick<FactDiskStorageDeps, "stat" | "mkdir" | "rename" | "fsync" | "write_file_stream" | "remove">

facts_dir

type string | undefined

source

type ReadableStream<Uint8Array<ArrayBufferLike>>

max_bytes

type number

embedded_threshold

type number

returns

Promise<StreamFactToDiskResult>

mutates

  • facts_dir filesystem —

StreamFactToDiskResult
#

db/fact_disk_storage.ts view source

StreamFactToDiskResult import type {StreamFactToDiskResult} from '@fuzdev/fuz_app/db/fact_disk_storage.js';

Outcome of streaming an upload to storage: the blake3:-prefixed fact hash, the bare-hex SHA-256, the byte count, and where the bytes landed. PgFactStore.put_stream turns this into the fact row insert.

hash

type FactHash

sha256

type string

size

type number

placement

type StreamPlacement

StreamPlacement
#

db/fact_disk_storage.ts view source

StreamPlacement

type { kind: "embedded"; bytes: Uint8Array<ArrayBufferLike>; } | { kind: "disk"; external_url: string & $brand<"FileFactUrl">; }

import type {StreamPlacement} from '@fuzdev/fuz_app/db/fact_disk_storage.js';

Where a streamed body landed — embedded carries the in-memory bytes (under the embedded threshold, bound for the PG fact.bytes column); disk means the bytes are already at <facts_dir>/<shard>/<rest> and the row carries the file: URL.

sweep_orphan_temps
#

db/fact_disk_storage.ts view source

(deps: Pick<FactDiskStorageDeps, "stat" | "readdir" | "remove">, facts_dir: string, options?: { max_age_ms?: number | undefined; log?: Pick<Logger, "warn"> | undefined; } | undefined): Promise<...> import {sweep_orphan_temps} from '@fuzdev/fuz_app/db/fact_disk_storage.js';

Reap stale temp files left under <facts_dir>/.tmp/ by a hard crash (SIGKILL / OOM / host crash) mid-write — the finally cleanup in the writers above never ran. Removes .tmp entries whose mtime is older than max_age_ms (so an in-flight upload isn't yanked out from under itself). The TS twin of the Rust sweep_orphan_temps; call on startup + on an interval.

Best-effort: a missing .tmp/ dir (no oversize upload has ever run) is a no-op; a runtime that doesn't report mtime_ms (a mock) leaves every temp untouched; a per-file stat/remove failure is logged and skipped rather than aborting the sweep. Returns the count removed.

deps

type Pick<FactDiskStorageDeps, "stat" | "readdir" | "remove">

facts_dir

type string

options?

type { max_age_ms?: number | undefined; log?: Pick<Logger, "warn"> | undefined; }
optional

returns

Promise<number>

mutates

  • facts_dir filesystem —

write_fact_bytes_to_disk
#

db/fact_disk_storage.ts view source

(deps: Pick<FactDiskStorageDeps, "stat" | "mkdir" | "rename" | "write_file" | "fsync" | "remove">, facts_dir: string, hash: FactHash, bytes: Uint8Array<...>): Promise<...> import {write_fact_bytes_to_disk} from '@fuzdev/fuz_app/db/fact_disk_storage.js';

Write fully-buffered bytes for hash to the canonical <facts_dir>/<shard>/<rest> path, then publish via commit_temp_to_cas (fsync'd temp + atomic rename, dedup-aware). The buffering twin of stream_fact_to_disk, used by PgFactStore.put for oversize sync bytes. Returns the file: external_url for the fact row.

deps

type Pick<FactDiskStorageDeps, "stat" | "mkdir" | "rename" | "write_file" | "fsync" | "remove">

facts_dir

type string

hash

type FactHash

bytes

type Uint8Array<ArrayBufferLike>

returns

Promise<FileFactUrl>

mutates

  • facts_dir filesystem —

Depends on
#

Imported by
#