Storage and ingestion

Files get into the corpus in two ways. Both land in the same storage folder: web.storage_dir, or CORPUSOS_STORAGE_DIR, by default <corpus>/storage.

storage/
├── hr/                    ← a source: everything in it is ingested as source "hr"
│   ├── policies/handbook.pdf
│   └── notes.md
└── finance/
    └── budget-2026.xlsx

Way in	How
Directly	Copy files into storage/<source>/: with cp or rsync, from a synced drive, or into a bucket mounted there (gcloud storage cp, aws s3 cp). Then press Sync in the web app, call POST /api/sync, or set web.sync_interval_minutes.
Through the API or the web app	POST /api/storage/<source> with the files (admins), or the Storage tab. The sync starts by itself.

A sync runs the full pipeline on each source folder:

• Unchanged files are a no-op.
• Changed files get a new revision.
• Files no longer in the folder are deleted from search, SQL, document lookups and exports, along with any archive members or attachments inside them.

Source names are lower-case slugs, such as team-docs. Paths can't contain .. or hidden segments. Uploads are limited to web.max_upload_mb per file.

Access per source

New documents get security.defaults (owner and access list). Override per tenant or per source:

defaults: {acl: [admin@example.com], owner: admin@example.com}
sources:
  personal/hr: {acl: [admin@example.com, hr-team]}
  personal/handbook: {acl: ["*"]}           # everyone who can sign in

Access changes take effect on the next sync, without reprocessing. They also reach files inside archives and email attachments.

Running on network storage

The corpus tree (raw bytes, canonical documents, tables, chunks) can live on network storage, such as a Cloud Storage FUSE, NFS or SMB mount. The SQLite catalog can't, because it needs real file locks. Set CORPUSOS_CATALOG_PATH to a path on local disk:

• On start, the catalog is restored from <corpus>/.corpusos/catalog.db when the local copy is missing.
• After every ingest, approval and rejection, a consistent snapshot is written back.

Run one writer at a time against a corpus: for example a single instance of the web app.