CorpusOS architecture¶

Operating principle¶

Keep the original evidence. Make every derived artifact reproducible. Make every agent answer traceable to the exact authorized source.

Source discovery        macOS · Windows · Linux · NAS mounts · synced cloud drives · Git checkouts
        │                (Phase 0: `inventory`, read-only)
Immutable raw store     original bytes · SHA-256 · locator_ref · revision · owner · ACL ref
        │
Extraction pipeline     detect → parse → OCR/ASR → layout → metadata → PII → screening → classify → quality gates
        │                └─ quarantine (no parser / failed gate)    └─ review hold (high risk)
Canonical model         document.json (schema 1.0.0) · content.md · metadata.json (operator-only)
        │
  ┌─────┼────────────────────────┬─────────────────────────┐
RAG index                    Typed datasets            Model datasets
chunks · BM25 · vectors · RRF  Parquet · DuckDB          golden queries · SFT pairs · held-out split
  └─────┴────────────────────────┴─────────────────────────┘
        │
Agent layer             retrieval router · evidence verdict · CLI · HTTP · MCP (stdio, remote with OAuth) · live ACL · audit log
Web app                 UI · /api · /docs · sign-in (Firebase sessions, proxy JWT, API keys) · storage folder · ingest jobs · review
Platform exports        audience-scoped knowledge files for ChatGPT, Claude, Copilot Studio, Bedrock, Vertex AI, OpenAI
        │
Quality loop            sample → inspect → eval → failures → regressions · scorecard (release gates)

Module map¶

Module	Responsibility
`models.py`	The canonical metadata contract (strict Pydantic, schema 1.0.0)
`schema.py`	JSON Schema export (`schemas/`) and validation of stored records
`ids.py`, `naming.py`	The three IDs; the filename convention, portability checks and collision detection
`config.py`, `layout.py`	Five YAML files with `${VAR}` interpolation and a config digest; the storage layout
`detect.py`, `parsers/`	Type detection from bytes; adapters: built-ins, Office (openpyxl, python-pptx), Docling, Tika, OCR, Whisper
`tabular.py`, `structured.py`	Typed tables (types, units, formulas); Parquet writer; sandboxed read-only SQL
`security.py`, `screening.py`	Source policies, PII, sensitivity; instruction-injection, hidden-text and invisible-character screening
`quality.py`, `classify.py`	Quality gates; domain and document type
`tokenizers.py`, `chunking.py`	Tokenizer-aware, structure-aware chunking with section IDs
`retrieval.py`	Embedders, hybrid search with reciprocal rank fusion, router, evidence sufficiency
`catalog.py`	Transactional SQLite catalog: live ACL resolution, FTS5, vectors, run accounting, reviews, failures, audit
`pipeline.py`	`Corpus`: ingest, canonicalize, hold for review, activate, quarantine, exports of state
`access.py`	Read-only agent operations: ask/search, documents, evidence, revision diff, lineage, SQL
`lifecycle.py`	Review decisions, retention purge, deletion propagation to exports
`evaluation.py`, `learning.py`, `inventory.py`, `exports.py`	Metrics and scorecard; improvement loop; Phase 0 inventory; RAG and SFT exports
`api.py`, `mcp_server.py`, `cli.py`	Interfaces

How the blueprint maps to the code¶

Blueprint	Implementation	Status
§1 three data products	`canonical/` + index · `structured/` + SQL · `evaluations/` + `exports/`	done
§2 target architecture	diagram above	done
§3 stack	Python/Pydantic/Typer, Docling, Tika, Tesseract, faster-whisper, DuckDB/PyArrow, Presidio, FastAPI, MCP	done, with the deviations below
§4 format semantics	parsers per format, including xlsx formulas/units and pptx notes/order	done; PDF/Word/images/audio need their extras
§5 naming, layout, IDs	`naming.py`, `layout.py`, `ids.py`	done
§6 metadata contract	strict schema 1.0.0, JSON Schema files, `validate`, `locator_ref` / `acl_ref`	done
§7 chunking	token-bounded structure-aware chunks, peer merge, section IDs, parent-child `get_evidence`	done
§8 retrieval and agents	router, hybrid RRF, evidence verdict, all named tools on HTTP and MCP; remote MCP with OAuth and API keys; knowledge exports per platform	done; reranking not yet
§9 quality, security, operations	review before indexing, screening, live ACL, deletion propagation, retention purge, eval metrics, scorecard	done; Ragas answer metrics not yet
§10 learning loop	`sample`, `failures`, `to-regression`, `eval`, `scorecard`	done
§11 Phase 0–4	inventory · canonical processing · retrieval · agent integration · continuous quality	built as a single-machine pilot
§11 Phase 5	packaging connectors, policies and datasets for several products and tenants	not started

Design decisions and deviations¶

SQLite catalog and vector store by default, not PostgreSQL + pgvector. One folder that works offline on any OS, with no services. FTS5 provides BM25. Vectors are stored as float32 blobs and searched exactly by cosine, which is fine for a pilot of thousands of chunks. All SQL lives in catalog.py, behind one class, so a PostgreSQL/pgvector backend is a contained change. It is the first roadmap item, because brute-force vector search does not scale.
Raw storage has a revision level (raw/<tenant>/<source>/<document-id>/<revision-id>/). Without it, a new revision would overwrite the evidence of the previous one.
revision_id is scoped to its document. It is a hash of document ID + content SHA-256. The raw content hash is kept separately in source.content_sha256. The blueprint's sha256:<hash> example would make two documents with identical bytes share a revision.
Policy is live; content is versioned. ACL, owner, training permission and retention are applied from the catalog on every query and overlaid on get_document. They are excluded from the config digest, so changing them does not reprocess the corpus. document.json keeps the policy as it was at processing time.
Idempotency comes from content. A revision is skipped only if its bytes, PIPELINE_VERSION and the config digest all match. Bump PIPELINE_VERSION when code changes alter derived artifacts. Two kinds of revision are always retried: those quarantined for lack of a parser, and those whose document reappears.
Review means "not indexed". A held revision is fully processed and written to canonical/, with its datasets on disk, but nothing is registered in any index until a person approves it. A newer revision that arrives during review supersedes the held one.
Tables are data, not prose. The index receives only a description of each table (columns, types, units, formulas, a few sample rows) that points to the SQL dataset.
The router is rules, not a model. Its decisions are explainable and testable. It suggests a strategy and always returns the evidence it found. The evidence verdict is a conservative heuristic (query-term coverage, plus exact identifiers that must be present), with a configurable threshold (retrieval.min_term_coverage).
Screening is heuristic. Instruction-like text, hidden HTML and invisible characters are caught by patterns. Agents must still treat every passage as untrusted data, which is why every hit is labelled untrusted_source_data and the MCP server's instructions say so.
Tokenizer. The default estimate-v1 tokenizer needs no download. Set tokenizer: tiktoken:o200k_base (needs the tokenizers extra) for exact counts. If an encoding cannot be loaded, the fallback is recorded on every chunk and reported as a run warning.
Remote MCP is a resource server only. It verifies tokens from an external identity provider and publishes Protected Resource Metadata; it never issues tokens or runs a sign-in flow. Tokens resolve to principals per request, and the ACL is checked live per call as over stdio. The transport is stateless (stateless_http, JSON responses), so replicas need no session affinity. API keys exist for clients that cannot do OAuth. They are stored as SHA-256 hashes and bypass no check.
Knowledge exports are scoped to an audience, not to a user. Uploaded files lose per-user access control, so an export takes only what everyone in the audience may read, below a classification ceiling and without risk flags. Limits differ by platform and change often. They live in one table in knowledge.py, with their source pages, and can be overridden from the CLI. Splitting and bundling count tokens with the estimate tokenizer, with a 10% margin. Deletions and revocations withdraw exported files (withdrawn.json) instead of rewriting bundles: a bundle that holds a withdrawn document is withdrawn whole, and the next export rebuilds it.
The web app delegates identity and serialises writes. Sign-in comes from Firebase Auth / Identity Platform or a proxy's signed JWT; CorpusOS never stores passwords. An identity is only an email: reads go through the same live-ACL checks as the CLI and MCP, and the admin flag adds upload, delete, sync and review, never extra reads. Ingest runs on one background worker with its own catalog connection, and review decisions take the same writer lock, so SQLite always has one writer. Both ways in (copying files into storage, and uploading through the API) land in one storage folder that a sync ingests, so deletion propagation is the ordinary "file left its source" path.
Network storage, local catalog. The corpus tree is write-once files and can live on object storage mounts; SQLite cannot. CORPUSOS_CATALOG_PATH keeps the catalog on local disk, restored from the tree on start and snapshotted (SQLite backup API) after every write, which is what makes a single Cloud Run instance on a bucket restartable.

Roadmap¶

Next	Scope
Retrieval scale	PostgreSQL + pgvector catalog backend; ANN indexes; optional cross-encoder reranking of fused candidates
Chunking	Docling `HybridChunker` as an alternative profile for Docling-parsed documents; compare profiles with `eval`
Evaluation	Ragas answer metrics (faithfulness, context precision/recall) over agent answers; per-format parser benchmarks for scans and media
Sources	Connectors for Google Drive, OneDrive, S3, IMAP and Git history; MinIO/S3 storage with the same logical layout
Versioning	DVC (`dvc add raw structured exports`) for portable dataset snapshots
Operations	Background workers only when the workload justifies them; no Kubernetes or dedicated vector cluster until then
Phase 5	Package connectors, policies, agents and datasets for reuse across Yarlis products and tenants