Yarlis CorpusOS¶

Universal data preparation platform for AI agents. macOS · Windows · Linux · open source (Apache-2.0) · cloud-agnostic.

CorpusOS discovers, classifies, extracts, normalizes, validates, versions and indexes your files, whatever their format, and turns them into trusted, searchable, permission-aware, traceable knowledge that any authorized LLM or agent can consume.

It is not a "convert everything to Markdown" tool. It produces three separate data products:

Product	What it is for	Where it lives
Knowledge corpus (RAG)	Answering questions from source-grounded passages with citations and access control	`canonical/`, `chunks/`, lexical + vector index
Structured data (SQL)	Exact numbers: typed tables with units and formulas kept	`structured/<dataset>/<version>/data.parquet`
Model datasets (SFT / eval)	Measuring and improving model behaviour, not storing changing facts	`evaluations/`, `exports/training/`, `exports/evaluation/`

A vector database is an index, not the source of truth. Fine-tuning is not a replacement for a governed knowledge repository.

Quick start¶

uv sync                                               # core + SQL + API + MCP + Office readers + tests
uv run corpusos inventory ~/Documents/projects        # Phase 0: read-only, writes nothing
uv run corpusos init corpus --name "My Knowledge" && cd corpus
uv run corpusos ingest ~/Documents/projects --source projects
uv run corpusos review list                           # documents held before indexing
uv run corpusos search "leave policy"                 # routed search with an evidence verdict
uv run corpusos sql 'SELECT Item, "Total (USD)" FROM personal_projects_finance_budget_budget'
uv run corpusos scorecard                             # release gates
uv run corpusos export knowledge -t chatgpt -a owner  # upload-ready files for an agent platform
uv run corpusos web                                   # the web app on http://127.0.0.1:8080

Running ingest again over unchanged files is a no-op. A changed file gets a new revision, and the earlier raw bytes stay where they were. A deleted file leaves the index, SQL, document lookups and exports immediately, together with the archive members and attachments inside it.

What happens to a file¶

discover → store raw bytes (immutable, SHA-256) → detect type from bytes → parse with the right adapter
→ PII scan / redact → screen for injected instructions and hidden text → classify → quality gates
→ canonical document (schema 1.0.0) → typed datasets → [human review if high risk] → chunks → embeddings → index

Quarantine. A file that fails a quality gate or has no installed parser goes to quarantine/ with the reason. Files that had no parser are retried on every run, so installing the suggested extra is enough to recover them.
Review. A document that is restricted (SSNs, card numbers, IBANs), contains instruction-like text ("ignore previous instructions…"), hides text from readers, or carries invisible characters is stored but not indexed until a person approves it. Configure the triggers in quality-gates.yaml under review.required_for.

Input	Treatment	Output
Markdown / text	Heading hierarchy, code fences, pipe tables, line provenance; legacy encodings detected	Structured JSON + Markdown
CSV / TSV	Typed columns (leading-zero IDs stay strings, `NA` is not null), units from headers	Parquet + `schema.json` + table description
Excel (`.xlsx`)	One dataset per sheet, cached values, column formulas, units, sheet/row provenance	Parquet per sheet
PowerPoint (`.pptx`)	Slide order, titles, bullets, tables, charts, speaker notes, slide provenance	Slide-level records
JSON / JSONL / logs	Field types and nesting kept; logs become timestamp/level/message events	Records + Parquet
HTML / XML	Hierarchy, lists, tables, links; scripts, nav and hidden text removed (hidden text reported); XML entities rejected	Structural records
Email (`.eml`)	Sender, date (as effective date), thread ID; attachments processed as child documents	Thread-aware messages
Source code	Python by AST symbol + imports; other languages by symbol boundaries	Symbol-aware chunks
ZIP / TAR	Safe extraction (no traversal, links or bombs), nested discovery	Manifest + child documents
PDF / Word / images	Docling (layout, tables, page + bbox provenance), Tika fallback, Tesseract OCR with a confidence gate	Structured JSON + Markdown
Audio / video	faster-whisper, timestamped segments	Timestamped transcript

corpusos parsers shows which engines are installed. Add the heavier ones with extras: uv sync --extra docling --extra ocr --extra media --extra tika --extra pii --extra tokenizers --extra embed.

Canonical metadata contract¶

Every document, whatever its format, is one document.json that conforms to a versioned, strict schema: schemas/document.schema.json (plus chunk.schema.json). Unknown fields are rejected, and corpusos validate re-checks every stored record.

Section	Contents
identity	`schema_version`, `document_id`, `revision_id`, `corpus_id`
`source`	connector, `locator_ref`, display path, original filename, MIME type, size, SHA-256, parent document
`security`	tenant, owner, classification, `acl_ref`, `training_allowed`, `retention_policy_id`, PII counts, risk flags
`processing`	parser and version, pipeline version, config digest, run, status
`content`	title, language, domain, type, blocks with `section_path` and provenance (page / slide / sheet / line / time), tables, media
`quality`	status, warnings, failures, checks, extraction confidence (which says nothing about whether the content is true)
`lifecycle`	active · superseded · pending_review · quarantined · rejected · deleted
`time`	source modified, ingested, effective date (only when the source states it)
`retrieval`	chunk profile and version, tokenizer, embedding model, index version

Agent-facing outputs never contain absolute source locators or ACL principal lists, only locator_ref and acl_ref. The absolute path stays in the catalog and in metadata.json for operators.

Retrieval for agents¶

search (CLI), GET /search (HTTP) and search_corpus (MCP) route each question:

Question	Route
"Explain our authentication architecture"	hybrid: BM25 + vectors fused with reciprocal rank fusion
"Find error code AUTH-1042"	keyword; the evidence is insufficient unless a passage contains `AUTH-1042`
"What was revenue in Q2?"	sql: lists the readable typed tables to query
"What changed between revisions?"	compare_revisions: block-level diff
"Show the architecture diagram"	assets: media references of matching documents
"Which source supports this?"	provenance: lineage of the top passage

Each response carries evidence_status: sufficient | insufficient, so an agent can answer "insufficient evidence" instead of inventing an answer. Passages are marked trust: untrusted_source_data.

Chunks are structure-aware and tokenizer-aware (default 512 tokens, minimum 64). They never mix sibling sections, and a short parent introduction is merged into its first subsection. Each chunk records its tokenizer and its section_id, so get_evidence can expand a precise hit to its parent section or its neighbours.

Interface	Tools
CLI	`search`, `show`, `evidence`, `compare`, `lineage`, `datasets`, `sql`
HTTP (`corpusos serve`)	`/search`, `/documents/{id}`, `/evidence/{chunk}`, `/documents/{id}/compare`, `/lineage/{id}`, `/tables`, `POST /sql`
MCP (`corpusos mcp -p alice`, or `--transport http` for remote clients)	`search_corpus`, `get_document`, `get_evidence`, `list_tables`, `query_table`, `compare_revisions`, `get_lineage`, plus `search` and `fetch` in the shape ChatGPT connectors expect

Every read interface is read-only; only admins write, through the web app's storage, sync and review endpoints. The caller's principal is checked against the live ACL on every call, and every call is audited. The stdio MCP principal is fixed when the server starts; the remote MCP principal comes from a verified OAuth token or API key on each request (see Agent platforms); the web app's principal comes from its sign-in (see Web app); and corpusos serve takes the principal from the X-CorpusOS-Principal header, so put an authenticating proxy in front of it before exposing it. SQL runs in an in-memory DuckDB that holds only the tables the principal may read, with file and network access disabled.

{ "mcpServers": { "corpusos": { "command": "uv", "args": ["run", "--directory", "/path/to/CorpusOS", "corpusos", "mcp", "-C", "/path/to/corpus", "-p", "owner"] } } }

Embeddings are off by default, which leaves search lexical only. In corpus.yaml, set embeddings.backend to sentence-transformers (needs the embed extra) or ollama (a local server), then run corpusos reindex. The hash backend is an offline, non-semantic baseline for testing.

Web app¶

corpusos web serves a browser UI at /, the HTTP API at /api (interactive reference at /api/reference) and the docs site at /docs/. People sign in, search with citations, browse the documents they may read, and run SQL over typed tables. Admins also upload files, sync storage, watch ingest jobs and work the review queue.

Sign-in (web.auth in security.yaml): firebase uses Firebase Auth or Identity Platform, with HttpOnly session cookies verified on every request, including the revocation check. jwt uses an authenticating proxy such as Google IAP or oauth2-proxy. header is for loopback development only. API keys (web.api_keys, hashes only) are for scripts.
Storage: files reach the corpus from one folder (web.storage_dir, default <corpus>/storage/<source>/). They get there directly (copy them, or mount a bucket) or through POST /api/storage/<source>. A sync, from the UI, from POST /api/sync or on a timer, ingests every source folder. A file removed from storage is deleted from the corpus.
Hosting: the Dockerfile builds the app with the docs site. The corpus tree can sit on a Cloud Storage or NFS mount, with the catalog on local disk (CORPUSOS_CATALOG_PATH); it is restored on start and snapshotted after each write.

The docs site covers the web app and API, storage and self-hosting.

Agent platforms¶

There are two ways to give ChatGPT, Claude, Copilot Studio, Bedrock or Vertex AI the corpus. Use the first wherever the platform allows it.

	Remote MCP server	Knowledge export (uploaded files)
Platforms	ChatGPT (connectors, deep research), Claude.ai and Claude Code (custom connectors), Copilot Studio (MCP tools), any MCP client	ChatGPT GPTs and Projects, Claude Projects, Copilot Studio file knowledge, Bedrock Knowledge Bases, Vertex AI Search / RAG Engine, OpenAI vector stores
Access control	the live ACL, per user, on every call	none: everyone who can use the agent reads every file
Deletions and revocations	effective on the next call	the copy stays until someone deletes it
Review gate, audit, exact SQL	yes	no

Remote MCP server¶

corpusos mcp --transport http serves streamable HTTP (stateless, JSON responses) at /mcp. Every request carries an OAuth 2.1 access token or an API key, which resolves to corpus principals; each tool call is then checked against the live ACL and audited, exactly as over stdio. Configure it in config/security.yaml under mcp:

mcp:
  public_url: https://corpus.example.com/mcp     # what clients connect to; required beyond loopback
  oauth:                                         # JWT access tokens from your identity provider
    issuer: https://login.microsoftonline.com/<tenant-id>/v2.0
    audience: api://corpusos                      # default: public_url
    required_scopes: [corpus.read]
    principal_claim: preferred_username           # or sub, email, oid
    group_claims: [groups, roles]
    principal_map: {"<entra-group-id>": finance}  # claim value -> corpus principal(s)
  api_keys:                                      # for clients that only send a key
    - {label: copilot-studio, sha256: <from corpusos mcp-key create>, principals: [copilot-agent]}

OAuth (ChatGPT, Claude.ai, Copilot Studio): the server is a resource server. It verifies the signature against the issuer's JWKS (discovered from its metadata), plus issuer, audience, expiry and scopes, and publishes RFC 9728 Protected Resource Metadata at /.well-known/oauth-protected-resource/mcp. A request without a valid token gets 401 with a WWW-Authenticate header pointing at it, and a token without the required scope gets 403. Sign-in, client registration and consent stay with the identity provider (Entra ID, Okta, Auth0, Keycloak, Google).
API keys (Copilot Studio's API key option): corpusos mcp-key create -p copilot-agent --label copilot-studio prints a key once. Only its SHA-256 goes into security.yaml. Clients send it as Authorization: Bearer <key> or X-API-Key: <key>, never in the URL.
Refusals: the server will not start beyond loopback without public_url and authentication, and public_url must be https. The Host and Origin headers are checked against public_url, loopback and allowed_hosts / allowed_origins. Run it behind a TLS-terminating proxy. On loopback without authentication it serves as --principal, for local testing only.
search and fetch return {results: [{id, title, url}]} and {id, title, text, url, metadata}, which ChatGPT needs for company knowledge and deep research. url comes from mcp.citation_url (default corpusos://<corpus>/<document>/<revision>); point it at your document viewer if you have one.

Client	Where to add it
ChatGPT	Apps & Connectors, in developer mode: create a connector with the `public_url` and OAuth
Claude.ai	Settings > Connectors > Add custom connector. Allow the callback `https://claude.ai/api/mcp/auth_callback` at your identity provider
Copilot Studio	Agent > Tools > Add a tool > Model Context Protocol, with OAuth 2.0 or an API key header `X-API-Key`

Knowledge exports¶

corpusos export knowledge --target chatgpt --audience sales-team
corpusos export knowledge -t bedrock -a '*' --max-classification public
corpusos export knowledge -t vertex -a finance --base-uri gs://my-bucket/finance-kb

--audience names the people who will use the agent. A document is exported only if all of these hold:

it is current, accepted and indexed;
its live ACL lets the audience read it (* in an ACL means everyone);
it is at or below --max-classification (default internal, so documents that PII made confidential stay out);
it has no risk flags (--allow-flagged lets in reviewed ones).

Each export goes to exports/knowledge/<target>/<date>__<audience>__v<NNN>/:

File	Contents
`READ-ME-FIRST.md`	what the platform will not enforce, upload steps, platform notes, limits applied, what was left out
`upload/`	exactly the files to upload, named `<date>__<domain>__<type>__<slug>__v<revision>.<ext>`
`export.json`	each file with the documents and revisions it holds, its SHA-256 and size
`excluded.json`	each document left out, and why
`changes.json`	files to add, upload in place of older copies, and delete on the platform, compared with the previous export for the same audience
`withdrawn.json`	files removed after a later deletion or revocation, which must also be deleted on the platform

Every file starts with the document title and a source line: path, document_id, revision_id, classification and effective date. Platforms without metadata still cite the right revision. Tables are written as Markdown with column types, units and formulas, and also as CSV where the platform reads CSV.

Target	Format	Limits applied	Metadata
`chatgpt`	`.md`	20 files per GPT, 2M tokens and 512 MB per file: documents are merged into per-domain bundles when they do not fit	in the text
`claude`	`.md` + `.csv`	30 MB per file	in the text
`copilot`	`.md` + `.csv`	500 files, 512 MB per file	in the text
`bedrock`	`.md`	50 MB per file	`<file>.metadata.json` sidecars with typed, filterable attributes (at most 10 KB)
`vertex`	`.txt` (Markdown inside)	10 MB per file (RAG Engine's limit for text)	`upload/metadata.jsonl` for Vertex AI Search, with `gs://` URIs from `--base-uri`
`openai`	`.md`	5M tokens and 512 MB per file	`attributes.jsonl` for `vector_stores.files.create` (16 keys, 512 characters)

The limits are the ones each platform documented on 2026-10-27. OpenAI's help pages could not be read directly, so the ChatGPT figures come from quotes of its knowledge article, and .md support in ChatGPT GPTs and Claude Projects is widely used but not listed in their help pages. A document too large for one file is split at headings, then paragraphs, then lines, into -part-NN files; table parts repeat the column header. Platforms change their limits, so --max-files, --max-file-mb and --max-file-tokens override them; each export records the limits it applied and their source pages.

Security, lifecycle and retention¶

Policies per source (security.yaml → defaults, sources["<tenant>/<source>"]): ACL, owner, training_allowed, retention policy. Policies apply live: revoking access takes effect on the next query, with no reprocessing.
Deletion propagation: chunks, embeddings, dataset registrations, chunks.jsonl and every file under exports/ are updated as soon as a deletion is detected. Each ingest also refreshes the acl_ref of exported records to the current policy. In knowledge exports, a file whose document was deleted, or whose audience lost access, is removed from upload/ and listed in withdrawn.json, so that the copy on the platform can be deleted too. Copies already taken outside the corpus folder (including exports written with --output) cannot be recalled.
Policy reaches nested files: a policy change also applies to the archive members and email attachments of an unchanged archive or email.
Retention: corpusos purge [--dry-run] removes the derived artifacts of deleted and superseded revisions after deleted_days / superseded_days, and the original bytes too when purge_raw: true.
PII: detected (regex by default, or Microsoft Presidio) and optionally redacted in derived artifacts; originals are never modified. Automated detection cannot guarantee that every sensitive value is found.

Quality and continuous improvement¶

ingest → sample → inspect → evaluate → classify failures → correct → reprocess → regression test

Command	Purpose
`corpusos inventory PATH`	Read-only Phase 0 inventory: kinds, sizes, parser coverage, duplicates, unportable names, collisions
`corpusos sample -n 10`	Documents to compare with their originals, plus a checklist in `evaluations/inspections/`
`corpusos failures add --category retrieval_miss --query … --expected …`	Classify what went wrong
`corpusos failures to-regression`	Turn reviewed failures into golden queries
`corpusos eval -k 10`	recall@k, precision@k, MRR, citation correctness, permission leakage, abstention accuracy
`corpusos scorecard`	The release gates below; exits non-zero until all pass
`corpusos export training`	Verified, training-allowed SFT pairs with a stable held-out split

The scorecard measures the proposed release gates. A gate without data is reported as not measured, never as passed.

Gate	Target	How it is measured
Inventoried files accounted for	100%	every discovered file has a recorded outcome in its run
Chunks with source provenance	100%	chunk → blocks → revision → raw bytes on disk
Unauthorized cross-tenant retrieval	0	ACL probes with every known principal + golden-query leakage probes
Silent ingestion failures	0	files without an outcome, runs that never finished
Parser correctness on golden fixtures	≥ 98%	`evaluations/parser-fixtures.jsonl`
Retrieval Recall@10	≥ 90%	`evaluations/golden-queries.jsonl`
High-risk documents reviewed before publication	100%	published revisions with risk flags all carry a reviewer
Repeated unchanged ingestion	no duplicates	duplicate revisions/chunks and needless reprocessing

Before processing a whole disk or NAS, pilot on about 100 representative files: clean documents, scans, nested spreadsheets, email attachments, corrupted files, multilingual documents and permission-restricted material.

Storage layout¶

corpus/
├── config/        corpus.yaml parsers.yaml security.yaml naming.yaml quality-gates.yaml
├── raw/           <tenant>/<source>/<document-id>/<revision-id>/original.<ext>
├── canonical/     <tenant>/<document-id>/<revision-id>/{document.json, content.md, metadata.json, assets/}
├── structured/    <dataset>/<version>/{schema.json, data.parquet}
├── chunks/        <corpus>/<profile>/chunks.jsonl
├── manifests/     sources / documents / revisions / runs / datasets .jsonl
├── evaluations/   golden-queries.jsonl, parser-fixtures.jsonl, curated-pairs.jsonl, inspections/, reports/
├── quarantine/    <run-id>/<document-id>.json
├── exports/       rag/ training/ evaluation/ knowledge/<target>/<date>__<audience>__v<NNN>/
├── storage/       <source>/...: files the web app and API ingest (default web.storage_dir)
└── .corpusos/     catalog.db: the transactional source of truth

ID	Meaning	Changes when
`document_id` (`doc_<ULID>`)	Logical document	Never: stable across revisions
`revision_id` (`rev_…`)	Exact bytes of one version	The source bytes change
`chunk_id` (`chk_…`)	One segment under one chunking profile	The revision, the profile version or the text changes

Generated artifacts follow <date>__<domain>__<document-type>__<short-slug>__v<revision>.<ext> (corpusos naming build|check). Originals are never renamed.

Development¶

uv sync && uv run pytest
uv run corpusos schema            # refresh schemas/ after changing models.py (a test checks they match)
uv run python tests/fixtures/make_fixtures.py   # regenerate the .xlsx/.pptx fixtures

Architecture, design decisions and the roadmap are in docs/architecture.md. Build the docs site with uvx --from mkdocs-material==9.5.39 sh deploy/build-docs.sh site. See CONTRIBUTING.md and SECURITY.md.

License¶

Apache License 2.0. See LICENSE.