# Local research archive **Experimental local research archive system.** Keep selected text, PDF and PNG/JPEG sources in an explicit local project. Dossiers bring notes and evidence together; claims record assertions; decisions explain the author's choices. The workspace preserves saved versions and exact quotations. Text sources can be analysed separately or compared with a manuscript. It does not implement the entire [RFC](README.md). Product version v0.3.0 describes development maturity before 1.0; it retains the features introduced under the historical v1.x–v3.x releases. Product, response and archive schema versions are independent. v0.3.0 reads research records/manifests v1–v7, including archives written by v3.0.1. A first manual manuscript-link save opts into v7; use v0.3.0 for those archives. Reading older archives alone does not upgrade them. Historical release notes describe the capabilities of those historical builds, not a numeric minimum for v0.3.0. | Implemented locally | Outside the current scope | | --- | --- | | Explicit project, immutable captures, paragraph citations | Embeddings, dense hybrid search, Qdrant, Haystack | | SQLite/FTS5 lexical search over passages and current authored records; auto-fresh transparent cache refresh | Remote web imports | | Local text / Markdown input, original bytes retained | Multi-tenant team services, cloud hosting | | Multi-file batch ingestion with `--progress` phase notifications | External web scrapers | | Section-bounded dossier reads (`--section`) | Audio/video transcription or image understanding | | Local PNG/JPEG captures and exact-version images in dossiers | Remote image downloads, image OCR or visual AI | | Decision-dossier review tracking (`review_needed`) | Automatic bidirectional Zotero synchronization | | Native PDF extraction with Poppler; scan OCR with local Tesseract or a configured worker | Automated factual proof or rewriting prose | | Safe Markdown & offline SVG diagram rendering (flowcharts, sequence) | Ambient configuration discovery | | BagIt-style archive export and restore with cryptographic verification | Ambient manuscript detection | | Native revisions, conflict reconciliation and atomic related-record change sets | | | Versioned source criticism context & core analysis adapter | | | Optional local Zotero catalogue browsing and selected PDF/text captures | | | Additive Zotero migration export and paired research/Zotero backup | | | Controlled withdrawal & physical purge with dry-run preview | | | Read-only HTML source dashboard with localized context | | | Integrity audit and snapshot conflict detection | | | Source listing/tagging, dossiers, scoped claims, evidence relations and authorial decisions | | | Structural decision-impact and editorial-review reports | | | Claim-evidence matrix export (`lixity research matrix`) | | | Claim timelines, authored causal links and keyword citation selection (historically v3.0.0; retained in v0.3.0) | | | Explicit manuscript quote–claim associations and structural anchor review (v0.3.0) | | | Cross-corpus linguistic grounding (`lixity research compare`) | | | Interactive web research panel in `lixity serve`: source/dossier details, search-to-evidence actions, dossier-linked claims, decisions | | The server dashboard offers interface text for seven languages. English and German have the deepest linguistic analysis heuristics; language selection does not translate archived quotations or turn lexical search into semantic search. The research CLI's help and errors are English. ## Research workflows ### Reading and CLI workflows (since v2.4.0) The v2.3.0 research CLI uses explicit `--project` arguments. Since v2.4.0, commands using an existing archive have a bounded shortcut: omit `--project` only when the current working directory is itself a validated, initialized research project. It does not search parent directories or use environment, global configuration or the last project opened in the browser. An explicit `--project` remains authoritative. `research init` still requires `--project`; restore commands keep their explicit `--to` destination. `schema` and `ocr-status` keep their existing behavior. Python and HTTP research APIs remain explicitly project-scoped. ```sh cd ./novel lixity research sources --format text lixity research search --query "reading room" lixity research link urn:uuid:YOUR-PASSAGE-UUID urn:uuid:YOUR-CLAIM-UUID \ --relation contextualizes --rationale "Background context for review" ``` `research link PASSAGE_ID CLAIM_ID` creates an evidence link through the existing writer. Pass the full IDs returned by source search and claim listing, in that order; abbreviations and reversed arguments are not inferred. The default relation is `supports`; alternatives are `contradicts`, `qualifies` and `contextualizes`. Optional `--rationale`, `--reviewer` and `--actor` record your interpretation and authorship. `link-evidence` remains available for creation, inspection and revisions, with its existing output and validation contracts. In an opened source reader, select text within one uniquely resolved retained paragraph to prepare the existing claim evidence-link form. Review the selected quote and complete paragraph context, choose a claim and relation, and save separately. The reference still identifies the complete retained source paragraph; it does not create a sentence-level passage ID or narrower citation selector. Cross-paragraph or ambiguous selections are rejected. The Passages view retains whole-passage actions for claims and dossiers, including as a touch fallback. Selecting text or preparing the draft writes no record and establishes no claim. In **Manuscript & Analysis**, open **Dossier alongside**, then choose a dossier. The view starts closed and does not select a dossier automatically. A dossier card can also open that chosen dossier beside the manuscript. The companion reader is read-only and does not edit either text, link evidence automatically or change the manuscript's statistical baseline. While open, it focuses the manuscript chapters, search and filters; other analysis panels are temporarily hidden. Closing it restores those panels and preserves the manuscript's search and open paragraphs. Its state lasts for the current page; refresh explicitly to read the latest saved dossier revision. Desktop reading areas sit beside one another with independent scrolling; screens at or below 900 pixels stack the dossier above the manuscript. The **Author review state** field describes your assessment of a claim's support: | English interface label | Stored value | | --- | --- | | Author assumption | `hypothetical` | | With evidence | `evidenced` | | Disputed | `disputed` | Stored values remain unchanged; neither the labels nor an evidence link establish historical certainty or a calculated probability. A chosen **With evidence** state does not require an evidence link or verify the source. Archive formats, immutable paragraph citations and existing JSON contracts remain unchanged by these workflow additions. The revision and scene workflows began in v1.24.1. New checkpoint imports and explicit author acknowledgements are available since 2.0.0; select that release explicitly. Upgrading a fixed tag does not choose a newer release. ### Associate a manuscript quote with a claim **Since v0.3.0:** in an opened manuscript paragraph, select text within that one paragraph and choose **Link selection to claim** (or press `Alt+L`). Review the exact quote, choose a current claim, explicitly permit local retention of the quote, then **Save association**. Selection, choosing a claim and previewing the form write nothing. Saving creates a distinct `manuscript_claim_link`, with an exact claim-revision pin and paragraph anchor. It does not edit prose, import the manuscript as a research source, create source evidence or establish factual support. This workflow requires an explicitly selected manuscript inside the initialized research project and a compatible native server; standalone HTML cannot save links. Only the selected quote, project-relative source path, raw-source SHA-256, paragraph SHA-256, source line range and Unicode-codepoint offsets are retained, plus an optional note and ordinary record metadata. The archive stores no full manuscript blob. Quotes are nonblank and limited to 10,000 codepoints; notes to 2,000 characters. Offsets refer to the displayed `ParagraphProfile.text`, with an exclusive `end`, rather than UTF-8 bytes or JavaScript UTF-16 units. The paragraph hash is SHA-256 of that displayed text encoded as UTF-8. Because that text removes inline Markdown (`*emphasis*`, links, inline code), quotes copied from the `.md` file must be stripped of formatting before saving; the raw-source SHA-256 still binds the file bytes. Cross-paragraph, ambiguous or stale selections fail. Paths escaping the project, including through symlinks, are rejected. Source bytes are checked again before publication; this optimistic guard does not lock an external editor. Saved quotes mark only exact resolved anchors, including paragraphs rendered later on opening. Manuscript search still checks complete text. Review reports changed source bytes, newer/withdrawn claims, unique moved paragraphs, adjusted paragraphs whose quote is unchanged, and ambiguous matches. Moved or adjusted candidates remain unresolved (and highlighted at their current position) until an explicit re-pin; ambiguous or edited quotes are never re-pinned. These flags request a human review, not a semantic contradiction check or factual verdict. Retraction and note/claim revisions use the existing record history and strict revision API. A uniquely resolvable anchor is re-pinned explicitly: `lixity research reconcile-manuscript --accept-moved` (or `POST /api/research-manuscript-reanchor`) writes a new link revision with the resolved line span, refreshed paragraph hash and offsets for adjusted paragraphs, and the current file digest; the earlier anchor stays in revision history. **Since v0.4.6:** the same flow is scriptable from the CLI. `--selection` takes the JSON selection object above (literal or a local file); `--expected-snapshot` / `--expected-source-sha256` default to one initial read and still fail when the bytes or the archive change before saving. ```sh lixity research find-anchor --project ./novel novel.md --query "exact quote" lixity research link-manuscript --project ./novel novel.md \ --claim-id CLAIM_ID --claim-revision 2 --quote "exact quote" --allow-retention lixity research manuscript-links --project ./novel lixity research review-manuscript --project ./novel novel.md lixity research reconcile-manuscript --project ./novel novel.md # Vorschau lixity research reconcile-manuscript --project ./novel novel.md --accept-moved ``` The manuscript path is accepted positionally (like `analyze`, `scenes`, `showing`) or as `--manuscript PATH`; pass exactly one of the two forms. A changed source-file digest (`source_changed`, for example after an edit in an unrelated chapter) is informational and does not mark an association for review while its paragraph hash, line span and quote still match exactly. An edit inside the paragraph but outside the quote reports `anchor_adjusted` and can be accepted with one command or click; editing the quote itself reports `anchor_changed` and requires a fresh association. Python integrations use the same API: ```python from lixity.research import api as research_api research_api.link_manuscript_claim( project, manuscript, claim_id, claim_revision=reviewed_claim_revision, selection={ "start_line": start_line, "end_line": end_line, "paragraph_sha256": paragraph_sha256, "start": start_codepoint, "end": end_codepoint, "quote": selected_quote, }, expected_project_id=project_id, expected_snapshot=snapshot, expected_source_sha256=source_sha256, allow_retention=True, note="Author's association for review", language="en", project_config=settings, ) research_api.list_manuscript_links(project, claim_id=claim_id) research_api.review_manuscript_links(project, manuscript, expected_project_id=project_id, language="en", project_config=settings) ``` The create/list/review envelopes are `research-manuscript-link-local/1`, `research-manuscript-links-local/1` and `research-manuscript-review-local/1`. Lists omit retracted links unless `include_retracted=True`. Review rows preserve exact/current claim revisions and the pinned claim's dossier association, with `flags`, `anchor_state`, `candidate_lines` and nullable resolved line fields. A compatible native server provides guarded manuscript review/list reads and an explicit save route; see the [HTTP contract](../AGENTS.md#manual-manuscript-associations). The native research Review tab lists the saved associations with their anchor state and flags; its section stays hidden without a selected manuscript. There is no new manuscript-link CLI command. The first successful, retention-confirmed save uses `research-local/7` and promotes the manifest to `research-manifest-local/7`. v0.3.0 reads v1–v7; historical v3.0.1 cannot read v7. Back up before opting in and use v0.3.0 for every reader and writer afterward. Reading/listing/reviewing an older archive does not change its bytes. Audit, history and export/restore preserve accepted link records; exported quotes remain private manuscript material, while the manuscript itself is not included as a source blob. Restoring links without the manuscript leaves their anchors unavailable until the source exists at the saved project-relative path. ### Claims linked to a dossier Both the research dossier reader and **Dossier alongside** show the current claims explicitly linked to the dossier. **Open claim** uses the existing record viewer and saved-version history. The reverse list reads current claim records; it does not reconstruct a historical claim list for an older dossier revision. Each claim can show an active saved manuscript-quote count. The count joins explicit links by claim ID and the linked claim revision's pinned dossier ID, from the same archive snapshot as the claim list. A changed snapshot or failed read makes the count unavailable. Zero means no matching active saved link in that snapshot; it does not establish that the claim is unused in the manuscript. Newer claim/dossier revisions do not silently change an older link's association. Counts include links pinned to other saved versions of the same logical dossier; the reader marks that version difference explicitly. ### Claim chronology and cause/effect links (since v3.0.0) These additions were introduced in historical v3.0.0 and are retained in v0.3.0; they are not part of historical v2.4.0. In **Claims**, choose **Timeline** to organize the current claim revisions by their recorded `time_period`. **List** returns to the ordinary claim cards. Each dated claim shows its placed calendar range, both sections carry a count, and the view names the recognized formats and how to place a claim. Every period label is preserved and the record's creation date is never substituted. Opening or filtering the timeline changes no archive records or index. The chronology places valid four-digit CE years (`0001`–`9999`), ISO months (`YYYY-MM`), ISO calendar dates (`YYYY-MM-DD`) and ranges whose endpoints have the same precision, for example `1912`, `1912-03`, `1912–1914`, `1912-05-01` and `1912-05-01 / 1912-05-03`. Ranges accept `–`, `—`, `/`, `..` and, since v0.4.7, a plain hyphen between two four-digit years (`1924-1926`, spaces optional); ISO month/day hyphens are never split. Missing periods, invalid dates/reversed ranges and labels such as `autumn 1912` or `44 BCE` remain visible under **Unplaced periods** with a reason; free-text periods such as `present day` stay there on purpose. Temporal placement does not verify chronology or establish a historical fact. Creating or revising a claim reports an unplaceable period in the envelope's `warnings` list, and the native claim form shows the accepted formats under the period field. ```sh lixity research timeline --project ./sample-project lixity research timeline --project ./sample-project --format text lixity research timeline --project ./sample-project --format md --dossier-id DOSSIER_ID ``` JSON uses `research-timeline-local/1` with `project_id`, `snapshot`, `dated` and `unplaced`. Entries retain claim ID/revision, title, statement, confidence, original period, place, actors and any dossier association. Placed entries add `period: {start, end, precision}`; unplaced entries add `reason: missing|unrecognized|invalid`. This is a projection of current claims, not a reconstruction of earlier timelines or a factual score. Under **Cause and effect claims**, choose the proposed cause and effect by their displayed claim labels, review the saved versions and optionally explain your interpretation, then select **Save cause/effect link**. The link is a separate authored `ClaimRelation`, distinct from passage-to-claim evidence links. `causes` is the stored relation; `results_in` is an input alias in the same cause-to-effect direction. Self-links are rejected. Cycles are allowed as explicit author assertions; no transitive or reverse links are inferred. ```sh # Replace placeholders with complete claim IDs from research claim. lixity research relate --project ./sample-project FROM_CLAIM TO_CLAIM \ --relation causes --rationale "Author interpretation for review" lixity research relate --project ./sample-project lixity research relate --project ./sample-project --claim-id CLAIM_ID --include-retracted lixity research relate --project ./sample-project --relation-id RELATION_ID --inspect lixity research relate --project ./sample-project --relation-id RELATION_ID --history ``` New links pin the chosen claim revisions. The CLI/Python API use current revisions when no explicit pins are supplied; `--from-claim-revision N`, `--to-claim-revision N` and `--expected-snapshot DIGEST` let automation bind a reviewed choice. A later claim edit does not retarget a link: its earlier endpoint labels remain visible and `review_needed` marks newer endpoint revisions. Creation returns `research-claim-relation-local/1`; listing returns `research-claim-relations-local/1` with the project, snapshot, archive schema and exact endpoint/current revision information. Retracted links are omitted unless `--include-retracted` is supplied. Use the existing **Edit**/**Saved versions** controls or generic revision commands to correct endpoints, change rationale or retract an interpretation. For example: ```sh lixity research relate --project ./sample-project --relation-id RELATION_ID --update \ --expected-snapshot SNAPSHOT_SHA256 --expected-revision 1 \ --change-kind supersession --reason "Author withdrew this interpretation" \ --state retracted ``` Read `snapshot` and `record.revision` from `--inspect` immediately before preparing an update. Retraction appends a revision; earlier active links remain in history. An unchanged endpoint keeps its existing pin; repinning requires an explicit endpoint revision. Three-way previews and atomic revision batches accept `kind: "claim_relation"` through the same revision machinery as other authored records. Archive integrity and linked evidence do not verify causality. **Archive compatibility:** the first successful explicit causal-link write uses `research-local/6` and promotes the manifest to `research-manifest-local/6`. It preserves existing source, claim, citation and revision bytes. Merely reading the timeline or relations does not upgrade an archive. Historical readers before v3.0.0 reject a v6 manifest; v0.3.0 supports it. Back up the complete archive before opting in and use a build that explicitly supports schema v6 for every reader/writer afterward. Existing v1–v5 archives remain readable without conversion. `research export`/`restore` retain their existing bundle format and preserve the complete causal-link history. Do not edit HEAD or manifest files to downgrade. ### Choose citations by keywords (since v3.0.0) In the evidence-link form, select a claim and choose **Find source passage**. This opens the existing Search view with **Sources** selected and keeps your claim choice. Search keywords, read the matches and choose **Use for claim** to return to the draft. Review the full retained paragraph, choose an evidence relation and save explicitly. Search and draft preparation accept no claim and write no authored record. Duplicate titles are not treated as identity. The terminal can also resolve a paragraph without typing its UUID: ```sh lixity research cite --project ./sample-project --query "reading room" # For multiple matches, repeat the same query and displayed snapshot: lixity research cite --project ./sample-project --query "reading room" \ --pick 2 --expected-snapshot SNAPSHOT_SHA256 ``` Exactly one match returns the existing full `research-citation-local/1` citation. Zero or multiple matches return the unchanged source-search `research-search-local/1` JSON with exit code `1`; multiple matches also print safe numbered choices on stderr. No best-ranked hit is accepted silently. `--pick` selects one of at most 20 displayed matches (`1`–`20`) and requires the same query plus its displayed snapshot. Changed snapshots, unavailable sources and out-of-range choices fail; search again before choosing. Ordinary search may refresh its disposable index, but citation selection creates no claim/link or source capture. `--passage PASSAGE_ID` remains precise retrieval of an exact retained citation, including historical captures. It is mutually exclusive with `--query`; `--pick` and `--expected-snapshot` apply only to query selection. Both routes identify the complete retained paragraph with its source version, hash and character offsets. They do not create sentence-level citations, translate quotations or change lexical search into semantic search. The native evidence draft binds its selected project, reviewed snapshot and claim revision. If these change before saving, review the current context rather than silently linking a newer claim. Python `link_evidence` accepts optional `expected_project_id`, `expected_snapshot` and `expected_claim_revision` guards; HTTP evidence creation accepts `project_id`, `research_token`, `expected_snapshot` and `expected_claim_revision`. Supplied guards reject changed reviewed context; guarded links also reject withdrawn evidence. Source withdrawal does not erase historical citations. **API change in v3.0.0:** `list_claims`, CLI `research claim` lists and HTTP `GET /api/research/claims` consistently return `research-claims-local/2` with `claims`, `project_id` and `snapshot` from one snapshot. Consumers of the earlier `research-claims-local/1` envelope must update; there is no alternate list shape. The stored claim records and their revision history are unchanged. Source-only and unified search retain their existing v1/v2 envelopes. ### Continue an interrupted source import For a larger collection, opt into a checkpoint through the existing ingest command. Ordinary single- and multi-file ingestion keep their earlier output. ```sh lixity research ingest --project ./novel --file source-a.pdf source-b.txt \ --checkpoint library.json --allow-retention --progress lixity research ingest --project ./novel --checkpoint library.json \ --resume --allow-retention --progress ``` The checkpoint remembers the ordered files and capture settings. Resume checks their bytes, the project, OCR configuration and source identity, then skips verified completed captures. Complete prepared extraction is reused after an interruption or failed publication; interrupted or unrecorded extraction may still run again. Changed files, order or settings require a fresh checkpoint. If another writer changed the research snapshot, inspect the current archive and explicitly acknowledge its current digest with `--expected-snapshot DIGEST` when resuming. This permits the reviewed retry; it does not rebase authored decisions or accept a changed/purged source version. A lost completion receipt is recovered from exact accepted record identities, avoiding duplicate captures. Relative names resolve inside `/.lixity/research/imports/`; absolute paths must remain there and end in `.json`. One checkpoint accepts 1–1,000 files and at most 64 MiB of JSON preparation/metadata. One large or heavily escaped capture can exceed that budget; smaller batches do not always resolve it. Oversized preparation is rejected before publication and preserves the prior readable checkpoint. Original source files must remain present and unchanged. `--dry-run` is unavailable for checkpoints; ordinary ingest previews write none. Pending preparation contains private source text and is outside the accepted research archive and its export bundles. Remove it explicitly when unnecessary: ```sh lixity research ingest --project ./novel --discard-checkpoint library.json ``` Discard removes checkpoint preparation and receipts, not accepted research records. Completed payloads are cleaned up after publication. Purge coordinates with matching checkpoints; a busy checkpoint blocks that purge. Checkpoints are loaded before source matching, so any unreadable or malformed checkpoint can block purge until explicit discard. Deleted captures cannot be restored silently through resume. Progress phases go to stderr; final JSON uses `research-checkpoint-ingest-local/1`, or `research-checkpoint-discard-local/1` for discard. CLI exit status is zero only for a complete checkpoint import. ### Read reports in the terminal ```sh lixity research sources --project ./novel --format text lixity research sources --project ./novel --format md --limit 25 --offset 25 lixity research audit --project ./novel --format text ``` Sources and audit retain their complete JSON output by default. Human source lists show 50 entries unless you set `--limit` (1–100) and `--offset`; full IDs remain visible. Audit summaries explain their scope and bound long findings; use JSON for the full findings. `--pager` opens a pager only when explicitly requested with a human format and both input/output are interactive terminals. Piped output does not start one. Imported titles and values remain inert text. For automation, request JSON explicitly with `--format json` or `--json` on sources, audit, review, decision-impact, compare and matrix. Human-facing defaults remain command-specific; `lixity about --json` reports the parser's supported formats and defaults. The local HTTP server exposes its supported route registry at `GET /api`, even before a project is loaded. HTTP research responses are JSON; the NDA draft endpoint explicitly downloads PDF or UTF-8 text. Since v2.4.0, the manuscript PDF endpoint is also a binary attachment workflow, separate from research source import and OCR; see [manuscript PDF downloads](../USAGE.md#download-a-manuscript-pdf-since-v240). ### Update one reviewed record Ordinary `dossier`, `claim`, `link-evidence`, `decision` and `relate` commands with `--update` require `--expected-revision`, `--change-kind` and `--reason`. The archive-wide `--expected-snapshot` is optional for these ordinary updates. When omitted, the CLI reads the current record, checks that its revision equals the one you reviewed and submits that read's snapshot to the unchanged strict backend writer. An earlier unrelated write no longer invalidates that update; a changed target record or a writer race still fails, without retry, merge or rebase. A supplied snapshot remains a strict archive-wide guard, even when the target record itself has not changed. ```sh lixity research claim --project ./novel --claim-id CLAIM_ID --update \ --expected-revision 1 --change-kind correction --reason "Reviewed wording" \ --statement "A revised author assertion" ``` `--diff` previews the same change as a terminal unified diff and never writes: dossier body edits render line by line, other fields as before/after values. It computes the diff from the current revision, so it shows what an `--update` would submit next, including merged section content. ```sh lixity research dossier --project ./novel --dossier-id DOSSIER_ID --diff \ --section "Smuggling route" --file revised-section.md ``` Unchanged association IDs retain their exact saved pins. With an omitted snapshot, adding/changing an association requires its explicit reviewed revision: `--dossier-revision`, `--claim-revision`, `--from-claim-revision`, `--to-claim-revision`, or a `--dossier-revisions` JSON mapping for added decision dossiers. Nonexistent pins fail. Immutable passage IDs need no separate revision flag. Explicit-snapshot updates retain their existing association behavior. Python/HTTP revision APIs, image attachments, acknowledgements and batch writes still require their documented exact snapshot/revision guards. Use the preview workflow below when a conflict requires reconciliation. ### Resolve conflicting drafts The revision editor keeps the draft and reason after a conflict and prepares a comparison of its base revision, the current record and the draft. Under **Review concurrent changes**, inspect the preview and explicitly choose current or draft values for overlapping fields. **Use reviewed changes** updates the draft; save separately. Preparing or selecting a resolution writes nothing. Another change before saving still rejects the save and keeps the draft. CLI users can prepare a draft with `--prepare --base-revision N` instead of `--update`. Resolve overlapping fields with repeated `--resolve FIELD=current` or `FIELD=mine`. Submit the resulting changes with the returned snapshot/revision tokens through the existing strict update command. Association IDs and revision pins are compared together; unchanged associations keep their historical pins. Prepared decision lists include `dossier_revisions`, a map from selected dossier IDs to existing revision numbers. Keep this map when applying the preview so a restored draft link retains the revision you inspected. ### Work on one dossier section In a dossier, choose **Edit section** beside an unambiguous heading. The existing revision editor opens that section's content, including its subsections. It keeps your draft after a failed save. Concurrent changes in other sections remain intact; overlapping edits need an explicit choice in the preview before saving. Use `dossier --dossier-id ID --section "Heading"` to read a bounded section; `--update --file section.md --section "Heading"` replaces only that section's content, with the usual snapshot, revision and reason requirements. Both reading and replacing a section include its subsections. Headings inside fenced code are ignored; repeated heading names reject ambiguous reads/updates. Other text, line endings and evidence references are preserved. Prepared section drafts start from the selected base revision, never a silently substituted current body. The Python equivalent `prepare_dossier_section(project, id, base_revision=N, section="Heading", content="Draft")` prepares a read-only, section-bounded three-way comparison. Save its full merged `changes` with the returned current snapshot and revision through `revise_record`. `get_dossier(..., revision=N)` reads the exact saved version; `editable_sections` lists real, unique headings, excluding code and the generated preamble label. ### Keep a visual reference in a dossier Choose **Add image**, select a local PNG or JPEG and describe what the image shows. Previewing stays in the browser. Optional details include a caption, source title, original source-page link, provenance note and an existing section. Confirm that you may retain the file, then save. The image and dossier revision are saved together; a rejected or stale save accepts neither. Cancel discards the preview. Click an embedded image to enlarge it; Escape closes the view. Earlier dossier versions keep their exact image version even after a source is replaced. Withdrawn, purged or missing images show an unavailable placeholder instead of changing to a different image. Source-page links remain links: Lixity never fetches them as images. Images have no extracted text, passages or linguistic scores. Retain an authorized local file yourself; a Wikimedia category or file-description page is not an image file. Text/PDF imports and OCR remain separate workflows. Generated images are imaginative visual references, not documentary evidence of how a real place looked. Identify generation or reconstruction in the caption and `provenance_note`, and retain the actual source of documentary photographs. A saved image, evidence link or author-selected claim status does not verify a historical, legal or biographical assertion. Character choices and stylistic intentions remain author decisions; external factual claims need suitable evidence. Limits are 16 MiB per image and 64 million pixels; image-specific HTTP JSON is limited to 32 MiB. Static PNG and supported 8-bit Huffman JPEG containers are checked for signatures, structure and dimensions. These checks are not a complete pixel decode or evidence of factual authenticity. Animated PNG, SVG and raw HTML images are outside this feature. The browser's decoder can still reject damaged pixel data. Original bytes, rather than a recompressed copy, enter the existing archive and its backup/audit lifecycle. CLI capture uses the ordinary import command: ```sh lixity research ingest --project ./novel --file ./reference.png --allow-retention ``` Programmatic capture is `ingest_image(project, bytes, filename="reference.png", allow_retention=True)`. `attach_dossier_image` additionally requires the dossier ID and exact `expected_snapshot` / `expected_revision`; it performs the joint save. Its Markdown image destination is `lixity:image/SOURCE_ID/VERSION_ID`, with canonical UUID URNs. For a joint capture and attachment, inspect the current dossier first and use its returned snapshot and revision: ```sh lixity research dossier --project ./novel --dossier-id ID --summary lixity research dossier-image --project ./novel --dossier-id ID \ --file ./reference.jpg --expected-snapshot DIGEST --expected-revision N \ --allow-retention --alt "Describe the reference" --section "Location" \ --context '{"provenance_note":"Generated visual reference; not a documentary photograph."}' ``` Omit `--section` to append at the end. Optional caption, source title, original source-page URL, change reason/kind and actor match the Python API. `--context` accepts a JSON object or a local JSON-object file, using the ordinary import convention. It creates no text passages and invokes no OCR. Success is the API's JSON receipt; errors go to stderr with a nonzero exit status. Stale edits fail without an accepted image or dossier revision; refresh and review before retrying. Image version purge records retain the exact source identity under `research-local/2`; older 2.0.0 writers reject that record. Existing v1 records stay unchanged. Back up before writing with a newer release, and keep compatible writers for that archive. Unprovable old associations remain missing. ### Review decisions and save related revisions together **Review** lists structural maintenance candidates. A decision's **Affected dossiers** disclosure shows its explicit direct or claim-based links, section outlines and pinned/current revision numbers. CLI equivalents: ```sh lixity research decision-impact --project ./novel --decision-id ID lixity research review --project ./novel --format md ``` These reports flag dated decisions, old pins, withdrawn links and unlinked decisions for human inspection. They do not detect semantic contradictions, establish historical truth, find every affected passage, or rewrite documents. Old pins can be intentional. The integrity audit remains a separate check. In the revision editor, choose **Add to change set**, then close the editor to select another record. **Review change set** shows every queued draft, its base values and reason. You can edit or remove individual drafts, then check and save the whole set. A failed or stale save keeps the drafts for explicit review. Drafts live only in that browser tab: reloading removes unsaved changes. If a network failure leaves the result uncertain, inspect history before retrying. CLI users can prepare the same group in a JSON file, then inspect it: ```sh lixity research revise-batch --project ./novel --file changes.json lixity research revise-batch --project ./novel --file preview.json \ --apply --expected-snapshot DIGEST ``` The file contains an operations list, or the saved preparation envelope `research-revision-batch-local/1`. Each operation has `kind`, `id`, `expected_revision`, `changes`, `change_kind` and `reason`. Preparation writes nothing. Apply validates every operation and publishes one archive commit; an invalid operation or stale snapshot accepts none. Duplicate record IDs and references to revisions that would be created by that same batch are rejected. Existing evidence and explicitly pinned references stay intact. ### Mark a decision as applied to a reviewed dossier After reviewing the decision and its linked dossier, choose **Mark as applied** for that pair. This records your assessment for the exact saved versions; it does not inspect a manuscript or establish semantic consistency. Later changes to either record make the acknowledgement stale and request review again. **Needs another check** explicitly reopens the same pair. After your own successful acknowledgement, the UI updates matching review controls and counts so you can continue with the next pair. An intervening external change still requires an explicit refresh; it is never silently accepted on your behalf. The first acknowledgement adds a `decision_acknowledgement` record under `research-local/5` and a `research-manifest-local/5` snapshot. Earlier source, claim, dossier, decision and citation bytes remain unchanged. Older applications reject that upgraded snapshot; back up the archive and use compatible writers. Reading an old archive alone does not upgrade it. CLI equivalents require the current snapshot and both current version numbers: ```sh lixity research mark-applied --project ./novel --decision-id DECISION \ --dossier-id DOSSIER --expected-snapshot DIGEST \ --expected-decision-revision 2 --expected-dossier-revision 4 lixity research reopen --project ./novel --decision-id DECISION \ --dossier-id DOSSIER --expected-snapshot DIGEST \ --expected-decision-revision 2 --expected-dossier-revision 4 ``` Both commands accept an optional short `--note`. A stale snapshot/version or foreign target writes nothing. Events form an explicit supersedes chain rather than relying on wall-clock ordering. Acknowledgement can clear its matching dated review request; outdated pins and withdrawn-source warnings remain visible. ### Use local OCR without a worker service Install Poppler, Tesseract and the language data you need on the server computer. Select the backend explicitly in that service's environment, then restart it: ```sh export LIXITY_OCR_BACKEND=tesseract export LIXITY_OCR_LANGUAGES=deu+eng lixity research ocr-status --probe ``` `LIXITY_OCR_BACKEND` accepts `native`, `worker` or `tesseract`. If unset, the existing configured worker takes precedence; otherwise native Poppler remains the default. Explicit `worker_cmd`/`--worker-cmd` takes precedence over the environment. Tesseract's default language is `eng`; extra languages need installed language data. Diagnostics report requested, available and missing languages. Nothing is downloaded or started automatically. Local Tesseract uses 300 dpi page images and a bounded invocation deadline; the external worker keeps its existing 150 dpi protocol. Failures reject the capture unless native fallback is explicitly allowed; inspect all returned warnings and compare quotations with the scan. A synthetic probe confirms execution, not recognition accuracy. New Tesseract captures identify `tesseract-cli/1`. Every new native Poppler capture, including an explicitly allowed fallback, identifies `poppler-native/1`. Both require `research-local/4` and a v4 manifest. Older readers reject that archive; use compatible readers and keep a backup. Existing v1–v3 records and cached extraction provenance stay unchanged. The external worker retains its existing implementation identifiers. ## Zotero Desktop bridge (since v1.19.0) Zotero can manage the library, bibliographic metadata and media; Lixity retains selected evidence captures and manages analysis, dossiers, claims and decisions. This optional CLI/Python/dashboard integration is available since v1.19.0. It adds no Python dependencies. Start Zotero on the same computer and enable **Settings → Advanced → Allow other applications on this computer to communicate with Zotero**. No account, cloud API key or plugin is required. The bridge reads the local API; it never writes Zotero's database or library. ```bash # Preview pages; use the returned next_start as --start for further pages. lixity research zotero --project ./novel --library users/0 --collections lixity research zotero --project ./novel --library users/0 --query "history" --limit 20 # Replace illustrative keys and SERVER_ID with preview results. lixity research zotero --project ./novel --library users/0 --collection-key ABCDEFGH lixity research zotero --project ./novel --library users/0 --item-key ABCDEFGH lixity research zotero-ingest --project ./novel --library users/0 --attachment-key JKLMNPQR --expected-server-id SERVER_ID --allow-retention --dry-run lixity research zotero-ingest --project ./novel --library users/0 --attachment-key JKLMNPQR --expected-server-id SERVER_ID --allow-retention lixity research reindex --project ./novel ``` Every call requires an explicit project and library. Use `users/0` for the local user library or `groups/` for a locally available group library. Preview returns `research-zotero-local/1`, metadata and an instance `server_id`. An item preview includes a page of attachments. Collections are paged in batches of 100; item pages accept a limit of 1–100. `next_start` is a continuation hint, not a total count; a final full page may lead to an empty page. External metadata and notes remain untrusted evidence. Capture selects one locally available PDF, UTF-8 text or Markdown attachment. The existing ingestion pipeline retains stable bytes, the parent title and origin URL. `provenance_note` records a bounded summary of creators, date, parent and attachment versions. `context.external_reference` separately records provider, server identity, library, parent/attachment keys and versions. This is not a lossless bibliography export. Invalid URLs or overlong metadata fail without publishing a capture. Language comes from the flag or project, not Zotero metadata. Images, audio and video remain in Zotero; Lixity does not transcribe or index them. Scanned PDFs require locally installed Tesseract or a configured OCR worker. See [local OCR setup](#use-local-ocr-without-a-worker-service). A repeated capture of the same instance/library/attachment finds its existing active source. Identical bytes and context are a no-op, rather than a duplicate. If the capture changed, explicitly pass its Lixity `--source-id` to refresh it. The attachment identity must match; existing source criticism and tags are preserved. Changed captures preserve earlier citations. A changed research snapshot fails the operation instead of overwriting concurrent work. Titles and languages retain the existing source-refresh restrictions. There is no automatic background synchronization or content-based deduplication across different items. Structured identity and safe refresh require the server identity provided by Zotero 10 or later. An older client supporting the local endpoints may create an unstructured CLI/Python capture, but safe refresh and identity deduplication are unavailable. The browser requires a previewed server identity for capture. The research dashboard lets users choose a library and collection, search items, inspect attachments, open originals in Zotero and capture or refresh selected PDF/text attachments after confirming retention. It marks captured attachments and metadata-version changes. This indication is not a comparison of file bytes. **Open in Zotero** uses the `zotero://` URL protocol: the Zotero installation registers that handler for the desktop account, and the running or newly started Zotero app selects the item. If a click has no effect, the protocol handler is missing — for example with a manually installed or portable copy; update or reinstall Zotero's desktop integration. This link is independent of the local API. When every active source has a structured external reference, the dashboard hides the separate local-import form. Native CLI/API import remains available for standalone projects and recovery; retained evidence history is not removed. The bridge uses GET requests to `127.0.0.1:23119`, with a 10-second socket timeout, bounded responses, no proxies or redirects. Files must resolve to local regular files; download cloud-only attachments in Zotero first. Dry runs require retention consent and validate/extract input, but leave the research store unchanged. Temporary capture files are removed afterward. Do not grant retention merely because a file is visible in Zotero: visibility is not redistribution permission. Python entrypoints are `lixity.research.zotero.collections`, `.browse` and `.ingest`, with explicit `project` and `library` arguments. They are independent of manuscript analysis. Standard research export includes retained captures and provenance, not the external library. Use the paired backup below for both stores. ### Literature Acquisition Workflow (URL → Zotero Reference → Attachment → Evidence) Literature moves through four distinct, verifiable states: 1. **Reference saved**: Register the bibliographic reference in Zotero (via browser connector, DOI translator, or manual entry). Metadata exists in Zotero, but this does not imply a local PDF or retained copy. 2. **Attachment available**: Download the PDF/full text inside Zotero (`contentType: application/pdf`, `linkMode: imported_file`). Verify attachment availability in Lixity with `lixity research zotero --library users/0 --item-key `. 3. **Evidence captured**: Ingest the local attachment into the Lixity research store with `lixity research zotero-ingest --attachment-key --expected-server-id --allow-retention [--progress]`. Lixity retains the PDF bytes, extracts passages, and establishes an immutable source version. 4. **Passage cited / Claim linked**: Search passages (`lixity research search --query "..."`) and cite verified passages into dossiers (`--evidence `) or claims (`lixity research link-evidence`). ```bash # 1 & 2. Find saved reference in Zotero and inspect available local attachments lixity research zotero --project ./novel --library users/0 --query "customs ledger" lixity research zotero --project ./novel --library users/0 --item-key # 3. Ingest selected attachment into Lixity with visible progress lixity research zotero-ingest --project ./novel --library users/0 \ --attachment-key --expected-server-id --allow-retention --progress # Or batch-ingest all eligible PDF/text attachments of a parent item: lixity research zotero-ingest --project ./novel --library users/0 \ --item-key --expected-server-id --allow-retention # 4. Search refreshed index and cite verified passage into a dossier lixity research search --project ./novel --query "port authority" lixity research dossier --project ./novel --title "Harbor Authority" \ --file ./notes.md --evidence ``` ### Additive migration to Zotero ```sh lixity research zotero-export --project ./novel --output ./zotero-import --allow-retention --dry-run lixity research zotero-export --project ./novel --output ./zotero-import --allow-retention ``` The destination must not exist. Export produces `library.ris`, `files/` and `migration.json` (`research-zotero-export-local/1`). It selects the newest active capture for each active source; withdrawn sources/captures are excluded. Original bytes, source IDs, context and SHA-256 hashes are retained in the bundle. RIS contains title, language, tags, origin URL and provenance notes, with an explicit `lixity-source:` tag and attachment file URI. Keep the reviewed bundle at its exported location while importing RIS: attachment URIs are absolute. Import `library.ris` through Zotero's import workflow and check its attachments. This is additive: it neither deletes old Lixity evidence nor reconciles duplicate Zotero imports. To bind an existing Lixity source, run `zotero-ingest` with its `--source-id` and the imported attachment key. The parent must retain the exported migration tag and the attachment bytes must exactly match the current capture. Binding updates capture provenance while preserving source IDs and older citations. A filename, title or similar text is never enough to establish this identity. Before retiring duplicate workflows, compare active source counts and bindings, audit the research archive, resolve representative old citations, and verify a paired restore. A successful export is not proof that a project's migration has finished. Preserve the pre-migration backup until restoration has been tested. ### Paired Zotero/research backup and restore Close Zotero before starting; `--confirm-zotero-closed` is the caller's explicit confirmation, not automatic process detection. Specify the **data directory** containing `zotero.sqlite`, not the application profile. ```sh lixity research zotero-backup --project ./novel --data-dir /path/to/Zotero --output ./paired-backup --confirm-zotero-closed lixity research zotero-restore --from ./paired-backup --to ./restored-pair lixity research audit --project ./restored-pair/project lixity research reindex --project ./restored-pair/project ``` Both destinations must be new; existing directories are never overwritten. The bundle contains `research.tar.gz`, a copy of the Zotero data directory, and `BACKUP.json` (`research-zotero-backup-local/1`) with SHA-256 checksums and sizes. Backup audits research and checks that Zotero files remain unchanged during copying. Restore checks the complete inventory, checksums and research archive before publishing `project/` and `zotero/` beneath the new destination. It does not switch a running Zotero instance or restart Lixity. Symlinks and special files are rejected; the paired inventory is limited to 10 GiB. External linked attachments, manuscripts and application profiles are excluded. Use Zotero stored attachments for self-contained media backup, and back up excluded project material separately. Verify restored Zotero data in a separate instance before adopting it; byte verification alone does not establish application-level usability or that an external attachment was included. #### Backup Scope Comparison | Scope / Content | `research export` | `research zotero-backup` | |---|---|---| | Retained source bytes & extracted passages | Yes | Yes | | Authored claims, evidence links, dossiers, decisions | Yes | Yes | | Full research revision history & snapshot digests | Yes | Yes | | Zotero database (`zotero.sqlite`) & metadata | No | Yes | | Zotero stored PDF/text originals (`storage/`) | No | Yes | | Zotero external linked files (outside data dir) | No | No (store as internal attachment to include) | | Manuscript prose & application browser profiles | No | No (maintain project/profile backup separately) | ## Try it Follow [installation](../INSTALLATION.md#cli-current-release) or use a development checkout (`make install-dev`), then activate `.venv` if applicable. Requires SQLite with FTS5; no additional Python dependencies or models. PDF extraction has separate prerequisites described below. ```bash lixity research init --project ./novel --title "Novel research" --language en lixity research ingest --project ./novel --file ./notes.txt --allow-retention --dry-run lixity research ingest --project ./novel --file ./notes.txt --context ./context.json --allow-retention lixity research reindex --project ./novel lixity research sources --project ./novel lixity research search --project ./novel --query "reading room" --limit 5 lixity research audit --project ./novel ``` The listing command is `sources`. Output choices are described in [Read reports in the terminal](#read-reports-in-the-terminal). `ocr-status` always returns JSON and does not accept `--json` or `--language`; its CLI guidance is English, while the browser localizes it in seven languages. Use named `--project` and `--file` arguments as shown above. Use a `passage_id` from search and a `source_id` from ingestion: ```bash lixity research cite --project ./novel --passage urn:uuid:YOUR-PASSAGE-UUID lixity research analyze --project ./novel --source-id urn:uuid:YOUR-SOURCE-UUID lixity research dashboard --project ./novel --source-id urn:uuid:YOUR-SOURCE-UUID > source.html lixity research compare --project ./novel --source-id urn:uuid:YOUR-SOURCE-UUID --manuscript ./novel.md lixity research dossier --project ./novel --title "Reading Room Notes" --file "Notes about a possible 1924 opening." --evidence urn:uuid:YOUR-PASSAGE-UUID lixity research dossier list --project ./novel lixity research dossier list --project ./novel --format table # ID, title, revision, date lixity research claim --project ./novel --title "Archive Founding" --statement "Founded in 1924." --confidence evidenced lixity research claim list --project ./novel lixity research link-evidence --project ./novel --claim-id urn:uuid:YOUR-CLAIM-UUID --passage-id urn:uuid:YOUR-PASSAGE-UUID --relation supports lixity research decision --project ./novel --title "Move date to 1914" --rationale "Plot tension" --claim-id urn:uuid:YOUR-CLAIM-UUID --deviation-from-fact lixity research decision --project ./novel lixity research matrix --project ./novel --format md lixity research matrix --project ./novel --format csv --output ./claim-matrix.csv lixity research withdraw --project ./novel --source-id urn:uuid:YOUR-SOURCE-UUID --reason "License revoked" lixity research purge --project ./novel --source-id urn:uuid:YOUR-SOURCE-UUID --dry-run lixity research purge --project ./novel --source-id urn:uuid:YOUR-SOURCE-UUID --reason "GDPR deletion" lixity research ingest --project ./novel --file ./revised-notes.txt \ --source-id urn:uuid:YOUR-SOURCE-UUID --allow-retention lixity research reindex --project ./novel lixity research schema # Or launch a foreground research server (stop with Ctrl+C): lixity serve --research-project ./novel --no-project ``` Replace the uppercase placeholders with returned IDs. Search uses the newest capture of each source. Earlier passage IDs still cite their original bytes. For the managed app, use `lixity service start --open` and **Open Project** to select the initialized archive; see the [lifecycle reference](../INSTALLATION.md#running-the-server). Repeating a refresh with unchanged bytes and context is a no-op. For native `ingest`, omitting `--source-id` creates a distinct source; identical text is not proof of identity. Zotero capture instead uses its structured external identity to recognize an already captured attachment. `hypothetical`, `evidenced` and `disputed` are user-selected claim labels, not computed probabilities or factual verdicts. A claim can exist without an evidence link. Links record a `supports`, `contradicts`, `qualifies` or `contextualizes` relation to a specific passage; the source and relation still need human review. Decisions are separate authorial records, optionally tied to a claim. An absent `--deviation-from-fact` flag means no deviation was marked, not that the decision was verified as factually faithful. `--allow-retention` confirms that you may keep a local copy. It does **not** grant copyright permission, authorize redistribution or accept a factual claim. Passages remain `unreviewed`. Commands normally emit JSON; `dashboard` emits HTML, `matrix` defaults to Markdown with CSV/JSON options, and `compare` can emit Markdown with `--format md`. Exit codes are `0` success, `1` operational/integrity failure and `2` CLI usage error. A failed audit returns `ok: false` and exit code `1`. Other errors go to stderr without source contents. ## Search current authored records (since v1.19.0) The default CLI/Python/HTTP search still searches source passages and returns `research-search-local/1`. Select `--scope all`, `dossiers`, `claims` or `decisions` to include current authored records with a typed `research-search-local/2` response: ```sh lixity research reindex --project ./novel lixity research search --project ./novel --query "inspector" --scope all lixity research search --project ./novel --query "inspector" --scope decisions ``` Python: `api.search("./novel", "inspector", scope="all")`. HTTP: `POST /api/research-search` with `{"query":"inspector","scope":"all"}`. The dashboard defaults to **All records** and offers the same type filters. Its details button opens the existing record history with the current revision. Source hits have `kind: "passage"` and retain their citation fields. Authored hits have `kind`, `record_id`, `revision`, `title` and a plain-text `excerpt`; they have no `passage_id` and cannot be used as source citations. Claim hits include the author-selected `confidence`; decision hits include `deviation_from_fact`. A match does not make a claim verified or a proposal final. The shared disposable FTS5 catalogue indexes current dossier bodies/tags, claim statements/tags/scope, decision rationales/plot impact, and their titles. It does not index revision history or revision reasons. `all` also indexes source titles and current source passages. Titles receive extra ranking weight; scores are lexical relevance, not authority. Query words must all occur in one record; there is no semantic search, identity resolution or automatic contradiction resolution. The result limit applies across all matching types. Use type filters when numerous source hits obscure authored records. CLI search automatically refreshes stale or missing indexes by default to avoid interrupting research tasks with manual maintenance detours. Callers who require strict validation can pass `--strict` to fail fast when the index is not up to date. When a query produces zero passage hits under the default `sources` scope, the response warnings suggest trying `--scope all` to search authored records. ### Batch ingestion and non-interactive progress - **Batch ingest**: Pass multiple files to `lixity research ingest --project ./p --file a.pdf b.txt --allow-retention`. It returns `research-batch-ingest-local/1` detailing per-item outcomes (`succeeded`, `failed`, `items`), ensuring partial batch success is preserved even if one file fails validation. - **Progress reporting**: Add `--progress` to `ingest` or `zotero-ingest`. Phase notifications with elapsed seconds are streamed to `stderr`, keeping `stdout` strictly machine-readable JSON. These are phase-boundary notifications, not periodic heartbeats, JSONL events or a per-page stream during worker inference. ### Section-bounded dossier inspection and revision Large dossiers can be inspected and updated incrementally without transferring oversized JSON payloads: - `lixity research dossier --project ./p --dossier-id --summary`: returns metadata, section heading outline, citation count, and a text excerpt. - `lixity research dossier --project ./p --dossier-id --section "Timeline"`: returns only the named section. - `lixity research dossier --project ./p --dossier-id --update --section "Timeline" --file ./new_timeline.md ...`: updates only that section in the latest dossier body. - Decisions can explicitly link affected dossiers with `--dossier-id ` or `--dossiers `. If a linked decision is revised after the dossier, inspecting the dossier reports `review_needed: true` with `decision_reviews`, allowing authors to track plot and historical decisions without speculative auto-rewriting. ### Claim-evidence matrix export Authors can export a consolidated overview matrix connecting user claims, supporting and contradicting evidence citations, associated dossiers, and decisions: ```bash # Markdown table summary to stdout: lixity research matrix --project ./novel --format md # CSV export for spreadsheet review: lixity research matrix --project ./novel --format csv --output ./claim-matrix.csv # Machine-readable JSON: lixity research matrix --project ./novel --format json ``` The matrix highlights deliberate narrative deviations from fact (`deviation_from_fact: true`), aggregates evidence support counts (`+N / -N`), and is accessible via the web API (`GET /api/research/matrix?format=json|md|csv`) and the dashboard export button. ## Python API ```python from lixity.research import api api.init("./novel", title="Novel research", language="en") capture = api.ingest("./novel", "notes.txt", allow_retention=True, context={"genre": "diary", "tags": ["archive"]}) api.reindex("./novel") sources = api.list_sources("./novel") results = api.search("./novel", "reading room", limit=5) if results["hits"]: citation = api.cite("./novel", results["hits"][0]["passage_id"]) dossier = api.create_dossier( "./novel", title="Reading Room Record", body="Notes about a possible 1924 opening.", evidence_ids=[results["hits"][0]["passage_id"]], tags=["milestone"], ) claim = api.create_claim( "./novel", title="Reading Room 1924", statement="The reading room was opened in 1924.", confidence="evidenced", ) link = api.link_evidence( "./novel", claim_id=claim["claim_id"], passage_id=results["hits"][0]["passage_id"], relation="supports", ) decision = api.record_decision( "./novel", title="Shift date to 1914", rationale="Dramatic tension before war.", claim_id=claim["claim_id"], deviation_from_fact=True, ) dossiers = api.list_dossiers("./novel") claims = api.list_claims("./novel") decisions = api.list_decisions("./novel") analysis = api.analyze_source("./novel", capture["source_id"]) html = api.source_dashboard("./novel", capture["source_id"]) comparison = api.compare_source("./novel", capture["source_id"], "novel.md") api.withdraw("./novel", capture["source_id"], reason="License revoked") preview = api.purge("./novel", capture["source_id"], dry_run=True) report = api.audit("./novel") ``` The project argument is mandatory. Research never discovers a project from a manuscript or reads analysis thresholds. `lixity build` remains unchanged and never imports sources or refreshes research indexes. Source analysis uses the shared engine on the selected archived source version, with its recorded language and explicit threshold overrides. Its interpretation is `source_internal`, with `context_status: user_supplied_unverified`. Manuscript project thresholds and corpus settings are not silently inherited. Genre, period, place, translation and original-language metadata supply context; they do not validate the source or turn historical travel accounts and monographs into the manuscript's stylistic baseline. Retaining or searching a source does not change manuscript analysis. In the local server, **Open Project** takes a path to an existing workspace folder or manuscript file on the server's filesystem and attaches that workspace's `research/` archive when present. **Browse files and folders** lets you browse the server computer's paths from its home directory, move with **Home** or **Parent folder**, select a manuscript or the current folder, and then confirm with **Open**; a typed path remains available. This is not a native OS file dialog or a browser upload. An initialized research-only folder opens without a manuscript. Source and record management remain available there; manuscript comparison needs a loaded manuscript. **New Project → Import Manuscript** accepts selected or dropped `.md`/`.txt` bytes and creates a separate project. Use Open Project when returning to an existing research archive. Source ingestion in the research panel has its own explicit local retention confirmation. Select a source or dossier for full details and verified citations. Search results can feed a selected passage into **Use for claim** or **Use for dossier**; the claim form can also associate an existing dossier. A dossier's **Add claim** and **Add decision** actions select that dossier by title in the existing forms; other unsaved text stays intact. Optional scope and technical fields are collapsed until needed. These actions record the author's links and notes, not a factual verdict. **Since v1.23.0:** Research & Dossiers is a separate dashboard view; New Project, Open Project and optional guidance remain accessible from every view. Local source and dossier filters match case-insensitive substrings in loaded titles, tags and IDs; dossiers also match excerpts and section names. These filters send no request while typing and do not search retained source text. Use the Search tab for full-text archive search. Filtering keeps loaded details and leaves association dropdowns complete. Selecting a manuscript under Project & Settings sends no request; **Continue to import…** opens the existing import dialog, and confirmation creates a separate project. Use Open Project when returning to this archive. ## Native editing and history Version 1.17.0 adds explicit revisions to the research workflow. In the dashboard, choose **Edit** on a dossier, claim, evidence link or decision. Change the fields, select **Correction** for a transcription/wording correction or **Supersession** for a changed interpretation or authorial choice, and give a reason. Save keeps the record's ID and adds one immutable revision. **History** opens earlier accepted versions; it does not restore or overwrite them. An edit does not create an additional active dossier. Existing associations remain pinned to the versions they originally referenced. For example, changing a claim does not make its old supporting link evidence for the revised statement; the UI shows the earlier claim revision for review. Creating a new association uses the target's current revision. Submitting an unchanged association keeps its original pin. To review an association against a revised claim or dossier, explicitly change its associated revision in the editor. The earlier association remains inspectable in history. The editor preserves its draft when validation, a request or a conflict fails. If any archive write occurred since the editor was opened, saving returns a conflict. **Reload current** explicitly discards the draft and loads the current record; nothing is silently merged or retried. Cancel closes the editor without writing. The same operations extend the existing commands: ```sh lixity research dossier --project ./novel --dossier-id DOSSIER_ID --inspect lixity research dossier --project ./novel --dossier-id DOSSIER_ID --update \ --expected-snapshot SNAPSHOT_SHA256 --expected-revision 1 \ --change-kind correction --reason "Correct a transcription" \ --title "Revised reading room notes" --file "The corrected working note." lixity research dossier --project ./novel --dossier-id DOSSIER_ID --history lixity research dossier --project ./novel --dossier-id DOSSIER_ID --revision 1 ``` Use the `snapshot` and `record.revision` returned by `--inspect`. The equivalent commands are `claim --claim-id`, `link-evidence --evidence-link-id` and `decision --decision-id`, with the same update/history flags and their existing content flags. Omitted update fields stay unchanged. An empty optional value clears it; empty comma-separated tags/actors/evidence clear those lists. `--no-deviation-from-fact` explicitly clears the decision flag. Use `--claim-revision N` on evidence links or decisions, or `--dossier-revision N` on claims, to select an exact associated revision during an update. Leaving that option out preserves an unchanged association's pin. ```python current = api.get_record("./novel", "dossier", dossier_id) updated = api.revise_record( "./novel", "dossier", dossier_id, changes={"body": "The corrected working note."}, expected_snapshot=current["snapshot"], expected_revision=current["record"]["revision"], change_kind="correction", reason="Correct a transcription", ) history = api.record_history("./novel", "dossier", dossier_id) original = api.get_record("./novel", "dossier", dossier_id, revision=1) ``` Source refresh never rewrites authored records. A newer-capture notice compares explicit passage citations with the source's latest available capture; the original quote remains visible. Copied body text alone does not establish an association with a source, and neither filenames nor titles are used to guess one. Existing historical imports can therefore remain unchanged until the author explicitly revises them. Historical dossier and evidence-link views resolve their pinned passages. Claim and decision views show the current evidence links to their pinned claim revision, with each link's ID and revision; they are not a reconstruction of the entire archive at an earlier date. ## Storage, lifecycle and recovery - `research/project.json`: immutable initial project configuration. - `research/revisions/`: immutable, schema-checked records. - `research/blobs/`: SHA-256-addressed original text/PDF bytes and extracted text; original bytes are not normalized. - `research/manifests/`: complete snapshots; `research/HEAD.json` selects one. - `.lixity/research/`: disposable catalogue and writer lock. ### Withdrawal vs. purge - **Withdrawal (`withdraw`)**: Preserves the audit trail and archived source text, records a tombstone in the manifest, excludes the source or version from default search, and marks existing citations as `availability: "withdrawn"`. Analysis fails closed for withdrawn sources. - **Purge (`purge`)**: Permanently removes source records, extraction records, and passage records from the store and unlinks unshared original blobs. Citations for purged passages fail closed with `ResearchError`. Supports `--dry-run` to preview affected records and blobs before deletion. Authored dossiers and evidence links remain for their audit trail, but their purged citations are marked `availability: "purged"` and expose no deleted quotation text. ### Cross-corpus comparison and evidence grounding - **Linguistic comparison (`compare`)**: Connects an archived research source with a literary manuscript without modifying either document. Computes: 1. Lexical overlap and alignment (Jaccard similarity, Szymkiewicz–Simpson overlap coefficient, top shared terms, exclusive source terms). The term lists are capped by `--top-n` (default 20; 1–100), which changes how much of the table is reported. 2. Signed keyness differential using a full term/nonterm $2\times2$ Dunning $G^2$ log-likelihood table; values can change from development builds. 3. Register and stylistic contrast (sentence length ASL delta, dialogue ratio delta, lexical diversity Guiraud's $R$ and Yule's $K$ deltas, staccato and kaskade deltas). 4. Per-chapter evidence grounding (mapping occurrences of source vocabulary and top key terms across individual manuscript chapters, reporting grounding density per 1,000 words). Comparison requires an explicit source and manuscript, or a deliberate Grounding action for the currently loaded manuscript. It returns a separate report and does not recalibrate either text's fingerprint. Grounding here means vocabulary occurrence, not semantic entailment, chronology validation or proof that a claim is supported. Source integrity and exact quotations establish what was retained, not its factual truth. Reports can be emitted as JSON (default) or formatted Markdown: ```bash # Output structured Markdown report to terminal or file lixity research compare --project ./novel --source-id --manuscript ./novel.md --format md lixity research compare --project ./novel --source-id --manuscript ./novel.md --format md --output ./grounding-report.md ``` Comparison counts, register contrasts and chapter grounding share the core prose scope: body paragraphs, list items and blockquotes; code, comments and footnote definitions are excluded. When chapters exist, only their bodies contribute, excluding headings, front matter and the configured appendix. Otherwise the remaining document prose is used with the same exclusions. A cross-language comparison still returns numeric overlap but sets `meta.lexical_comparable: false` and includes `"cross_language_lexical_comparison"` in `comparison_limits`; its lexical scores should not be interpreted as directly comparable vocabulary coverage. Comparison uses the source's language and the explicit manuscript language (English by default), with code-default corpus parsing. It does not inherit a manuscript project's custom `chapter_regex` or `appendix_marker`; inspect its chapter trace before comparing it with a differently configured dashboard. Guiraud and register deltas remain sensitive to length, language and discourse mode; matching a source's values is not a writing-quality objective. Back up the **entire `research/` directory**, preferably while no writer is running. Restore it to an explicit project root, run `audit`, then `reindex`. Never restore only SQLite. Do not commit private source text to a public repository: both blobs and passage records contain it. ## Export and restoration Lixity provides explicit commands to package and restore authoritative research stores with cryptographic checksum verification: ```sh lixity research export --project ./novel --output ./backup-novel.tar.gz lixity research restore --from ./backup-novel.tar.gz --to ./restored-novel ``` The archive packages all accepted records, revisions, snapshot manifests, original source blobs, and project configuration into a gzip-compressed tar archive. It includes an explicit `EXPORT_MANIFEST.json` with SHA-256 digests and byte lengths for every file. Disposable search indexes (`catalogue.sqlite3`) and lock files are excluded. Restoration requires a clean target directory (it refuses to overwrite an existing research store or non-empty project directory). Before placing the store, restoration verifies: 1. Member path safety (rejects path traversal, absolute paths, symlinks, and device nodes). 2. SHA-256 checksum and byte length of every extracted file against `EXPORT_MANIFEST.json`. 3. Snapshot rules, contiguous revision histories, and project identities. 4. Moves the verified store into place atomically. After restoring, audit and rebuild the search index: ```sh lixity research audit --project ./restored-novel lixity research reindex --project ./restored-novel ``` Writes use an OS lock, immutable publication and atomic HEAD replacement. Uncommitted objects left after interruption are ignored, not selected by date. Locks release when the process exits. A stale expected HEAD fails rather than overwriting another ingestion. Interrupted initialization may leave an incomplete directory: preserve it for inspection; `init` refuses to overwrite it. Files are flushed before publication; directory syncing is used on POSIX. If an interrupted authored edit leaves a valid next-revision file outside the accepted snapshot, the next save preserves its bytes under a content-addressed `.unpublished` filename in that revision directory before retrying. It is a recovery artifact, not an accepted revision or an automatic merge. Accepted history is never replaced; malformed or foreign files stop the save. ## Origin URL and capture provenance Version 1.18.0 adds an optional **Origin URL** field to the browser source import, `--origin-url` to the CLI and `origin_url=` to `api.ingest`. The value is retained as `SourceVersion.context.origin_url`, independently of title, tags and `provenance_note`. Source details, source listings and citation context expose it. In the `research sources` JSON response, the first source's current URL is at `.sources[0].context.origin_url`. Since v1.19.0, refreshing identical PDF bytes reuses the retained extraction after verifying the original blob, extracted text and passage spans. Changing only context, including the origin URL or Zotero binding, creates a new capture and passage records without rerunning OCR. Identical bytes and context remain a no-op. Earlier citations remain valid. Version 1.18.0 reruns extraction for this operation; changed PDF bytes still require extraction in v1.19.0. ```sh lixity research ingest --project ./novel --file ./notes.txt --allow-retention \ --origin-url https://example.org/archive/notes ``` ```python api.ingest("./novel", "notes.txt", allow_retention=True, origin_url="https://example.org/archive/notes", context={"tags": ["archive"], "provenance_note": "Author-supplied context."}) ``` The URL must be absolute HTTP(S), at most 2,000 characters, without credentials, whitespace or control characters. It is stored as submitted, never requested or used to accept a claim. Prefer stable source addresses over signed links with private query tokens. The browser displays it as escaped text. Failed validation keeps the draft; successful import clears the field with the rest of the form. On refresh, an omitted `origin_url` retains the context selected by the existing API: supplied `context`, or the latest capture's context when omitted. An explicit URL overrides `context.origin_url`. Changing only the URL creates a new capture; identical bytes and context remain a no-op. To clear a URL deliberately, supply a context mapping with `origin_url: null` (and any other metadata to retain). Earlier citations resolve their own capture and retain the original URL. URLs, filenames and titles never establish source identity automatically. **Compatibility:** source versions carrying a URL without a structured external reference use `research-local/2`, still at record revision 1, with a v2-or-later manifest. Zotero bindings in v1.19.0 require record and manifest v3. Their sequence counts source captures; it is separate from authored-record revision history. URL-free v1 records omit the new field entirely and retain their old encoding. Reading an old archive does not migrate it. Lixity 1.17.0 and earlier reject URL-bearing source records, including when they already support authored v2 revisions. Back up before this write, use compatible builds for every writer, and restore the complete pre-upgrade backup to roll back. Do not edit schemas or delete provenance fields by hand. ## Self-hosted PDF and OCR extraction PDF ingestion retains the original PDF bytes and extracted UTF-8 text as separate content-addressed blobs: ```sh lixity research ingest --project ./novel --file ./document.pdf --allow-retention ``` Version 1.18.0 also accepts native binary PDF upload through the browser file input. It reads the selected file as base64 for transport and passes the decoded bytes through the same import API as text and server-local files. The browser shows reading/extraction/completion states and prevents duplicate submits while waiting. This is not a streamed per-page progress feed. Successful local and Zotero captures also show returned warnings in the import status. Read those messages; a successful capture is not proof of OCR accuracy. ### Native PDFs versus scans Install local Poppler tools: `poppler-utils` on Debian/Ubuntu, or `poppler` with Homebrew. Both `pdftotext` and `pdftoppm` must be visible on the server's PATH. `pdftoppm` rasterizes physical pages at 150 dpi for native/worker extraction and 300 dpi for Tesseract; `pdftotext` extracts text layers. Without page rasterization, the configured scan worker cannot receive normal page inputs. Native text extraction may still work when the page count can be determined; `partial` does not indicate a complete OCR setup. An image-only scan has no usable text layer. Select [local Tesseract](#use-local-ocr-without-a-worker-service) or a configured self-hosted worker; Lixity does not install their engines, language data or models. Run diagnostics in the same environment as the server: ```sh lixity research ocr-status [--worker-cmd PATH] [--probe] ``` Pass `--probe` (or query `GET /api/research/ocr-status?probe=1`) to execute a synthetic request to the selected worker or Tesseract backend and report execution and latency. Native mode has no OCR worker to probe. A successful probe upgrades only `ready` to `ready (probed)`; it cannot override an invalid timeout/backend configuration or missing rasterizer (`partial`). Inspect `probe.ok` and configuration/dependency fields together. The diagnostic CLI still exits `0` when the probe fails, so exit status alone is not a health gate. The same information is available at `GET /api/research/ocr-status` and in `GET /api/research/status`. The dashboard maps these codes to localized labels: | Code | Configuration detected | Action | | --- | --- | --- | | `ready` / `ready (probed)` | Selected worker or Tesseract with requested language data, plus pdftoppm | Test an authorized scan and inspect extracted text | | `native_only` | Native extraction with both Poppler tools | Native PDFs work; select Tesseract or a worker for scans | | `partial` | Extraction/OCR engine available, pdftoppm missing | Install the rasterizer before multi-page use | | `misconfigured_worker` | Worker absent/not executable, invalid timeout, or failed probe | Check worker path, permissions, service environment and `LIXITY_OCR_TIMEOUT` | | `misconfigured_backend` | Invalid backend/native timeout, or unavailable Tesseract/language data | Check `LIXITY_OCR_BACKEND`, `LIXITY_OCR_LANGUAGES`, timeout and local installation | | `missing_dependencies` | No complete usable setup | Install Poppler and select a scan backend when needed | Configuration diagnostics inspect executable availability, not model weights, snapshot integrity, GPU resources or recognition accuracy. A synthetic probe checks only that request, not real-model or GPU validation. The worker model/recipe identifiers are configuration constants, not attestations. `ready` therefore means configured, not an end-to-end health check; use `--probe` to verify live execution. Native extraction refuses an undeterminable page count. If Poppler's `pdfinfo` is absent and the page tree is not readable in the raw file -- typically because a modern producer stored it in a compressed object stream -- the capture is refused with an explicit error rather than silently retaining only the first page. Install Poppler or configure `LIXITY_OCR_WORKER` to ingest such a file. Pass `--fallback` (or set `LIXITY_OCR_FALLBACK=1`) to permit native `pdftotext` extraction if worker execution fails/times out or Tesseract extraction fails. The result records a warning; review mixed/scanned pages for missing text. Without explicit fallback, these failures reject the capture. ### Worker interface Read [Unlimited-OCR integration and deployment boundaries](OCR_INTEGRATION.md) before provisioning inference. Baidu's official entrypoints do not implement this custom worker protocol; a separate adapter is required. Set `LIXITY_OCR_WORKER` to one executable path or a command name on PATH. It is invoked directly as `[worker, request_file]`, without shell parsing or embedded arguments. Use an absolute path without `~` for portable service configuration. `LIXITY_OCR_TIMEOUT` sets the worker subprocess deadline in seconds: an integer from 1 to 3600, default 120. This is a total invocation deadline, not per page. Set the variable in the service's environment and restart the server; exporting it in another terminal does not update an existing process. The temporary request JSON contains: ```json { "model_snapshot": "07dea832e22aefee32ad281d4b80551282e1c168", "recipe_revision": "d49ff64afffc1f47ab563dc1c589bc2f78808fa4", "pdf_path": "/absolute/path/to/input.pdf", "pages": [{"page_number": 1, "sha256": ""}] } ``` Those are the implementation's Baidu Unlimited-OCR integration pins (recipe date 2026-07-29). The worker must arrange its own runtime and model provisioning; this interface does not prove that an arbitrary executable uses those weights. The request includes PDF access and page checksums, not image paths or image bytes. A worker that needs raster images must obtain them from the local PDF. Return JSON on stdout; send logs to stderr: ```json {"blocks": [{"page_number": 1, "text": "Synthetic reading room note."}], "warnings": []} ``` Since v1.19.0, worker responses are validated before accepting a capture. Every block needs a nonempty string `text` and an integer `page_number` from the rendered request pages; at most 5,000 blocks are accepted. Optional `box` contains four finite numbers. Optional `confidence` must be a finite number from 0 to 1; omission or `null` means **unknown**, never an implicit score of 1.0. A supplied score is worker-reported, not an independently calibrated accuracy estimate. Every requested page must be covered by a block or listed in optional `completed_pages`. Use that integer list for pages successfully inspected with no readable text, such as a blank page; never use it to conceal a failed page. For example, a two-page document with a blank second page may return: ```json {"blocks": [{"page_number": 1, "text": "Synthetic note."}], "completed_pages": [2], "warnings": []} ``` Warnings, when supplied, must be a list of strings. Blocks are ordered by page number while retaining their supplied order within each page; their trimmed text is joined with blank lines. Coverage validates what the worker reports, not whether it recognized every word. A completely textless document still cannot be ingested as a searchable source. A configured worker's nonzero exit or timeout fails the import by default; explicit `--fallback` / `LIXITY_OCR_FALLBACK` can permit native text extraction with a warning. Invalid block structure or missing reported page coverage still fails closed. There is no silent native-text fallback. With no backend or worker configured, native text-layer extraction remains available and can omit scanned pages; inspect mixed PDFs for completeness. A successful native import is not an OCR-success claim. For local scans, use the built-in Tesseract backend described above. Custom model deployment belongs to a separately maintained adapter implementing this worker protocol. The former experimental CPU bridge was removed in 2.1.0; its limited experiment is recorded in [deployment boundaries](OCR_INTEGRATION.md#historical-cpu-experiment). ### What is actually retained `SourceVersion.blob` points to original PDF bytes; `Extraction.text_blob` points to extracted UTF-8 text. Passages cite exact character spans in that text. The extraction boundary computes page images, boxes and warnings, but the current archive does **not** persist those as audited page/coordinate/confidence records. The `research-ingest-local/1` response now includes runtime `warnings: list[str]` for committed, dry-run and unchanged results. Clean native/text extraction and reused retained extraction return `[]`; an unchanged response does not recover warnings from an earlier import. Successful `research-batch-ingest-local/1` items preserve warnings, as do Zotero single/batch captures alongside the existing explicit-capture notice. Existing keys, types and schema identifiers are unchanged. Save diagnostics separately if needed: these warnings are not persisted in source or extraction records. Likewise, it does not persist an attestation of which model ran. Inspect the original PDF for visual verification and retain operational diagnostics separately when needed. Core import is local; a user-configured worker has its own network behavior and must be deployed according to the project's privacy requirements. ## Safe Markdown and offline diagram rendering Dossier body text supports a safe Markdown subset and offline SVG diagram rendering directly in the browser without external dependencies, CDNs, or network access: - **Safe Markdown**: Paragraphs, headings (h1–h6), bold, italic, blockquotes, ordered/unordered lists, code spans, code blocks, and GFM tables. Raw HTML is strictly escaped to prevent cross-site scripting (XSS). - **Offline diagrams**: Fenced code blocks with `mermaid` syntax are parsed and converted to native inline SVGs by Lixity's built-in vector generator. Supported types include flowcharts (`graph TD`, `graph LR`) and sequence diagrams (`sequenceDiagram`). - **Source inspectability**: Any rendered diagram provides a collapsible "Show diagram source" control to view the raw specification. ## Limits The local archive uses versioned `research-local/*` records and operation-specific envelopes, not the RFC's illustrative `research/1` bundle. Do not assume every command or stored record has the same schema version. Structured Zotero captures require `research-local/3` and v3 manifests; unified search uses `research-search-local/2`, while source-only search retains v1. Since v3.0.0, causal claim relations add `research-local/6` and an opt-in v6 manifest; historical readers before v3.0.0 reject it, while v0.3.0 supports it. See [chronology and causal-link compatibility](#claim-chronology-and-causeeffect-links-since-v300). Timeline reads, citation selection and the claims-list API change do not rewrite existing v1–v5 archives. Native editing in version 1.17.0 adds `research-local/2` for authored revisions and `research-manifest-local/2` for snapshots containing them. The first accepted edit upgrades that snapshot; reading a revision-1 archive does not rewrite any files. Older Lixity versions reject the new snapshot. Back up the whole archive before upgrading, use the same supported version for all writers, and restore the full backup if reverting to 1.16.0. Never edit HEAD or revision files to downgrade. Unknown fields and versions are rejected. Source identity metadata cannot yet be edited; refresh creates a new source version, not a changed old record. The 25,000-entry snapshot limit includes retained revisions. Status counts show current logical records; the audit's `records` count includes their history. Text inputs are at most 2 MiB and must be valid UTF-8 without NUL bytes. PDF inputs are at most 50 MiB. Both routes allow at most 5,000 nonempty extracted passages. Search uses literal Unicode words joined with AND; it is not stemming, translation or semantic search. BM25 scores are ranks, not truth probabilities. Language tags do not translate quotations. Empty queries, stale indexes and missing FTS5 support produce errors, not silent fallbacks. Source analysis emits `original_language_supplied` when source context names an original language; that free-text field alone does not establish historical language variety. This replaces the earlier experimental `historical_language` limitation code. Audits check retained records, hashes and citations, not factual accuracy, permissions or historical completeness. A citation's `availability` indicates whether its source is currently available or withdrawn; a purged passage no longer resolves. The web panel displays those evidence states for linked passages. The [licensing rules](../LICENSING.md) still apply; imported sources keep their own rights.