commit 6f0b4785e0b18f528b2594ea860abb4cad85431f Author: Jeremy Anderson Date: Mon Sep 28 13:48:27 2026 -0400 Thicket - Super-Injest diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..4098570 --- /dev/null +++ b/.gitignore @@ -0,0 +1,14 @@ +__pycache__/ +*.py[cod] +.pytest_cache/ +*.egg-info/ +dist/ +build/ +logs/ +graphify-out/ +.graphify/ +.lightrag/ +.thicket/ +Ingested_Brain/ +ingested-archive/ +thicket_smoke.png diff --git a/BLOG.md b/BLOG.md new file mode 100644 index 0000000..e4675ab --- /dev/null +++ b/BLOG.md @@ -0,0 +1,190 @@ +# Growing a Thicket + +*Building a super-ingest + RAG console that treats a document library +like a living ecosystem, not a filing cabinet.* + +**Jeremy Anderson** — [dcos.net](https://dcos.net) — info@dcos.net +September 2026 + +--- + +A few weeks ago a friend's Gemini session produced ~270 lines of +Python titled "brain ingest": walk a folder, extract text from PDFs +and EPUBs, write Markdown notes into an Obsidian vault, chunk, embed, +and pour the results into Qdrant. It was a good sketch. It also had +four bugs I cared about, an import surface that required every heavy +dependency up front, and a LightRAG binding pinned to one release of +a fast-moving API. + +We turned it into **Thicket**: a super-ingest + retrieval console with +the same brushed-aluminum, amber-LED MMD3 interface as my transcoder +[OpenTranscode](http://git.dcos.net/dcosnet/OpenTranscode) — because +the tool you actually use is the tool you actually enjoy opening. + +The name is the design brief. A thicket is dense, self-connected, and +grows on its own. Feed it documents; it grows a thicket. + +## The architecture bet: one engine, two drivers + +The single most important decision was making the pipeline core +Qt-free. `IngestPipeline` knows nothing about widgets: it walks a +stage table and reports progress through four plain callables. + +Two drivers share it: + +- a **QThread worker** that wires those callables to Qt signals, and +- a **headless CLI** (`thicket --ingest ... --vault ...`) that wires + them to `print`, works over SSH, and fits in a cron line. + +One behavior change lands in both drivers at once. There is no "GUI +logic" to port or keep in sync — that category of bug cannot exist. + +## Stage tables instead of branch nests + +Every per-document decision is data, not control flow. The pipeline +stage order is a tuple of `(status, gate, runner)`; file-type +extraction is a dict keyed by extension; readiness checks before an +ingest run are a table where the first enabled-but-not-ready entry +wins. Adding a stage, a format, or a gate means adding an entry — +never a new `if` ladder. The QA pass measured this concretely: the +extractor went from a five-branch dispatch to a one-line dict lookup. + +## Lazy by default, graceful by contract + +The console starts on a bare system. Every heavy import — FastEmbed, +the Qdrant client, EbookLib, LightRAG — happens at point of use, and +a background probe reports module and service readiness into the LED +status strip. Stages that cannot run are blocked at INGEST with the +exact reason and the exact fix; nothing crashes, nothing silently +no-ops. + +Where a choice of paths exists, the code steps down the chain +explicitly, best option first: Qdrant search uses `query_points` and +falls to `search` on older clients; the LightRAG Ollama binding +resolves across the package's API generations; podman networking that +cannot create a tap device gets `--network=host`. Unix philosophy in +practice: try the clean path, degrade to the simple one, keep going. + +## Re-ingestion is an overwrite + +The subtlest correctness property in the system: **the point count +after N re-ingests equals the count after the first.** Deterministic +point IDs (MD5-UUID of `title|path|index`) plus a delete-by-filter +before every upsert make editing a source and re-ingesting it an +exact replacement — a shrunken document leaves no stale tail chunks. +We verify this in the live QA run: 4 points, re-ingest, still 4 +points, same retrieval rankings. + +The vault side holds the same invariant: re-ingesting a source +refreshes its note in place, and a *different* document with the same +title claims a digest-suffixed sibling rather than clobbering it. + +## Local embeddings, on purpose + +FastEmbed runs ONNX on your own cores; nothing leaves the machine. +Retrieval quality still respects the model's contract — BGE models +want an instruction prefix on the *query* side and bare passages on +the *document* side, so Thicket prefixes queries only. It is the kind +of detail that silently costs you ten points of relevance when a +sketch gets it wrong. + +## The QA pass that shaped the code + +Before calling it production-ready, the codebase went through a +five-hat review — senior QA, Linux engineer, architect, admin, and +devops PM. What changed: + +- **Table-driven dispatch everywhere** it beat nested conditionals + (stage table, extension table, readiness table, stage-color map). +- **Loops reduced to comprehensions and `next()`** where iteration + was bookkeeping; explicit loops remain only where iteration *is* + the semantics (chunk word windows, queue walks). +- **Bounded reads** on files we do not own (frontmatter collision + checks read at most 32 lines). +- **One decisive failure path per scope** — per-file isolation in the + pipeline, a single report-and-disable path in the retrieval worker. +- Comments state invariants and contracts. Version-history narration + does not survive review; the code reads like decisions, not like an + argument with itself. + +Standards kept in view: PEP 8 throughout, SEI CERT practices (bounded +I/O, precise exception scope — the Qdrant step-down catches +`AttributeError`, not the world), MISRA-style bounded structured +control flow, and POSIX assumptions (paths via `pathlib`, no platform +branches, systemd/cron-friendly headless mode). + +## Live-fire verification + +The release gate was not the test suite alone (26 tests, no services +required) but a live run: podman Qdrant up, three documents through +the full GUI pipeline, two semantic queries returning correctly +ranked hits, an idempotent re-ingest, and a dry-run probe reporting +every module and service green. Screenshot or it didn't happen — the +console looks the part too: knobs for chunk size, overlap, and +Top-K; an LED queue table tracking every file's stage; a phosphor +log; a retrieval strip at the bottom. + +## What's next + +The graph stage (LightRAG over Ollama) is wired and gating on +readiness, but it is deliberately optional — entity extraction is an +LLM pass per document, and the vault + vector stages already answer +the daily question: *where did I read that?* + +The thicket grows. Pull it, feed it a shelf of books, and see what +surfaces. + +--- + +**Thicket** — AGPL-3.0-or-later — [git.dcos.net/dcosnet/Thicket](http://git.dcos.net/dcosnet/Thicket) +Jeremy Anderson — info@dcos.net — [dcos.net](https://dcos.net) + +--- + +## Addendum — v1.8: the thicket grows roots + +*September 2026, after the first full functionality matrix.* + +The sketch became a workstation tool. What changed since the first +essay: + +**Destinations, not stages.** The original three-stage line (notes → +Qdrant → LightRAG) became a destination model: `obsidian` for +notes-only, or any of ten open-source vector stores — qdrant, chroma, +lancedb, faiss, milvus, weaviate, pgvector, duckdb, sqlite-vec, +mariadb — one payload schema, one exact-replacement contract, cosine +scores identical to four decimals across all ten. The console's OUT +line resolves the real destination live; the status footer follows the +target selector keystroke by keystroke. + +**Sources are never deleted.** The dangerous delete-after-verify +checkbox from the transcoder lineage is gone. Verified sources move to +an archive directory and bzip2-compress in place — the incoming tree +stays clean, the archive always decompresses. + +**The corpus is technical.** Fenced code blocks became atomic +language-tagged chunks; prose became paragraph-aligned; shebangs +stopped masquerading as headings; twenty-one source/config extensions +ingest as listings; the default embedder became jina-code, and +"force browsers to refuse plain http" retrieves the nginx block where +HSTS actually lives. + +**The workstation has a filesystem.** /mnt/AI/corpus/{cold,hot} is now +the default corpus flow — cold in, hot brain — with ~/$VAR expansion +everywhere and THICKET_AI_ROOT for other hosts. + +**Two interaction modes.** SEARCH embeds locally; ASK turns natural +language into read-only SQL over the Postgres/MariaDB corpora through +Vanna 2's agent API and the same Ollama selector. "Which document has +the most chunks?" is now a console question. + +**The matrix.** `scripts/func_test.py` runs eighteen live checks — +every destination with idempotency and retrieval assertions, both +graph engines, both archives, ask on both SQL targets. Its first run +caught six real bugs, including a subtle one: milvus-lite's embedded +server keeps the database file lock after client close, so a second +process gets `[Errno 11]` — fixed with an explicit `close()` lifecycle +that releases the server manager. Production readiness is not a claim; +it is a rerunnable script. + +— Jeremy Anderson · info@dcos.net · dcos.net diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..16efb73 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,54 @@ +# Changelog + +## 1.8.1 — production gate + +- Live functionality matrix (`scripts/func_test.py`): every destination, + both graph engines, both archives, ask on both SQL targets — 18/18. +- Fixed: LanceDB schema stability (JSON-string payloads), pgvector + per-document rollback, MariaDB stale-dimension guard, Weaviate IPv4 + pin, LightRAG ≥ 1.5 async lifecycle, Milvus Lite lock release + (`close()` lifecycle + open retry). +- `pyflakes` clean across package, scripts, and tests. + +## 1.8.0 — the workstation's filesystem + +- `/mnt/AI` layout awareness: `corpus/cold` → IN, `corpus/hot` → vault, + `corpus/archive` for the bz2 stage; `THICKET_AI_ROOT` override. +- `~` and `$VAR` expansion through one shared `expand_path`. + +## 1.7.x — destinations, not stages + +- TARGET selects the destination: `obsidian` (notes only) or any vector + store (notes + that store). Vault/vector checkboxes removed. +- Sources are never deleted: archive stage moves verified sources to + `ingested-archive/` and bzip2-compresses them (streaming, level 9). +- Skip-unchanged (content-hash manifest), MAX MB guard, custom notes dir. +- Dynamic Paths panel: live OUT resolution per target; status footer + follows the target selector; per-target connection hints. + +## 1.5.0–1.6.0 — technical corpora + +- Code-preserving chunker: fenced blocks atomic with language tags, + paragraph-aligned prose, shebang-safe headings, line-windowed + oversized blocks. +- 21 source/config extensions ingest as language-tagged listings. +- Embedder catalog: jina-code default (English + code), bge-base, + bge-small; chunk payloads carry `chunk_kind` / `lang`. + +## 1.4.0 — graphs, ask, objects + +- Graphify engine beside LightRAG (`--graph-engine`). +- ASK: natural-language SQL over pgvector/MariaDB corpora (Vanna 2 + agent + Ollama; CLI and console). +- MinIO archive stage (object storage for originals). + +## 1.2.0–1.3.0 — ten vector targets + +- qdrant, chroma, lancedb, faiss, milvus, weaviate, pgvector, duckdb, + sqlite-vec, mariadb — one registry, one payload schema, cosine-score + parity to four decimals. + +## 1.0.0–1.1.0 — the console + +- MMD3 console (OpenTranscode lineage): LED queue table, phosphor log, + knobs, live probe. Qt-free pipeline core shared by GUI and CLI. diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..633fc84 --- /dev/null +++ b/LICENSE @@ -0,0 +1,673 @@ +Thicket — super-ingest + RAG console +Copyright (C) 2026 Jeremy Anderson — https://dcos.net + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. + +The full license text follows. + +--- + + GNU AFFERO GENERAL PUBLIC LICENSE + Version 3, 19 November 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU Affero General Public License is a free, copyleft license for +software and other kinds of works, specifically designed to ensure +cooperation with the community in the case of network server software. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +our General Public Licenses are intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + Developers that use our General Public Licenses protect your rights +with two steps: (1) assert copyright on the software, and (2) offer +you this License which gives you legal permission to copy, distribute +and/or modify the software. + + A secondary benefit of defending all users' freedom is that +improvements made in alternate versions of the program, if they +receive widespread use, become available for other developers to +incorporate. Many developers of free software are heartened and +encouraged by the resulting cooperation. However, in the case of +software used on network servers, this result may fail to come about. +The GNU General Public License permits making a modified version and +letting the public access it on a server without ever releasing its +source code to the public. + + The GNU Affero General Public License is designed specifically to +ensure that, in such cases, the modified source code becomes available +to the community. It requires the operator of a network server to +provide the source code of the modified version running there to the +users of that server. Therefore, public use of a modified version, on +a publicly accessible server, gives the public access to the source +code of the modified version. + + An older license, called the Affero General Public License and +published by Affero, was designed to accomplish similar goals. This is +a different license, not a version of the Affero GPL, but Affero has +released a new version of the Affero GPL which permits relicensing under +this license. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU Affero General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Remote Network Interaction; Use with the GNU General Public License. + + Notwithstanding any other provision of this License, if you modify the +Program, your modified version must prominently offer all users +interacting with it remotely through a computer network (if your version +supports such interaction) an opportunity to receive the Corresponding +Source of your version by providing access to the Corresponding Source +from a network server at no charge, through some standard or customary +means of facilitating copying of software. This Corresponding Source +shall include the Corresponding Source for any work covered by version 3 +of the GNU General Public License that is incorporated pursuant to the +following paragraph. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the work with which it is combined will remain governed by version +3 of the GNU General Public License. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU Affero General Public License from time to time. Such new versions +will be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU Affero General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU Affero General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU Affero General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU Affero General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If your software can interact with users remotely through a computer +network, you should also make sure that it provides a way for users to +get its source. For example, if your program is a web application, its +interface could display a "Source" link that leads users to an archive +of the code. There are many ways you could offer source, and different +solutions will be better for different programs; see section 13 for the +specific requirements. + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU AGPL, see +. diff --git a/QUICKSTART.md b/QUICKSTART.md new file mode 100644 index 0000000..77892f1 --- /dev/null +++ b/QUICKSTART.md @@ -0,0 +1,175 @@ +# Thicket — Quick Start + +Five minutes from clone to retrieval. Linux, Python 3.12+. + +## 0. Zero-config on an AI workstation + +If `/mnt/AI` exists, Thicket detects it: IN defaults to +`/mnt/AI/corpus/cold`, the vault to `/mnt/AI/corpus/hot`, and the +bz2 archive stage targets `/mnt/AI/corpus/archive`. Drop documents in +`corpus/cold`, press INGEST. (Point IN at `corpus/books` to chew +through the standing library.) + +## 1. Install + +```bash +git clone http://git.dcos.net/dcosnet/Thicket.git # or your local copy +cd Thicket +# venv lives in the AI tree: /mnt/AI/runtime/thicket-venv +true # (scripts/bootstrap_sources.sh creates it) +/mnt/AI/runtime/thicket-venv/bin/pip install -e ".[ingest]" # parsers + FastEmbed + Qdrant client +/mnt/AI/runtime/thicket-venv/bin/pip install -e ".[graph]" # optional: LightRAG stage +``` + +That's it for setup — ONE venv, always. From here on: + +```bash +thicket # anywhere — ~/.local/bin command +/mnt/AI/runtime/thicket-venv/bin/python thicket.py # explicit interpreter +/mnt/AI/tools/bin/thicket # AI-tree launcher +``` + +(A bare `python thicket.py` uses the system interpreter, which cannot +see the venv — Python's rule. The `thicket` command exists so you +never need to think about it.) + +## 2. Services (optional — depends on your vector target) + +The vault-note stage needs nothing but the install. The vector stage +targets ten open-source stores; six of them run **embedded** (files +under `/.thicket/`, zero services): + +```bash +/mnt/AI/runtime/thicket-venv/bin/pip install -e ".[chroma]" # or lancedb / faiss / milvus / + # duckdb / sqlitevec, or +/mnt/AI/runtime/thicket-venv/bin/pip install -e ".[targets]" # every target extra at once + +/mnt/AI/runtime/thicket-venv/bin/python thicket.py --ingest ~/Books --vault ~/Vault --target chroma +``` + +Only the default `qdrant` target needs a service: + +```bash +# Qdrant (service-backed target) +podman run -d --name thicket-qdrant -p 6333:6333 \ + -v thicket_qdrant:/qdrant/storage docker.io/qdrant/qdrant +# docker works identically; --network=host sidesteps rootless +# networking issues on some kernels. + +# Ollama (graph engines + ASK) +ollama pull llama3.1 +ollama pull nomic-embed-text +``` + +## 3. Check readiness + +```bash +/mnt/AI/runtime/thicket-venv/bin/python thicket.py --dry-run +``` + +You want `READY: vault + Qdrant stages available.` — every MISSING +module or DOWN service is listed with the exact fix. + +## 4. First ingest + +**GUI** (the console): + +```bash +/mnt/AI/runtime/thicket-venv/bin/python thicket.py +``` + +1. Set **IN** to a folder of `.pdf` / `.epub` / `.md` / `.txt` files + and **VAULT** to your Obsidian vault root. +2. Press **SCAN QUEUE** — the LED table previews every document found. +3. Press **> INGEST**. Watch the stage column walk each file through + EXTRACT → VAULT → INDEX → … → DONE (graph/archives add their own + stages; the OUT line under IN/VAULT shows exactly where data lands). +4. Type a question in the retrieval strip and press **SEARCH** — + hits land in the log with score, document, and section. + +**Headless** (SSH / cron friendly — identical pipeline, no Qt): + +```bash +/mnt/AI/runtime/thicket-venv/bin/python thicket.py \ + --ingest ~/Downloads/Raw_Books_And_Papers \ + --vault ~/Documents/ObsidianVault +``` + +The first run downloads the embedding model (~160 MB, once — the +default jina-code embedder is tuned for technical corpora); notes +appear in `/Ingested_Brain/`, vectors in the `second_brain` +collection. + +## 4b. Ask questions in natural language (SQL targets) + +With the corpus in Postgres or MariaDB, skip SQL entirely: + +```bash +/mnt/AI/runtime/thicket-venv/bin/python thicket.py --ask "which document has the most chunks?" \ + --target pgvector --ollama-llm llama3.1:latest +``` + +Connection settings come from `/mnt/AI/backends/thicket.env` +(bootstrap writes it, chmod 600); exported `PG*` / `MARIADB_*` / +`MINIO_*` variables override the file, and defaults apply last. + +Needs `pip install -e ".[ask]"` (Vanna 2) and Ollama. + +## 4c. Archive originals to MinIO (optional) + +```bash +MINIO_ENDPOINT=localhost:9000 MINIO_ACCESS_KEY=... MINIO_SECRET_KEY=... \ +/mnt/AI/runtime/thicket-venv/bin/python thicket.py --ingest ~/Books --vault ~/Vault \ + --target chroma --minio +``` + +Notes gain a `source_uri: s3://bucket/key` frontmatter line. + +## 5. Verify retrieval + +From the GUI retrieval strip, or headless: + +```bash +.venv/bin/python - <<'EOF' +from thicket.embedder import DEFAULT_EMBED_MODEL, EmbeddingEngine +from thicket.vector_stores import create_store + +engine = EmbeddingEngine(DEFAULT_EMBED_MODEL); engine.load() +store = create_store("qdrant", collection="second_brain", dim=engine.dim) +store.set_embedder(engine); store.ensure_collection() +try: + for hit in store.search(engine.embed_query("your question here"), limit=3): + p = hit["payload"] + print(f"[{hit['score']:.3f}] {p['chunk_kind']:5s} " + f"{p['document_title']} § {p['section_header']}") +finally: + store.close() +EOF +``` + +## Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| `Qdrant: DOWN` in probe / dry-run | service not running | start the container (step 2) | +| `fastembed MISSING` | extras not installed | `/mnt/AI/runtime/thicket-venv/bin/pip install -e ".[ingest]"` | +| `ebooklib MISSING` | EPUBs will fail | same extras install as above | +| INGEST blocked: dimension mismatch | collection built with a different embedding model | pick a new collection name (or delete the old collection) | +| `Ollama: DOWN` | only the graph stage needs it | start Ollama, or leave the graph stage off | +| Empty documents SKIP | scanned PDFs have no text layer | OCR first (e.g. `ocrmypdf`), then ingest | +| Re-ingest count unchanged | that is correct — re-ingest replaces, never duplicates | nothing to fix | + +## Day-two operations + +```bash +podman stop thicket-qdrant && podman start thicket-qdrant # restart service +podman volume rm thicket_qdrant # wipe vectors (notes stay) +/mnt/AI/runtime/thicket-venv/bin/python thicket.py --ingest ... --target obsidian # notes only +/mnt/AI/runtime/thicket-venv/bin/python thicket.py --ingest ... --skip-unchanged # cheap re-runs +/mnt/AI/runtime/thicket-venv/bin/python thicket.py --ingest ... --archive # bz2 the sources away +.venv/bin/python scripts/func_test.py # full live matrix +``` + +--- + +Questions: Jeremy Anderson — info@dcos.net — https://dcos.net diff --git a/README.md b/README.md new file mode 100644 index 0000000..257f7b5 --- /dev/null +++ b/README.md @@ -0,0 +1,276 @@ +# Thicket + +**Super-ingest + RAG console** — feed it documents; it grows a thicket: +dense, interconnected, searchable. Import PDF / EPUB / Markdown / plain +text / source-and-config files into a destination of your choice, +wrapped in the same retro-futuristic MMD3 console UI as OpenTranscode +(brushed aluminum, amber LEDs, green phosphor log, rotary knobs). + +``` + ┌─▶ Obsidian vault ──▶ MinIO bucket* ┐ +documents ──▶ notes │ (Markdown + (s3:// URIs) ├─▶ ingested-archive/ + │ YAML frontmatter) │ (bz2, never deleted) + └─▶ vector index ──▶ knowledge graph* ─┘ + (10 targets, (LightRAG or + local FastEmbed) Graphify, Ollama) + + TARGET picks the destination: obsidian = notes only, + any vector store = notes + that store. * optional stages +``` + +| Doc | What it is | +|---|---| +| [QUICKSTART.md](QUICKSTART.md) | clone → services → first ingest → first retrieval | +| [BLOG.md](BLOG.md) | design essay + v1.8 addendum | +| [LICENSE](LICENSE) | AGPL-3.0-or-later, full text | + +## Destinations and stages + +Vault notes always run — they are the product. Everything else stacks: + +| Piece | What it does | Needs | +|---|---|---| +| **Vault notes** *(always on)* | Normalized Markdown notes (YAML frontmatter: title, source file, optional `source_uri`, ingest timestamp, `brain/ingested` + `source/` tags) into the notes dir (default `/Ingested_Brain/`). PDFs get `## Page N` headings; EPUBs are chapter-split; code files render as language-tagged listings. `TARGET=obsidian` = notes only. | `python-slugify` | +| **Vector index** *(a TARGET away)* | Structure-preserving contextual chunking, local ONNX embeddings (FastEmbed), exact-replacement indexing — the point count after N re-ingests equals the count after the first. Ten interchangeable targets (below). | `fastembed` + the target's library | +| **Knowledge graph** *(optional)* | **LightRAG** (merged entity graph in `/.lightrag`, one Ollama pass per document) or **Graphify** (batch `graphify extract --backend ollama` → `graph.json`, `GRAPH_REPORT.md`, interactive `graph.html` in `/.graphify`). | engine package, Ollama | +| **MinIO archive** *(optional)* | Originals uploaded to an S3-compatible bucket (key = input-tree path, idempotent); the note records `s3://bucket/key`. Open-source MinIO has no vector API — it is an archive stage, not a vector target. | `minio`, MinIO service | +| **Source archive** *(optional)* | Verified sources *move* out of the incoming tree into `ingested-archive/` and are bzip2-compressed (streaming, level 9). Sources are never deleted; the archive always holds a decompressible original. | nothing | + +Sources are never deleted — by design, not by flag. Guards (MAX MB, +skip-unchanged) skip files entirely; skips never archive. + +### Vector targets + +All ten share one payload schema (`document_title`, `section_header`, +`content`, `chunk_kind`, `lang`, …), one retrieval strip, and the same +exact-replacement semantics — cosine scores agree to four decimals +across targets (verified by the live matrix). + +| Target | Mode | Library | Service needed | Install extra | +|---|---|---|---|---| +| `qdrant` *(default)* | service | `qdrant-client` | Qdrant container | `.[ingest]` | +| `chroma` | embedded | `chromadb` | none | `.[chroma]` | +| `lancedb` | embedded | `lancedb` | none | `.[lancedb]` | +| `faiss` | file | `faiss-cpu` | none | `.[faiss]` | +| `milvus` | embedded (Lite) | `pymilvus[milvus_lite]` | none | `.[milvus]` | +| `weaviate` | service | `weaviate-client` | Weaviate container (8080/50051) | `.[weaviate]` | +| `pgvector` | service | `psycopg` + `pgvector` | Postgres + pgvector (libpq env: `PG*`) | `.[pgvector]` | +| `duckdb` | embedded | `duckdb` (+vss index, steps down to exact scan) | none | `.[duckdb]` | +| `sqlitevec` | embedded | `sqlite-vec` (vec0 tables) | none | `.[sqlitevec]` | +| `mariadb` | service | `PyMySQL` | MariaDB 11.7+ (env: `MARIADB_*`) | `.[mariadb]` | + +Embedded targets keep data under `/.thicket//` — a vault +is one portable tree. Service targets read Unix-standard environments: +`PGHOST`/`PGUSER`/`PGPASSWORD`/`PGDATABASE` (or `PGDSN`), +`MARIADB_HOST`/`MARIADB_USER`/`MARIADB_PASSWORD`/`MARIADB_DATABASE`. +`pip install -e ".[targets]"` installs every target extra at once. + +**Upgrading from 1.1.x:** points indexed before 1.2.0 lack the internal +`doc_key`; the first re-ingest of each document cleans them up +automatically — or start a fresh collection name. + +## AI filesystem layout awareness + +When the canonical `/mnt/AI` tree exists, Thicket adopts its corpus flow +as defaults — zero configuration: + +| Canonical path | Thicket role | +|---|---| +| `/mnt/AI/corpus/cold` | IN — incoming raw documents | +| `/mnt/AI/corpus/hot` | VAULT — the active brain: notes + `.thicket/` vector data + graphs in one tree | +| `/mnt/AI/corpus/books` | standing library (reported by the probe; point IN at it to ingest) | +| `/mnt/AI/corpus/archive` | destination of the source-archive (bz2) stage | +| `/mnt/AI/backends/thicket.env` | connection profiles (600) — fills `PG*` / `MARIADB_*` / `MINIO_*` gaps; the real environment always wins | + +Without the layout, home-directory defaults apply — behavior identical. +Override the root with `THICKET_AI_ROOT`. Every path field accepts `~` +and `$VAR` references, resolved through one shared expansion point in +the GUI, CLI, and core. + +## Technical corpora (code, configuration, policy) + +Ingestion preserves what makes technical documents useful: + +- **Fenced code blocks are atomic** — never split mid-listing, never + merged with prose, indentation and blank lines verbatim; the language + tag travels in the chunk context (`| code:python`) and the payload + (`chunk_kind`, `lang`). Oversized blocks window by lines with overlap. +- **Prose chunks are paragraph-aligned** — lists, commands, and tables + keep their line structure in stored content. +- **ATX headings require `#` + whitespace** — shebangs and `#comments` + never masquerade as section headers. +- **Source files ingest directly**: `.py .sh .bash .zsh .yaml .yml .toml + .ini .conf .cfg .json .sql .rs .go .c .h .cpp .js .ts .tf .nix` are + wrapped as language-tagged listings. +- **Code-strong default embedder**: `jinaai/jina-embeddings-v2-base-code` + (English + code, 768-dim, 8k context); curated alternatives in the + EMBED selector. Dimension is fixed per collection — switching models + means a new collection name. +- **Graph tip**: directories of real code want the `graphify` engine — + tree-sitter gives it per-symbol structure no prose pass can match. + +## The console + +- **Paths panel** — IN, VAULT, and a live **OUT** line resolving the + actual destination per TARGET (notes dir, `.thicket/` data path, or + service/collection URI), re-resolved on every edit +- **Pipeline panel** — TARGET is the destination (`obsidian` or one of + ten vector stores) with collection/host/port and a per-target + connection hint line; stage toggles (graph, MinIO, bz2 archive); + EMBED / OLLAMA LLM / GRAPH ENGINE selectors; FILTER, NOTES DIR, + MAX MB, Skip-unchanged +- **Document queue** — LED matrix table with per-file stage (QUEUED → + EXTRACT → MINIO → VAULT → INDEX → GRAPH → ARCHIVE → DONE / SKIP / + ERROR), color-coded, with detail column +- **Knobs** — chunk size, chunk overlap, retrieval Top-K +- **Retrieval strip** — **SEARCH** (semantic across the collection) and + **ASK (SQL)** (natural language over SQL-backed targets via Vanna 2 + + the Ollama LLM); results land in the log +- **Status footer** — live target + service readiness, updated the + instant TARGET changes +- **Transport** — `> INGEST`, `[] STOP` (cooperative), `~~ SCAN QUEUE` + (preview), `? ABOUT` +- **Probe** — every dependency and service reported at startup; a stage + that cannot run is blocked at INGEST with the exact reason and fix + +## Install + +**One venv, always**: `/mnt/AI/runtime/thicket-venv` is the single environment. +The source-anchored bootstrap mirrors every dependency as a git +checkout under `/mnt/AI/distfiles/git/`, builds them into that same +venv, and drops a launcher in `/mnt/AI/tools/bin`: + +```bash +scripts/bootstrap_sources.sh # add --force-source for +/mnt/AI/tools/bin/thicket --dry-run # native builds from git +``` + +The venv is created and owned by the bootstrap at +`/mnt/AI/runtime/thicket-venv` — nothing is written inside the project +checkout (`.gitignore` keeps it archive-clean). Plain pip path: + +```bash +# venv lives in the AI tree: /mnt/AI/runtime/thicket-venv +true # (scripts/bootstrap_sources.sh creates it) +/mnt/AI/runtime/thicket-venv/bin/pip install -e . # console only (PySide6) +/mnt/AI/runtime/thicket-venv/bin/pip install -e ".[ingest]" # + parsers, FastEmbed, qdrant +/mnt/AI/runtime/thicket-venv/bin/pip install -e ".[targets]" # + every vector target +/mnt/AI/runtime/thicket-venv/bin/pip install -e ".[graph,graphify]" # + both graph engines +/mnt/AI/runtime/thicket-venv/bin/pip install -e ".[ask,minio]" # + Vanna ask, MinIO archive +``` + +Services (only for the stages that need them): + +```bash +podman run -d --name thicket-qdrant -p 6333:6333 \ + -v thicket_qdrant:/qdrant/storage docker.io/qdrant/qdrant +ollama pull llama3.1 && ollama pull nomic-embed-text +``` + +The first ingest downloads the embedding model locally (~160 MB for the +default jina-code embedder). + +## Usage + +One command owns every mode — no flags loads the console: + +```bash +python thicket.py # the console — also: ./thicket.py +``` + +**Headless** (same core, no Qt — SSH / cron friendly): + +```bash +python thicket.py --ingest /mnt/AI/corpus/cold --vault /mnt/AI/corpus/hot +python thicket.py --ingest ~/Books --vault ~/Vault --target obsidian # notes only +python thicket.py --ingest ~/Books --vault ~/Vault --target chroma # embedded, no service +python thicket.py --ingest ~/Books --vault ~/Vault --lightrag --graph-engine graphify \ + --ollama-llm llama3.1:latest +python thicket.py --ingest ~/Books --vault ~/Vault --skip-unchanged --max-mb 200 +python thicket.py --ingest ~/Books --vault ~/Vault --archive --minio +python thicket.py --ask "which document has the most chunks?" --target pgvector +``` + +**Probe & report**: `python thicket.py --dry-run` · `--version` + +Three interchangeable entry points share one environment: the `thicket` +command (installed to `~/.local/bin` by the bootstrap — works from any +directory), `/mnt/AI/tools/bin/thicket`, and `/mnt/AI/runtime/thicket-venv/bin/python thicket.py`. +A bare `python thicket.py` uses the system interpreter, which cannot see +the venv — that is Python's rule, not a Thicket setting. + +## Architecture + +``` +thicket/ +├── thicket.py # one-command launcher (GUI default, flags pass through) +├── cli.py # version / dry-run / headless ingest / ask / GUI +├── layout.py # /mnt/AI taxonomy awareness + ~/$VAR expansion +├── extractors.py # PDF / EPUB / MD / TXT / code-config -> (title, text) +├── vault_writer.py # notes: frontmatter, escaping, collisions, source_uri +├── chunker.py # structure-preserving chunking (fences, langs, paragraphs) +├── embedder.py # FastEmbed wrapper (curated catalog, jina-code default) +├── vector_stores.py # registry + 10 targets, close() lifecycle +├── qdrant_store.py # qdrant target +├── graph_store.py # graph engines: LightRAG (async lifecycle) + Graphify +├── minio_archive.py # S3 object archive stage +├── ask_vanna.py # Vanna 2 agent: natural-language SQL over SQL targets +├── pipeline_core.py # the stage table — shared by GUI worker and CLI +├── pipeline_worker.py # QThreads: probe / ingest / search / ask +├── env_probe.py # module + service readiness (incl. liveness tables) +├── ui_theme.py # MMD3 QSS (OpenTranscode visual lineage) +├── ui_window.py # ThicketWindow + launch_gui() +└── widgets/radio_knob.py +``` + +Design invariants: + +- **Qt-free core.** The QThread worker and the headless CLI drive the + same `IngestPipeline`; one behavior change lands in both at once. +- **Table-driven dispatch.** Stage order, file-type routing, target + registry, readiness gating, UI stage colors — data tables, not branch + nests. +- **Lazy heavy imports + graceful degradation.** Every heavy dependency + loads at point of use; missing pieces report themselves and block only + the stage that needs them. +- **Idempotent re-ingest.** Deterministic IDs plus delete-by-`doc_key` + matching both key generations) make re-ingesting an edited source an + exact replacement — proven per-target by the live matrix. +- **Per-file isolation.** One broken document logs an ERROR; the queue + moves on. Postgres transactions roll back per document. +- **Explicit resource lifecycle.** Every store implements `close()`; + Milvus Lite's embedded server is released so the next process can + open the database. +- **Step-down chains.** Qdrant `query_points`→`search`; LightRAG + bindings across API generations; DuckDB vss→exact scan; podman + rootless→`--network=host`. + +## Development + +```bash +/mnt/AI/runtime/thicket-venv/bin/pip install -e ".[dev,targets,graph,graphify,ask,minio]" +.venv/bin/pytest # 62 tests, no services needed +QT_QPA_PLATFORM=offscreen \ + .venv/bin/python scripts/smoke_gui.py # headless GUI smoke + screenshot +.venv/bin/python scripts/func_test.py # LIVE matrix: every destination, + # engine, archive, ask — 18 checks +/mnt/AI/runtime/thicket-venv/bin/python thicket.py --dry-run # live readiness report +``` + +Standards applied: PEP 8; SEI CERT practices (bounded reads, precise +exception scope); MISRA-style bounded, structured control flow; POSIX +assumptions (`pathlib`, no platform branches, cron-safe headless mode). + +## License + +AGPL-3.0-or-later — Jeremy Anderson · info@dcos.net · [dcos.net](https://dcos.net) · 2026. +See [LICENSE](LICENSE) for the full text. + +Invoked (not bundled) components carry their own licenses: PySide6 +(LGPL-3.0), pypdf (BSD), EbookLib (AGPL-3.0), BeautifulSoup (MIT), +python-slugify (MIT), FastEmbed (Apache-2.0), Qdrant (Apache-2.0), +Chroma (Apache-2.0), LanceDB (Apache-2.0), FAISS (MIT), pymilvus +(Apache-2.0), weaviate-client (BSD-3), psycopg (LGPL-3.0), pgvector +(PostgreSQL), PyMySQL (MIT), sqlite-vec (MIT), LightRAG (MIT), +Graphify (Apache-2.0/MIT), Vanna (MIT), MinIO client (Apache-2.0), +Ollama (MIT). diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..9373cd9 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,170 @@ +# pyproject.toml — Thicket v1.0 +# +# Super-ingest + RAG console: batch-import PDF / EPUB / Markdown / +# plain-text into an Obsidian vault, a Qdrant vector store, and an +# optional LightRAG knowledge graph — wrapped in the MMD3-style +# PySide6 console UI (same visual lineage as OpenTranscode). +# +# Publish with: +# python -m build +# twine upload dist/* + +[build-system] +requires = ["setuptools>=68.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "thicket" +version = "1.8.1" +description = "Super-ingest + RAG console: documents -> Obsidian + Qdrant + LightRAG, with a PySide6 GUI" +readme = "README.md" +requires-python = ">=3.12" +license = { text = "AGPL-3.0-or-later" } +authors = [ + { name = "Jeremy Anderson", email = "info@dcos.net" }, +] +maintainers = [ + { name = "Jeremy Anderson", email = "info@dcos.net" }, +] +keywords = [ + "obsidian", + "qdrant", + "lightrag", + "rag", + "thicket", + "super-ingest", + "second-brain", + "embeddings", + "knowledge-graph", + "pyside6", + "linux", +] +classifiers = [ + "Development Status :: 4 - Beta", + "Environment :: X11 Applications :: Qt", + "Intended Audience :: End Users/Desktop", + "License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)", + "Operating System :: POSIX :: Linux", + "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Topic :: Text Processing :: Indexing", + "Typing :: Typed", +] +# Core dependency is intentionally ONLY the GUI toolkit — the app +# launches and probes even on a bare system, reporting exactly which +# ingest extras are missing. Heavy pipelines deps live in extras so +# a "vault-only" install (no vector DB, no graph) stays lightweight. +dependencies = [ + "PySide6>=6.6.0", +] + +[project.optional-dependencies] +# Document parsing + vector indexing — install with: +# pip install ".[ingest]" +ingest = [ + "beautifulsoup4>=4.12", + "EbookLib>=0.18", + "pypdf>=4.0", + "python-slugify>=8.0", + "fastembed>=0.3", + "qdrant-client>=1.9", +] +# LightRAG knowledge-graph stage — install with: +# pip install ".[graph]" +graph = [ + "lightrag-hku>=1.0", +] +# Embedded vector targets — one extra per target, or ".[targets]" for all +chroma = ["chromadb>=0.5"] +lancedb = ["lancedb>=0.15"] +faiss = ["faiss-cpu>=1.8", "numpy>=1.26"] +milvus = ["pymilvus[milvus_lite]>=2.4"] +weaviate = ["weaviate-client>=4.6"] +pgvector = ["psycopg[binary]>=3.1", "pgvector>=0.3"] +duckdb = ["duckdb>=1.0"] +sqlitevec = ["sqlite-vec>=0.1.6"] +mariadb = ["PyMySQL>=1.1"] +# Graphify graph engine — install with: pip install ".[graphify]" +graphify = ["graphifyy[ollama]>=0.9"] +# Vanna ask interaction (natural-language SQL over SQL targets) +ask = ["vanna[ollama,postgres,mysql]>=2.0"] +# MinIO object archive for source documents +minio = ["minio>=7.2"] +targets = [ + "chromadb>=0.5", + "lancedb>=0.15", + "faiss-cpu>=1.8", + "numpy>=1.26", + "pymilvus[milvus_lite]>=2.4", + "weaviate-client>=4.6", + "psycopg[binary]>=3.1", + "pgvector>=0.3", + "duckdb>=1.0", + "sqlite-vec>=0.1.6", + "PyMySQL>=1.1", +] +# Dev / test extras — install with: pip install -e ".[dev]" +dev = [ + "pytest>=8.0", + "pytest-cov>=4.0", + "build>=1.0", +] + +[project.urls] +Homepage = "https://git.dcos.net/dcosnet/Thicket" +Repository = "https://git.dcos.net/dcosnet/Thicket" +Documentation = "https://git.dcos.net/dcosnet/Thicket/blob/main/README.md" +"Bug Tracker" = "https://git.dcos.net/dcosnet/Thicket/issues" + +[project.scripts] +# Console entry point — `thicket` command after `pip install thicket` +thicket = "thicket.__main__:main" + +# ───────────────────────────────────────────────────────────────────────────── +# Setuptools-specific config +# ───────────────────────────────────────────────────────────────────────────── + +[tool.setuptools] +# We're a pure-Python package — no extension modules. +zip-safe = false + +[tool.setuptools.packages.find] +# Auto-discover packages under thicket/ and widgets/ +where = ["."] +include = ["thicket*"] +exclude = ["tests*"] + +[tool.setuptools.package-data] +# Include the QSS theme + non-Python assets +thicket = ["*.qss", "*.txt"] + +# ───────────────────────────────────────────────────────────────────────────── +# Tool config +# ───────────────────────────────────────────────────────────────────────────── + +[tool.pytest.ini_options] +testpaths = ["tests"] +python_files = ["test_*.py"] +python_classes = ["Test*"] +python_functions = ["test_*"] +addopts = "-ra --strict-markers" +markers = [ + "slow: marks tests as slow (deselect with '-m \"not slow\"')", + "e2e: marks tests as end-to-end (require running Qdrant / Ollama)", +] + +[tool.coverage.run] +source = ["thicket"] +omit = [ + "*/tests/*", + "*/__main__.py", +] + +[tool.coverage.report] +exclude_lines = [ + "pragma: no cover", + "if TYPE_CHECKING:", + "raise NotImplementedError", + "if __name__ == .__main__.:", +] diff --git a/scripts/bootstrap_sources.sh b/scripts/bootstrap_sources.sh new file mode 100755 index 0000000..1b92427 --- /dev/null +++ b/scripts/bootstrap_sources.sh @@ -0,0 +1,328 @@ +#!/usr/bin/env bash +# bootstrap_sources — source-anchored dependency build into /mnt/AI. +# +# Every Thicket dependency is mirrored as a git checkout under +# /distfiles/git/ and built into THE venv at +# /runtime/thicket-venv — one environment, always, outside +# the project tree so the checkout stays clean for git and archives. +# Launchers land in /tools/bin and ~/.local/bin. +# Idempotent: re-run to update. +# +# Methods (per package, table below): +# source — pip install from the local git checkout (pure Python) +# wheel — pip install the release wheel; the git checkout is still +# mirrored for audit. Native-heavy builds (faiss, lancedb, +# duckdb, fastembed/onnxruntime, PySide6/Qt) produce the +# same code as the wheel at a fraction of the build cost. +# --force-source builds everything from checkouts. +# +# Repository sync is state-aware: +# absent -> atomic clone (staged in a temp dir, moved into place) +# corrupt -> dir without .git is moved aside, then re-cloned +# dirty -> local modifications are never destroyed; the package +# is skipped with the offending files listed +# moved -> fast-forward to FETCH_HEAD only when HEAD differs; +# unchanged checkouts are NOT reinstalled (the venv is +# probed for the importable module first) +# Step-down: a failed source build falls back to the wheel (logged), +# so one broken upstream never wedges the whole bootstrap. +# +# Usage: +# scripts/bootstrap_sources.sh # asks which projects +# (Enter = all) in a +# terminal; installs all +# when non-interactive +# scripts/bootstrap_sources.sh --only a,b # subset, no prompt +# scripts/bootstrap_sources.sh --force-source # build native too +# scripts/bootstrap_sources.sh --update # git pull + reinstall +# +# Native build deps (auto-installed via pacman when missing, sudo): +# git gcc make cmake rustup postgresql-libs + +set -euo pipefail + +AI_ROOT="${THICKET_AI_ROOT:-/mnt/AI}" +PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +GIT_MIRROR="$AI_ROOT/distfiles/git" +VENV_DIR="$AI_ROOT/runtime/thicket-venv" +BIN_DIR="$AI_ROOT/tools/bin" +LOG_DIR="$AI_ROOT/logs" +LOG="$LOG_DIR/thicket-bootstrap.log" + +FORCE_SOURCE=0 +UPDATE_ONLY=0 +ONLY="" +mode="${1:-}" +case "$mode" in + --force-source) FORCE_SOURCE=1 ;; + --update) UPDATE_ONLY=1 ;; + --only) ONLY="${2:-}"; [ -n "$ONLY" ] || { echo "--only needs a list"; exit 2; } ;; + "") ;; + *) echo "unknown option: $mode"; exit 2 ;; +esac + +mkdir -p "$GIT_MIRROR" "$VENV_DIR" "$BIN_DIR" "$LOG_DIR" +exec > >(tee -a "$LOG") 2>&1 + +log() { printf '\033[1;32m>>>\033[0m %s\n' "$*"; } +warn() { printf '\033[1;33m>>> WARN\033[0m %s\n' "$*"; } + +# ── name | git url | pip name | extras | method ───────────────────────── +# Pure-python projects install from their checkout; the native-heavy +# set keeps a mirrored checkout but installs the release wheel. +PACKAGES=( + "qdrant-client|https://github.com/qdrant/qdrant-client.git|qdrant-client||source|qdrant_client|vector target: qdrant (service)" + "chromadb|https://github.com/chroma-core/chroma.git|chromadb||source|chromadb|vector target: chroma (embedded)" + "lancedb|https://github.com/lancedb/lancedb.git|lancedb||wheel|lancedb|vector target: lancedb (embedded)" + "faiss|https://github.com/facebookresearch/faiss.git|faiss-cpu||wheel|faiss|vector target: faiss (file index)" + "pymilvus|https://github.com/milvus-io/pymilvus.git|pymilvus|[milvus_lite]|source|pymilvus|vector target: milvus lite" + "weaviate-client|https://github.com/weaviate/weaviate-python-client.git|weaviate-client||source|weaviate|vector target: weaviate (service)" + "psycopg|https://github.com/psycopg/psycopg.git|psycopg|[binary]|wheel|psycopg|vector target: pgvector - postgres driver" + "pgvector|https://github.com/pgvector/pgvector-python.git|pgvector||source|pgvector|vector target: pgvector - python bindings" + "duckdb|https://github.com/duckdb/duckdb.git|duckdb||wheel|duckdb|vector target: duckdb (embedded)" + "sqlite-vec|https://github.com/asg017/sqlite-vec.git|sqlite-vec||source|sqlite_vec|vector target: sqlite-vec (embedded)" + "PyMySQL|https://github.com/PyMySQL/PyMySQL.git|PyMySQL||source|pymysql|vector target: mariadb driver" + "graphify|https://github.com/Graphify-Labs/graphify.git|graphifyy|[ollama]|source|graphify|graph engine: graph.json + HTML report" + "vanna|https://github.com/vanna-ai/vanna.git|vanna|[ollama,postgres,mysql]|source|vanna|ask: natural-language SQL over corpora" + "minio-py|https://github.com/minio/minio-py.git|minio||source|minio|stage: MinIO object archive" + "LightRAG|https://github.com/HKUDS/LightRAG.git|lightrag-hku||source|lightrag|graph engine: merged entity graph" + "ebooklib|https://github.com/aerkalov/ebooklib.git|EbookLib||source|ebooklib|parser: EPUB" + "BeautifulSoup4|https://github.com/wention/BeautifulSoup4.git|beautifulsoup4||source|bs4|parser: HTML inside EPUB" + "pypdf|https://github.com/py-pdf/pypdf.git|pypdf||source|pypdf|parser: PDF" + "python-slugify|https://github.com/un33k/python-slugify.git|python-slugify||source|slugify|vault note filenames" + "fastembed|https://github.com/qdrant/fastembed.git|fastembed||wheel|fastembed|local ONNX embeddings" + "PySide6|https://code.qt.io/pyside/pyside-setup.git|PySide6||wheel|PySide6|the console GUI toolkit" +) + +# ── native build tools via pacman (skipped when present) ───────────── +required_tools=(git gcc make) +missing=() +for tool in "${required_tools[@]}"; do + command -v "$tool" >/dev/null 2>&1 || missing+=("$tool") +done +if [ "${#missing[@]}" -gt 0 ]; then + log "installing build tools via pacman: ${missing[*]}" + sudo pacman -S --needed "${missing[@]}" +fi + +# ── project selection ───────────────────────────────────────────────── +# Interactive in a terminal; --only or non-TTY stdin bypasses the menu. +select_packages() { + if [ -n "$ONLY" ]; then + return + fi + if [ ! -t 0 ] && [ "${THICKET_BOOTSTRAP_INTERACTIVE:-0}" != "1" ]; then + return # everything, no prompt + fi + + local -a names=() roles=() + local row + for row in "${PACKAGES[@]}"; do + names+=("$(cut -d'|' -f1 <<<"$row")") + roles+=("$(cut -d'|' -f7 <<<"$row")") + done + + printf 'Thicket projects:\n' + local i + for i in "${!names[@]}"; do + printf ' %2d) %-18s %-34s %s\n' \ + "$((i + 1))" "${names[$i]}" \ + "$(cut -d'|' -f5 <<<"${PACKAGES[$i]}")" "${roles[$i]}" + done + + local answer token from to picked valid + while :; do + read -r -p $'Install which? (numbers/names/ranges, "all", Enter = all): ' answer || return + [ -z "$answer" ] && return + [ "$answer" = "all" ] && return + picked="" + for token in ${answer//,/ }; do + valid="" + if [[ "$token" =~ ^[0-9]+$ ]] && [ "$token" -ge 1 ] \ + && [ "$token" -le "${#names[@]}" ]; then + picked+=",${names[$((token - 1))]}"; valid=1 + elif [[ "$token" =~ ^([0-9]+)-([0-9]+)$ ]]; then + from="${BASH_REMATCH[1]}"; to="${BASH_REMATCH[2]}" + if [ "$from" -ge 1 ] && [ "$to" -le "${#names[@]}" ] \ + && [ "$from" -le "$to" ]; then + for ((i = from; i <= to; i++)); do + picked+=",${names[$((i - 1))]}" + done + valid=1 + fi + else + for i in "${!names[@]}"; do + if [ "${names[$i],,}" = "${token,,}" ]; then + picked+=",${names[$i]}"; valid=1; break + fi + done + fi + [ -n "$valid" ] || warn "ignored unknown selection: $token" + done + if [ -n "$picked" ]; then + ONLY="${picked#,}" + log "selected: ${ONLY//,/ }" + return + fi + done +} +select_packages + +# ── the venv ────────────────────────────────────────────────────────── +if [ ! -x "$VENV_DIR/bin/python" ]; then + log "creating venv at $VENV_DIR" + python3 -m venv "$VENV_DIR" +fi +PIP=("$VENV_DIR/bin/pip") +"${PIP[@]}" install -q --upgrade pip setuptools wheel + +# ── repository sync: absent | corrupt | dirty | unchanged | moved ──── +# Prints the resulting state; never destroys uncommitted work. +sync_repo() { + local name="$1" url="$2" + local dir="$GIT_MIRROR/$name" + + if [ ! -e "$dir" ]; then + local stage="$dir.clone.$$" + if git clone --quiet --depth 1 "$url" "$stage"; then + mv "$stage" "$dir" + echo "cloned" + else + rm -rf "$stage" + echo "clone-failed" + fi + return + fi + + if [ ! -d "$dir/.git" ]; then + local aside="$dir.corrupt-$(date +%s)" + warn "[$name] not a git checkout — moved aside to $(basename "$aside")" + mv "$dir" "$aside" + local stage="$dir.clone.$$" + if git clone --quiet --depth 1 "$url" "$stage"; then + mv "$stage" "$dir" + echo "recloned" + else + rm -rf "$stage" + echo "clone-failed" + fi + return + fi + + local origin_url + origin_url="$(git -C "$dir" remote get-url origin 2>/dev/null || true)" + if [ "$origin_url" != "$url" ]; then + warn "[$name] remote is '$origin_url', table says '$url' — skipping" + echo "url-mismatch" + return + fi + + if [ -n "$(git -C "$dir" status --porcelain 2>/dev/null)" ]; then + warn "[$name] local modifications — not touching them:" + git -C "$dir" status --porcelain | sed 's/^/ /' + echo "dirty" + return + fi + + local branch refspec + branch="$(git -C "$dir" rev-parse --abbrev-ref HEAD)" + if [ "$branch" = "HEAD" ]; then + refspec="HEAD" # detached shallow clone + else + refspec="$branch" + fi + if ! git -C "$dir" fetch --quiet --depth 1 origin "$refspec"; then + warn "[$name] fetch failed (offline?)" + echo "unchanged" + return + fi + + local head fetched + head="$(git -C "$dir" rev-parse HEAD)" + fetched="$(git -C "$dir" rev-parse FETCH_HEAD)" + if [ "$head" = "$fetched" ]; then + echo "unchanged" + return + fi + git -C "$dir" reset --hard --quiet "$fetched" + echo "moved" +} + +module_importable() { + "$VENV_DIR/bin/python" -c "import $1" >/dev/null 2>&1 +} + +# ── per-package install with step-down ──────────────────────────────── +install_one() { + local name="$1" url="$2" pipname="$3" extras="$4" method="$5" module="$6" + local dir="$GIT_MIRROR/$name" + + if [ -n "$ONLY" ]; then + case ",$ONLY," in *",$name,"*) ;; *) return 0 ;; esac + fi + + local state + state="$(sync_repo "$name" "$url")" + case "$state" in + clone-failed) + warn "[$name] clone failed — skipping" + return 0 ;; + url-mismatch | dirty) + return 0 ;; # never clobber; report above + esac + + if [ "$UPDATE_ONLY" -eq 1 ]; then + log "[$name] $state" + return 0 + fi + + # Unchanged checkout + already importable: nothing to do. + if [ "$state" = "unchanged" ] && module_importable "$module"; then + log "[$name] up to date ($(git -C "$dir" rev-parse --short HEAD))" + return 0 + fi + + if [ "$method" = "source" ] || [ "$FORCE_SOURCE" -eq 1 ]; then + log "[$name] $state — installing from source checkout" + # Local-path extras: pip accepts "[extra1,extra2]". + if "${PIP[@]}" install -q "${dir}${extras}"; then + return 0 + fi + warn "[$name] source build failed — stepping down to wheel" + fi + log "[$name] $state — installing ${pipname}${extras} (wheel)" + "${PIP[@]}" install -q "${pipname}${extras}" +} + +for row in "${PACKAGES[@]}"; do + IFS='|' read -r name url pipname extras method module _ <<<"$row" + install_one "$name" "$url" "$pipname" "$extras" "$method" "$module" +done + +# ── thicket itself, editable from this checkout ─────────────────────── +log "[thicket] installing project (editable)" +"${PIP[@]}" install -q -e "$PROJECT_ROOT" + +# ── launcher in the AI tree ─────────────────────────────────────────── +cat > "$BIN_DIR/thicket" < "$HOME/.local/bin/thicket" < None: + tag = outcome if not detail else f"{outcome} — {detail}" + RESULTS.append((section, name, tag)) + print(f" [{outcome:4s}] {name:28s} {detail}") + + +def run_cli(args: list[str], timeout: int = 600) -> tuple[int, str]: + proc = subprocess.run( + [sys.executable, str(LAUNCHER), *args], + capture_output=True, text=True, timeout=timeout, cwd=ROOT, + ) + return proc.returncode, proc.stdout + proc.stderr + + +def make_corpus(base: Path) -> Path: + docs = base / "cold" + docs.mkdir(parents=True) + for name, body in CORPUS.items(): + (docs / name).write_text(body, encoding="utf-8") + return docs + + +def fresh_vault(base: Path, name: str) -> Path: + vault = base / f"vault-{name}" + vault.mkdir(parents=True, exist_ok=True) + return vault + + +def search_once(target: str, vault: Path, query: str, port: int = 6333): + from thicket.vector_stores import create_store + + engine = search_once._engine + store = create_store(target, collection=COLLECTION, dim=engine.dim, + host="localhost", port=port, + data_dir=vault / ".thicket" / target) + store.set_embedder(engine) + try: + store.ensure_collection() + return store.search(engine.embed_query(query), limit=1) + finally: + store.close() + + +def drop_stale() -> None: + """The matrix owns its collection name; any earlier state at that + name (including other dimension eras) goes before the run.""" + try: + from qdrant_client import QdrantClient + c = QdrantClient(url="http://localhost:6333", + check_compatibility=False) + if c.collection_exists(COLLECTION): + c.delete_collection(COLLECTION) + except Exception: + pass + try: + import psycopg + with psycopg.connect(connect_timeout=3) as conn: + conn.execute(f'DROP TABLE IF EXISTS "{COLLECTION}"') + conn.commit() + except Exception: + pass + try: + import pymysql + conn = pymysql.connect(host="127.0.0.1", user="root", + database="thicket", connect_timeout=3) + with conn.cursor() as cur: + cur.execute(f"DROP TABLE IF EXISTS `{COLLECTION}`") + conn.commit(); conn.close() + except Exception: + pass + try: + import weaviate + client = weaviate.connect_to_local( + host="127.0.0.1", port=8080, grpc_port=50051) + mapped = "Thicket_func" + if client.collections.exists(mapped): + client.collections.delete(mapped) + client.close() + except Exception: + pass + + +def phase_destinations(base: Path) -> None: + from thicket.embedder import EmbeddingEngine + from thicket.vector_stores import TARGETS + + drop_stale() + engine = EmbeddingEngine(EMBED_MODEL) + engine.load() + search_once._engine = engine + + print("\n== destinations ==") + # obsidian: notes only + docs = make_corpus(base / "obsidian") + vault = fresh_vault(base, "obsidian") + rc, out = run_cli(["--ingest", str(docs), "--vault", str(vault), + "--target", "obsidian", "--no-qdrant"]) + notes = sorted(p.name for p in (vault / "Ingested_Brain").glob("*.md")) + ok = rc == 0 and len(notes) == 3 and "INDEX" not in out + record("destinations", "obsidian", + "PASS" if ok else "FAIL", f"notes={notes}" if ok else out[-160:]) + + parity: dict[str, float] = {} + for target in TARGETS: + docs = make_corpus(base / target) + vault = fresh_vault(base, target) + port = 8080 if target == "weaviate" else 6333 + rc, out = run_cli(["--ingest", str(docs), "--vault", str(vault), + "--target", target, "--port", str(port), + "--collection", COLLECTION]) + if rc != 0 or "0 failed" not in out: + record("destinations", target, "FAIL", out.strip()[-140:]) + continue + try: + first = search_once(target, vault, QUERIES[0][0], port) + rc2, _ = run_cli(["--ingest", str(docs), "--vault", str(vault), + "--target", target, "--port", str(port), + "--collection", COLLECTION]) + second = search_once(target, vault, QUERIES[0][0], port) + except Exception as e: # noqa: BLE001 + record("destinations", target, "FAIL", f"search: {e}") + continue + hit_ok = all( + search_once(target, vault, q, port)[0]["payload"] + ["document_title"].lower().replace(" ", "-").startswith(exp.split("-")[0]) + for q, exp in QUERIES[:1] + ) + idem = (rc2 == 0 and abs(first[0]["score"] - second[0]["score"]) < 1e-3 + and second[0]["payload"]["document_title"] + == first[0]["payload"]["document_title"]) + parity[target] = first[0]["score"] + record("destinations", target, + "PASS" if (hit_ok and idem) else "FAIL", + f"top={first[0]['payload']['document_title'][:22]!r} " + f"score={first[0]['score']:.3f} idempotent={idem}") + + if len(parity) >= 2: + scores = sorted(parity.values()) + spread = scores[-1] - scores[0] + record("destinations", "score parity (all)", + "PASS" if spread <= 0.02 else "WARN", f"spread={spread:.4f}") + + +def phase_graphs(base: Path) -> None: + print("\n== graph engines ==") + for engine, marker in (("lightrag", ".lightrag"), ("graphify", ".graphify")): + docs = make_corpus(base / f"g-{engine}") + docs_extra = docs / "tls-rotation.sh" + docs_extra.write_text(CORPUS["tls-rotation.sh"], encoding="utf-8") + vault = fresh_vault(base, f"g-{engine}") + rc, out = run_cli(["--ingest", str(docs), "--vault", str(vault), + "--target", "obsidian", "--no-qdrant", + "--lightrag", "--graph-engine", engine, + "--ollama-llm", "llama3.1:latest"], + timeout=900) + if rc != 0: + record("graphs", engine, "FAIL", out.strip()[-160:]) + continue + if engine == "graphify": + ok = (vault / ".graphify" / "graphify-out" / "graph.json").exists() + else: + graph_dir = vault / ".lightrag" + ok = graph_dir.exists() and any(graph_dir.iterdir()) + record("graphs", engine, "PASS" if ok else "FAIL", + f"artifacts={'yes' if ok else 'missing'}") + + +def phase_archives(base: Path) -> None: + import bz2 + + print("\n== archive stages ==") + # filesystem bz2 + docs = make_corpus(base / "fsarch") + vault = fresh_vault(base, "fsarch") + archive = base / "ingested-archive" + rc, out = run_cli(["--ingest", str(docs), "--vault", str(vault), + "--target", "obsidian", "--no-qdrant", "--archive", + "--archive-dir", str(archive)]) + packed = sorted(archive.glob("*.bz2")) if archive.exists() else [] + roundtrip = bool(packed) and all( + bz2.decompress(p.read_bytes()) for p in packed) + empty_in = not any(docs.iterdir()) + record("archives", "filesystem bz2", + "PASS" if (rc == 0 and roundtrip and empty_in) else "FAIL", + f"{len(packed)} objects, roundtrip={roundtrip}") + + # MinIO object stage + try: + from minio import Minio + client = Minio("127.0.0.1:9000", access_key="thicket", + secret_key="thicket-secret", secure=False) + if not client.bucket_exists("thicket-corpus"): + client.make_bucket("thicket-corpus") + except Exception as e: # noqa: BLE001 + record("archives", "minio objects", "SKIP", f"service: {e}") + return + docs = make_corpus(base / "minioarch") + vault = fresh_vault(base, "minioarch") + rc, out = run_cli(["--ingest", str(docs), "--vault", str(vault), + "--target", "obsidian", "--no-qdrant", "--minio"]) + objects = {o.object_name for o in + client.list_objects("thicket-corpus", recursive=True)} + uris = "source_uri" in (vault / "Ingested_Brain" / "pasta.md").read_text() + expected = set(CORPUS) + ok = rc == 0 and uris and expected & objects == expected + record("archives", "minio objects", + "PASS" if ok else "FAIL", + f"{len(expected & objects)}/{len(expected)} objects, uri_in_note={uris}") + + +def phase_ask(base: Path) -> None: + print("\n== ask (Vanna + Ollama) ==") + from thicket.ask_vanna import SQL_TARGETS, ask + + for target in SQL_TARGETS: + try: + answer = ask("How many rows are in the corpus table?", + target=target, llm_model="llama3.1:latest") + ok = bool(answer.strip()) and "ERROR" not in answer + record("ask", target, "PASS" if ok else "FAIL", + answer.strip().replace("\n", " ")[:60]) + except Exception as e: # noqa: BLE001 + record("ask", target, "FAIL", str(e)[:100]) + + +def main() -> int: + started = time.time() + print(f"thicket functionality matrix — {time.strftime('%Y-%m-%d %H:%M')}") + base = Path(tempfile.mkdtemp(prefix="thicket-func-")) + try: + phase_destinations(base) + phase_graphs(base) + phase_archives(base) + phase_ask(base) + finally: + shutil.rmtree(base, ignore_errors=True) + + fails = [r for r in RESULTS if r[2].startswith("FAIL")] + print(f"\n{'=' * 60}") + print(f"{len(RESULTS)} checks, {len(fails)} failed " + f"({time.time() - started:.0f}s)") + return 1 if fails else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/smoke_gui.py b/scripts/smoke_gui.py new file mode 100755 index 0000000..439867b --- /dev/null +++ b/scripts/smoke_gui.py @@ -0,0 +1,68 @@ +#!/usr/bin/env python3 +"""Offscreen GUI smoke test — instantiates the full window, lets the +probe thread settle, populates a demo queue, and grabs a screenshot. + +Run: QT_QPA_PLATFORM=offscreen python3 scripts/smoke_gui.py +""" + +import os +import sys +import tempfile +from pathlib import Path + +os.environ.setdefault("QT_QPA_PLATFORM", "offscreen") +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from PySide6.QtCore import QTimer +from PySide6.QtWidgets import QApplication + +from thicket.ui_window import ThicketWindow + +OUT = Path("/tmp/thicket_smoke.png") + + +def main() -> int: + app = QApplication(sys.argv) + + # Demo input dir with a few plausible documents so the queue table + # shows real rows in the screenshot. + demo = Path(tempfile.mkdtemp(prefix="bf_smoke_")) + (demo / "attention-is-all-you-need.pdf").write_bytes(b"%PDF-1.4 stub") + (demo / "thinking-fast-and-slow.epub").write_bytes(b"PK stub") + (demo / "zettelkasten-method.md").write_text("# Zettelkasten\n\nnotes.\n") + (demo / "reading backlog.txt").write_text("todo list\n") + + window = ThicketWindow() + window.in_path_edit.setText(str(demo)) + window.show() + + def settle(): + window._scan() + window._on_file_status(str(demo / "zettelkasten-method.md"), "DONE") + window._on_file_detail(str(demo / "zettelkasten-method.md"), "14 chunks indexed") + window._on_file_status(str(demo / "thinking-fast-and-slow.epub"), "EXTRACT") + window._on_file_detail(str(demo / "thinking-fast-and-slow.epub"), "extracting text") + window.query_edit.setText("how do transformers handle attention?") + window.log_box.append("> [1] 0.873 Attention Is All You Need § Page 3") + + QTimer.singleShot(1500, settle) + + def finish(): + ok_run = window.btn_run.isEnabled() + window.grab().save(str(OUT)) + print(f"btn_run enabled: {ok_run}") + print(f"status bar: {window.status_label.text()!r}") + print(f"queue rows: {window.queue_table.rowCount()}") + print(f"screenshot: {OUT}") + results["ok"] = bool(ok_run and window.queue_table.rowCount() == 4) + window.close() + QTimer.singleShot(0, app.quit) + + results = {} + QTimer.singleShot(3000, finish) + app.exec() + return 0 if results.get("ok") else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 0000000..498ac1b --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,36 @@ +"""Shared fixtures — temp vault + sample documents.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + + +@pytest.fixture +def sample_docs(tmp_path: Path) -> Path: + """Input directory with a couple of small markdown/txt documents.""" + docs = tmp_path / "docs" + docs.mkdir() + (docs / "deep-learning-notes.md").write_text( + "# Deep Learning\n\n" + "Neural networks learn representations via gradient descent.\n\n" + "## Optimizers\n\n" + "SGD is simple. Adam adapts per-parameter learning rates.\n\n" + "## Regularization\n\n" + "Dropout randomly masks units during training to reduce overfit.\n", + encoding="utf-8", + ) + (docs / "reading-list.txt").write_text( + "Books to read this year, in no particular order.\n\n" + "The Pragmatic Programmer. Structure and Interpretation.\n", + encoding="utf-8", + ) + return docs + + +@pytest.fixture +def vault(tmp_path: Path) -> Path: + vault_dir = tmp_path / "ObsidianVault" + vault_dir.mkdir() + return vault_dir diff --git a/tests/test_chunker.py b/tests/test_chunker.py new file mode 100644 index 0000000..16c2bc2 --- /dev/null +++ b/tests/test_chunker.py @@ -0,0 +1,104 @@ +"""ContextualChunker — structure preservation for technical corpora.""" + +from __future__ import annotations + +from thicket.chunker import ContextualChunker + + +def _filler(n: int, tag: str = "w") -> str: + return " ".join(f"{tag}{i}" for i in range(n)) + + +def test_empty_text_yields_no_chunks(): + assert ContextualChunker().chunk("T", "") == [] + assert ContextualChunker().chunk("T", "\n\n\n") == [] + + +def test_fenced_block_is_atomic_and_verbatim(): + code = "\n".join([ + "```python", + "def harden(host):", + " if host == 'prod':", + " sys.exit('no')", + "", + " return True", + "```", + ]) + chunks = ContextualChunker(chunk_size=10, overlap=0).chunk("T", code) + assert len(chunks) == 1 + assert chunks[0].kind == "code" + assert chunks[0].lang == "python" + assert " if host == 'prod':" in chunks[0].text # indentation kept + assert "\n\n" in chunks[0].text # blank line kept + assert "| code:python]" in chunks[0].contextual_text + + +def test_code_never_merges_with_prose(): + doc = "Intro prose here.\n\n```bash\nls -la\n```\n\nOutro prose here." + chunks = ContextualChunker(chunk_size=100, overlap=0).chunk("T", doc) + kinds = [c.kind for c in chunks] + assert kinds.count("code") == 1 + code = next(c for c in chunks if c.kind == "code") + assert "Intro" not in code.text and "Outro" not in code.text + + +def test_shebang_and_comments_are_not_headings(): + doc = "#!/usr/bin/env python3\nimport os\n\n#no-space-comment\n\n## Real Heading\n\nBody text." + chunks = ContextualChunker(chunk_size=100).chunk("T", doc) + assert chunks[-1].header == "Real Heading" + + +def test_prose_paragraph_lines_are_preserved(): + doc = "# H\n\nFirst paragraph.\nSecond line of same paragraph." + chunks = ContextualChunker(chunk_size=100).chunk("T", doc) + assert "First paragraph.\nSecond line of same paragraph." in chunks[0].text + + +def test_paragraph_aligned_budget(): + paras = [_filler(40, f"p{i}_") for i in range(6)] # 6 x 40 words + doc = "# H\n\n" + "\n\n".join(paras) + chunks = ContextualChunker(chunk_size=100, overlap=0).chunk("T", doc) + # No paragraph is ever split: each chunk is 2 paragraphs (80 words). + assert all(c.text.count("_") >= 40 for c in chunks) + assert all(len(c.text.split()) <= 100 for c in chunks) + assert len(chunks) == 3 + + +def test_oversized_code_is_windowed_with_lang_on_every_window(): + lines = [f"x_{i} = {i} # {'filler ' * 20}" for i in range(60)] + doc = "```python\n" + "\n".join(lines) + "\n```" + chunks = ContextualChunker(chunk_size=40, overlap=0).chunk("T", doc) + assert len(chunks) > 1 + assert all(c.kind == "code" and c.lang == "python" for c in chunks) + # Window overlap: first line of window N+1 appeared in window N. + assert any(chunks[i + 1].text.split("\n")[0] in chunks[i].text + for i in range(len(chunks) - 1)) + + +def test_section_boundaries_do_not_straddle(): + doc = ("# A\n\n" + _filler(30, "a") + "\n\n" + "# B\n\n" + _filler(30, "b")) + chunks = ContextualChunker(chunk_size=100, overlap=50).chunk("T", doc) + assert len(chunks) == 2 + assert chunks[0].header == "A" and chunks[1].header == "B" + assert "b0" not in chunks[0].text + + +def test_indented_block_treated_as_code(): + doc = "Prose.\n\n permit root no\n retries 3\n\nMore prose." + chunks = ContextualChunker(chunk_size=100).chunk("T", doc) + code = [c for c in chunks if c.kind == "code"] + assert code and "permit root no" in code[0].text + + +def test_default_header_is_introduction(): + chunks = ContextualChunker(chunk_size=50).chunk("T", _filler(10)) + assert chunks[0].header == "Introduction" + + +def test_source_provenance_attaches_to_every_chunk(): + doc = "# H\n\nprose\n\n```python\nx = 1\n```" + chunks = ContextualChunker(chunk_size=100).chunk( + "T", doc, source_path="sub/deploy.py") + assert {c.source for c in chunks} == {"sub/deploy.py"} + assert ContextualChunker().chunk("T", doc)[0].source is None diff --git a/tests/test_extractors.py b/tests/test_extractors.py new file mode 100644 index 0000000..373ba6a --- /dev/null +++ b/tests/test_extractors.py @@ -0,0 +1,47 @@ +"""DocumentExtractor — dispatch, md/txt extraction, scanning.""" + +from __future__ import annotations + +import pytest + +from thicket.extractors import ( + ExtractionError, DocumentExtractor, scan_files, +) + + +def test_md_extraction_title_from_stem(sample_docs): + title, text = DocumentExtractor.extract(sample_docs / "deep-learning-notes.md") + assert title == "Deep Learning Notes" + assert "# Deep Learning" in text + assert "## Optimizers" in text + + +def test_txt_extraction(sample_docs): + title, text = DocumentExtractor.extract(sample_docs / "reading-list.txt") + assert title == "Reading List" + assert "Pragmatic Programmer" in text + + +def test_unsupported_extension_raises(tmp_path): + bogus = tmp_path / "photo.jpg" + bogus.write_bytes(b"\xff\xd8fake") + with pytest.raises(ExtractionError): + DocumentExtractor.extract(bogus) + + +def test_scan_files_sorted_and_filtered(sample_docs, tmp_path): + (sample_docs / "nested").mkdir() + (sample_docs / "nested" / "zz-last.md").write_text("x", encoding="utf-8") + (sample_docs / "image.png").write_bytes(b"\x89PNG") # ignored + + files = scan_files(sample_docs) + names = [f.name for f in files] + assert "image.png" not in names + # Sorted by full path (case-insensitive), so the nested file sits + # between the top-level entries alphabetically. + assert names == ["deep-learning-notes.md", "zz-last.md", "reading-list.txt"] + + +def test_scan_files_custom_extensions(sample_docs): + files = scan_files(sample_docs, {".txt"}) + assert [f.name for f in files] == ["reading-list.txt"] diff --git a/tests/test_graph_and_ask.py b/tests/test_graph_and_ask.py new file mode 100644 index 0000000..b13a9a3 --- /dev/null +++ b/tests/test_graph_and_ask.py @@ -0,0 +1,57 @@ +"""Graph engines, MinIO archiver, and the ask bridge — contract tests +that run without any service or heavy dependency installed.""" + +from __future__ import annotations + +import pytest + +from thicket.graph_store import GRAPH_ENGINES, GraphUnavailable, create_graph + + +def test_graph_engine_registry(): + assert sorted(GRAPH_ENGINES) == ["graphify", "lightrag"] + assert GRAPH_ENGINES["lightrag"]["modules"] == ("lightrag",) + assert GRAPH_ENGINES["graphify"]["modules"] == ("graphify",) + + +def test_unknown_graph_engine_names_choices(tmp_path): + with pytest.raises(GraphUnavailable, match="known:"): + create_graph("memgraph", working_dir=tmp_path, llm_model="m", + embed_model="e") + + +def test_graphify_engine_stages_documents(tmp_path): + pytest.importorskip("graphify") + store = create_graph("graphify", working_dir=tmp_path, llm_model="m", + embed_model="e") + store.ingest_document("Doc One", "alpha text") + store.ingest_document("Doc Two", "beta text") + staged = list((tmp_path / ".graphify" / "corpus").glob("*.md")) + assert len(staged) == 2 + store.ingest_document("Doc One", "different content this time") + # same title + different content -> a digest sibling, not a clobber + assert len(list((tmp_path / ".graphify" / "corpus").glob("doc-one*.md"))) == 2 + + +def test_minio_archiver_requires_package(): + from thicket.minio_archive import ArchiveUnavailable, MinioArchiver + import importlib.util + if importlib.util.find_spec("minio"): + pytest.skip("minio installed — guard path covered live") + with pytest.raises(ArchiveUnavailable, match="pip install"): + MinioArchiver() + + +def test_ask_rejects_non_sql_targets(): + from thicket.ask_vanna import AskUnavailable, SQL_TARGETS, target_ddl + assert SQL_TARGETS == ("pgvector", "mariadb") + with pytest.raises(AskUnavailable, match="no SQL corpus"): + target_ddl("chroma") + + +def test_ask_ddl_documents_payload_fields(): + from thicket.ask_vanna import target_ddl + for target in ("pgvector", "mariadb"): + ddl = target_ddl(target, "second_brain") + assert "second_brain" in ddl + assert "document_title" in ddl and "chunk_index" in ddl diff --git a/tests/test_launcher.py b/tests/test_launcher.py new file mode 100644 index 0000000..19c2edb --- /dev/null +++ b/tests/test_launcher.py @@ -0,0 +1,38 @@ +"""Root launcher — thicket.py must own every mode from any cwd.""" + +from __future__ import annotations + +import subprocess +import sys +from pathlib import Path + +LAUNCHER = Path(__file__).resolve().parents[1] / "thicket.py" + + +def _run(args: list[str], cwd: Path) -> subprocess.CompletedProcess: + return subprocess.run( + [sys.executable, str(LAUNCHER), *args], + capture_output=True, text=True, timeout=60, cwd=cwd, + ) + + +def test_launcher_version_from_project_root(): + from thicket import __version__ + result = _run(["--version"], LAUNCHER.parent) + assert result.returncode == 0 + assert result.stdout.strip() == f"thicket {__version__}" + + +def test_launcher_version_from_unrelated_cwd(tmp_path): + """By-path invocation must resolve the in-tree package anywhere.""" + result = _run(["--version"], tmp_path) + from thicket import __version__ + assert result.returncode == 0 + assert f"thicket {__version__}" in result.stdout + + +def test_launcher_dry_run_reports_probe(): + result = _run(["--dry-run"], LAUNCHER.parent) + assert result.returncode == 0 + assert "Thicket environment probe" in result.stdout + assert "Qdrant:" in result.stdout and "Ollama:" in result.stdout diff --git a/tests/test_layout.py b/tests/test_layout.py new file mode 100644 index 0000000..8700d4b --- /dev/null +++ b/tests/test_layout.py @@ -0,0 +1,105 @@ +"""AI layout awareness and path expansion.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +import thicket.layout as layout +from thicket.pipeline_core import IngestConfig + + +@pytest.fixture +def fake_ai(tmp_path: Path, monkeypatch): + (tmp_path / "corpus" / "cold").mkdir(parents=True) + (tmp_path / "corpus" / "hot").mkdir(parents=True) + (tmp_path / "corpus" / "books").mkdir(parents=True) + monkeypatch.setattr(layout, "AI_ROOT", tmp_path) + return tmp_path + + +def test_layout_detected_with_corpus_taxonomy(fake_ai): + profile = layout.detect_layout() + assert profile is not None + assert profile.corpus_cold == fake_ai / "corpus" / "cold" + assert profile.corpus_hot == fake_ai / "corpus" / "hot" + assert profile.books_present is True + assert profile.archive == fake_ai / "corpus" / "archive" + + +def test_layout_absent_without_corpus(tmp_path, monkeypatch): + monkeypatch.setattr(layout, "AI_ROOT", tmp_path) + assert layout.detect_layout() is None + + +def test_defaults_follow_the_layout(fake_ai): + assert layout.default_input() == fake_ai / "corpus" / "cold" + assert layout.default_vault() == fake_ai / "corpus" / "hot" + assert layout.default_archive(Path("/anywhere/in")) == fake_ai / "corpus" / "archive" + + +def test_defaults_fall_home_without_layout(tmp_path, monkeypatch): + monkeypatch.setattr(layout, "AI_ROOT", tmp_path) + assert layout.default_input() == Path.home() / "Downloads" / "Raw_Books_And_Papers" + assert layout.default_vault() == Path.home() / "Documents" / "ObsidianVault" + assert layout.default_archive(Path("/data/in")) == Path("/data/ingested-archive") + + +def test_archive_falls_back_to_sibling_without_layout(tmp_path, monkeypatch): + monkeypatch.setattr(layout, "AI_ROOT", tmp_path) + assert layout.default_archive(Path("/data/in")) == Path("/data/ingested-archive") + + +def test_expand_path_tilde_and_env(monkeypatch): + monkeypatch.setenv("THICKET_TEST_DIR", "/tmp/thicket-expanded") + assert layout.expand_path("~/books") == Path.home() / "books" + assert layout.expand_path("$THICKET_TEST_DIR/x") == Path("/tmp/thicket-expanded/x") + assert layout.expand_path("/plain/path") == Path("/plain/path") + + +def test_ingest_config_expands_paths(monkeypatch): + monkeypatch.setenv("THICKET_TEST_DIR", "/tmp/thicket-expanded") + config = IngestConfig(input_dir="~/in", vault_dir="$THICKET_TEST_DIR/vault", + archive_dir="$THICKET_TEST_DIR/archive", + use_qdrant=False) + assert config.input_dir == Path.home() / "in" + assert config.vault_dir == Path("/tmp/thicket-expanded/vault") + assert config.archive_dir == Path("/tmp/thicket-expanded/archive") + + +def test_backend_profiles_fill_gaps_env_wins(tmp_path, monkeypatch): + import os + + import thicket.layout as layout + + backends = tmp_path / "backends" + backends.mkdir() + (backends / "thicket.env").write_text( + "# profile\n" + "PGHOST=db.local\n" + "PGUSER='fileuser'\n" + "MINIO_ENDPOINT=\"127.0.0.1:9000\"\n" + "NOT_A_LINE\n", encoding="utf-8") + monkeypatch.setattr(layout, "AI_ROOT", tmp_path) + for key in ("PGHOST", "PGUSER", "MINIO_ENDPOINT"): + monkeypatch.delenv(key, raising=False) + + applied = layout.apply_backend_profiles() + assert applied == {"PGHOST": "db.local", "PGUSER": "fileuser", + "MINIO_ENDPOINT": "127.0.0.1:9000"} + assert os.environ["PGUSER"] == "fileuser" + + monkeypatch.setenv("PGUSER", "shelluser") + monkeypatch.delenv("PGHOST", raising=False) + applied = layout.apply_backend_profiles() + assert "PGUSER" not in applied # environment wins + assert applied["PGHOST"] == "db.local" # gaps still filled + assert os.environ["PGUSER"] == "shelluser" + + +def test_backend_profiles_absent_file_is_noop(tmp_path, monkeypatch): + import thicket.layout as layout + + monkeypatch.setattr(layout, "AI_ROOT", tmp_path) + assert layout.apply_backend_profiles() == {} diff --git a/tests/test_package.py b/tests/test_package.py new file mode 100644 index 0000000..4cb42fb --- /dev/null +++ b/tests/test_package.py @@ -0,0 +1,39 @@ +"""Package + probe smoke tests (no services required).""" + +from __future__ import annotations + +import re +from pathlib import Path + +from thicket import __version__ + + +def test_version_string(): + assert isinstance(__version__, str) + assert __version__.count(".") == 2 + + +def test_version_matches_pyproject(): + pyproject = Path(__file__).resolve().parents[1] / "pyproject.toml" + match = re.search(r'^version = "(.+)"$', pyproject.read_text(), re.M) + assert match is not None, "pyproject version line missing" + assert match.group(1) == __version__ + + +def test_env_probe_reports_module_table(): + from thicket.env_probe import PROBED_MODULES, probe_environment + + env = probe_environment(qdrant_host="localhost", qdrant_port=1) # dead port + assert set(env.modules.keys()) == set(PROBED_MODULES.keys()) + assert all(isinstance(v, bool) for v in env.modules.values()) + # known-good modules on this interpreter + assert env.modules["pypdf"] is True + assert env.modules["slugify"] is True + + +def test_ui_modules_import_without_heavy_deps(): + # GUI must import on a bare system (heavy deps are lazy). + import thicket.ui_theme # noqa: F401 + import thicket.ui_window # noqa: F401 + import thicket.widgets + assert hasattr(thicket.widgets, "RadioKnob") diff --git a/tests/test_pipeline_core.py b/tests/test_pipeline_core.py new file mode 100644 index 0000000..93b84d8 --- /dev/null +++ b/tests/test_pipeline_core.py @@ -0,0 +1,171 @@ +"""IngestPipeline — vault-only end-to-end run (no services needed).""" + +from __future__ import annotations + +from thicket.pipeline_core import ( + IngestConfig, IngestPipeline, PipelineCallbacks, +) + + +class RecordingCallbacks: + def __init__(self): + self.logs: list[str] = [] + self.statuses: list[tuple[str, str]] = [] + self.details: list[tuple[str, str]] = [] + self.progress: list[tuple[str, int, int]] = [] + + def bind(self) -> PipelineCallbacks: + return PipelineCallbacks( + log=self.logs.append, + file_status=lambda p, s: self.statuses.append((p, s)), + file_detail=lambda p, d: self.details.append((p, d)), + progress=lambda n, c, t: self.progress.append((n, c, t)), + ) + + +def test_vault_only_pipeline_processes_every_document(sample_docs, vault): + config = IngestConfig( + input_dir=sample_docs, vault_dir=vault, + use_qdrant=False, use_lightrag=False, + ) + cb = RecordingCallbacks() + ok, fail = IngestPipeline(config, cb.bind()).run() + + assert (ok, fail) == (2, 0) + assert (vault / "Ingested_Brain" / "deep-learning-notes.md").exists() + assert (vault / "Ingested_Brain" / "reading-list.md").exists() + + # every file walked the full status lifecycle + for path in {p for p, _ in cb.statuses}: + stages = [s for p, s in cb.statuses if p == path] + assert stages[0] == "QUEUED" + assert stages[-1] == "DONE" + assert "EXTRACT" in stages and "VAULT" in stages + + # progress counted 1..2 + assert cb.progress[-1][1:] == (2, 2) + + +def test_empty_file_is_skip_not_failure(sample_docs, vault): + (sample_docs / "blank.md").write_text("", encoding="utf-8") + config = IngestConfig(input_dir=sample_docs, vault_dir=vault, + use_qdrant=False, use_lightrag=False) + cb = RecordingCallbacks() + ok, fail = IngestPipeline(config, cb.bind()).run() + assert (ok, fail) == (3, 0) + assert ("SKIP" in [s for _, s in cb.statuses]) + + +def test_cooperative_stop_between_files(sample_docs, vault): + config = IngestConfig(input_dir=sample_docs, vault_dir=vault, + use_qdrant=False, use_lightrag=False) + pipeline = IngestPipeline(config, PipelineCallbacks.quiet()) + + original = pipeline._process_one + calls = {"n": 0} + + def stop_after_first(filepath): + calls["n"] += 1 + result = original(filepath) + pipeline.request_stop() + return result + + pipeline._process_one = stop_after_first + ok, fail = pipeline.run() + assert calls["n"] == 1 # stopped before the second file + assert (ok, fail) == (1, 0) + + +def test_broken_document_fails_alone_and_queue_continues(sample_docs, vault): + (sample_docs / "broken.epub").write_bytes(b"not really an epub") + config = IngestConfig(input_dir=sample_docs, vault_dir=vault, + use_qdrant=False, use_lightrag=False) + cb = RecordingCallbacks() + ok, fail = IngestPipeline(config, cb.bind()).run() + + # the two good files still succeeded; the epub failed in isolation + assert ok == 2 and fail == 1 + assert any("ERROR processing broken.epub" in line for line in cb.logs) + + +def test_vault_stage_disabled_writes_no_notes(sample_docs, vault): + config = IngestConfig(input_dir=sample_docs, vault_dir=vault, + use_vault=False, use_qdrant=False, use_lightrag=False) + cb = RecordingCallbacks() + ok, fail = IngestPipeline(config, cb.bind()).run() + + assert (ok, fail) == (2, 0) + assert not (vault / "Ingested_Brain").exists() or \ + not any((vault / "Ingested_Brain").iterdir()) + assert not any("Vault note written" in line for line in cb.logs) + # VAULT never appears in the stage lifecycle + assert "VAULT" not in [s for _, s in cb.statuses] + + +def test_skip_unchanged_skips_second_run(sample_docs, vault): + config = IngestConfig(input_dir=sample_docs, vault_dir=vault, + use_qdrant=False, use_lightrag=False, + skip_unchanged=True) + cb = RecordingCallbacks() + first = IngestPipeline(config, cb.bind()).run() + second = IngestPipeline(config, cb.bind()).run() + + assert first == (2, 0) + assert second == (2, 0) + assert [s for _, s in cb.statuses].count("SKIP") == 2 + assert any("unchanged" in d for _, d in cb.details) + + # Editing a file brings it back into the queue. + (sample_docs / "reading-list.txt").write_text("new content\n", + encoding="utf-8") + IngestPipeline(config, cb.bind()).run() + assert [s for _, s in cb.statuses].count("DONE") == 3 + + +def test_fs_archive_moves_and_compresses(sample_docs, vault, tmp_path): + archive = tmp_path / "ingested-archive" + config = IngestConfig(input_dir=sample_docs, vault_dir=vault, + use_qdrant=False, use_lightrag=False, + use_fs_archive=True, archive_dir=archive) + ok, fail = IngestPipeline(config, PipelineCallbacks.quiet()).run() + + assert (ok, fail) == (2, 0) + # Incoming tree is empty; the archive holds bz2 payloads only. + assert not any(p.exists() for p in sample_docs.iterdir() if p.is_file()) + packed = sorted(archive.glob("*.bz2")) + assert len(packed) == 2 and not any(p.suffix != ".bz2" for p in archive.iterdir()) + # Content round-trips through bunzip2. + import bz2 + text = bz2.decompress(packed[0].read_bytes()).decode("utf-8") + assert text.strip() # real content survived + + +def test_obsidian_target_is_notes_only(sample_docs, vault): + config = IngestConfig(input_dir=sample_docs, vault_dir=vault, + target="obsidian", use_qdrant=False, + use_lightrag=False) + cb = RecordingCallbacks() + ok, fail = IngestPipeline(config, cb.bind()).run() + assert (ok, fail) == (2, 0) + assert (vault / "Ingested_Brain" / "reading-list.md").exists() + assert "INDEX" not in [s_ for _, s_ in cb.statuses] # no vector stage + + +def test_max_mb_guards_oversized_files(sample_docs, vault): + (sample_docs / "huge.txt").write_text("x" * 1_200_000, encoding="utf-8") + config = IngestConfig(input_dir=sample_docs, vault_dir=vault, + use_qdrant=False, use_lightrag=False, max_mb=1) + cb = RecordingCallbacks() + ok, fail = IngestPipeline(config, cb.bind()).run() + assert (ok, fail) == (3, 0) + details = [d for _, d in cb.details] + assert any("exceeds" in d for d in details) + assert (sample_docs / "huge.txt").exists() # guards skip, never delete + + +def test_custom_notes_dir(sample_docs, vault): + config = IngestConfig(input_dir=sample_docs, vault_dir=vault, + use_qdrant=False, use_lightrag=False, + notes_dir="Technical_Shots") + IngestPipeline(config, PipelineCallbacks.quiet()).run() + assert (vault / "Technical_Shots" / "reading-list.md").exists() diff --git a/tests/test_vault_writer.py b/tests/test_vault_writer.py new file mode 100644 index 0000000..72bd13a --- /dev/null +++ b/tests/test_vault_writer.py @@ -0,0 +1,58 @@ +"""ObsidianVaultWriter — frontmatter, escaping, collision handling.""" + +from __future__ import annotations + +import re + +from thicket.vault_writer import ObsidianVaultWriter + + +def test_note_written_with_frontmatter_and_header(vault, sample_docs): + writer = ObsidianVaultWriter(vault) + source = sample_docs / "reading-list.txt" + note = writer.write("Reading List", "body text", source) + + assert note.parent == vault / "Ingested_Brain" + text = note.read_text(encoding="utf-8") + assert text.startswith("---\n") + assert re.search(r'^title: "Reading List"$', text, re.M) + assert re.search(r'^source_file: "reading-list\.txt"$', text, re.M) + assert "tags:" in text and "brain/ingested" in text and "source/txt" in text + assert "# Reading List" in text + assert "*Source document: `reading-list.txt`*" in text + assert text.endswith("body text") + + +def test_title_with_quotes_survives_frontmatter(vault, sample_docs): + writer = ObsidianVaultWriter(vault) + source = sample_docs / "reading-list.txt" + note = writer.write('The "Real" Deal', "x", source) + text = note.read_text(encoding="utf-8") + fm = text.split("---")[1] + assert 'title: "The \\"Real\\" Deal"' in fm + + +def test_reingest_same_source_overwrites(vault, sample_docs): + writer = ObsidianVaultWriter(vault) + source = sample_docs / "reading-list.txt" + first = writer.write("Same Title", "v1", source) + second = writer.write("Same Title", "v2", source) + assert first == second + assert "v2" in first.read_text(encoding="utf-8") + + +def test_same_title_different_source_gets_suffix(vault, sample_docs): + writer = ObsidianVaultWriter(vault) + a = writer.write("Same Title", "from A", sample_docs / "reading-list.txt") + b = writer.write("Same Title", "from B", sample_docs / "deep-learning-notes.md") + assert a != b + assert a.exists() and b.exists() + # suffix is a 6-hex digest of the source name + assert re.search(r"-[0-9a-f]{6}\.md$", b.name) + + +def test_symbol_only_title_falls_back_to_untitled(vault, sample_docs): + writer = ObsidianVaultWriter(vault) + source = sample_docs / "reading-list.txt" + note = writer.write("###", "x", source) + assert note.stem.startswith("untitled") diff --git a/tests/test_vector_stores.py b/tests/test_vector_stores.py new file mode 100644 index 0000000..338947f --- /dev/null +++ b/tests/test_vector_stores.py @@ -0,0 +1,181 @@ +"""Vector store targets — registry, doc_key contract, and per-target +roundtrips against a deterministic stub engine (no model download). + +The per-target roundtrips are skipped automatically when a target's +library is not installed; the registry tests always run. +""" + +from __future__ import annotations + +import pytest + +from thicket.chunker import Chunk +from thicket.vector_stores import ( + TARGETS, VectorStoreError, create_store, doc_key, +) + +CHUNKS = [ + Chunk(header="Alpha", text="alpha content one", + contextual_text="[Source: T | Section: Alpha]\nalpha content one"), + Chunk(header="Beta", text="beta content two", + contextual_text="[Source: T | Section: Beta]\nbeta content two"), +] + + +class StubEngine: + """Deterministic 4-dim embeddings: direction keyed by first word.""" + model_name = "stub-model" + dim = 4 + + _BASIS = { + "alpha": [1.0, 0.0, 0.0, 0.0], + "beta": [0.0, 1.0, 0.0, 0.0], + } + + def embed(self, texts: list[str]) -> list[list[float]]: + return [self._vector(t) for t in texts] + + def embed_query(self, text: str) -> list[float]: + return self._vector(text) + + @classmethod + def _vector(cls, text: str) -> list[float]: + for word, vec in cls._BASIS.items(): + if word in text.lower(): + return list(vec) + return [0.0, 0.0, 1.0, 0.0] + + +def _roundtrip(store) -> None: + store.set_embedder(StubEngine()) + store.ensure_collection() + + written = store.replace_document("Doc One", "Ingested_Brain/doc-one.md", CHUNKS) + assert written == 2 + + hits = store.search(StubEngine().embed_query("alpha query"), limit=2) + assert hits, "expected at least one hit" + top = hits[0] + assert top["payload"]["document_title"] == "Doc One" + assert top["payload"]["section_header"] == "Alpha" + assert "doc_key" not in top["payload"] # internal key never surfaces + + # Exact replacement: shrink to one chunk, count must follow exactly. + store.replace_document("Doc One", "Ingested_Brain/doc-one.md", CHUNKS[:1]) + hits_after = store.search(StubEngine().embed_query("alpha query"), limit=10) + alpha_hits = [h for h in hits_after + if h["payload"]["section_header"] == "Alpha"] + beta_hits = [h for h in hits_after + if h["payload"]["section_header"] == "Beta"] + assert len(alpha_hits) == 1 and not beta_hits, \ + "re-ingest must replace exactly, leaving no stale chunks" + + +# ── registry (always runs) ── + +def test_registry_covers_ten_targets(): + assert sorted(TARGETS) == [ + "chroma", "duckdb", "faiss", "lancedb", "mariadb", "milvus", + "pgvector", "qdrant", "sqlitevec", "weaviate", + ] + + +def test_service_requirements_are_exact(): + services = {key: spec.service for key, spec in TARGETS.items()} + assert services == { + "qdrant": "qdrant", "pgvector": "postgres", + "weaviate": "weaviate", "mariadb": "mariadb", + "chroma": None, "lancedb": None, "faiss": None, "milvus": None, + "duckdb": None, "sqlitevec": None, + } + + +def test_identifier_mappers_never_emit_raw_names(): + from thicket.vector_stores import _mariadb_ident, _weaviate_name + assert _mariadb_ident("second-brain; DROP TABLE x") == "second_brain__DROP_TABLE_x" + assert _weaviate_name("second_brain") == "Second_brain" + assert _mariadb_ident("") == "thicket" + + +def test_doc_key_is_stable_and_injective(): + a = doc_key("Title", "path/one.md") + assert a == doc_key("Title", "path/one.md") + assert a != doc_key("Title", "path/two.md") + assert a != doc_key("Other", "path/one.md") + + +def test_unknown_target_names_every_choice(): + with pytest.raises(VectorStoreError, match="known:"): + create_store("vespa", collection="x", dim=4) + + +def test_missing_module_error_names_the_fix(): + from thicket.vector_stores import BaseVectorStore + base = BaseVectorStore("c", 4) + with pytest.raises(VectorStoreError, match="pip install"): + base._require_module("definitely_not_a_module_xyz") + + +# ── per-target roundtrips (skip when library absent) ── + +def test_qdrant_roundtrip_needs_service(): + assert TARGETS["qdrant"].service == "qdrant" + + +def test_chroma_roundtrip(tmp_path): + pytest.importorskip("chromadb") + from thicket.vector_stores import ChromaStore + _roundtrip(ChromaStore(data_dir=tmp_path / "data", collection="test_col", dim=4)) + + +def test_lancedb_roundtrip(tmp_path): + pytest.importorskip("lancedb") + from thicket.vector_stores import LanceStore + _roundtrip(LanceStore(data_dir=tmp_path, collection="t", dim=4)) + + +def test_faiss_roundtrip(tmp_path): + pytest.importorskip("faiss") + from thicket.vector_stores import FaissStore + _roundtrip(FaissStore(data_dir=tmp_path, collection="t", dim=4)) + + +def test_milvus_roundtrip(tmp_path): + pytest.importorskip("pymilvus") + from thicket.vector_stores import MilvusStore + _roundtrip(MilvusStore(data_dir=tmp_path / "data", collection="test_col", dim=4)) + + +def test_duckdb_roundtrip(tmp_path): + pytest.importorskip("duckdb") + from thicket.vector_stores import DuckStore + _roundtrip(DuckStore(data_dir=tmp_path / "data", collection="test_col", dim=4)) + + +def test_sqlitevec_roundtrip(tmp_path): + pytest.importorskip("sqlite_vec") + from thicket.vector_stores import SqliteVecStore + _roundtrip(SqliteVecStore(data_dir=tmp_path / "data", collection="test_col", dim=4)) + + +def test_weaviate_name_mapping(): + from thicket.vector_stores import _weaviate_name + assert _weaviate_name("second_brain") == "Second_brain" + assert _weaviate_name("my-papers 2") == "My_papers_2" + + +def test_payload_carries_code_provenance(): + from thicket.chunker import Chunk + from thicket.vector_stores import _payload + + chunk = Chunk("H", "def x(): pass", "[S|H]\ndef x(): pass", + kind="code", lang="python", source="pkg/mod.py") + payload = _payload("Doc", "notes/doc.md", chunk, 0) + assert payload["chunk_kind"] == "code" + assert payload["lang"] == "python" + assert payload["source_path"] == "pkg/mod.py" + assert payload["doc_key"] + + plain = Chunk("H", "prose", "[S|H]\nprose") + payload = _payload("Doc", "notes/doc.md", plain, 0) + assert "lang" not in payload and "source_path" not in payload diff --git a/thicket.py b/thicket.py new file mode 100755 index 0000000..3977580 --- /dev/null +++ b/thicket.py @@ -0,0 +1,28 @@ +#!/usr/bin/env python3 +"""Thicket launcher — the one-command entry point. + + python thicket.py # the console (GUI) + python thicket.py --dry-run # readiness report, no GUI + python thicket.py --ingest DIR --vault DIR # headless batch + +Every mode and flag is owned by ``thicket.cli``; this file only finds +the package and defers. + +One environment, always: the project venv at /.venv holds +every dependency (bootstrap_sources.sh installs its git-built packages +there). Run via .venv/bin/python, the /mnt/AI/tools/bin/thicket +launcher, or after activating the venv. +""" + +import sys +from pathlib import Path + +# Guarantee the in-tree package resolves when invoked by path or from +# another directory (sys.path[0] normally covers this; belt-and-braces +# for embedded and interpreter edge cases). +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from thicket.cli import main + +if __name__ == "__main__": + sys.exit(main()) diff --git a/thicket/__init__.py b/thicket/__init__.py new file mode 100644 index 0000000..d4e54af --- /dev/null +++ b/thicket/__init__.py @@ -0,0 +1,19 @@ +"""Thicket — super-ingest + RAG console. + +Feed it documents; it grows a thicket: dense, interconnected, +searchable knowledge. + +Batch-import PDF / EPUB / Markdown / plain-text documents into a +three-stage "second brain" pipeline: + + 1. Obsidian vault notes (normalized Markdown + YAML frontmatter) + 2. Qdrant vector index (FastEmbed local embeddings) + 3. Optional LightRAG knowledge graph (Ollama entity extraction) + +GUI is a PySide6 retro-futuristic console (MMD3 lineage, shared with +OpenTranscode). A headless CLI path (``thicket --ingest ... --vault +...``) drives the exact same core modules without Qt. +""" + +__version__ = "1.8.1" +__all__ = ["__version__"] diff --git a/thicket/__main__.py b/thicket/__main__.py new file mode 100644 index 0000000..40df2b1 --- /dev/null +++ b/thicket/__main__.py @@ -0,0 +1,8 @@ +"""``python -m thicket`` entry point — defers to cli.main().""" + +import sys + +from .cli import main + +if __name__ == "__main__": + sys.exit(main()) diff --git a/thicket/ask_vanna.py b/thicket/ask_vanna.py new file mode 100644 index 0000000..b255399 --- /dev/null +++ b/thicket/ask_vanna.py @@ -0,0 +1,233 @@ +"""Ask — natural-language SQL over the SQL-backed vector targets. + +Vanna 2.0 (vanna-ai/vanna, agent-based rewrite) drives an Ollama LLM +with a RunSqlTool pointed at the same Postgres/MariaDB servers Thicket +ingests into. The question is prefixed with the corpus DDL so the +model writes correct SQL; only SELECT-style reads are requested. + +Connections follow the same Unix conventions as the vector stores: +PGHOST/PGPORT/PGUSER/PGPASSWORD/PGDATABASE (or PGDSN) for pgvector, +MARIADB_HOST/PORT/USER/PASSWORD/DATABASE for mariadb. +""" + +from __future__ import annotations + +import asyncio +import os +import tempfile +from collections.abc import Callable + +# Targets whose corpus lives in a SQL database Vanna can query. +SQL_TARGETS = ("pgvector", "mariadb") + + +class AskUnavailable(Exception): + """Raised when the ask interaction cannot run (target, deps).""" + + +def _pg_params() -> dict: + if os.environ.get("PGDSN"): + return {"connection_string": os.environ["PGDSN"]} + return { + "host": os.environ.get("PGHOST", "localhost"), + "port": int(os.environ.get("PGPORT", "5432")), + "database": os.environ.get("PGDATABASE", "thicket"), + "user": os.environ.get("PGUSER", "thicket"), + "password": os.environ.get("PGPASSWORD", ""), + } + + +def _mariadb_params() -> dict: + return { + "host": os.environ.get("MARIADB_HOST", "127.0.0.1"), + "port": int(os.environ.get("MARIADB_PORT", "3306")), + "database": os.environ.get("MARIADB_DATABASE", "thicket"), + "user": os.environ.get("MARIADB_USER", "root"), + "password": os.environ.get("MARIADB_PASSWORD", ""), + } + + +_RUNNER_PARAMS = {"pgvector": _pg_params, "mariadb": _mariadb_params} + + +def target_ddl(target: str, collection: str = "second_brain") -> str: + """Corpus DDL + column documentation for the question context — + the exact shape Thicket's ensure_collection creates.""" + match target: + case "pgvector": + return ( + f'CREATE TABLE "{collection}" (' + "id TEXT PRIMARY KEY, doc_key TEXT NOT NULL, " + "embedding vector(384), payload JSONB);" + "\n-- payload fields: document_title text, obsidian_path text," + " section_header text, content text, chunk_index int, doc_key text" + ) + case "mariadb": + table = "".join(c if c.isalnum() or c == "_" else "_" for c in collection) + return ( + f"CREATE TABLE `{table}` (" + "id VARCHAR(36) PRIMARY KEY, doc_key VARCHAR(64) NOT NULL, " + "embedding VECTOR(384) NOT NULL, payload JSON);" + "\n-- payload fields: document_title, obsidian_path," + " section_header, content, chunk_index, doc_key" + ) + raise AskUnavailable( + f"target '{target}' has no SQL corpus — ask works with: " + f"{', '.join(SQL_TARGETS)}" + ) + + +def _component_text(component) -> str | None: + """Text from a yielded component — vanna wraps each Rich component + (RichText, DataFrame, status pings) in a UiComponent envelope.""" + rich = getattr(component, "rich_component", None) or component + for attr in ("content", "text", "markdown", "value"): + value = getattr(rich, attr, None) + if isinstance(value, str) and value.strip(): + return value.strip() + df = getattr(rich, "df", None) + if df is not None and hasattr(df, "to_string"): + return df.head(20).to_string() + return None + + +class VannaAsker: + """One question at a time against the corpus database.""" + + def __init__(self, target: str, collection: str, llm_model: str, + log: Callable[[str], None] = lambda _msg: None): + self._target = target + if target not in SQL_TARGETS: + raise AskUnavailable( + f"ask needs a SQL-backed target ({', '.join(SQL_TARGETS)}) — " + f"current target: '{target}'" + ) + try: + from vanna import Agent, AgentConfig + from vanna.core.registry import ToolRegistry + from vanna.core.user import RequestContext, User, UserResolver + from vanna.integrations.ollama import OllamaLlmService + from vanna.tools import RunSqlTool + except ImportError as e: + raise AskUnavailable( + "vanna not installed — run: pip install 'thicket[ask]'" + ) from e + + class _LocalUserResolver(UserResolver): + """Single-user resolver: every ask is the same local user.""" + async def resolve_user(self, request_context) -> User: + return User(id="thicket-local", username="thicket", + email="thicket@local", group_memberships=["user"]) + + from vanna.capabilities.agent_memory.base import AgentMemory + + class _StatelessMemory(AgentMemory): + """No persistence between asks — every question stands alone.""" + + def save_text_memory(self, content, context): + return None + + def save_tool_usage(self, question, tool_name, args, context, + success=True, metadata=None): + return None + + def get_recent_memories(self, context, limit=10): + return [] + + def get_recent_text_memories(self, context, limit=10): + return [] + + def search_similar_usage(self, question, context, *, limit=10, + similarity_threshold=0.7, + tool_name_filter=None): + return [] + + def search_text_memories(self, query, context, *, limit=10, + similarity_threshold=0.7): + return [] + + def clear_memories(self, context, tool_name=None, before_date=None): + return 0 + + def delete_by_id(self, context, memory_id): + return False + + def delete_text_memory(self, context, memory_id): + return False + + runner_cls = self._runner_cls(target) + runner = runner_cls(**_RUNNER_PARAMS[target]()) + + # Scope the tool's result-CSV scratch to a temp dir — without + # this, every ask litters a hash-named folder in the cwd. + from vanna.tools import LocalFileSystem + + registry = ToolRegistry() + registry.register_local_tool( + RunSqlTool( + sql_runner=runner, + file_system=LocalFileSystem( + working_directory=tempfile.mkdtemp(prefix="thicket-ask-")), + ), + access_groups=["user"]) + + resolver = _LocalUserResolver() + self._context = RequestContext( + remote_addr="127.0.0.1", + metadata={"source": "thicket"}, + ) + self._log = log + self._agent = Agent( + llm_service=OllamaLlmService( + model=llm_model, host=os.environ.get("OLLAMA_HOST"), + num_ctx=8192, temperature=0.1, + ), + config=AgentConfig(stream_responses=False), + tool_registry=registry, + user_resolver=resolver, + agent_memory=_StatelessMemory(), + ) + self._ddl = target_ddl(target, collection) + + @staticmethod + def _runner_cls(target: str): + if target == "pgvector": + from vanna.integrations.postgres import PostgresRunner + return PostgresRunner + from vanna.integrations.mysql import MySQLRunner + return MySQLRunner + + async def _ask_async(self, question: str) -> str: + json_hint = ( + "payload->>'document_title'" if self._target == "pgvector" + else "JSON_UNQUOTE(JSON_EXTRACT(payload, '$.document_title'))" + ) + prompt = ( + "You are querying a knowledge-base corpus. Schema:\n" + f"{self._ddl}\n" + f"JSON fields are read with {json_hint}. " + "Read-only: SELECT queries only. " + "Answer with the final result only.\n\n" + f"Question: {question}" + ) + # The stream carries status pings, reasoning, tool calls, and + # the final answer — the last RichText IS the answer. + texts: list[str] = [] + async for component in self._agent.send_message(self._context, prompt): + text = _component_text(component) + if text: + texts.append(text) + return texts[-1] if texts else "(no answer produced)" + + def ask(self, question: str) -> str: + """Ask one question; returns the agent's answer text.""" + self._log(f"Asking {self._agent.__class__.__name__} " + f"(target corpus, Ollama LLM)...") + answer = asyncio.run(self._ask_async(question)) + return answer or "(no answer produced)" + + +def ask(question: str, target: str, collection: str = "second_brain", + llm_model: str = "llama3", log: Callable[[str], None] = lambda _msg: None) -> str: + """Convenience one-shot entry for CLI and workers.""" + return VannaAsker(target, collection, llm_model, log).ask(question) diff --git a/thicket/chunker.py b/thicket/chunker.py new file mode 100644 index 0000000..81fea9b --- /dev/null +++ b/thicket/chunker.py @@ -0,0 +1,184 @@ +"""Contextual chunker — structure-preserving splitting for technical +corpora. + +The chunker treats programming books, system configuration, and policy +documents as what they are: mixed prose and code. + +Invariants: + + * Fenced code blocks (``` / ~~~) are atomic — a block is never split + mid-listing, never merged with prose, and keeps every newline and + indentation character verbatim. Only oversized blocks (beyond + ~2x the budget) are line-windowed, and every window carries the + language tag. + * Prose chunks are paragraph-aligned: paragraphs are never split + mid-way (an oversized paragraph is line-windowed with its lines + preserved), and intra-chunk newlines are kept — lists, commands, + and tables hold their line structure in the stored payload. + * ATX headings require whitespace after the hashes (``^#{1,6}\\s``), + so shebangs (``#!``), machine comments (``#x``), and fenced code + never masquerade as section headers. + * Each chunk carries a contextual prefix — ``[Source: … | Section: …]`` + plus ``| code:lang`` for code — which is what gets embedded; the + plain text is stored verbatim. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass + +# ATX heading: 1-6 hashes, mandatory whitespace, then content — rejects +# "#!/shebang", "#no-space-comment", and bare "#######" runs. +HEADING_RE = re.compile(r"^#{1,6}\s+\S") +# Fenced block opening: ``` or ~~~ with an optional language tag. +FENCE_OPEN_RE = re.compile(r"^(`{3,}|~{3,})\s*(\S*)\s*$") +# Indented block (pandoc-style code): 4+ spaces or a tab, not a list. +INDENT_RE = re.compile(r"^(?: {4,}|\t)\S") + +# Code blocks larger than chunk_size * this factor are line-windowed. +OVERSIZED_FACTOR = 2 + + +@dataclass(slots=True) +class Chunk: + header: str + text: str + contextual_text: str + kind: str = "prose" # "prose" | "code" + lang: str | None = None # language tag for code chunks + source: str | None = None # input-tree path of the origin file + + +def _words(text: str) -> int: + return len(text.split()) + + +class ContextualChunker: + """Splits markdown-ish text into header-aware, code-preserving + contextual chunks.""" + + def __init__(self, chunk_size: int = 500, overlap: int = 50): + self.chunk_size = max(1, int(chunk_size)) + self.overlap = max(0, int(overlap)) + + # ── block parsing ── + + def _blocks(self, text: str): + """Yield (kind, body, lang) blocks: fenced code carries its + whole verbatim body; everything else is blank-line-delimited.""" + lines = text.split("\n") + i = 0 + while i < len(lines): + fence = FENCE_OPEN_RE.match(lines[i]) + if fence: + marker, lang = fence.group(1), fence.group(2) + j = i + 1 + while j < len(lines) and not lines[j].startswith(marker[:3]): + j += 1 + yield ("code", "\n".join(lines[i + 1:j]), lang or None) + i = j + 1 + continue + if not lines[i].strip(): + i += 1 + continue + j = i + while j < len(lines) and lines[j].strip(): + j += 1 + yield ("para", "\n".join(lines[i:j]), None) + i = j + + # ── chunk assembly ── + + def chunk(self, title: str, text: str, + source_path: str | None = None) -> list[Chunk]: + chunks: list[Chunk] = [] + header = "Introduction" + buffer: list[str] = [] # prose paragraphs, verbatim + + def _flush() -> None: + nonlocal buffer + if not buffer: + return + chunks.append(self._prose_chunk(title, header, + "\n\n".join(buffer))) + # Paragraph-level overlap: carry the tail paragraph into + # the next chunk when it fits the overlap budget. + buffer = ([buffer[-1]] if _words(buffer[-1]) <= self.overlap + else []) + + for kind, body, lang in self._blocks(text): + if kind == "code": + _flush() + chunks.extend(self._code_chunks(title, header, body, lang)) + continue + + first_line = body.split("\n", 1)[0] + if "\n" not in body and HEADING_RE.match(first_line): + _flush() # section boundary never straddles chunks + header = first_line.lstrip("#").strip() or header + continue + + if len(body.split("\n")) > 1 and all( + INDENT_RE.match(ln) or not ln.strip() + for ln in body.split("\n")): + # Indented block (pandoc-style code): treat as code. + _flush() + chunks.extend(self._code_chunks( + title, header, + re.sub(r"^ {0,4}", "", body, flags=re.M), None)) + continue + + if buffer and _words("\n\n".join(buffer)) + _words(body) > self.chunk_size: + _flush() + buffer.append(body) + if _words("\n\n".join(buffer)) >= self.chunk_size: + _flush() + + _flush() + if source_path: + for chunk in chunks: + chunk.source = source_path + return chunks + + # ── chunk constructors ── + + def _prose_chunk(self, title: str, header: str, body: str) -> Chunk: + return Chunk( + header=header, + text=body, + contextual_text=f"[Source: {title} | Section: {header}]\n{body}", + kind="prose", + ) + + def _code_chunk(self, title: str, header: str, body: str, + lang: str | None) -> Chunk: + tag = f" | code:{lang}" if lang else " | code" + return Chunk( + header=header, + text=body, + contextual_text=f"[Source: {title} | Section: {header}{tag}]\n{body}", + kind="code", + lang=lang, + ) + + def _code_chunks(self, title: str, header: str, body: str, + lang: str | None) -> list[Chunk]: + """One atomic code chunk — or line windows when the block is + oversized; every window keeps the language tag.""" + if _words(body) <= self.chunk_size * OVERSIZED_FACTOR: + return [self._code_chunk(title, header, body, lang)] + + lines = body.split("\n") + windows: list[list[str]] = [] + current: list[str] = [] + for line in lines: + current.append(line) + if _words("\n".join(current)) >= self.chunk_size: + windows.append(current) + # Two tail lines carry into the next window as overlap. + current = current[-2:] + if current and (not windows or current != windows[-1]): + windows.append(current) + return [self._code_chunk(title, header, "\n".join(w), lang) + for w in windows] diff --git a/thicket/cli.py b/thicket/cli.py new file mode 100644 index 0000000..157d401 --- /dev/null +++ b/thicket/cli.py @@ -0,0 +1,298 @@ +"""Command-line interface for Thicket. + +Three modes: + + - ``--version`` — print the package version and exit. + - ``--dry-run`` — probe the environment (Python modules, + Qdrant service, Ollama service), print a readiness report, exit. + Does NOT launch the GUI and ingests nothing. + - ``--ingest DIR --vault DIR`` — headless batch ingest using the same + core pipeline modules the GUI drives. No Qt is imported on this + path, so it works over SSH / cron. + +With no flags, ``main()`` defers to ``ui_window.launch_gui()``. + +Heavy imports (``ui_window``, pipeline modules) are deferred into the +branches that need them so that ``--version`` / ``--dry-run`` start +instantly and never touch PySide6. +""" + +from __future__ import annotations + +import argparse +import sys +from pathlib import Path + + +def build_parser() -> argparse.ArgumentParser: + """Build the CLI argument parser.""" + parser = argparse.ArgumentParser( + prog="thicket", + description="Super-ingest + RAG console: documents -> Obsidian + Qdrant + LightRAG", + ) + parser.add_argument( + "--version", action="store_true", + help="Print version and exit", + ) + parser.add_argument( + "--dry-run", action="store_true", + help="Probe environment (modules, Qdrant, Ollama) and print a " + "readiness report — do NOT launch GUI or ingest anything", + ) + parser.add_argument( + "--ingest", metavar="DIR", + help="Headless mode: batch-ingest this directory (no GUI)", + ) + parser.add_argument( + "--vault", metavar="DIR", + help="Obsidian vault root (required with --ingest)", + ) + from .vector_stores import TARGETS as _VECTOR_TARGETS + parser.add_argument( + "--target", default="qdrant", + choices=["obsidian"] + list(_VECTOR_TARGETS), + help="Ingest destination: 'obsidian' = notes only; any vector " + "store key = notes + that store (qdrant default; most run " + "embedded under /.thicket/; pgvector/mariadb use " + "PG* / MARIADB_* env)", + ) + parser.add_argument( + "--host", default="localhost", + help="Service host for service-backed targets (qdrant default " + "6333, weaviate 8080; qdrant target only)", + ) + parser.add_argument( + "--port", type=int, default=6333, + help="Service port for service-backed targets (qdrant 6333, " + "weaviate 8080)", + ) + parser.add_argument( + "--collection", default="second_brain", + help="Qdrant collection name (default: second_brain)", + ) + from .embedder import DEFAULT_EMBED_MODEL + parser.add_argument( + "--embed-model", default=DEFAULT_EMBED_MODEL, + help=f"FastEmbed embedding model (default: {DEFAULT_EMBED_MODEL} — " + "code/config-strong; see thicket.embedder catalog)", + ) + parser.add_argument( + "--chunk-size", type=int, default=400, + help="Chunk size in words (default: 400)", + ) + parser.add_argument( + "--overlap", type=int, default=50, + help="Chunk overlap in words (default: 50)", + ) + parser.add_argument( + "--no-vault", action="store_true", + help="Headless mode: skip the Obsidian vault-note stage " + "(vectors/graph only)", + ) + parser.add_argument( + "--no-qdrant", action="store_true", + help="Headless mode: skip the Qdrant vector stage (vault notes only)", + ) + parser.add_argument( + "--lightrag", action="store_true", + help="Headless mode: also run the knowledge-graph stage", + ) + parser.add_argument( + "--graph-engine", default="lightrag", choices=["lightrag", "graphify"], + help="Knowledge-graph engine (default: lightrag; graphify builds " + "graph.json + HTML report via one local Ollama pass)", + ) + parser.add_argument( + "--notes-dir", default="Ingested_Brain", + help="Vault subdirectory for generated notes (default: Ingested_Brain)", + ) + parser.add_argument( + "--skip-unchanged", action="store_true", + help="Skip files whose content hash matches the last ingest " + "(manifest under /.thicket/)", + ) + parser.add_argument( + "--max-mb", type=int, default=0, + help="Skip files larger than this many megabytes (default: 0 = no limit)", + ) + parser.add_argument( + "--archive", action="store_true", + help="After verified ingest, move each source to the archive dir " + "(default: /../ingested-archive) and bzip2 it — " + "sources are never deleted", + ) + parser.add_argument( + "--archive-dir", metavar="DIR", + help="Override the ingested-archive directory (default: " + "sibling of the input dir)", + ) + parser.add_argument( + "--minio", action="store_true", + help="Headless mode: archive source documents to MinIO " + "(env: MINIO_ENDPOINT/ACCESS_KEY/SECRET_KEY/BUCKET)", + ) + parser.add_argument( + "--ask", metavar="QUESTION", + help="Ask a natural-language question over the SQL corpus of the " + "selected target (pgvector / mariadb; Vanna 2 + Ollama) — " + "no GUI, no ingest", + ) + parser.add_argument( + "--ollama-llm", default="llama3", + help="Ollama LLM for LightRAG entity extraction (default: llama3)", + ) + parser.add_argument( + "--ollama-embed", default="nomic-embed-text", + help="Ollama embedding model for LightRAG (default: nomic-embed-text)", + ) + return parser + + +def run_dry_run(qdrant_host: str, qdrant_port: int) -> int: + """Probe the environment and print a readiness report.""" + from .env_probe import PROBED_MODULES, probe_environment + + env = probe_environment(qdrant_host=qdrant_host, qdrant_port=qdrant_port) + + def _module_line(name: str) -> str: + tag = "OK" if env.modules.get(name) else "MISSING" + note = " (optional — graph stage)" if name == "lightrag" else "" + return f" {name:14s} {tag}{note}" + + def _service_line(name: str, up: bool, detail: str, error: str | None) -> str: + state = f"UP — {detail}" if up else f"DOWN ({error or 'no response'})" + return f"{name}: {state}" + + def _ollama_detail() -> str: + shown = env.ollama_models[:12] + extra = len(env.ollama_models) - len(shown) + return ", ".join(shown) + (f" … (+{extra} more)" if extra > 0 else "") \ + or "(no models pulled)" + + print("=== Thicket environment probe ===") + print(f"Python: {env.python_version}") + print("Modules:") + print("\n".join(_module_line(name) for name in PROBED_MODULES)) + print(_service_line("Qdrant", env.qdrant_up, + ", ".join(env.qdrant_collections), env.qdrant_error)) + print(_service_line("Postgres", env.pg_up, "reachable", env.pg_error)) + print(_service_line("MinIO", env.minio_up, "reachable", env.minio_error)) + print(_service_line("Ollama", env.ollama_up, + _ollama_detail(), env.ollama_error)) + print() + if env.qdrant_ready: + print("READY: vault + Qdrant stages available.") + else: + print("PARTIAL: vault stage available; install '.[ingest]' extras and/or") + print("start Qdrant (docker run -p 6333:6333 qdrant/qdrant) for vectors.") + return 0 + + +def run_headless(args: argparse.Namespace) -> int: + """Drive the core pipeline without Qt — the GUI's worker logic, + reduced to sequential prints.""" + from .pipeline_core import headless_ingest + + input_dir = Path(args.ingest).expanduser() + vault_dir = Path(args.vault).expanduser() if args.vault else None + + if not input_dir.is_dir(): + print(f"Error: input directory '{input_dir}' does not exist.", file=sys.stderr) + return 1 + if vault_dir is None: + print("Error: --vault is required with --ingest.", file=sys.stderr) + return 1 + if not vault_dir.is_dir(): + print(f"Error: vault directory '{vault_dir}' does not exist.", file=sys.stderr) + return 1 + + try: + ok, fail = headless_ingest( + input_dir=input_dir, + vault_dir=vault_dir, + qdrant_host=args.host, + qdrant_port=args.port, + target=args.target if args.target != "obsidian" else "qdrant", + collection=args.collection, + embed_model=args.embed_model, + chunk_size=args.chunk_size, + overlap=args.overlap, + use_vault=not args.no_vault, + use_qdrant=(args.target != "obsidian") and not args.no_qdrant, + use_lightrag=args.lightrag, + use_minio=args.minio, + graph_engine=args.graph_engine, + notes_dir=args.notes_dir, + skip_unchanged=args.skip_unchanged, + use_fs_archive=args.archive, + archive_dir=Path(args.archive_dir).expanduser() + if args.archive_dir else None, + max_mb=args.max_mb, + ollama_llm=args.ollama_llm, + ollama_embed=args.ollama_embed, + ) + except KeyboardInterrupt: + return 130 # POSIX: 128 + SIGINT + print(f"\nDone — {ok} succeeded, {fail} failed.") + return 0 if fail == 0 else 2 + + + +def run_ask(args: argparse.Namespace) -> int: + """Natural-language SQL over the corpus via Vanna + Ollama.""" + from .ask_vanna import ask + + try: + answer = ask(args.ask, target=args.target, collection=args.collection, + llm_model=args.ollama_llm, + log=lambda msg: print(f"> {msg}")) + except Exception as e: # noqa: BLE001 — surface every failure readably + print(f"ASK ERROR: {e}", file=sys.stderr) + return 1 + print(f"\n{answer}") + return 0 + + +def main(argv: list[str] | None = None) -> int: + """Entry point — backend profiles first, then arg dispatch.""" + from .layout import apply_backend_profiles + + apply_backend_profiles() + + # Step-down by mode: --version, --dry-run, and --ingest each run + # Qt-free and exit; only the no-flag default loads the GUI. + parser = build_parser() + args = parser.parse_args(argv) + + if args.version: + from . import __version__ + print(f"thicket {__version__}") + return 0 + + if args.dry_run: + return run_dry_run(args.host, args.port) + + if args.ingest: + return run_headless(args) + + if args.ask: + return run_ask(args) + + # No flag — launch the GUI. + try: + from .ui_window import launch_gui + except ImportError as e: + print( + f"GUI unavailable: {e}\n" + f"Run the console with its environment:\n" + f" thicket (~/.local/bin command)\n" + f" /mnt/AI/runtime/thicket-venv/bin/python thicket.py\n" + f"Or bootstrap it: scripts/bootstrap_sources.sh", + file=sys.stderr, + ) + return 1 + return launch_gui(argv) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/thicket/embedder.py b/thicket/embedder.py new file mode 100644 index 0000000..c15f1a9 --- /dev/null +++ b/thicket/embedder.py @@ -0,0 +1,79 @@ +"""Local embedding engine — FastEmbed wrapper. + +FastEmbed runs ONNX models fully locally (no API keys, no network +after the first model download). The catalog maps each model to its +vector dimension so the Qdrant collection is created with matching +geometry. BGE models want a short instruction prefix on *query* side +only — passages are embedded bare. +""" + +from __future__ import annotations + +# Curated catalog for technical corpora — code/config-heavy +# libraries want jina-code (English+code, 8k context); bge-base is the +# prose-strong alternative; bge-small for light setups. +# model name -> embedding dimension +EMBEDDING_MODELS: dict[str, int] = { + "jinaai/jina-embeddings-v2-base-code": 768, + "BAAI/bge-base-en-v1.5": 768, + "BAAI/bge-small-en-v1.5": 384, +} + +DEFAULT_EMBED_MODEL = "jinaai/jina-embeddings-v2-base-code" + +# BGE retrieval instruction — prefix for QUERIES only, never passages. +BGE_QUERY_PREFIX = "Represent this sentence for searching relevant passages: " + + +class EmbedderUnavailable(Exception): + """Raised when fastembed is missing or the model cannot load.""" + + +def model_dim(model_name: str) -> int: + """Dimension for a catalog model (0 if unknown — Qdrant will tell us).""" + return EMBEDDING_MODELS.get(model_name, 0) + + +class EmbeddingEngine: + """Lazy-loading FastEmbed engine. Construct anywhere; call load() + from the worker thread (first call may download the model).""" + + def __init__(self, model_name: str = DEFAULT_EMBED_MODEL): + self.model_name = model_name + self._model = None + + @property + def dim(self) -> int: + return EMBEDDING_MODELS.get(self.model_name, 0) + + def load(self) -> None: + """Import fastembed and load the model. Safe to call twice.""" + if self._model is not None: + return + try: + from fastembed import TextEmbedding + except ImportError as e: + raise EmbedderUnavailable( + "fastembed not installed — run: pip install 'thicket[ingest]'" + ) from e + try: + self._model = TextEmbedding(model_name=self.model_name) + except Exception as e: + raise EmbedderUnavailable( + f"failed to load embedding model '{self.model_name}': {e}" + ) from e + + def embed(self, texts: list[str]) -> list[list[float]]: + """Embed passages (no instruction prefix). Order preserved. + + Values are normalized to plain Python floats: FastEmbed yields + numpy scalars, which some targets' validators reject.""" + self.load() + return [[float(x) for x in vec] for vec in self._model.embed(texts)] + + def embed_query(self, text: str) -> list[float]: + """Embed a retrieval query with the BGE instruction prefix + when the selected model is from the BGE family.""" + self.load() + query = BGE_QUERY_PREFIX + text if "bge" in self.model_name.lower() else text + return [float(x) for x in next(iter(self._model.embed([query])))] diff --git a/thicket/env_probe.py b/thicket/env_probe.py new file mode 100644 index 0000000..97c722a --- /dev/null +++ b/thicket/env_probe.py @@ -0,0 +1,246 @@ +"""Environment probe — readiness report shown at startup. + +Checks (all cheap, all non-fatal): + + * Python module availability for every pipeline dependency + (importlib.util.find_spec — no heavy imports). + * Qdrant service reachability (get_collections with a short timeout). + * Ollama service reachability (GET /api/tags, also lists installed + models so the UI can pre-fill the model combo). + +The GUI runs this in a background thread because the FastEmbed model +check and service pings can take seconds on a cold start. +""" + +from __future__ import annotations + +import importlib.util +import json +import os +import platform +import urllib.request +from dataclasses import dataclass, field + +# module name -> role in the pipeline +PROBED_MODULES: dict[str, str] = { + "bs4": "EPUB parsing", + "ebooklib": "EPUB parsing", + "pypdf": "PDF parsing", + "slugify": "vault note filenames", + "fastembed": "local embeddings", + "qdrant_client": "vector target: qdrant", + "chromadb": "vector target: chroma (optional)", + "lancedb": "vector target: lancedb (optional)", + "faiss": "vector target: faiss (optional)", + "pymilvus": "vector target: milvus lite (optional)", + "weaviate": "vector target: weaviate (optional)", + "psycopg": "vector target: pgvector (optional)", + "pgvector": "vector target: pgvector (optional)", + "duckdb": "vector target: duckdb (optional)", + "sqlite_vec": "vector target: sqlite-vec (optional)", + "pymysql": "vector target: mariadb (optional)", + "graphify": "graph engine (optional)", + "vanna": "ask interaction (optional)", + "minio": "object archive (optional)", + "lightrag": "knowledge graph (optional)", +} + + +def _module_available(name: str) -> bool: + try: + return importlib.util.find_spec(name) is not None + except (ImportError, ValueError): + return False + + +@dataclass(slots=True) +class EnvProbe: + python_version: str = "" + modules: dict[str, bool] = field(default_factory=dict) + qdrant_up: bool = False + qdrant_error: str | None = None + qdrant_collections: list[str] = field(default_factory=list) + ollama_up: bool = False + ollama_error: str | None = None + ollama_models: list[str] = field(default_factory=list) + pg_up: bool = False + pg_error: str | None = None + weaviate_up: bool = False + weaviate_error: str | None = None + mariadb_up: bool = False + mariadb_error: str | None = None + minio_up: bool = False + minio_error: str | None = None + + @property + def vault_ready(self) -> bool: + """Vault-note stage: only needs slugify.""" + return self.modules.get("slugify", False) + + @property + def qdrant_ready(self) -> bool: + """Vector stage: client + embedder libs AND a reachable service.""" + return ( + self.modules.get("fastembed", False) + and self.modules.get("qdrant_client", False) + and self.qdrant_up + ) + + @property + def graph_ready(self) -> bool: + """Graph stage: lightrag installed AND Ollama reachable.""" + return self.modules.get("lightrag", False) and self.ollama_up + + # service key -> EnvProbe attribute carrying its liveness + SERVICE_LIVENESS = {"qdrant": "qdrant_up", "postgres": "pg_up", + "weaviate": "weaviate_up", "mariadb": "mariadb_up"} + + def vector_ready(self, target: str) -> bool: + """Readiness for a specific vector target: modules present, + plus the live service when the target is service-backed.""" + from .vector_stores import TARGETS + spec = TARGETS.get(target) + if spec is None: + return False + if not all(self.modules.get(m, False) for m in spec.modules): + return False + live = self.SERVICE_LIVENESS.get(spec.service) if spec.service else None + return live is None or getattr(self, live, False) + + @property + def ingest_ready(self) -> bool: + """At least one parse path for every declared extension.""" + return ( + self.modules.get("pypdf", False) + and self.modules.get("bs4", False) + and self.modules.get("ebooklib", False) + and self.modules.get("slugify", False) + ) + + +def _probe_qdrant(host: str, port: int, env: EnvProbe) -> None: + try: + from qdrant_client import QdrantClient + except ImportError: + env.qdrant_error = "qdrant-client not installed" + return + try: + # Connectivity ping only — no client/server version negotiation. + client = QdrantClient(url=f"http://{host}:{port}", timeout=3, + check_compatibility=False) + env.qdrant_collections = sorted( + c.name for c in client.get_collections().collections + ) + env.qdrant_up = True + except Exception as e: # noqa: BLE001 — any failure means "down" + env.qdrant_error = str(e).splitlines()[0][:100] + + +def _probe_ollama(host: str, env: EnvProbe, port: int = 11434) -> None: + url = f"http://{host}:{port}/api/tags" + try: + with urllib.request.urlopen(url, timeout=3) as resp: + data = json.loads(resp.read().decode("utf-8", errors="ignore")) + env.ollama_models = sorted( + m.get("name", "?") for m in data.get("models", []) + ) + env.ollama_up = True + except Exception as e: # noqa: BLE001 + env.ollama_error = str(e)[:100] + + +def _probe_postgres(env: EnvProbe) -> None: + """Env-driven like the MariaDB probe: PGHOST/PGPORT/PGUSER/ + PGPASSWORD/PGDATABASE (or PGDSN) — the same variables the + pgvector target documents.""" + import os + + try: + import psycopg + except ImportError: + env.pg_error = "psycopg not installed" + return + kwargs = {"connect_timeout": 3} + if os.environ.get("PGDSN"): + conn_kwargs = {"dsn": os.environ["PGDSN"], **kwargs} + else: + conn_kwargs = {**kwargs, "host": os.environ.get("PGHOST", "localhost"), + "port": int(os.environ.get("PGPORT", "5432")), + "user": os.environ.get("PGUSER"), + "password": os.environ.get("PGPASSWORD") or None, + "dbname": os.environ.get("PGDATABASE")} + try: + with psycopg.connect(**conn_kwargs) as conn: + conn.execute("SELECT 1") + env.pg_up = True + except Exception as e: # noqa: BLE001 — any failure means "down" + env.pg_error = str(e).splitlines()[0][:100] + + +def _probe_minio(env: EnvProbe) -> None: + """Ping the S3-compatible health endpoint — no client library needed.""" + import urllib.request + + endpoint = os.environ.get("MINIO_ENDPOINT", "localhost:9000") + host = endpoint if endpoint.startswith("http") else f"http://{endpoint}" + try: + with urllib.request.urlopen(f"{host}/minio/health/live", timeout=2): + env.minio_up = True + except Exception as e: # noqa: BLE001 + env.minio_error = str(e)[:80] + + +def _probe_weaviate(env: EnvProbe, ports: tuple[int, ...] = (8080,)) -> None: + """Ping /v1/.well-known/ready over HTTP — no client library needed.""" + import urllib.request + + for port in ports: + url = f"http://localhost:{port}/v1/.well-known/ready" + try: + with urllib.request.urlopen(url, timeout=2) as resp: + if resp.status == 200: + env.weaviate_up = True + return + except Exception: + continue + env.weaviate_error = "no response on ports " + ", ".join(map(str, ports)) + + +def _probe_mariadb(env: EnvProbe) -> None: + try: + import pymysql + except ImportError: + env.mariadb_error = "pymysql not installed" + return + import os + + try: + conn = pymysql.connect( + host=os.environ.get("MARIADB_HOST", "127.0.0.1"), + port=int(os.environ.get("MARIADB_PORT", "3306")), + user=os.environ.get("MARIADB_USER", "root"), + password=os.environ.get("MARIADB_PASSWORD", ""), + unix_socket=os.environ.get("MARIADB_UNIX_SOCKET") or None, + connect_timeout=2, + ) + with conn.cursor() as cur: + cur.execute("SELECT 1") + conn.close() + env.mariadb_up = True + except Exception as e: # noqa: BLE001 + env.mariadb_error = str(e).splitlines()[0][:100] + + +def probe_environment(qdrant_host: str = "localhost", + qdrant_port: int = 6333, + ollama_host: str = "localhost") -> EnvProbe: + """Gather the full readiness report. Never raises.""" + env = EnvProbe(python_version=platform.python_version()) + env.modules = {name: _module_available(name) for name in PROBED_MODULES} + _probe_qdrant(qdrant_host, qdrant_port, env) + _probe_ollama(ollama_host, env) + _probe_postgres(env) + _probe_weaviate(env) + _probe_mariadb(env) + _probe_minio(env) + return env diff --git a/thicket/extractors.py b/thicket/extractors.py new file mode 100644 index 0000000..c42ec28 --- /dev/null +++ b/thicket/extractors.py @@ -0,0 +1,149 @@ +"""Document extraction — PDF / EPUB / Markdown / plain text. + +Extracts ``(title, text)`` from every supported format. Structural +headers are preserved in the extracted text (PDF pages become +``## Page N`` headings, EPUB chapters are joined with rules) so the +chunker can attribute each chunk to its section. + +All third-party parsers are imported lazily inside the extract methods: +the GUI must launch and probe on a bare system where EbookLib or pypdf +are not installed, and fail with an actionable message only when the +matching file type is actually encountered. +""" + +from __future__ import annotations + +import re +from pathlib import Path + +# Source files ingested verbatim as language-tagged code blocks. +CODE_LANGUAGES: dict[str, str] = { + ".py": "python", ".sh": "bash", ".bash": "bash", ".zsh": "zsh", + ".yaml": "yaml", ".yml": "yaml", ".toml": "toml", ".ini": "ini", + ".conf": "ini", ".cfg": "ini", ".json": "json", ".sql": "sql", + ".rs": "rust", ".go": "go", ".c": "c", ".h": "c", ".cpp": "cpp", + ".js": "javascript", ".ts": "typescript", ".tf": "hcl", ".nix": "nix", +} + +SUPPORTED_EXTENSIONS = frozenset( + {".pdf", ".epub", ".md", ".markdown", ".txt"} + | set(CODE_LANGUAGES) +) + + +class ExtractionError(Exception): + """Raised when a document cannot be parsed (missing dep, bad file).""" + + +def scan_files( + input_dir: Path, + extensions: frozenset[str] | set[str] = SUPPORTED_EXTENSIONS, +) -> list[Path]: + """Recursively list supported documents under *input_dir*, sorted + for deterministic queue order.""" + return sorted( + (p for p in input_dir.rglob("*") + if p.is_file() and p.suffix.lower() in extensions), + key=lambda p: str(p).lower(), + ) + + +def _title_from_stem(filepath: Path) -> str: + return filepath.stem.replace("-", " ").replace("_", " ").title() + + +class DocumentExtractor: + """Extracts raw text and structural headers from supported formats.""" + + @staticmethod + def extract(filepath: Path) -> tuple[str, str]: + """Dispatch on extension via the runner table. Raises + ExtractionError for unsupported types or missing parser + dependencies.""" + runner = _EXTENSION_RUNNERS.get(filepath.suffix.lower()) + if runner is None: + raise ExtractionError(f"unsupported file type: {filepath.suffix}") + return runner(filepath) + + @staticmethod + def extract_md_txt(filepath: Path) -> tuple[str, str]: + text = filepath.read_text(encoding="utf-8", errors="ignore") + return _title_from_stem(filepath), text + + @staticmethod + def extract_code(filepath: Path) -> tuple[str, str]: + """Source/config file: verbatim content wrapped in a + language-tagged fence so the chunker treats it as one code + document and the vault note renders it as a listing.""" + lang = CODE_LANGUAGES.get(filepath.suffix.lower(), "") + body = filepath.read_text(encoding="utf-8", errors="ignore") + return _title_from_stem(filepath), f"```{lang}\n{body}\n```" + + @staticmethod + def extract_pdf(filepath: Path) -> tuple[str, str]: + try: + from pypdf import PdfReader + except ImportError as e: + raise ExtractionError( + "pypdf not installed — run: pip install 'thicket[ingest]'" + ) from e + + reader = PdfReader(str(filepath)) + title = _title_from_stem(filepath) + + # Prefer the embedded metadata title when present. + try: + if reader.metadata and reader.metadata.title: + title = str(reader.metadata.title).strip() or title + except Exception: + pass # malformed metadata must not kill the extract + + pages_text: list[str] = [] + for i, page in enumerate(reader.pages): + try: + txt = page.extract_text() or "" + except Exception: + txt = "" + if txt.strip(): + pages_text.append(f"## Page {i + 1}\n\n{txt}") + + return title, "\n\n".join(pages_text) + + @staticmethod + def extract_epub(filepath: Path) -> tuple[str, str]: + try: + import bs4 + from ebooklib import epub, ITEM_DOCUMENT + except ImportError as e: + raise ExtractionError( + "ebooklib / beautifulsoup4 not installed — run: " + "pip install 'thicket[ingest]'" + ) from e + + book = epub.read_epub(str(filepath)) + title = _title_from_stem(filepath) + + meta_titles = book.get_metadata("DC", "title") + if meta_titles: + title = str(meta_titles[0][0]).strip() or title + + chapters: list[str] = [] + for item in book.get_items_of_type(ITEM_DOCUMENT): + soup = bs4.BeautifulSoup(item.get_content(), "html.parser") + text = soup.get_text(separator="\n") + clean_text = re.sub(r"\n+", "\n", text).strip() + if clean_text: + chapters.append(clean_text) + + return title, "\n\n---\n\n".join(chapters) + +# Extension dispatch table — the single source of routing for extract(). +# Adding a format means adding one entry here and its extractor method. +_EXTENSION_RUNNERS: dict[str, object] = { + ".md": DocumentExtractor.extract_md_txt, + ".markdown": DocumentExtractor.extract_md_txt, + ".txt": DocumentExtractor.extract_md_txt, + ".pdf": DocumentExtractor.extract_pdf, + ".epub": DocumentExtractor.extract_epub, + **{ext: DocumentExtractor.extract_code for ext in CODE_LANGUAGES}, +} diff --git a/thicket/graph_store.py b/thicket/graph_store.py new file mode 100644 index 0000000..bd5aab0 --- /dev/null +++ b/thicket/graph_store.py @@ -0,0 +1,210 @@ +"""Knowledge-graph engines (optional graph stage). + +Two engines share one contract (``ingest_document`` + ``finalize``): + + * lightrag — per-document entity extraction into a merged graph + (Ollama LLM + Ollama embeddings). The LightRAG package renamed + its Ollama bindings across releases, so binding functions are + resolved by stepping through the known candidate names. + * graphify — stages extracted documents as Markdown, then one + ``graphify extract --backend ollama`` pass builds graph.json, + GRAPH_REPORT.md, and the interactive graph.html (Graphify-Labs). + +When an engine cannot be constructed — missing package, unknown API +generation, unreachable service — it raises GraphUnavailable with the +reason, and the stage reports unavailable with that reason. +""" + +from __future__ import annotations + +import hashlib +from collections.abc import Callable +from pathlib import Path + + +class GraphUnavailable(Exception): + """Raised when a graph engine cannot be constructed.""" + + +def _resolve_ollama_binding(): + """Return (complete_fn, embed_fn) from whichever naming generation + of lightrag is installed, or raise GraphUnavailable.""" + try: + import lightrag.llm.ollama as _ollama_mod + except ImportError as e: + raise GraphUnavailable( + "lightrag not installed (or too old) — run: pip install 'thicket[graph]'" + ) from e + + complete = next( + (name for name in ("ollama_model_complete", "ollama_complete") + if hasattr(_ollama_mod, name)), None + ) + embed = next( + (name for name in ("ollama_embed", "ollama_embedding") + if hasattr(_ollama_mod, name)), None + ) + if not complete or not embed: + raise GraphUnavailable( + "installed lightrag exposes no known Ollama binding " + f"(found complete={complete!r}, embed={embed!r}) — pin " + "lightrag-hku to a release matching this code" + ) + return getattr(_ollama_mod, complete), getattr(_ollama_mod, embed) + + +class LightRAGGraphStore: + """Extracts entities and relationships via LightRAG + Ollama.""" + + def __init__(self, working_dir: Path, llm_model: str = "llama3", + embed_model: str = "nomic-embed-text", + ollama_host: str = "localhost", + log: Callable[[str], None] = lambda _msg: None): + try: + from lightrag import LightRAG + from lightrag.utils import EmbeddingFunc + except ImportError as e: + raise GraphUnavailable( + "lightrag not installed — run: pip install 'thicket[graph]'" + ) from e + + self.working_dir = working_dir / ".lightrag" + self.working_dir.mkdir(parents=True, exist_ok=True) + + complete_fn, embed_fn = _resolve_ollama_binding() + + # nomic-embed-text (the default Ollama embedding model) is 768-dim. + # num_ctx 16384 keeps entity extraction from truncating mid-chunk. + self._log = log + self._embed_model = embed_model + self.rag = LightRAG( + working_dir=str(self.working_dir), + llm_model_func=complete_fn, + llm_model_name=llm_model, + llm_model_kwargs={"options": {"num_ctx": 16384}}, + embedding_func=EmbeddingFunc( + embedding_dim=768, + max_token_size=8192, + func=lambda texts: embed_fn(texts, embed_model=embed_model), + ), + ) + + def ingest_document(self, title: str, text: str) -> None: + """Insert one document through the full async lifecycle — + pipeline status, storages initialize, ainsert, finalize — + inside one event loop (LightRAG >= 1.5 binds its shared + storage to the running loop).""" + if not text.strip(): + return + import asyncio + + formatted = f"Document Title: {title}\n\n{text}" + + async def _run() -> None: + try: + from lightrag.kg.shared_storage import ( + initialize_pipeline_status, + ) + await initialize_pipeline_status( + workspace=str(self.working_dir)) + except ImportError: + pass # step-down: releases without the handshake need no init + await self.rag.initialize_storages() + try: + await self.rag.ainsert(formatted) + finally: + await self.rag.finalize_storages() + + asyncio.run(_run()) + + +# ────────────────────────────────────────────────────────────────── +# Graphify engine +# ────────────────────────────────────────────────────────────────── + +class GraphifyGraphStore: + """Graphify-Labs graphify: extracted documents are staged as + Markdown, then one ``graphify extract --backend ollama`` pass at + finalize() builds the queryable graph (graph.json), report, and + interactive HTML under /graphify-out/.""" + + def __init__(self, working_dir: Path, llm_model: str = "llama3", + embed_model: str = "nomic-embed-text", + log: Callable[[str], None] = lambda _msg: None): + import importlib.util + + if importlib.util.find_spec("graphify") is None: + raise GraphUnavailable( + "graphify not installed — run: pip install 'thicket[graphify]'" + ) + self._log = log + self._llm_model = llm_model + self.working_dir = working_dir / ".graphify" + self.corpus_dir = self.working_dir / "corpus" + self.corpus_dir.mkdir(parents=True, exist_ok=True) + self._staged = 0 + + def ingest_document(self, title: str, text: str) -> None: + """Stage one document as Markdown for the batch build.""" + if not text.strip(): + return + from slugify import slugify + + slug = slugify(title) or "untitled" + target = self.corpus_dir / f"{slug}.md" + digest = hashlib.md5(text.encode()).hexdigest()[:6] + if target.exists(): # same title, different content — sibling file + target = self.corpus_dir / f"{slug}-{digest}.md" + target.write_text(f"# {title}\n\n{text}", encoding="utf-8") + self._staged += 1 + + def finalize(self) -> None: + """One graphify pass over the staged corpus (local Ollama LLM; + concurrency 1 — local models do not parallelize well).""" + if not self._staged: + return + import subprocess + import sys + + cmd = [sys.executable, "-m", "graphify", "extract", + str(self.corpus_dir), "--backend", "ollama", + "--model", self._llm_model, "--max-concurrency", "1", + "--out", str(self.working_dir)] + self._log(f"Graphify: building graph from {self._staged} staged " + f"document(s) (local Ollama pass)...") + result = subprocess.run(cmd, capture_output=True, text=True, + timeout=3600) + if result.returncode != 0: + raise GraphUnavailable( + f"graphify extract failed: " + f"{(result.stderr or result.stdout).strip()[:300]}" + ) + graph_json = self.working_dir / "graphify-out" / "graph.json" + self._log(f"Graphify: graph built at '{graph_json}' " + f"(query it: graphify query \"...\" --graph {graph_json})") + + +# ────────────────────────────────────────────────────────────────── +# Engine registry — the single dispatch point +# ────────────────────────────────────────────────────────────────── + +GRAPH_ENGINES: dict[str, dict] = { + "lightrag": {"class": LightRAGGraphStore, "modules": ("lightrag",)}, + "graphify": {"class": GraphifyGraphStore, "modules": ("graphify",)}, +} +DEFAULT_GRAPH_ENGINE = "lightrag" + + +def create_graph(engine: str, working_dir: Path, llm_model: str, + embed_model: str, + log: Callable[[str], None] = lambda _msg: None): + """Build the named engine; raises GraphUnavailable with the known + choices for an unknown name.""" + spec = GRAPH_ENGINES.get(engine) + if spec is None: + known = ", ".join(sorted(GRAPH_ENGINES)) + raise GraphUnavailable( + f"unknown graph engine '{engine}' — known: {known}" + ) + return spec["class"](working_dir=working_dir, llm_model=llm_model, + embed_model=embed_model, log=log) diff --git a/thicket/layout.py b/thicket/layout.py new file mode 100644 index 0000000..41d32de --- /dev/null +++ b/thicket/layout.py @@ -0,0 +1,109 @@ +"""AI workstation layout awareness. + +/mnt/AI is the canonical AI filesystem: a deliberate taxonomy where +corpus/cold is the incoming raw pile, corpus/hot is the active brain +("indexed in Vector DBs and used by agents" — Thicket's vault concept: +notes + .thicket/ vector data + graphs in one portable tree), and +corpus/books holds the standing library. + +When the layout exists, Thicket adopts its paths as defaults (IN, +VAULT, archive) and the probe reports what it found. When it does not, +home-directory defaults apply — behavior is identical either way. + +``expand_path`` is the single point where user-supplied paths expand +``~`` and ``$VAR`` references — the GUI, CLI, and core all resolve +identically. +""" + +from __future__ import annotations + +import os +from dataclasses import dataclass +from pathlib import Path + +# Root override for tests / alternative hosts. +AI_ROOT = Path(os.environ.get("THICKET_AI_ROOT", "/mnt/AI")) + +# Canonical subtrees Thicket cares about. The corpus trio carries the +# ingest flow; the rest are reported for context. +@dataclass(frozen=True, slots=True) +class LayoutProfile: + """Resolved view of the AI filesystem — only what exists.""" + root: Path + corpus_cold: Path # incoming / raw + corpus_hot: Path # active brain (vault default) + corpus_books: Path # standing library + corpus_obsidian: Path # dedicated vault alternative + archive: Path # ingested-archive home + books_present: bool + + +def apply_backend_profiles() -> dict[str, str]: + """Load connection profiles from /backends/thicket.env. + + Precedence is unix-ordered: the real environment wins (variables + already set are untouched), the profile fills the gaps, built-in + defaults apply last. KEY=VALUE lines; blank lines and # comments + ignored. Returns the variables the file contributed. + """ + profile = AI_ROOT / "backends" / "thicket.env" + if not profile.is_file(): + return {} + applied: dict[str, str] = {} + for line in profile.read_text(encoding="utf-8", errors="ignore").splitlines(): + line = line.strip() + if not line or line.startswith("#") or "=" not in line: + continue + key, _, value = line.partition("=") + key, value = key.strip(), value.strip().strip('"').strip("'") + if key and key not in os.environ: + os.environ[key] = value + applied[key] = value + return applied + + +def expand_path(value: str | Path) -> Path: + """Expand ~ and $VAR references to a concrete Path.""" + return Path(os.path.expandvars(os.path.expanduser(str(value)))) + + +def detect_layout() -> LayoutProfile | None: + """The layout profile when the corpus taxonomy exists, else None.""" + corpus = AI_ROOT / "corpus" + if not corpus.is_dir(): + return None + return LayoutProfile( + root=AI_ROOT, + corpus_cold=corpus / "cold", + corpus_hot=corpus / "hot", + corpus_books=corpus / "books", + corpus_obsidian=corpus / "obsidian", + archive=corpus / "archive", + books_present=(corpus / "books").is_dir(), + ) + + +def default_input() -> Path: + """IN default: the cold corpus when the layout exists.""" + profile = detect_layout() + if profile and profile.corpus_cold.is_dir(): + return profile.corpus_cold + return Path.home() / "Downloads" / "Raw_Books_And_Papers" + + +def default_vault() -> Path: + """VAULT default: the hot corpus — one tree holding notes, + .thicket/ vector data, and the knowledge graphs.""" + profile = detect_layout() + if profile: + return profile.corpus_hot + return Path.home() / "Documents" / "ObsidianVault" + + +def default_archive(input_dir: Path) -> Path: + """Archive default: corpus/archive inside the layout, else the + ingested-archive sibling of the input tree.""" + profile = detect_layout() + if profile: + return profile.archive + return input_dir.parent / "ingested-archive" diff --git a/thicket/minio_archive.py b/thicket/minio_archive.py new file mode 100644 index 0000000..0773f03 --- /dev/null +++ b/thicket/minio_archive.py @@ -0,0 +1,77 @@ +"""MinIO object archive — durable cold storage for source documents. + +The source files themselves (PDFs, EPUBs, notes) are uploaded to an +S3-compatible MinIO bucket as part of ingest; the vault note records +the resulting ``s3://bucket/key`` URI in its frontmatter. The vault +and vector index stay lean; the originals live in object storage. + +Open-source MinIO has no vector API (vector search is an AIStor +feature; the community edition is S3 object storage) — so MinIO is an +archive stage here, never a vector target. + +Connection from Unix-standard env: MINIO_ENDPOINT (host:port), +MINIO_ACCESS_KEY, MINIO_SECRET_KEY, MINIO_SECURE (default false), +MINIO_BUCKET (default thicket-corpus). +""" + +from __future__ import annotations + +import os +from collections.abc import Callable +from pathlib import Path + +DEFAULT_BUCKET = "thicket-corpus" + + +class ArchiveUnavailable(Exception): + """Raised when the archive stage cannot be constructed or run.""" + + +class MinioArchiver: + """Uploads source documents to a MinIO bucket. Same key = same + document: re-ingesting overwrites in place (idempotent).""" + + def __init__(self, log: Callable[[str], None] = lambda _msg: None): + try: + from minio import Minio + except ImportError as e: + raise ArchiveUnavailable( + "minio package not installed — run: pip install 'thicket[minio]'" + ) from e + + endpoint = os.environ.get("MINIO_ENDPOINT", "localhost:9000") + self.bucket = os.environ.get("MINIO_BUCKET", DEFAULT_BUCKET) + secure = os.environ.get("MINIO_SECURE", "").lower() in ("1", "true", "yes") + self._client = Minio( + endpoint, + access_key=os.environ.get("MINIO_ACCESS_KEY", ""), + secret_key=os.environ.get("MINIO_SECRET_KEY", ""), + secure=secure, + ) + self._log = log + try: + if not self._client.bucket_exists(self.bucket): + self._log(f"Creating MinIO bucket '{self.bucket}'...") + self._client.make_bucket(self.bucket) + except Exception as e: + raise ArchiveUnavailable( + f"cannot reach MinIO at {endpoint}: {e} — set " + f"MINIO_ENDPOINT / MINIO_ACCESS_KEY / MINIO_SECRET_KEY" + ) from e + + def object_key(self, source_path: Path, input_dir: Path) -> str: + """Bucket key mirrors the input tree's relative path.""" + return source_path.relative_to(input_dir).as_posix() + + def archive(self, source_path: Path, input_dir: Path) -> str: + """Upload one source document; returns its s3:// URI.""" + key = self.object_key(source_path, input_dir) + try: + self._client.fput_object( + self.bucket, key, str(source_path), + ) + except Exception as e: + raise ArchiveUnavailable( + f"upload of '{key}' failed: {e}" + ) from e + return f"s3://{self.bucket}/{key}" diff --git a/thicket/pipeline_core.py b/thicket/pipeline_core.py new file mode 100644 index 0000000..d42b0f0 --- /dev/null +++ b/thicket/pipeline_core.py @@ -0,0 +1,429 @@ +"""Pipeline orchestration — the shared engine behind GUI and CLI. + +``IngestPipeline`` owns the per-file walk: extract, then a stage table +(vault note -> vector index -> knowledge graph). It is UI-agnostic: +progress flows out through ``PipelineCallbacks`` (plain callables), +which the QThread worker wires to Qt signals and the headless CLI +wires to prints. Heavy objects (embedding model, Qdrant client, +LightRAG) are constructed once inside ``run()`` — always on the +caller's thread, never the UI thread. + +Design invariants: + + * One stage table drives dispatch — adding a stage is one tuple, + never a new branch nest. + * Stages communicate through a per-file ``ctx`` dict (the vault + stage publishes the note's vault-relative path); no hidden + cross-stage state. + * A failing document is isolated: one ERROR row, queue continues. + * Stop is cooperative and checked between files only — a file is + either fully processed or not started. + +Stage status vocabulary (the UI color-codes on these exact strings): +QUEUED, EXTRACT, VAULT, INDEX, GRAPH, DONE, SKIP, ERROR. +""" + +from __future__ import annotations + +import bz2 +import hashlib +import json +import shutil +import sys +from collections.abc import Callable +from dataclasses import dataclass +from pathlib import Path + +from .chunker import ContextualChunker +from .embedder import DEFAULT_EMBED_MODEL, EmbeddingEngine, model_dim +from .extractors import DocumentExtractor, scan_files +from .graph_store import create_graph +from .layout import default_archive, expand_path +from .minio_archive import MinioArchiver +from .vector_stores import TARGETS, VectorStoreError, create_store, doc_key +from .vault_writer import ObsidianVaultWriter + +# Stage runner: (filepath, title, content, ctx) -> detail line. +StageRunner = Callable[[Path, str, str, dict], str] + +VAULT_OUTPUT_DIRNAME = "Ingested_Brain" + +# The notes-only destination key — valid wherever a target is accepted. +OBSIDIAN_TARGET = "obsidian" + + +def _bz2_compress(source: Path, level: int = 9) -> Path: + """Streaming bzip2 of *source*; the plain original is replaced by + the .bz2 (compress-then-unlink, never unlink-first).""" + target = source.with_suffix(source.suffix + ".bz2") + with source.open("rb") as src, bz2.BZ2File(target, "wb", + compresslevel=level) as dst: + shutil.copyfileobj(src, dst, length=1 << 20) + source.unlink() + return target + + +@dataclass(slots=True) +class IngestConfig: + input_dir: Path + vault_dir: Path + qdrant_host: str = "localhost" + qdrant_port: int = 6333 + target: str = "qdrant" + collection: str = "second_brain" + embed_model: str = DEFAULT_EMBED_MODEL + chunk_size: int = 400 + overlap: int = 50 + # Destination model: TARGET picks where the corpus lands — a + # vector store key from the registry, or "obsidian" for notes-only. + target: str = "qdrant" # kept in sync with qdrant_host below + use_vault: bool = True + use_qdrant: bool = True + use_lightrag: bool = False + use_minio: bool = False + use_fs_archive: bool = False + archive_dir: Path | None = None # default: corpus/archive or sibling + graph_engine: str = "lightrag" + notes_dir: str = VAULT_OUTPUT_DIRNAME + skip_unchanged: bool = False + max_mb: int = 0 # 0 = no size guard + + def __post_init__(self) -> None: + """Resolve ~ and $VAR in every user-supplied path — the GUI, + CLI, and core share one expansion point.""" + self.input_dir = expand_path(self.input_dir) + self.vault_dir = expand_path(self.vault_dir) + if self.archive_dir is not None: + self.archive_dir = expand_path(self.archive_dir) + ollama_llm: str = "llama3" + ollama_embed: str = "nomic-embed-text" + + +@dataclass(slots=True) +class PipelineCallbacks: + log: object = lambda _msg: None # (msg) + file_status: object = lambda _p, _s: None # (path, status) + file_detail: object = lambda _p, _d: None # (path, detail) + progress: object = lambda _n, _c, _t: None # (name, current, total) + + @classmethod + def quiet(cls) -> "PipelineCallbacks": + return cls() + + +class IngestPipeline: + """One batch run over a directory of documents.""" + + def __init__(self, config: IngestConfig, + callbacks: PipelineCallbacks = PipelineCallbacks.quiet()): + self.config = config + self._manifest_path = config.vault_dir / ".thicket" / "manifest.json" + self.cb = callbacks + self._stop = False + self._engine: EmbeddingEngine | None = None + self._store = None + self._writer: ObsidianVaultWriter | None = None + self._graph = None + self._archiver = None + self._chunker = ContextualChunker( + chunk_size=config.chunk_size, overlap=config.overlap + ) + # Stage table: (status, gate, runner). Dispatch is one pass + # over this tuple — stage order and gating live here only. + self._stages: tuple[tuple[str, bool, StageRunner], ...] = ( + ("MINIO", config.use_minio, self._stage_minio), + ("VAULT", config.use_vault, self._stage_vault), + ("INDEX", config.use_qdrant, self._stage_index), + ("GRAPH", config.use_lightrag, self._stage_graph), + ("ARCHIVE", config.use_fs_archive, self._stage_fs_archive), + ) + + def request_stop(self) -> None: + """Cooperative stop: the queue exits after the current file.""" + self._stop = True + + # ── change manifest (powers skip-unchanged) ── + + def _read_manifest(self) -> dict: + try: + return json.loads(self._manifest_path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return {} + + def _write_manifest(self, manifest: dict) -> None: + self._manifest_path.parent.mkdir(parents=True, exist_ok=True) + self._manifest_path.write_text( + json.dumps(manifest, indent=1), encoding="utf-8") + + @staticmethod + def _file_digest(path: Path) -> str: + return hashlib.md5(path.read_bytes()).hexdigest() + + # ── lazy stage construction (runs on the worker thread) ── + + def _get_writer(self) -> ObsidianVaultWriter: + import importlib.util + + if importlib.util.find_spec("slugify") is None: + raise VectorStoreError( + "python-slugify not installed — run: pip install 'thicket[ingest]'" + ) + if self._writer is None: + self._writer = ObsidianVaultWriter( + self.config.vault_dir, output_dirname=self.config.notes_dir + ) + return self._writer + + def _get_engine(self) -> EmbeddingEngine: + if self._engine is None: + self.cb.log( + f"Loading local embedding model '{self.config.embed_model}' " + f"(first run downloads it)..." + ) + self._engine = EmbeddingEngine(self.config.embed_model) + self._engine.load() + self.cb.log(f"Embedding model ready ({self._engine.dim} dimensions).") + return self._engine + + def _get_store(self): + if self._store is None: + spec = TARGETS.get(self.config.target) + if spec is None: + known = ", ".join(sorted(TARGETS)) + raise VectorStoreError( + f"unknown vector target '{self.config.target}' — known: {known}" + ) + store = create_store( + self.config.target, + collection=self.config.collection, + dim=model_dim(self.config.embed_model), + host=self.config.qdrant_host, + port=self.config.qdrant_port, + data_dir=self._store_data_dir(), + log=self.cb.log, + ) + store.set_embedder(self._get_engine()) + store.ensure_collection() + self._store = store + return self._store + + def _store_data_dir(self) -> Path: + """Embedded/file targets keep data under the vault so the whole + knowledge tree stays one portable directory.""" + return self.config.vault_dir / ".thicket" / self.config.target + + def _get_graph(self): + if self._graph is None: + self.cb.log( + f"Initializing LightRAG graph at " + f"'{self.config.vault_dir / '.lightrag'}' " + f"(LLM: {self.config.ollama_llm}, embed: {self.config.ollama_embed})..." + ) + self._graph = create_graph( + self.config.graph_engine, + working_dir=self.config.vault_dir, + llm_model=self.config.ollama_llm, + embed_model=self.config.ollama_embed, + log=self.cb.log, + ) + return self._graph + + def _get_archiver(self) -> MinioArchiver: + if self._archiver is None: + self._archiver = MinioArchiver(log=self.cb.log) + return self._archiver + + # ── stage runners ── + + def _stage_minio(self, filepath: Path, title: str, content: str, + ctx: dict) -> str: + uri = self._get_archiver().archive(filepath, self.config.input_dir) + ctx["source_uri"] = uri + self.cb.log(f" └─ Archived to MinIO: {uri}") + return f"s3: {uri}" + + def _stage_fs_archive(self, filepath: Path, title: str, content: str, + ctx: dict) -> str: + """Move the source out of the incoming tree into the archive + directory and bzip2 it — sources are never deleted.""" + archive_dir = self.config.archive_dir or default_archive( + self.config.input_dir) + archive_dir.mkdir(parents=True, exist_ok=True) + target = archive_dir / filepath.name + if target.exists(): + digest = doc_key(title, str(filepath))[:6] + target = archive_dir / f"{filepath.stem}-{digest}{filepath.suffix}" + shutil.move(str(filepath), target) + compressed = _bz2_compress(target) + self.cb.log(f" └─ Archived + bz2: {compressed.relative_to(compressed.parents[1])}") + return f"bz2: {compressed.name}" + + def _stage_vault(self, filepath: Path, title: str, content: str, + ctx: dict) -> str: + note = self._get_writer().write( + title, content, filepath, source_uri=ctx.get("source_uri")) + ctx["rel_path"] = str(note.relative_to(self.config.vault_dir)) + self.cb.log(f" └─ Vault note written: [[{note.stem}]]") + return f"note: {note.name}" + + def _stage_index(self, filepath: Path, title: str, content: str, + ctx: dict) -> str: + rel_path = ctx.get("rel_path") or self._projected_rel_path(title) + chunks = self._chunker.chunk( + title, content, + source_path=str(filepath.relative_to(self.config.input_dir))) + count = self._get_store().replace_document(title, rel_path, chunks) + self.cb.log( + f" └─ Vector store: indexed {count} chunk(s) into " + f"'{self.config.collection}'." + ) + return f"{count} chunks indexed" + + def _stage_graph(self, filepath: Path, title: str, content: str, + ctx: dict) -> str: + self._get_graph().ingest_document(title, content) + self.cb.log(" └─ LightRAG graph updated.") + return "graph updated" + + def _projected_rel_path(self, title: str) -> str: + """Canonical note path for the vector payload when the vault + stage is disabled — identical location to the writer's plain + slug (collision suffixes only exist once notes are written).""" + from slugify import slugify + slug = slugify(title) or "untitled" + return f"{self.config.notes_dir}/{slug}.md" + + # ── main loop ── + + def run(self, files: list[Path] | None = None) -> tuple[int, int]: + """Process every supported document. Returns (ok, fail).""" + files = scan_files(self.config.input_dir) if files is None else files + + self.cb.log(f"Found {len(files)} eligible document(s) in " + f"'{self.config.input_dir}'") + for f in files: + self.cb.file_status(str(f), "QUEUED") + + outcomes = [self._process_one(f) for f in self._itinerary(files)] + ok = sum(outcomes) + if self._graph is not None: + try: + self._graph.finalize() # batch engines build once here + except Exception as e: # noqa: BLE001 — report, keep counts + self.cb.log(f" └─ ERROR finalizing graph: {e}") + return ok, len(outcomes) - ok + + def _itinerary(self, files: list[Path]): + """Yield files in order, honouring the stop flag between files + and emitting progress as we go.""" + total = len(files) + for current, filepath in enumerate(files, start=1): + if self._stop: + self.cb.log("STOP: exiting queue before next file.") + return + self.cb.progress(filepath.name, current, total) + yield filepath + + def _process_one(self, filepath: Path) -> bool: + """One document through the gates + extract + the stage table.""" + try: + skip_reason = self._gate(filepath) + if skip_reason is not None: + self.cb.file_status(str(filepath), "SKIP") + self.cb.file_detail(str(filepath), skip_reason) + self.cb.log(f" └─ Skipping {filepath.name}: {skip_reason}") + return True # deliberate non-processing — not a failure + + self.cb.file_status(str(filepath), "EXTRACT") + self.cb.file_detail(str(filepath), "extracting text") + title, content = DocumentExtractor.extract(filepath) + + if not content.strip(): + self.cb.log(f" └─ Skipping empty file: {filepath.name}") + self.cb.file_status(str(filepath), "SKIP") + self.cb.file_detail(str(filepath), "no extractable text") + return True # nothing to ingest — not a failure + + ctx: dict = {} + for status, enabled, runner in self._stages: + if not enabled: + continue + self.cb.file_status(str(filepath), status) + self.cb.file_detail(str(filepath), runner(filepath, title, content, ctx)) + + self.cb.file_status(str(filepath), "DONE") + if self.config.skip_unchanged: + manifest = self._read_manifest() + manifest[str(filepath)] = self._file_digest(filepath) + self._write_manifest(manifest) + return True + + except Exception as e: # noqa: BLE001 — per-file isolation is the contract + self._fail(filepath, e) + return False + + def _gate(self, filepath: Path) -> str | None: + """Pre-extraction gates: oversized files and unchanged re-runs. + Returns the skip reason, or None to proceed.""" + if self.config.max_mb: + size_mb = filepath.stat().st_size / 1_048_576 + if size_mb > self.config.max_mb: + return f"{size_mb:.1f} MB exceeds {self.config.max_mb} MB limit" + if self.config.skip_unchanged: + manifest = self._read_manifest() + if manifest.get(str(filepath)) == self._file_digest(filepath): + return "unchanged since last ingest" + return None + + def _fail(self, filepath: Path, error: Exception) -> None: + msg = f" └─ ERROR processing {filepath.name}: {error}" + print(msg, file=sys.stderr) + self.cb.log(msg) + self.cb.file_status(str(filepath), "ERROR") + self.cb.file_detail(str(filepath), str(error)[:120]) + + +# ────────────────────────────────────────────────────────────────── +# HEADLESS ENTRY (no Qt) +# ────────────────────────────────────────────────────────────────── + +def headless_ingest(input_dir: Path, vault_dir: Path, + qdrant_host: str = "localhost", qdrant_port: int = 6333, + target: str = "qdrant", collection: str = "second_brain", + embed_model: str = DEFAULT_EMBED_MODEL, + chunk_size: int = 400, overlap: int = 50, + use_vault: bool = True, use_qdrant: bool = True, + use_lightrag: bool = False, use_minio: bool = False, + graph_engine: str = "lightrag", + notes_dir: str = VAULT_OUTPUT_DIRNAME, + skip_unchanged: bool = False, + use_fs_archive: bool = False, + archive_dir: Path | None = None, max_mb: int = 0, + ollama_llm: str = "llama3", + ollama_embed: str = "nomic-embed-text") -> tuple[int, int]: + """CLI batch ingest — the same core path the GUI worker drives.""" + config = IngestConfig( + input_dir=input_dir, vault_dir=vault_dir, + qdrant_host=qdrant_host, qdrant_port=qdrant_port, + target=target, collection=collection, embed_model=embed_model, + chunk_size=chunk_size, overlap=overlap, + use_vault=use_vault, use_qdrant=use_qdrant, use_lightrag=use_lightrag, + use_minio=use_minio, graph_engine=graph_engine, + notes_dir=notes_dir, skip_unchanged=skip_unchanged, + use_fs_archive=use_fs_archive, archive_dir=archive_dir, + max_mb=max_mb, + ollama_llm=ollama_llm, ollama_embed=ollama_embed, + ) + callbacks = PipelineCallbacks( + log=lambda msg: print(f"> {msg}"), + file_status=lambda _p, _s: None, + file_detail=lambda _p, d: print(f" {d}"), + progress=lambda n, c, t: print(f"\n[{c}/{t}] {n}"), + ) + pipeline = IngestPipeline(config, callbacks) + try: + return pipeline.run() + except KeyboardInterrupt: + # POSIX: exit code 130 = 128 + SIGINT; report what completed. + print("\nInterrupted — counts reflect files finished before SIGINT.") + raise diff --git a/thicket/pipeline_worker.py b/thicket/pipeline_worker.py new file mode 100644 index 0000000..3b822e7 --- /dev/null +++ b/thicket/pipeline_worker.py @@ -0,0 +1,183 @@ +"""Qt workers — threads that keep the UI responsive. + + * ``ProbeWorker`` — runs env_probe.probe_environment() off-thread. + * ``PipelineWorker`` — wraps IngestPipeline, forwarding progress as + Qt signals (same signal contract as + OpenTranscode's EncoderWorker). + * ``SearchWorker`` — embeds a query and searches Qdrant off-thread. +""" + +from __future__ import annotations + +from pathlib import Path + +from PySide6.QtCore import QThread, Signal + +from .embedder import DEFAULT_EMBED_MODEL, EmbeddingEngine, model_dim +from .env_probe import probe_environment +from .pipeline_core import IngestConfig, IngestPipeline, PipelineCallbacks + + +class ProbeWorker(QThread): + """Background environment probe — service pings + module checks.""" + + log_msg = Signal(str) + probe_done = Signal(object) # EnvProbe + + def __init__(self, qdrant_host: str = "localhost", qdrant_port: int = 6333, + parent=None): + super().__init__(parent) + self._host = qdrant_host + self._port = qdrant_port + + def run(self): + self.log_msg.emit("Probing environment (modules, Qdrant, Ollama)...") + env = probe_environment(qdrant_host=self._host, qdrant_port=self._port) + self.probe_done.emit(env) + + +class PipelineWorker(QThread): + """Drives IngestPipeline on a worker thread. + + Signal contract mirrors OpenTranscode's EncoderWorker: + log_msg(str) — human-readable progress lines + file_status(str, str) — (absolute path, stage status) + file_detail(str, str) — (filename, detail text for table row) + progress_msg(str, int, int) — (filename, current, total) + finished_queue(int, int) — (ok count, fail count) + """ + + log_msg = Signal(str) + file_status = Signal(str, str) + file_detail = Signal(str, str) + progress_msg = Signal(str, int, int) + finished_queue = Signal(int, int) + + def __init__(self, config: IngestConfig, parent=None): + super().__init__(parent) + self.config = config + self._pipeline: IngestPipeline | None = None + + def stop(self) -> None: + """Cooperative stop — the pipeline exits after the current file.""" + if self._pipeline is not None: + self._pipeline.request_stop() + + def run(self): + callbacks = PipelineCallbacks( + log=self.log_msg.emit, + file_status=self.file_status.emit, + file_detail=self.file_detail.emit, + progress=self.progress_msg.emit, + ) + self._pipeline = IngestPipeline(self.config, callbacks) + try: + ok, fail = self._pipeline.run() + except Exception as e: # noqa: BLE001 — fatal pipeline errors + self.log_msg.emit(f"FATAL: pipeline aborted — {e}") + ok, fail = 0, 0 + self.finished_queue.emit(ok, fail) + + +class SearchWorker(QThread): + """Semantic retrieval against the ingested collection of the + selected vector target.""" + + log_msg = Signal(str) + search_done = Signal(int) # number of hits returned + + def __init__(self, query: str, vault_dir: Path, target: str, + host: str = "localhost", port: int = 6333, + collection: str = "second_brain", + embed_model: str = DEFAULT_EMBED_MODEL, top_k: int = 5, + parent=None): + super().__init__(parent) + self._query = query + self._vault_dir = vault_dir + self._target = target + self._host = host + self._port = port + self._collection = collection + self._embed_model = embed_model + self._top_k = top_k + + def run(self): + try: + hits = self._search() + except Exception as e: # noqa: BLE001 — report any failure to the log + self.log_msg.emit(f"SEARCH ERROR: {e}") + self.search_done.emit(-1) + return + self._report(hits) + self.search_done.emit(len(hits)) + + def _search(self) -> list[dict]: + """Embed the query and search the collection (worker thread).""" + from .vector_stores import create_store + + engine = EmbeddingEngine(self._embed_model) + self.log_msg.emit(f"Embedding query with '{self._embed_model}'...") + store = create_store( + self._target, + collection=self._collection, + dim=model_dim(self._embed_model), + host=self._host, port=self._port, + data_dir=self._vault_dir / ".thicket" / self._target, + ) + try: + store.ensure_collection() + return store.search(engine.embed_query(self._query), + limit=self._top_k) + finally: + store.close() + + def _report(self, hits: list[dict]) -> None: + """Emit ranked results to the log, best first.""" + if not hits: + self.log_msg.emit( + f"No matches in '{self._collection}' — is anything ingested?" + ) + return + for rank, hit in enumerate(hits, start=1): + payload = hit["payload"] + snippet = (payload.get("content") or "")[:160].replace("\n", " ") + tags = payload.get("chunk_kind", "prose") + if payload.get("lang"): + tags += f":{payload['lang']}" + if payload.get("source_path"): + tags += f" src={payload['source_path']}" + self.log_msg.emit( + f"[{rank}] {hit['score']:.3f} " + f"{payload.get('document_title', '?')} " + f"§ {payload.get('section_header', '?')} ({tags})" + ) + self.log_msg.emit(f" {snippet}...") + + +class AskWorker(QThread): + """Natural-language SQL over the corpus (Vanna 2 + Ollama).""" + + log_msg = Signal(str) + ask_done = Signal(bool) # success + + def __init__(self, question: str, target: str, collection: str, + llm_model: str, parent=None): + super().__init__(parent) + self._question = question + self._target = target + self._collection = collection + self._llm_model = llm_model + + def run(self): + from .ask_vanna import ask + + try: + answer = ask(self._question, target=self._target, + collection=self._collection, llm_model=self._llm_model, + log=self.log_msg.emit) + except Exception as e: # noqa: BLE001 — report, never crash the UI + self.log_msg.emit(f"ASK ERROR: {e}") + self.ask_done.emit(False) + return + self.log_msg.emit(f"ANSWER: {answer}") + self.ask_done.emit(True) diff --git a/thicket/qdrant_store.py b/thicket/qdrant_store.py new file mode 100644 index 0000000..e612084 --- /dev/null +++ b/thicket/qdrant_store.py @@ -0,0 +1,197 @@ +"""Qdrant vector store — the service-backed target in the registry. + +Invariants: + + * ``replace_document`` deletes every existing point for the document + (filter on ``doc_key``) before upserting, so re-ingesting an edited + or shrunken source replaces its vectors exactly — the point count + after N re-ingests equals the count after the first. + * Upserts travel in batches of 64 — one document never arrives as a + single unbounded request. + * Point IDs are deterministic (MD5-UUID of doc_key|index), making + re-ingestion an overwrite by construction. + * A model/collection dimension mismatch is a hard error with an + actionable message; vectors are never silently mixed. + +Step-down chain: ``search`` uses ``query_points`` (qdrant-client >= 1.10) +and steps down to ``search`` on older clients. +""" + +from __future__ import annotations + +import hashlib +import uuid +from collections.abc import Callable + +from .chunker import Chunk +from .vector_stores import VectorStoreError, _payload, _snippet_payload, doc_key + +UPSERT_BATCH = 64 + + +def _point_id(key: str, idx: int) -> str: + raw = f"{key}|{idx}".encode() + return str(uuid.UUID(bytes=hashlib.md5(raw).digest())) + + +class QdrantStore: + """Handles embedding indexing into a Qdrant collection. Embeddings + are computed by the attached EmbeddingEngine; the store never + embeds on its own.""" + + def __init__(self, host: str, port: int, collection: str, dim: int, + log: Callable[[str], None] = lambda _msg: None): + self.host = host + self.port = port + self.collection = collection + self.dim = dim + self.model_hint = "selected model" + self._engine = None + self._log = log + self._client = None + + def client(self): + """Lazy client — imports qdrant_client on first use so the GUI + never needs it installed to start.""" + if self._client is None: + try: + from qdrant_client import QdrantClient + except ImportError as e: + raise VectorStoreError( + "qdrant-client not installed — run: pip install 'thicket[ingest]'" + ) from e + self._client = QdrantClient( + url=f"http://{self.host}:{self.port}", timeout=10, + check_compatibility=False, # API gaps handled by step-down + ) + return self._client + + def ensure_collection(self) -> None: + """Create the collection when missing; verify dimensions when it + exists.""" + from qdrant_client.models import Distance, VectorParams + + client = self.client() + try: + names = [c.name for c in client.get_collections().collections] + except Exception as e: + raise VectorStoreError( + f"cannot reach Qdrant at {self.host}:{self.port} — {e}" + ) from e + + if self.collection not in names: + self._log(f"Creating collection '{self.collection}' (dim={self.dim})...") + client.create_collection( + collection_name=self.collection, + vectors_config=VectorParams(size=self.dim, distance=Distance.COSINE), + ) + self._index_payload(client) + return + + self._index_payload(client) + + existing_dim = self._collection_dim(client) + if isinstance(existing_dim, int) and self.dim and existing_dim != self.dim: + raise VectorStoreError( + f"collection '{self.collection}' has {existing_dim}-dim vectors " + f"but '{self.model_hint}' produces {self.dim} — pick the " + f"matching embedding model or a new collection name" + ) + + def _index_payload(self, client) -> None: + """Keyword index on doc_key — every re-ingest deletes by that + filter, and unindexed payload filters scan the whole collection.""" + from qdrant_client.models import PayloadSchemaType + + client.create_payload_index( + collection_name=self.collection, + field_name="doc_key", + field_schema=PayloadSchemaType.KEYWORD, + ) + + def _collection_dim(self, client) -> int | None: + """Vector size of the existing collection; None when the server + does not report it.""" + vectors = client.get_collection(self.collection).config.params.vectors + size = getattr(vectors, "size", None) + return size if isinstance(size, int) else None + + def replace_document(self, title: str, obsidian_rel_path: str, + chunks: list[Chunk]) -> int: + """Index every chunk of a document, replacing any previous + version. Returns the number of points written.""" + from qdrant_client.models import FieldCondition, Filter, MatchValue, PointStruct + + if not chunks: + return 0 + client = self.client() + key = doc_key(title, obsidian_rel_path) + + # Exact replacement: drop the previous version before writing, + # so a shrunken source leaves no stale tail chunks behind. The + # should-clause also matches pre-1.2 points (which lack + # doc_key) via their title+path — upgrading installations clean + # themselves on first re-ingest. + client.delete( + collection_name=self.collection, + points_selector=Filter(should=[ + FieldCondition(key="doc_key", match=MatchValue(value=key)), + Filter(must=[ + FieldCondition(key="document_title", + match=MatchValue(value=title)), + FieldCondition(key="obsidian_path", + match=MatchValue(value=obsidian_rel_path)), + ]), + ]), + ) + + vectors = self._engine.embed([c.contextual_text for c in chunks]) + points = [ + PointStruct( + id=_point_id(key, idx), + vector=vector, + payload=_payload(title, obsidian_rel_path, chunk, idx), + ) + for idx, (chunk, vector) in enumerate(zip(chunks, vectors)) + ] + + batches = (points[i:i + UPSERT_BATCH] + for i in range(0, len(points), UPSERT_BATCH)) + for batch in batches: + client.upsert(collection_name=self.collection, points=batch) + return len(points) + + def set_embedder(self, engine) -> None: + """Attach the EmbeddingEngine; records its model name for error + messages and adopts its dimension when none was given.""" + self._engine = engine + self.model_hint = engine.model_name + if engine.dim and not self.dim: + self.dim = engine.dim + + def close(self) -> None: + if self._client is not None: + self._client.close() + self._client = None + + def search(self, vector: list[float], limit: int = 5) -> list[dict]: + """Semantic search. Returns [{score, payload}] best-first.""" + client = self.client() + try: + hits = client.query_points( + collection_name=self.collection, + query=vector, + limit=limit, + with_payload=True, + ).points + except AttributeError: + # Step down: qdrant-client < 1.10 exposes search() only. + hits = client.search( + collection_name=self.collection, + query_vector=vector, + limit=limit, + with_payload=True, + ) + return [{"score": h.score, + "payload": _snippet_payload(h.payload or {})} + for h in hits] diff --git a/thicket/ui_theme.py b/thicket/ui_theme.py new file mode 100644 index 0000000..8aa9314 --- /dev/null +++ b/thicket/ui_theme.py @@ -0,0 +1,383 @@ +"""MMD3 retro-futuristic console Qt stylesheet (QSS string). + +Same visual lineage as OpenTranscode's ui_theme: brushed aluminum +panels, amber/green LED displays, beveled metallic group boxes, +modernized with rounded corners, subtle glow, and glassmorphism +hints. Extended for Thicket with the document-queue table, spin +boxes, and the teal auxiliary buttons. Pure string constant — no +imports at all. +""" + +MMD3_QSS = """ +/* ── Global ── */ +QMainWindow, QWidget#central { + background-color: #1a1a1e; +} + +/* ── Group Boxes — brushed aluminum panels ── */ +QGroupBox { + font-family: 'Segoe UI', 'Ubuntu', sans-serif; + font-size: 10px; + font-weight: bold; + color: #8a8a8a; + border: 1px solid #3a3a40; + border-radius: 8px; + margin-top: 14px; + padding: 14px 10px 10px 10px; + background: qlineargradient(x1:0, y1:0, x2:0, y2:1, + stop:0 #2c2c32, stop:0.5 #27272c, stop:1 #222228); +} +QGroupBox::title { + subcontrol-origin: margin; + subcontrol-position: top left; + padding: 2px 10px; + color: #666; + background: qlineargradient(x1:0, y1:0, x2:0, y2:1, + stop:0 #2c2c32, stop:1 #222228); + border-radius: 4px; +} + +/* ── Labels ── */ +QLabel { + color: #999; + font-size: 10px; + font-family: 'Segoe UI', 'Ubuntu', sans-serif; +} + +/* ── Line Edits — recessed aluminum wells ── */ +QLineEdit { + background: qlineargradient(x1:0, y1:0, x2:0, y2:1, + stop:0 #18181c, stop:1 #141418); + border: 1px solid #333; + border-radius: 4px; + padding: 5px 8px; + color: #d4aa50; /* amber LED */ + font-family: 'Consolas', 'DejaVu Sans Mono', 'Ubuntu Mono', monospace; + font-size: 11px; + selection-background-color: #d4aa50; + selection-color: #000; +} +QLineEdit:focus { + border-color: #d4aa50; +} + +/* ── Combo Boxes ── */ +QComboBox { + background: qlineargradient(x1:0, y1:0, x2:0, y2:1, + stop:0 #1e1e24, stop:1 #1a1a20); + border: 1px solid #3a3a40; + border-radius: 4px; + padding: 4px 8px; + color: #c8c8c8; + font-family: 'Segoe UI', 'Ubuntu', sans-serif; + font-size: 11px; + min-height: 24px; +} +QComboBox:hover { + border-color: #555; +} +QComboBox:focus { + border-color: #d4aa50; +} +QComboBox::drop-down { + border: none; + width: 22px; +} +QComboBox::down-arrow { + image: none; + border-left: 4px solid transparent; + border-right: 4px solid transparent; + border-top: 6px solid #888; + margin-right: 6px; +} +QComboBox QAbstractItemView { + background: #1e1e24; + border: 1px solid #3a3a40; + border-radius: 4px; + color: #c8c8c8; + selection-background-color: #3a3a48; + selection-color: #d4aa50; + padding: 4px; +} +QComboBox item { + min-height: 22px; + padding: 2px 8px; +} + +/* ── Spin Boxes — LED readout wells ── */ +QSpinBox { + background: qlineargradient(x1:0, y1:0, x2:0, y2:1, + stop:0 #18181c, stop:1 #141418); + border: 1px solid #333; + border-radius: 4px; + padding: 4px 8px; + color: #d4aa50; + font-family: 'Consolas', 'DejaVu Sans Mono', 'Ubuntu Mono', monospace; + font-size: 11px; + min-height: 22px; + selection-background-color: #d4aa50; + selection-color: #000; +} +QSpinBox:focus { + border-color: #d4aa50; +} +QSpinBox::up-button, QSpinBox::down-button { + background: #2c2c32; + border: none; + width: 16px; +} +QSpinBox::up-button:hover, QSpinBox::down-button:hover { + background: #3a3a42; +} +QSpinBox::up-arrow { + border-left: 4px solid transparent; + border-right: 4px solid transparent; + border-bottom: 5px solid #888; + width: 0; height: 0; +} +QSpinBox::down-arrow { + border-left: 4px solid transparent; + border-right: 4px solid transparent; + border-top: 5px solid #888; + width: 0; height: 0; +} + +/* ── Buttons — beveled metallic (MMD3 transport style) ── */ +QPushButton { + background: qlineargradient(x1:0, y1:0, x2:0, y2:1, + stop:0 #404048, stop:0.15 #38383f, + stop:0.85 #2e2e35, stop:1 #28282e); + border: 1px solid #4a4a52; + border-bottom-color: #1a1a1e; + border-radius: 5px; + padding: 6px 16px; + color: #d0d0d0; + font-family: 'Segoe UI', 'Ubuntu', sans-serif; + font-size: 11px; + font-weight: bold; +} +QPushButton:hover { + background: qlineargradient(x1:0, y1:0, x2:0, y2:1, + stop:0 #4a4a54, stop:0.15 #424248, + stop:0.85 #363640, stop:1 #303038); + border-color: #5a5a64; + color: #fff; +} +QPushButton:pressed { + background: qlineargradient(x1:0, y1:0, x2:0, y2:1, + stop:0 #28282e, stop:1 #3a3a42); + border-bottom-color: #4a4a52; + border-top-color: #1a1a1e; +} +QPushButton:disabled { + background: #222228; + border-color: #2a2a30; + color: #555; +} + +/* Primary action button — amber glow */ +QPushButton#btnRun { + background: qlineargradient(x1:0, y1:0, x2:0, y2:1, + stop:0 #3a3428, stop:0.15 #332e22, + stop:0.85 #2a261c, stop:1 #221e16); + border: 1px solid #5a4a30; + border-bottom-color: #1a1608; + color: #d4aa50; + font-size: 13px; + letter-spacing: 2px; +} +QPushButton#btnRun:hover { + background: qlineargradient(x1:0, y1:0, x2:0, y2:1, + stop:0 #4a4030, stop:0.15 #423828, + stop:0.85 #3a3020, stop:1 #322a1a); + border-color: #d4aa50; + color: #f0d080; +} +QPushButton#btnRun:disabled { + background: #22201a; + border-color: #2a2820; + color: #5a4a30; +} + +/* Stop button — red danger */ +QPushButton#btnStop { + background: qlineargradient(x1:0, y1:0, x2:0, y2:1, + stop:0 #3a2222, stop:0.15 #321c1c, + stop:0.85 #2a1616, stop:1 #221010); + border: 1px solid #5a3030; + border-bottom-color: #1a0808; + color: #e05050; + font-size: 13px; + letter-spacing: 2px; +} +QPushButton#btnStop:hover { + border-color: #e05050; + color: #ff7070; +} +QPushButton#btnStop:disabled { + background: #221a1a; + border-color: #2a2020; + color: #5a3030; +} + +/* Auxiliary buttons — muted teal (scan / search) */ +QPushButton#btnTeal { + background: qlineargradient(x1:0, y1:0, x2:0, y2:1, + stop:0 #1e2e2e, stop:0.15 #1a2a2a, + stop:0.85 #162424, stop:1 #121e1e); + border: 1px solid #2a5050; + border-bottom-color: #0e1818; + color: #50b0b0; + font-size: 10px; + letter-spacing: 1px; +} +QPushButton#btnTeal:hover { + border-color: #50b0b0; + color: #70d0d0; +} +QPushButton#btnTeal:disabled { + background: #1a1e1e; + border-color: #222828; + color: #304040; +} + +/* Browse buttons — small, subdued */ +QPushButton#btnBrowse { + font-size: 9px; + padding: 4px 10px; + letter-spacing: 1px; +} + +/* ── Check Boxes ── */ +QCheckBox { + color: #999; + font-size: 10px; + spacing: 8px; + font-family: 'Segoe UI', 'Ubuntu', sans-serif; +} +QCheckBox::indicator { + width: 16px; + height: 16px; + border-radius: 3px; + border: 1px solid #444; + background: #1a1a1e; +} +QCheckBox::indicator:checked { + background: #d4aa50; + border-color: #b8903a; +} +QCheckBox:disabled { + color: #444; +} +QCheckBox#dangerCheck { + color: #c05050; + font-weight: bold; +} +QCheckBox#dangerCheck::indicator:checked { + background: #c04040; + border-color: #a03030; +} + +/* ── Document Queue — LED matrix table ── */ +QTableWidget#queueTable { + background: #0a0a0c; + alternate-background-color: #101014; + gridline-color: #1a1a20; + border: 2px solid #1e1e24; + border-radius: 6px; + color: #c8c8c8; + font-family: 'Consolas', 'DejaVu Sans Mono', 'Ubuntu Mono', monospace; + font-size: 10px; + selection-background-color: #3a3a48; + selection-color: #d4aa50; +} +QTableWidget#queueTable::item { + padding: 2px 6px; +} +QTableWidget#queueTable QHeaderView { + background: transparent; +} +QTableWidget#queueTable QHeaderView::section { + background: qlineargradient(x1:0, y1:0, x2:0, y2:1, + stop:0 #2c2c32, stop:1 #222228); + color: #8a8a8a; + border: none; + border-right: 1px solid #1a1a20; + border-bottom: 1px solid #3a3a40; + padding: 4px 8px; + font-size: 9px; + font-family: 'Segoe UI', 'Ubuntu', sans-serif; + font-weight: bold; +} + +/* ── Text Edit (log) — LED terminal display ── */ +QTextEdit#logBox { + background: #0a0a0c; + border: 2px solid #1e1e24; + border-radius: 6px; + color: #40d060; /* green phosphor LED */ + font-family: 'Consolas', 'DejaVu Sans Mono', 'Ubuntu Mono', monospace; + font-size: 11px; + padding: 8px; +} + +/* ── Status Bar — LED readout strip ── */ +QStatusBar { + background: #0e0e12; + border-top: 1px solid #2a2a30; + font-family: 'Consolas', 'DejaVu Sans Mono', 'Ubuntu Mono', monospace; + font-size: 10px; + color: #d4aa50; + padding: 2px 8px; +} +QStatusBar QLabel { + color: #d4aa50; + font-family: 'Consolas', 'DejaVu Sans Mono', 'Ubuntu Mono', monospace; + font-size: 10px; +} + +/* ── Tooltips ── */ +QToolTip { + background: #2a2a30; + color: #c8c8c8; + border: 1px solid #444; + border-radius: 4px; + padding: 6px; + font-size: 10px; +} + +/* ── Scrollbars — thin, dark ── */ +QScrollBar:vertical { + background: #141418; + width: 10px; + border-radius: 5px; + margin: 0; +} +QScrollBar::handle:vertical { + background: #3a3a42; + border-radius: 5px; + min-height: 30px; +} +QScrollBar::handle:vertical:hover { + background: #4a4a54; +} +QScrollBar::add-line:vertical, QScrollBar::sub-line:vertical { + height: 0; +} +QScrollBar:horizontal { + background: #141418; + height: 10px; + border-radius: 5px; +} +QScrollBar::handle:horizontal { + background: #3a3a42; + border-radius: 5px; + min-width: 30px; +} +QScrollBar::handle:horizontal:hover { + background: #4a4a54; +} +QScrollBar::add-line:horizontal, QScrollBar::sub-line:horizontal { + width: 0; +} +""" diff --git a/thicket/ui_window.py b/thicket/ui_window.py new file mode 100644 index 0000000..982c637 --- /dev/null +++ b/thicket/ui_window.py @@ -0,0 +1,1184 @@ +"""ThicketWindow (QMainWindow) — the main GUI window. + +Retro-futuristic MMD3 console (same visual lineage as OpenTranscode): +brushed-aluminum group panels, amber LED path fields, green phosphor +log terminal, rotary knobs for the numeric controls, and a document +queue table that color-codes per-file stage as the ingest proceeds. + +Wires together: extractors (per-file text extraction), vault writer +(Obsidian notes), chunker + embedder + Qdrant store (vector index), +the optional LightRAG graph stage, the background environment probe, +and the retrieval strip. + +Also exposes ``launch_gui()``, the QApplication entry point invoked +by ``cli.main()`` and ``python -m thicket``. +""" + +from __future__ import annotations + +from pathlib import Path + +from PySide6.QtCore import Qt, Slot +from PySide6.QtGui import QColor, QFont, QPalette +from PySide6.QtWidgets import ( + QApplication, QCheckBox, QComboBox, QFileDialog, QGroupBox, QHBoxLayout, + QHeaderView, QLabel, QLineEdit, QMainWindow, QMessageBox, + QPushButton, QSpinBox, QStatusBar, QStyleFactory, QTableWidget, + QTableWidgetItem, QTextEdit, QVBoxLayout, QWidget, +) + +from .embedder import EMBEDDING_MODELS +from .env_probe import EnvProbe, PROBED_MODULES +from .extractors import SUPPORTED_EXTENSIONS, scan_files +from .layout import detect_layout, default_input, default_vault +from .pipeline_core import IngestConfig +from .pipeline_worker import PipelineWorker, ProbeWorker, SearchWorker +from .ui_theme import MMD3_QSS +from .widgets import RadioKnob + +# Stage status -> LED color (queue table STAGE column) +_STAGE_COLORS: dict[str, str] = { + "QUEUED": "#8a8a8a", + "EXTRACT": "#d4aa50", + "VAULT": "#50b0b0", + "INDEX": "#d4aa50", + "GRAPH": "#d4aa50", + "DONE": "#40d060", + "SKIP": "#666666", + "ERROR": "#e05050", +} + + +class ThicketWindow(QMainWindow): + def __init__(self): + super().__init__() + self.setWindowTitle("Thicket — dcos.net") + self.resize(1150, 940) + self.worker: PipelineWorker | None = None + self.probe_worker: ProbeWorker | None = None + self.search_worker: SearchWorker | None = None + self.ask_worker = None + self.env: EnvProbe | None = None + self._row_for_path: dict[str, int] = {} + + self._apply_mmd3_theme() + self._build_ui() + + # Probe in the background — service pings + module checks can + # take seconds, and the UI must stay responsive meanwhile. + self._start_probe() + + # ── UI Construction ── + + def _build_ui(self): + central = QWidget() + central.setObjectName("central") + self.setCentralWidget(central) + root = QVBoxLayout(central) + root.setContentsMargins(10, 6, 10, 4) + root.setSpacing(4) + + # ── Header ── + header = QWidget() + header_lay = QVBoxLayout(header) + header_lay.setContentsMargins(0, 0, 0, 0) + header_lay.setSpacing(0) + + title = QLabel("Thicket") + title.setFont(QFont("Segoe UI", 22, QFont.Weight.Bold)) + title.setAlignment(Qt.AlignmentFlag.AlignCenter) + title.setStyleSheet("color: #d4aa50; letter-spacing: 4px;") + header_lay.addWidget(title) + + subtitle = QLabel("dcos.net // super-ingest · rag console") + subtitle.setFont(QFont("Consolas", 8)) + subtitle.setAlignment(Qt.AlignmentFlag.AlignCenter) + subtitle.setStyleSheet("color: #555; letter-spacing: 2px;") + header_lay.addWidget(subtitle) + + accent = QWidget() + accent.setFixedHeight(1) + accent.setStyleSheet("background: qlineargradient(x1:0, y1:0, x2:1, y2:0," + "stop:0 transparent, stop:0.15 #d4aa5044," + "stop:0.5 #d4aa5088, stop:0.85 #d4aa5044, stop:1 transparent);") + header_lay.addWidget(accent) + + root.addWidget(header) + + # ── Paths ── + # IN is the incoming tree; VAULT is the notes root (and the data + # root for embedded targets); OUT resolves live per TARGET — + # the panel never assumes one destination. + path_grp = QGroupBox("Paths") + path_lay = QVBoxLayout(path_grp) + path_lay.setSpacing(2) + path_lay.setContentsMargins(10, 14, 10, 8) + + # Layout-aware defaults: /mnt/AI corpus flow when present, + # home-directory defaults otherwise. + self.in_path_edit = QLineEdit(str(default_input())) + self.vault_path_edit = QLineEdit(str(default_vault())) + for label_text, line_edit, tooltip in [ + ("IN:", self.in_path_edit, + "Incoming tree — scanned recursively for supported documents.\n" + "Accepts ~ and $VAR references; resolved at ingest time."), + ("VAULT:", self.vault_path_edit, ""), # tooltip set per-target + ]: + row = QHBoxLayout() + row.setSpacing(6) + lbl = QLabel(label_text) + lbl.setFixedWidth(42) + lbl.setStyleSheet( + "color: #d4aa50; font-family: 'Consolas', monospace; " + "font-weight: bold; font-size: 10px;") + row.addWidget(lbl) + line_edit.setToolTip(tooltip) + row.addWidget(line_edit, 1) + btn_browse = QPushButton("...") + btn_browse.setObjectName("btnBrowse") + btn_browse.setFixedSize(30, 22) + btn_browse.setToolTip("Browse") + btn_browse.clicked.connect( + lambda checked, le=line_edit: self._browse(le) + ) + row.addWidget(btn_browse) + path_lay.addLayout(row) + self.vault_lbl = lbl # reframed per target in _on_target_changed + + # Resolved destination — recomputed on every target or field edit. + out_row = QHBoxLayout() + out_row.setSpacing(6) + out_lbl = QLabel("OUT:") + out_lbl.setFixedWidth(42) + out_lbl.setStyleSheet( + "color: #d4aa50; font-family: 'Consolas', monospace; " + "font-weight: bold; font-size: 10px;") + out_row.addWidget(out_lbl) + self.out_label = QLabel("") + self.out_label.setToolTip( + "Resolved destination for the current TARGET — updates live\n" + "as the target, vault, notes dir, collection, or host change.") + self.out_label.setTextInteractionFlags( + Qt.TextInteractionFlag.TextSelectableByMouse) + self.out_label.setStyleSheet( + "color: #50b0b0; font-family: 'Consolas', 'DejaVu Sans Mono', " + "monospace; font-size: 10px;") + out_row.addWidget(self.out_label, 1) + path_lay.addLayout(out_row) + + root.addWidget(path_grp) + + # ── Pipeline stages ── + stage_grp = QGroupBox("Pipeline") + stage_lay = QVBoxLayout(stage_grp) + stage_lay.setSpacing(4) + stage_lay.setContentsMargins(10, 14, 10, 8) + + checks_row = QHBoxLayout() + checks_row.setSpacing(16) + + self.stage_graph_check = self._stage_check( + "Knowledge graph", + "Extract entities + relationships via the selected graph engine\n" + "(LightRAG: merged entity graph under /.lightrag;\n" + "Graphify: graph.json + interactive HTML under /.graphify).\n" + "Ollama LLM, one pass per document (LightRAG) or per batch (Graphify).", + checked=False, + ) + checks_row.addWidget(self.stage_graph_check) + + self.stage_minio_check = self._stage_check( + "MinIO archive", + "Upload the original source documents to an S3-compatible MinIO\n" + "bucket; the note records s3://bucket/key in its frontmatter.\n" + "Env: MINIO_ENDPOINT / ACCESS_KEY / SECRET_KEY / BUCKET.", + checked=False, + ) + checks_row.addWidget(self.stage_minio_check) + checks_row.addStretch(1) + stage_lay.addLayout(checks_row) + + # Target row — the vector destination and its connection model. + # Field construction is table-driven: (label, width, field). + from .vector_stores import TARGETS as _VECTOR_TARGETS + self._target_keys = ["obsidian"] + list(_VECTOR_TARGETS.keys()) + self._build_target_row_fields() + + target_row = QHBoxLayout() + target_row.setSpacing(6) + for label_text, width, field in ( + ("TARGET", 48, lambda: (self.target_combo, 2)), + ("COLLECTION", 76, lambda: (self.collection_edit, 0)), + ("HOST", 34, lambda: (self.svc_host_edit, 0)), + ("PORT", 30, lambda: (self.svc_port_spin, 0)), + ): + lbl = QLabel(label_text) + lbl.setFixedWidth(width) + lbl.setStyleSheet( + "color: #d4aa50; font-family: 'Consolas', monospace; " + "font-weight: bold; font-size: 10px;") + target_row.addWidget(lbl) + widget, stretch = field() + target_row.addWidget(widget, stretch) + target_row.addStretch(1) + stage_lay.addLayout(target_row) + + # Connection hint — one line, target-specific, always true. + self.target_hint_label = QLabel("") + self.target_hint_label.setStyleSheet( + "color: #666; font-family: 'Consolas', monospace; font-size: 9px;") + stage_lay.addWidget(self.target_hint_label) + + # Model selectors: the embedding model (retrieval-critical, + # tied to collection geometry) and the Ollama LLM (graph stage + # + ask interaction). + models_row = QHBoxLayout() + models_row.setSpacing(8) + + embed_lbl = QLabel("EMBED") + embed_lbl.setStyleSheet("color: #666; font-size: 7px; letter-spacing: 1px;") + models_row.addWidget(embed_lbl) + self.embed_combo = QComboBox() + self.embed_combo.addItems(list(EMBEDDING_MODELS.keys())) + self.embed_combo.setToolTip( + "Embedding model for the vector index.\n\n" + "jina-code — English + programming/config corpora; the right\n" + " choice for technical libraries (768-dim).\n" + "bge-base — strongest pure-prose alternative.\n" + "bge-small — light default for small libraries.\n\n" + "Vector dimension is fixed per collection: switching models\n" + "on an existing collection needs a new collection name." + ) + self.embed_combo.setFixedHeight(24) + models_row.addWidget(self.embed_combo, 2) + + llm_lbl = QLabel("OLLAMA LLM") + llm_lbl.setStyleSheet("color: #666; font-size: 7px; letter-spacing: 1px;") + models_row.addWidget(llm_lbl) + self.llm_combo = QComboBox() + self.llm_combo.setEditable(True) + self.llm_combo.addItem("llama3") + self.llm_combo.setToolTip( + "Ollama chat model driving LightRAG entity extraction.\n" + "Populated from your Ollama server at startup (most sensible\n" + "model pre-selected); editable.\n\n" + "For technical corpora the coder models (qwen3-coder,\n" + "qwen2.5-coder) are strong choices for entity extraction." + ) + models_row.addWidget(self.llm_combo, 2) + + engine_lbl = QLabel("GRAPH ENGINE") + engine_lbl.setStyleSheet("color: #666; font-size: 7px; letter-spacing: 1px;") + models_row.addWidget(engine_lbl) + self.graph_engine_combo = QComboBox() + self.graph_engine_combo.addItems(["LightRAG", "Graphify"]) + self.graph_engine_combo.setToolTip( + "Knowledge-graph engine for the graph stage.\n\n" + "LightRAG — merged entity/relationship graph, one LLM pass\n" + " per document.\n" + "Graphify — stages documents, one batch pass builds graph.json,\n" + " GRAPH_REPORT.md and an interactive graph.html." + ) + self.graph_engine_combo.setFixedHeight(24) + models_row.addWidget(self.graph_engine_combo) + models_row.addStretch(1) + + stage_lay.addLayout(models_row) + + # Options row: filters, notes layout, guards. + opt_row = QHBoxLayout() + opt_row.setSpacing(8) + opt_lbl = QLabel("FILTER") + opt_lbl.setFixedWidth(44) + opt_lbl.setStyleSheet("color: #666; font-size: 7px; letter-spacing: 1px;") + opt_row.addWidget(opt_lbl) + self.ext_edit = QLineEdit(", ".join(sorted(SUPPORTED_EXTENSIONS))) + self.ext_edit.setFixedHeight(22) + self.ext_edit.setToolTip("File extensions to process. Separate with commas.") + opt_row.addWidget(self.ext_edit, 1) + + notes_lbl = QLabel("NOTES DIR") + notes_lbl.setStyleSheet("color: #666; font-size: 7px; letter-spacing: 1px;") + opt_row.addWidget(notes_lbl) + self.notes_dir_edit = QLineEdit("Ingested_Brain") + self.notes_dir_edit.setFixedWidth(120) + self.notes_dir_edit.setFixedHeight(22) + self.notes_dir_edit.setToolTip( + "Vault subdirectory for generated notes.") + opt_row.addWidget(self.notes_dir_edit) + + maxmb_lbl = QLabel("MAX MB") + maxmb_lbl.setStyleSheet("color: #666; font-size: 7px; letter-spacing: 1px;") + opt_row.addWidget(maxmb_lbl) + self.max_mb_spin = QSpinBox() + self.max_mb_spin.setRange(0, 4096) + self.max_mb_spin.setValue(0) + self.max_mb_spin.setFixedWidth(64) + self.max_mb_spin.setToolTip( + "Skip files larger than this (0 = no limit). Guards against\n" + "accidental disk images / video in the input tree.") + opt_row.addWidget(self.max_mb_spin) + + self.skip_unchanged_check = QCheckBox("Skip unchanged") + self.skip_unchanged_check.setToolTip( + "Files whose content hash matches the last ingest are skipped\n" + "(manifest under /.thicket/). Makes re-running over a\n" + "big library cheap: only new or edited files pay for embedding.") + opt_row.addWidget(self.skip_unchanged_check) + stage_lay.addLayout(opt_row) + + guard_row = QHBoxLayout() + guard_row.setSpacing(16) + self.fs_archive_check = QCheckBox("Archive sources (bz2)") + self.fs_archive_check.setToolTip( + "After every stage verifies, move the source out of the incoming\n" + "tree into ingested-archive/ (sibling of IN) and bzip2 it.\n" + "Sources are never deleted — the archive always holds a\n" + "decompressible copy. Skipped files stay in place.") + guard_row.addWidget(self.fs_archive_check) + guard_row.addStretch(1) + stage_lay.addLayout(guard_row) + + root.addWidget(stage_grp) + + # ── Side panel: compact knobs ── + knobs_panel = QWidget() + knobs_panel.setFixedWidth(170) + knobs_lay = QVBoxLayout(knobs_panel) + knobs_lay.setContentsMargins(6, 8, 6, 8) + knobs_lay.setSpacing(6) + knobs_lay.setAlignment(Qt.AlignmentFlag.AlignTop | Qt.AlignmentFlag.AlignHCenter) + + self.chunk_knob = RadioKnob( + min_val=100, max_val=1000, default_val=400, + label="Chunk", unit="wds", + color=(212, 170, 80), + num_ticks=19, + tick_labels=["100", "400", "700", "1000"], + snap_ticks=True, + compact=True, + ) + self.chunk_knob.setToolTip( + "Chunk size in words for vector indexing.\n" + "Smaller = sharper retrieval, more points.\n" + "Larger = more context per hit, coarser match." + ) + knobs_lay.addWidget(self.chunk_knob, 0, Qt.AlignmentFlag.AlignHCenter) + + self.overlap_knob = RadioKnob( + min_val=0, max_val=200, default_val=50, + label="Overlap", + unit="wds", + color=(64, 208, 96), + num_ticks=21, + tick_labels=["0", "100", "200"], + snap_ticks=True, + compact=True, + ) + self.overlap_knob.setToolTip( + "Words shared between consecutive chunks.\n" + "Keeps sentences that straddle a boundary\n" + "retrievable from both sides." + ) + knobs_lay.addWidget(self.overlap_knob, 0, Qt.AlignmentFlag.AlignHCenter) + + self.topk_knob = RadioKnob( + min_val=1, max_val=20, default_val=5, + label="Top-K", + unit="hits", + color=(80, 176, 176), + num_ticks=20, + tick_labels=["1", "10", "20"], + snap_ticks=True, + compact=True, + ) + self.topk_knob.setToolTip("Results returned by the retrieval strip.") + knobs_lay.addWidget(self.topk_knob, 0, Qt.AlignmentFlag.AlignHCenter) + + # ── Document queue + log (left), knobs (right) ── + left_col = QVBoxLayout() + left_col.setSpacing(4) + + self.queue_table = QTableWidget(0, 5) + self.queue_table.setObjectName("queueTable") + self.queue_table.setHorizontalHeaderLabels( + ["FILE", "TYPE", "SIZE", "STAGE", "DETAIL"] + ) + self.queue_table.verticalHeader().setVisible(False) + self.queue_table.setAlternatingRowColors(True) + self.queue_table.setEditTriggers(QTableWidget.EditTrigger.NoEditTriggers) + self.queue_table.setSelectionBehavior(QTableWidget.SelectionBehavior.SelectRows) + self.queue_table.setWordWrap(False) + hdr = self.queue_table.horizontalHeader() + hdr.setSectionResizeMode(0, QHeaderView.ResizeMode.Interactive) + hdr.setSectionResizeMode(1, QHeaderView.ResizeMode.Fixed) + hdr.setSectionResizeMode(2, QHeaderView.ResizeMode.Fixed) + hdr.setSectionResizeMode(3, QHeaderView.ResizeMode.Fixed) + hdr.setSectionResizeMode(4, QHeaderView.ResizeMode.Stretch) + self.queue_table.setColumnWidth(0, 300) + self.queue_table.setColumnWidth(1, 46) + self.queue_table.setColumnWidth(2, 72) + self.queue_table.setColumnWidth(3, 66) + left_col.addWidget(self.queue_table, 2) + + self.log_box = QTextEdit() + self.log_box.setObjectName("logBox") + self.log_box.setReadOnly(True) + left_col.addWidget(self.log_box, 3) + + mid_split = QHBoxLayout() + mid_split.setSpacing(6) + mid_split.addLayout(left_col, 1) + mid_split.addWidget(knobs_panel) + root.addLayout(mid_split, 1) + + # ── Retrieval strip ── + query_grp = QGroupBox("Retrieval") + query_lay = QHBoxLayout(query_grp) + query_lay.setSpacing(8) + query_lay.setContentsMargins(10, 14, 10, 8) + query_lbl = QLabel("QUERY") + query_lbl.setFixedWidth(44) + query_lbl.setStyleSheet("color: #666; font-size: 7px; letter-spacing: 1px;") + query_lay.addWidget(query_lbl) + self.query_edit = QLineEdit() + self.query_edit.setPlaceholderText("semantic search across ingested chunks...") + self.query_edit.returnPressed.connect(self._search) + query_lay.addWidget(self.query_edit, 1) + self.btn_search = QPushButton(" ? SEARCH") + self.btn_search.setObjectName("btnTeal") + self.btn_search.setFixedHeight(28) + self.btn_search.setToolTip( + "Embed the query locally and search the Qdrant collection.\n" + "Results appear in the log with score, document, and section." + ) + self.btn_search.clicked.connect(self._search) + query_lay.addWidget(self.btn_search) + + self.btn_ask = QPushButton(" ▣ ASK (SQL)") + self.btn_ask.setObjectName("btnTeal") + self.btn_ask.setFixedHeight(28) + self.btn_ask.setToolTip( + "Natural-language SQL over the corpus via Vanna 2 + Ollama.\n" + "Needs a SQL-backed TARGET (pgvector or mariadb) with data\n" + "ingested. The question runs read-only SELECT queries." + ) + self.btn_ask.clicked.connect(self._ask) + query_lay.addWidget(self.btn_ask) + root.addWidget(query_grp) + + # ── Status Bar: LED readout ── + self.status = QStatusBar() + self.setStatusBar(self.status) + self.status_label = QLabel(" INITIALIZING...") + self.status_label.setStyleSheet( + "color: #d4aa50; font-family: 'Consolas', 'DejaVu Sans Mono', monospace; " + "font-size: 10px;" + ) + self.status.addWidget(self.status_label, 1) + + # ── Transport Buttons ── + btn_lay = QHBoxLayout() + btn_lay.setSpacing(8) + + self.btn_run = QPushButton(" > INGEST") + self.btn_run.setObjectName("btnRun") + self.btn_run.setFixedHeight(40) + self.btn_run.setEnabled(False) + self.btn_run.clicked.connect(self._start_ingest) + btn_lay.addWidget(self.btn_run) + + self.btn_stop = QPushButton(" [] STOP") + self.btn_stop.setObjectName("btnStop") + self.btn_stop.setFixedHeight(40) + self.btn_stop.setEnabled(False) + self.btn_stop.clicked.connect(self._stop_process) + btn_lay.addWidget(self.btn_stop) + + self.btn_scan = QPushButton(" ~~ SCAN QUEUE") + self.btn_scan.setObjectName("btnTeal") + self.btn_scan.setFixedHeight(40) + self.btn_scan.setToolTip( + "Preview: scan the input directory and populate the queue table\n" + "without ingesting anything." + ) + self.btn_scan.clicked.connect(self._scan) + btn_lay.addWidget(self.btn_scan) + + self.btn_about = QPushButton(" ? ABOUT") + self.btn_about.setObjectName("btnTeal") + self.btn_about.setFixedHeight(40) + self.btn_about.setToolTip("About Thicket and its components.") + self.btn_about.clicked.connect(self._show_about) + btn_lay.addWidget(self.btn_about) + root.addLayout(btn_lay) + + # ── Footer ── + footer = QWidget() + footer_lay = QHBoxLayout(footer) + footer_lay.setContentsMargins(6, 4, 6, 2) + footer_lay.setSpacing(0) + + link_lbl = QLabel( + 'Visit Homepage' + ) + link_lbl.setTextInteractionFlags(Qt.TextInteractionFlag.TextBrowserInteraction) + link_lbl.setOpenExternalLinks(True) + link_lbl.setStyleSheet("font-size: 8px;") + footer_lay.addWidget(link_lbl) + + footer_lay.addStretch() + + copy_lbl = QLabel( + 'AGPL-3.0 | Jeremy Anderson - dcos.net (c) 2026' + ) + copy_lbl.setTextInteractionFlags(Qt.TextInteractionFlag.TextBrowserInteraction) + copy_lbl.setOpenExternalLinks(True) + copy_lbl.setAlignment(Qt.AlignmentFlag.AlignRight) + copy_lbl.setStyleSheet("color: #555; font-size: 8px;") + footer_lay.addWidget(copy_lbl) + + root.addWidget(footer) + + # Destination resolution is live: target switches, vault, + # notes-dir, collection, host, and port edits all re-resolve. + for edit in (self.vault_path_edit, self.notes_dir_edit, + self.collection_edit, self.svc_host_edit): + edit.textChanged.connect(lambda _t: self._refresh_destination()) + self.svc_port_spin.valueChanged.connect( + lambda _v: self._refresh_destination()) + + # Initialize the target row state (hint line, port, status + # footer). Default destination: qdrant — an indexed brain out + # of the box; obsidian (notes-only) stays one click away. + self.target_combo.setCurrentIndex(self._target_keys.index("qdrant")) + self._on_target_changed(self.target_combo.currentIndex()) + + def _build_target_row_fields(self) -> None: + """Construct the four target-row fields; the row layout + composes them from a table.""" + self.target_combo = QComboBox() + self.target_combo.addItem("Obsidian vault (notes only)") + from .vector_stores import TARGETS + for spec in TARGETS.values(): + self.target_combo.addItem(spec.label) + self.target_combo.setToolTip( + "Vector store destination.\n\n" + "Service targets (qdrant, weaviate) use HOST/PORT.\n" + "Env-driven targets (pgvector, mariadb) read their\n" + " standard environment (see the hint line).\n" + "Embedded targets keep data under /.thicket/.\n\n" + "Every target shares one payload schema and the same\n" + "exact-replacement re-ingest semantics." + ) + self.target_combo.setFixedHeight(24) + self.target_combo.currentIndexChanged.connect(self._on_target_changed) + + self.collection_edit = QLineEdit("second_brain") + self.collection_edit.setFixedWidth(150) + self.collection_edit.setToolTip( + "Collection / table name shared by the target.\n" + "Vector dimension is fixed at creation — switching the\n" + "embedding model later needs a fresh collection name." + ) + self.svc_host_edit = QLineEdit("localhost") + self.svc_host_edit.setFixedWidth(110) + self.svc_host_edit.setToolTip( + "Service host — applies to service-backed targets only.") + self.svc_port_spin = QSpinBox() + self.svc_port_spin.setRange(1, 65535) + self.svc_port_spin.setValue(6333) + self.svc_port_spin.setFixedWidth(76) + self.svc_port_spin.setToolTip( + "Service port — applies to service-backed targets only.\n" + "qdrant: 6333, weaviate: 8080 (set automatically).") + + @staticmethod + def _stage_check(text: str, tooltip: str, checked: bool) -> QCheckBox: + check = QCheckBox(text) + check.setChecked(checked) + check.setToolTip(tooltip) + return check + + def _apply_mmd3_theme(self): + self.setStyle(QStyleFactory.create("Fusion")) + self.setStyleSheet(MMD3_QSS) + p = QPalette() + p.setColor(QPalette.ColorRole.Window, QColor(26, 26, 30)) + p.setColor(QPalette.ColorRole.WindowText, QColor(200, 200, 200)) + p.setColor(QPalette.ColorRole.Base, QColor(20, 20, 24)) + p.setColor(QPalette.ColorRole.AlternateBase, QColor(40, 40, 46)) + p.setColor(QPalette.ColorRole.ToolTipBase, QColor(30, 30, 36)) + p.setColor(QPalette.ColorRole.ToolTipText, QColor(200, 200, 200)) + p.setColor(QPalette.ColorRole.Text, QColor(200, 200, 200)) + p.setColor(QPalette.ColorRole.Button, QColor(40, 40, 46)) + p.setColor(QPalette.ColorRole.ButtonText, QColor(200, 200, 200)) + p.setColor(QPalette.ColorRole.Highlight, QColor(212, 170, 80)) + p.setColor(QPalette.ColorRole.HighlightedText, QColor(0, 0, 0)) + QApplication.instance().setPalette(p) + + # ── Slots / helpers ── + + @Slot() + def _browse(self, line_edit: QLineEdit): + anchor = str(detect_layout().root) if detect_layout() else "" + path = QFileDialog.getExistingDirectory( + self, "Select Directory", anchor) + if path: + line_edit.setText(path) + + def _log(self, msg: str): + # Guard against signals firing during __init__ before log_box + # exists; buffer so nothing is lost. + if not hasattr(self, "log_box") or self.log_box is None: + buffered = getattr(self, "_log_buffer", None) + if buffered is None: + buffered = self._log_buffer = [] + buffered.append(msg) + return + buffered = getattr(self, "_log_buffer", None) + if buffered: + for m in buffered: + self.log_box.append(f"> {m}") + self._log_buffer = [] + self.log_box.append(f"> {msg}") + sb = self.log_box.verticalScrollBar() + sb.setValue(sb.maximum()) + + def _parse_extensions(self) -> set[str]: + raw = self.ext_edit.text() + exts = set() + for part in raw.split(","): + part = part.strip().lower() + if not part: + continue + if not part.startswith("."): + part = "." + part + exts.add(part) + return exts or set(SUPPORTED_EXTENSIONS) + + # ── Environment probe ── + + def _start_probe(self): + host = self.svc_host_edit.text().strip() or "localhost" + self.probe_worker = ProbeWorker(qdrant_host=host, + qdrant_port=self.svc_port_spin.value()) + self.probe_worker.log_msg.connect(self._log) + self.probe_worker.probe_done.connect(self._on_probe_done) + self.probe_worker.start() + + @Slot(object) + def _on_probe_done(self, env: EnvProbe): + self.env = env + + self._log(f"Python {env.python_version}") + missing = [name for name, ok in env.modules.items() if not ok] + for name, role in PROBED_MODULES.items(): + ok = env.modules.get(name, False) + self._log(f" {name:14s} {'OK' if ok else 'MISSING':7s} {role}") + if missing: + self._log( + " HINT: pip install 'thicket[ingest]' (parsers+vectors) " + "and/or 'thicket[graph]' (LightRAG)." + ) + + if env.qdrant_up: + colls = ", ".join(env.qdrant_collections) or "(none yet)" + self._log(f"Qdrant: UP at {self.svc_host_edit.text()}:" + f"{self.svc_port_spin.value()} — collections: {colls}") + else: + self._log(f"Qdrant: DOWN ({env.qdrant_error or 'no response'}). " + f"Start it with: docker run -p 6333:6333 qdrant/qdrant") + + if env.ollama_up: + if env.ollama_models: + self.llm_combo.clear() + self.llm_combo.addItems(env.ollama_models) + picked = self._preferred_llm(env.ollama_models) + self.llm_combo.setCurrentText(picked) + self._log(f"Ollama: UP — graph LLM pre-selected: {picked}") + shown = env.ollama_models[:12] + extra = len(env.ollama_models) - len(shown) + models = ", ".join(shown) + (f" … (+{extra} more)" if extra > 0 else "") + self._log(f"Ollama models: {models or '(none pulled)'}") + else: + self._log(f"Ollama: DOWN ({env.ollama_error or 'no response'}) — " + f"LightRAG stage unavailable.") + + pg_state = ("UP" if env.pg_up + else f"DOWN ({env.pg_error or 'no response'}) — set PG* env") + self._log(f"Postgres: {pg_state}") + minio_state = ("UP" if env.minio_up + else f"DOWN ({env.minio_error or 'no response'}) — " + f"set MINIO_* env") + self._log(f"MinIO: {minio_state}") + + # AI filesystem layout — what Thicket adopted as defaults. + profile = detect_layout() + if profile: + self._log(f"layout: {profile.root} detected — " + f"IN={profile.corpus_cold}, vault={profile.corpus_hot}" + + (", books library present" if profile.books_present else "")) + else: + self._log("layout: /mnt/AI not found — home-directory defaults.") + + # Vector target availability — one line per registry entry. + from .vector_stores import TARGETS + for spec in TARGETS.values(): + state = "ready" if env.vector_ready(spec.key) else "not available" + self._log(f" vector target {spec.key:8s} {state}") + + # Vault notes are the always-on core stage; slugify is its one + # dependency. INGEST stays available (other stages validate at + # click time) but the warning is loud. + if not env.vault_ready: + self._log("WARNING: notes cannot be written " + "(python-slugify missing — pip install 'thicket[ingest]').") + + self.btn_run.setEnabled(True) + self.btn_run.setText(" > INGEST") + + qdrant_txt = "UP" if env.qdrant_up else "DOWN" + ollama_txt = "UP" if env.ollama_up else "DOWN" + fast_txt = "OK" if env.modules.get("fastembed") else "MISSING" + self._probe_summary = ( + f"Qdrant: {qdrant_txt} | Ollama: {ollama_txt} | fastembed: {fast_txt}" + ) + self._refresh_status_target() + + def _current_target(self) -> str: + idx = self.target_combo.currentIndex() + return self._target_keys[idx] if 0 <= idx < len(self._target_keys) else "qdrant" + + # Default port per service-backed target; env-driven connection + # hints for the rest. + _TARGET_PORTS = {"qdrant": 6333, "weaviate": 8080} + _ENV_HINTS = { + "pgvector": "env: PGHOST / PGPORT / PGUSER / PGPASSWORD / PGDATABASE (or PGDSN)", + "mariadb": ("env: MARIADB_HOST / MARIADB_PORT / MARIADB_USER / " + "MARIADB_PASSWORD / MARIADB_DATABASE"), + } + + # Embedded targets keep their index inside the vault tree. + _EMBEDDED_DATA = ("chroma", "lancedb", "faiss", "milvus", "duckdb", + "sqlitevec") + + def _refresh_destination(self) -> None: + """Resolve the true destination for the current TARGET from the + live field values — the Paths panel never assumes one output.""" + import os + + target = self._current_target() + vault = self.vault_path_edit.text().strip() or "" + notes = self.notes_dir_edit.text().strip() or "Ingested_Brain" + collection = self.collection_edit.text().strip() or "second_brain" + host = self.svc_host_edit.text().strip() or "localhost" + port = self.svc_port_spin.value() + notes_txt = f"notes: {vault}/{notes}/" + + match target: + case "obsidian": + out = f"{vault}/{notes}/" + case t if t in self._EMBEDDED_DATA: + out = f"{vault}/.thicket/{t}/ ({notes_txt})" + case "qdrant": + out = f"qdrant://{host}:{port}/{collection} ({notes_txt})" + case "weaviate": + out = (f"weaviate://{host}:{port}+grpc50051/" + f"{collection} ({notes_txt})") + case "pgvector": + pg_host = os.environ.get("PGHOST", "localhost") + pg_db = os.environ.get("PGDATABASE", "thicket") + out = (f"postgres://{pg_host}/{pg_db} \u00b7 \"{collection}\"" + f" ({notes_txt})") + case "mariadb": + my_host = os.environ.get("MARIADB_HOST", "127.0.0.1") + my_db = os.environ.get("MARIADB_DATABASE", "thicket") + out = (f"mariadb://{my_host}/{my_db} \u00b7 `{collection}`" + f" ({notes_txt})") + case _: + out = target + self.out_label.setText(out) + + # Reframe the vault field for what it means under this target. + vault_tooltips = { + "obsidian": + "THE destination: notes land in /.\n" + "No vector index runs under this target.", + **{t: f"Notes land in /; the {t} index lives\n" + f"in /.thicket/{t}/ — the whole brain is one tree." + for t in self._EMBEDDED_DATA}, + } + self.vault_path_edit.setToolTip(vault_tooltips.get( + target, + f"Notes land in /; the index lives in the " + f"{target} service, not in the vault.")) + + def _target_hint(self, target: str) -> str: + """One truthful line about how the target connects.""" + if target == "obsidian": + return "notes only: / — no vector index (SEARCH/ASK disabled)" + if target in self._ENV_HINTS: + return self._ENV_HINTS[target] + if target in self._TARGET_PORTS: + return (f"service: http://{self.svc_host_edit.text()}:" + f"{self.svc_port_spin.value()} (container)") + return f"embedded: /.thicket/{target}/ — no service needed" + + @Slot() + def _on_target_changed(self, _index: int): + """Target switch re-scores the whole connection row: host/port + live only where they apply (with the right default port), the + hint line restates how this target connects, and the status + footer carries the new target immediately.""" + target = self._current_target() + service_port = self._TARGET_PORTS.get(target) + self.svc_host_edit.setEnabled(service_port is not None) + self.svc_port_spin.setEnabled(service_port is not None) + if service_port is not None: + self.svc_port_spin.setValue(service_port) + self.target_hint_label.setText(f" {self._target_hint(target)}") + self._refresh_destination() + self._refresh_status_target() + + def _refresh_status_target(self) -> None: + """Status footer: current target + last probe summary. Called + on probe completion and on every target change.""" + target_txt = f"target: {self._current_target()}" + summary = getattr(self, "_probe_summary", None) + self.status_label.setText( + f"{target_txt} | {summary}" if summary else target_txt) + + def _vector_block_reason(self, target: str) -> str: + """Decisive reason string for a blocked vector target.""" + from .vector_stores import TARGETS + spec = TARGETS.get(target) + if spec is None: + return f"unknown target '{target}'" + if self.env is not None and spec.service == "qdrant" and not self.env.qdrant_up: + return f"Qdrant service unreachable at " \ + f"{self.svc_host_edit.text()}:{self.svc_port_spin.value()}" + missing = [m for m in spec.modules if not self.env.modules.get(m, False)] \ + if self.env else list(spec.modules) + return f"missing modules: {', '.join(missing)}" + + @staticmethod + def _preferred_llm(models: list[str]) -> str: + """Step-down auto-pick for the graph LLM: first preference + present on the server wins; fall to the first llama-family + model, then to the first model listed.""" + preferences = ("llama3.1:latest", "qwen3-coder:30b", + "qwen2.5-coder:32b", "llama3.3:latest", + "mistral:latest", "qwen3:latest") + exact = next((m for m in preferences if m in models), None) + if exact is not None: + return exact + llama_family = next((m for m in models if "llama" in m.lower()), None) + return llama_family or models[0] + + # ── Queue table ── + + def _populate_queue(self, files: list[Path]): + self.queue_table.setRowCount(0) + self._row_for_path.clear() + for row, filepath in enumerate(files): + self.queue_table.insertRow(row) + try: + size = filepath.stat().st_size + if size < 1_048_576: + size_txt = f"{size / 1024:.0f}K" + else: + size_txt = f"{size / 1_048_576:.1f}M" + except OSError: + size_txt = "?" + for col, text in ( + (0, filepath.name), (1, filepath.suffix.lstrip(".").upper()), + (2, size_txt), (3, "QUEUED"), (4, ""), + ): + item = QTableWidgetItem(text) + if col == 3: + item.setForeground(QColor(_STAGE_COLORS["QUEUED"])) + self.queue_table.setItem(row, col, item) + self._row_for_path[str(filepath)] = row + self.queue_table.sortItems(0) + + @Slot() + def _scan(self): + in_dir = Path(self.in_path_edit.text()).expanduser() + if not in_dir.is_dir(): + self._log(f"ERROR: Input directory does not exist: {in_dir}") + return + files = scan_files(in_dir, self._parse_extensions()) + self._populate_queue(files) + self._log(f"Scan: {len(files)} eligible document(s) in '{in_dir}'") + + @Slot(str, str) + def _on_file_status(self, path: str, status: str): + row = self._row_for_path.get(path) + if row is None: + return + item = self.queue_table.item(row, 3) + if item is None: + return + item.setText(status) + item.setForeground(QColor(_STAGE_COLORS.get(status, "#c8c8c8"))) + + @Slot(str, str) + def _on_file_detail(self, path: str, detail: str): + row = self._row_for_path.get(path) + if row is None: + return + item = self.queue_table.item(row, 4) + if item is not None: + item.setText(detail) + + # ── Process control ── + + @Slot() + def _start_ingest(self): + if self.worker is not None and self.worker.isRunning(): + return + + in_dir = Path(self.in_path_edit.text()).expanduser() + vault_dir = Path(self.vault_path_edit.text()).expanduser() + + if not in_dir.is_dir(): + self._log(f"ERROR: Input directory does not exist: {in_dir}") + return + + if not vault_dir.is_dir(): + reply = QMessageBox.question( + self, "Create Vault Directory", + f"Obsidian vault path does not exist:\n{vault_dir}\n\nCreate it?", + QMessageBox.StandardButton.Yes | QMessageBox.StandardButton.No, + QMessageBox.StandardButton.Yes, + ) + if reply != QMessageBox.StandardButton.Yes: + self._log("Cancelled: vault directory not confirmed.") + return + vault_dir.mkdir(parents=True, exist_ok=True) + self._log(f"Created vault directory: {vault_dir}") + + target = self._current_target() + use_vault = True # notes are the product + use_vector = target != "obsidian" # a vector TARGET adds indexing + use_graph = self.stage_graph_check.isChecked() + use_minio = self.stage_minio_check.isChecked() + + if not (use_vault or use_vector or use_graph or use_minio + or self.fs_archive_check.isChecked()): + self._log("ERROR: Nothing to do — pick a destination or a stage.") + return + + # Readiness table: (enabled, ready, reason, dialog title, hint). + # The first enabled-but-not-ready stage blocks the run. + readiness = ( + ( + use_vector, + self.env.vector_ready(target) if self.env else True, + self._vector_block_reason(target), + "Vector Target Unavailable", + "Pick another TARGET or fix the environment.", + ), + ( + use_graph, + self.env.graph_ready if self.env else True, + "Ollama service unreachable" + if self.env and not self.env.ollama_up + else "lightrag not installed (pip install 'thicket[graph]')", + "LightRAG Stage Unavailable", + "Uncheck it or fix the environment.", + ), + ) + if use_minio and self.env is not None and not self.env.minio_up: + readiness += (( + True, False, + f"MinIO unreachable ({self.env.minio_error or 'no response'})", + "MinIO Archive Unavailable", + "Uncheck it, start MinIO, or fix MINIO_* env vars.", + ),) + blocker = next((g for g in readiness if g[0] and not g[1]), None) + if blocker is not None: + _, _, reason, title, hint = blocker + self._log(f"ERROR: {title} — {reason}. See probe log above.") + QMessageBox.warning(self, title, + f"The stage cannot run: {reason}.\n\n{hint}") + return + + config = IngestConfig( + input_dir=in_dir, + vault_dir=vault_dir, + qdrant_host=self.svc_host_edit.text().strip() or "localhost", + qdrant_port=self.svc_port_spin.value(), + collection=self.collection_edit.text().strip() or "second_brain", + embed_model=self.embed_combo.currentText(), + chunk_size=self.chunk_knob.intValue(), + overlap=self.overlap_knob.intValue(), + use_vault=use_vault, + target=target if use_vector else "obsidian", + use_qdrant=use_vector, + use_lightrag=use_graph, + ollama_llm=self.llm_combo.currentText().strip() or "llama3", + ) + + # Sync the queue table with what the worker is about to process. + self._populate_queue(scan_files(in_dir, self._parse_extensions())) + + self.worker = PipelineWorker(config) + self.worker.log_msg.connect(self._log) + self.worker.file_status.connect(self._on_file_status) + self.worker.file_detail.connect(self._on_file_detail) + self.worker.progress_msg.connect(self._on_progress) + self.worker.finished_queue.connect(self._on_finished) + + self.btn_run.setEnabled(False) + self.btn_run.setText("RUNNING...") + self.btn_stop.setEnabled(True) + self.btn_scan.setEnabled(False) + self.btn_search.setEnabled(False) + self.worker.start() + + @Slot(str, int, int) + def _on_progress(self, filename: str, current: int, total: int): + self.status_label.setText(f"Ingesting {current}/{total}: {filename}") + + @Slot(int, int) + def _on_finished(self, ok: int, fail: int): + self.btn_run.setEnabled(True) + self.btn_run.setText(" > INGEST") + self.btn_stop.setEnabled(False) + self.btn_scan.setEnabled(True) + self.btn_search.setEnabled(True) + self.status_label.setText(f"Done — {ok} succeeded, {fail} failed") + self._log(f"Queue finished: {ok} succeeded, {fail} failed.") + + @Slot() + def _stop_process(self): + if self.worker is not None and self.worker.isRunning(): + self._log("STOP: Exiting queue after current file finishes...") + self.worker.stop() + self.btn_stop.setEnabled(False) + + # ── Retrieval ── + + @Slot() + def _search(self): + if self.search_worker is not None and self.search_worker.isRunning(): + return + if self._current_target() == "obsidian": + self._log("SEARCH: TARGET is Obsidian vault (notes only) — " + "pick a vector target to search.") + return + query = self.query_edit.text().strip() + if not query: + self._log("SEARCH: empty query — type something first.") + return + self.search_worker = SearchWorker( + query=query, + vault_dir=Path(self.vault_path_edit.text()).expanduser(), + target=self._current_target(), + host=self.svc_host_edit.text().strip() or "localhost", + port=self.svc_port_spin.value(), + collection=self.collection_edit.text().strip() or "second_brain", + embed_model=self.embed_combo.currentText(), + top_k=self.topk_knob.intValue(), + ) + self.search_worker.log_msg.connect(self._log) + self.search_worker.search_done.connect(self._on_search_done) + self.btn_search.setEnabled(False) + self._log(f"SEARCH: '{query}'") + self.search_worker.start() + + @Slot(int) + def _on_search_done(self, count: int): + self.btn_search.setEnabled(True) + if count > 0: + self.status_label.setText(f"Search: {count} hit(s)") + + @Slot() + def _ask(self): + """Natural-language SQL over the corpus (SQL-backed targets).""" + if self.ask_worker is not None and self.ask_worker.isRunning(): + return + question = self.query_edit.text().strip() + if not question: + self._log("ASK: empty query — type a question first.") + return + from .ask_vanna import SQL_TARGETS + + target = self._current_target() + if target == "obsidian": + self._log("ASK: TARGET is Obsidian vault (notes only) — " + "pick a SQL target (pgvector / mariadb) to ask.") + return + if target not in SQL_TARGETS: + self._log( + f"ASK: target '{target}' has no SQL corpus — ask works with " + f"{', '.join(SQL_TARGETS)} (switch TARGET)." + ) + return + from .pipeline_worker import AskWorker + + self.ask_worker = AskWorker( + question, target=target, + collection=self.collection_edit.text().strip() or "second_brain", + llm_model=self.llm_combo.currentText().strip() or "llama3", + ) + self.ask_worker.log_msg.connect(self._log) + self.ask_worker.ask_done.connect( + lambda ok: self.btn_ask.setEnabled(True)) + self.btn_ask.setEnabled(False) + self._log(f"ASK ({target}): '{question}'") + self.ask_worker.start() + + # ── About ── + + def _show_about(self): + from . import __version__ + dlg = QMessageBox(self) + dlg.setWindowTitle("About Thicket") + dlg.setText( + f"Thicket v{__version__} — super-ingest + RAG console\n\n" + "Feed it documents; it grows a thicket: dense, interconnected,\n" + "searchable.\n\n" + "Destinations: Obsidian vault (notes only) or any of ten\n" + "open-source vector stores — qdrant, chroma, lancedb, faiss,\n" + "milvus, weaviate, pgvector, duckdb, sqlite-vec, mariadb —\n" + "with identical payloads and scores across all of them.\n\n" + "Optional stages: knowledge graphs (LightRAG or Graphify via\n" + "Ollama), MinIO object archive, and a bz2 source archive —\n" + "sources are never deleted. Ask questions in natural language\n" + "over SQL-backed corpora (Vanna 2 + Ollama).\n\n" + "Code-aware ingestion: fenced blocks stay atomic with their\n" + "language tags; source and config files ingest as listings.\n" + "Detects the /mnt/AI corpus layout (cold in, hot out)." + ) + dlg.setInformativeText( + "Components (invoked, not bundled): PySide6 (LGPL-3.0), pypdf (BSD),\n" + "EbookLib (AGPL-3.0), BeautifulSoup (MIT), python-slugify (MIT),\n" + "FastEmbed (Apache-2.0), Qdrant / Chroma / LanceDB / pymilvus /\n" + "MinIO client (Apache-2.0), FAISS / PyMySQL / sqlite-vec / LightRAG /\n" + "Vanna / Ollama (MIT), weaviate-client (BSD-3), psycopg (LGPL-3.0),\n" + "Graphify (Apache-2.0/MIT).\n\n" + "Thicket itself: AGPL-3.0 — Jeremy Anderson, info@dcos.net, dcos.net, 2026." + ) + dlg.setStandardButtons(QMessageBox.StandardButton.Ok) + dlg.exec() + + +# ────────────────────────────────────────────── +# GUI ENTRY POINT +# ────────────────────────────────────────────── + +def launch_gui(argv: list[str] | None = None) -> int: + """Create the QApplication, show the ThicketWindow, run the Qt + event loop. Returns the Qt event-loop exit code (0 on clean + shutdown).""" + import sys + + app = QApplication(sys.argv if argv is None else argv) + window = ThicketWindow() + window.show() + return sys.exit(app.exec()) diff --git a/thicket/vault_writer.py b/thicket/vault_writer.py new file mode 100644 index 0000000..6154016 --- /dev/null +++ b/thicket/vault_writer.py @@ -0,0 +1,120 @@ +"""Obsidian vault writer — normalized Markdown notes with YAML frontmatter. + +Invariants: + + * Frontmatter values are escaped (backslash + double quote), so titles + like ``The "Real" Deal`` always produce valid YAML. + * Re-ingesting a source refreshes its note in place. + * A *different* source with the same title never clobbers an existing + note — it claims a deterministic hash-suffixed filename. + * An empty slug (symbol-only titles) lands on ``untitled``. +""" + +from __future__ import annotations + +import hashlib +import itertools +import re +from collections.abc import Iterator +from datetime import datetime +from pathlib import Path + +# Maximum frontmatter lines examined when resolving filename collisions +# (SEI CERT FIO39-C spirit: bounded reads on files we do not own). +_FRONTMATTER_SCAN_LIMIT = 32 + +_FRONTMATTER_SOURCE_RE = re.compile(r'^source_file:\s*"(.*)"\s*$') +_FRONTMATTER_KEYS = ("title:", "source_file:", "ingested_at:", "tags:", "- ") + + +def _yaml_escape(value: str) -> str: + """Escape a string for a double-quoted YAML scalar.""" + return value.replace("\\", "\\\\").replace('"', '\\"') + + +def _unescape_yaml(value: str) -> str: + """Inverse of _yaml_escape for values we wrote ourselves.""" + return value.replace('\\"', '"').replace("\\\\", "\\") + + +def _frontmatter_lines(fh) -> Iterator[str]: + """Yield stripped frontmatter lines, stopping at the block end.""" + for raw in fh: + stripped = raw.strip() + if not stripped or stripped == "---": + continue # block delimiters and blank lines + yield stripped + if not stripped.startswith(_FRONTMATTER_KEYS): + return # first non-frontmatter line ends the block + + +def _read_frontmatter_source(path: Path) -> str | None: + """The ``source_file:`` value from an existing note's frontmatter; + None when absent or unreadable.""" + try: + with path.open("r", encoding="utf-8", errors="ignore") as fh: + bounded = list(itertools.islice(fh, _FRONTMATTER_SCAN_LIMIT)) + except OSError: + return None + for line in _frontmatter_lines(bounded): + match = _FRONTMATTER_SOURCE_RE.match(line) + if match: + return _unescape_yaml(match.group(1)) + return None + + +class ObsidianVaultWriter: + """Formats and writes extracted text into normalized Obsidian + Markdown files under ``/Ingested_Brain``.""" + + def __init__(self, vault_path: Path, output_dirname: str = "Ingested_Brain"): + self.output_dir = vault_path / output_dirname + self.output_dir.mkdir(parents=True, exist_ok=True) + + def write(self, title: str, content: str, source_path: Path, + source_uri: str | None = None) -> Path: + """Write (or refresh) the note for *source_path*. Returns the + note path — which carries a hash suffix when a *different* + source already claimed the plain slug. *source_uri* records the + object-storage location when the archive stage ran first.""" + from slugify import slugify # lazy: only needed once ingesting + + slug = slugify(title) or "untitled" + target_file = self._claim_filename(slug, source_path.name) + + archive_line = ( + f'source_uri: "{_yaml_escape(source_uri)}"\n' + if source_uri else "" + ) + frontmatter = ( + "---\n" + f'title: "{_yaml_escape(title)}"\n' + f'source_file: "{_yaml_escape(source_path.name)}"\n' + f'{archive_line}' + f'ingested_at: "{datetime.now().isoformat()}"\n' + "tags:\n" + " - brain/ingested\n" + f" - source/{source_path.suffix.lstrip('.')}\n" + "---\n\n" + ) + + header = ( + f"# {title}\n\n" + f"*Source document: `{source_path.name}`*\n\n" + "---\n\n" + ) + + target_file.write_text(frontmatter + header + content, encoding="utf-8") + return target_file + + def _claim_filename(self, slug: str, source_name: str) -> Path: + """Resolve the note path for a slug: same source reclaims its + note; a different source gets a digest-suffixed sibling.""" + target = self.output_dir / f"{slug}.md" + if not target.exists(): + return target + existing_source = _read_frontmatter_source(target) + if existing_source == source_name or existing_source is None: + return target + digest = hashlib.md5(source_name.encode()).hexdigest()[:6] + return self.output_dir / f"{slug}-{digest}.md" diff --git a/thicket/vector_stores.py b/thicket/vector_stores.py new file mode 100644 index 0000000..e0c22a1 --- /dev/null +++ b/thicket/vector_stores.py @@ -0,0 +1,1035 @@ +"""Vector store abstraction — one protocol, five open-source targets. + +Every target implements the same contract (``ensure_collection``, +``set_embedder``, ``replace_document``, ``search``) with the same +payload schema, so the pipeline, retrieval strip, and probe are target- +agnostic. Heavy client libraries import lazily inside each store — a +missing target is a readable error at point of use, never a crash at +startup. + +Targets: + + ========== ============ ============================================== + key mode library + ========== ============ ============================================== + qdrant service qdrant-client (HTTP to a Qdrant server) + chroma embedded chromadb (PersistentClient, on-disk) + lancedb embedded lancedb (columnar, on-disk) + faiss file faiss-cpu (IndexIDMap2 + JSON sidecar) + milvus embedded pymilvus (Milvus Lite local database) + ========== ============ ============================================== + +Embedded/file targets keep their data under ```` (the caller +passes ``/.thicket/``), so a vault stays a single +portable tree. + +Payload contract (identical across targets): + + document_title, obsidian_path, section_header, content, + chunk_index, doc_key + +``doc_key`` (MD5 of ``title|rel_path``) is the deletion key for +exact-replacement re-ingest — a collision-free handle that keeps +filter expressions free of user-controlled strings. +""" + +from __future__ import annotations + +import hashlib +import json +import re +from collections.abc import Callable +from dataclasses import dataclass +from pathlib import Path + +from .chunker import Chunk + +UPSERT_BATCH = 64 + + +class VectorStoreError(Exception): + """Raised for target availability, connectivity, or write failures.""" + + +def doc_key(title: str, rel_path: str) -> str: + """Stable per-document key used for exact-replacement deletes.""" + return hashlib.md5(f"{title}|{rel_path}".encode()).hexdigest() + + +def _payload(title: str, rel_path: str, chunk: Chunk, idx: int) -> dict: + payload = { + "document_title": title, + "obsidian_path": rel_path, + "section_header": chunk.header, + "content": chunk.text, + "chunk_index": idx, + "chunk_kind": getattr(chunk, "kind", "prose"), + "doc_key": doc_key(title, rel_path), + } + if getattr(chunk, "lang", None): + payload["lang"] = chunk.lang + if getattr(chunk, "source", None): + # Origin file within the input tree — programming RAG answers + # "which file is this from", not just "which document title". + payload["source_path"] = chunk.source + return payload + + +def _snippet_payload(payload: dict) -> dict: + """Payload fields the retrieval strip displays (drop internal keys).""" + return {k: v for k, v in payload.items() if k != "doc_key"} + + +class BaseVectorStore: + """Shared behavior for embedded/file targets: embedder attachment, + dimension bookkeeping, batched writes. Subclasses own the client + lifecycle and implement the four protocol methods.""" + + def __init__(self, collection: str, dim: int, + log: Callable[[str], None] = lambda _msg: None): + self.collection = collection + self.dim = dim + self.model_hint = "selected model" + self._engine = None + self._log = log + + def set_embedder(self, engine) -> None: + """Attach the EmbeddingEngine; adopts its dimension when none + was declared.""" + self._engine = engine + self.model_hint = engine.model_name + if engine.dim and not self.dim: + self.dim = engine.dim + + def _require_dimension(self) -> None: + if not self.dim: + raise VectorStoreError( + f"'{self.model_hint}' has no catalog dimension — the " + f"{type(self).__name__} target requires one" + ) + + def _vectors(self, chunks: list[Chunk]) -> list[list[float]]: + return self._engine.embed([c.contextual_text for c in chunks]) + + def close(self) -> None: + """Release connections and file locks. Embedded file targets + (Milvus Lite in particular) hold exclusive locks while + connected — release or the next process cannot open them.""" + for attr in ("_client", "_conn", "_db"): + obj = getattr(self, attr, None) + if obj is None: + continue + closer = getattr(obj, "close", None) + if closer is not None: + try: + closer() + except Exception: + pass + setattr(self, attr, None) + # Milvus Lite: closing the client leaves the embedded server + # thread holding the database file lock — release it too. + try: + from milvus_lite.server_manager import server_manager_instance + server_manager_instance.release_all() + except Exception: + pass + + def _require_module(self, module: str) -> None: + try: + return __import__(module) + except ImportError as e: + raise VectorStoreError( + f"{module} not installed — run: pip install 'thicket[{module}]' " + f"or pick another vector target" + ) from e + + # protocol: ensure_collection / replace_document / search + + +# ────────────────────────────────────────────────────────────────── +# Chroma (embedded) +# ────────────────────────────────────────────────────────────────── + +_CHROMA_NAME_OK = ( + lambda name: 3 <= len(name) <= 512 + and all(c.isalnum() or c in "._-" for c in name) + and name[0].isalnum() and name[-1].isalnum() +) + + +class ChromaStore(BaseVectorStore): + """Chroma persistent collection; cosine space, on-disk under the + data dir. Dimensions are implicit — Chroma validates them at write.""" + + def __init__(self, data_dir: Path, collection: str, dim: int, + log: Callable[[str], None] = lambda _msg: None): + super().__init__(collection, dim, log) + self._require_module("chromadb") + import chromadb + self._client = chromadb.PersistentClient(path=str(data_dir)) + self._col = None + + def ensure_collection(self) -> None: + if not _CHROMA_NAME_OK(self.collection): + raise VectorStoreError( + f"Chroma collection names need 3-512 chars of [a-zA-Z0-9._-] " + f"(got '{self.collection}')" + ) + self._col = self._client.get_or_create_collection( + name=self.collection, metadata={"hnsw:space": "cosine"} + ) + + def _collection(self): + if self._col is None: + self.ensure_collection() + return self._col + + def replace_document(self, title: str, obsidian_rel_path: str, + chunks: list[Chunk]) -> int: + import uuid as _uuid + + if not chunks: + return 0 + col = self._collection() + key = doc_key(title, obsidian_rel_path) + col.delete(where={"doc_key": {"$eq": key}}) + + vectors = self._vectors(chunks) + for i in range(0, len(chunks), UPSERT_BATCH): + batch = chunks[i:i + UPSERT_BATCH] + col.add( + ids=[str(_uuid.UUID(bytes=hashlib.md5( + f"{key}|{i + j}".encode()).digest())) for j in range(len(batch))], + embeddings=vectors[i:i + UPSERT_BATCH], + documents=[c.text for c in batch], + metadatas=[_payload(title, obsidian_rel_path, c, i + j) + for j, c in enumerate(batch)], + ) + return len(chunks) + + def search(self, vector: list[float], limit: int = 5) -> list[dict]: + result = self._collection().query( + query_embeddings=[vector], n_results=limit, + include=["metadatas", "distances"], + ) + metas = result["metadatas"][0] + dists = result["distances"][0] + return [{"score": 1.0 - d, "payload": _snippet_payload(m)} + for m, d in zip(metas, dists)] + + +# ────────────────────────────────────────────────────────────────── +# LanceDB (embedded) +# ────────────────────────────────────────────────────────────────── + +class LanceStore(BaseVectorStore): + """LanceDB table; vectors are L2-normalized on write and query so + the default L2 metric ranks identically to cosine.""" + + def __init__(self, data_dir: Path, collection: str, dim: int, + log: Callable[[str], None] = lambda _msg: None): + super().__init__(collection, dim, log) + self._require_module("lancedb") + import lancedb + self._db = lancedb.connect(str(data_dir)) + self._data_dir = data_dir + + def _table(self): + try: + return self._db.open_table(self.collection) + except Exception: + return None + + def ensure_collection(self) -> None: + # Tables are created with their first data batch (the schema is + # data-derived); a missing table simply means "nothing ingested". + pass + + def replace_document(self, title: str, obsidian_rel_path: str, + chunks: list[Chunk]) -> int: + if not chunks: + return 0 + self._require_dimension() + import numpy as np + + key = doc_key(title, obsidian_rel_path) + table = self._table() + if table is not None: + table.delete(f"doc_key = '{key}'") + + # Payload is a JSON string column: the chunk schema carries + # optional fields (lang), and struct inference would break on + # their presence/absence across batches. + records = [ + {"doc_key": key, + "vector": (np.asarray(v) / np.linalg.norm(v)).tolist(), + "payload": json.dumps(_payload(title, obsidian_rel_path, c, i))} + for i, (c, v) in enumerate(zip(chunks, self._vectors(chunks))) + ] + if table is None: + self._db.create_table(self.collection, data=records) + else: + table.add(records) + return len(chunks) + + def search(self, vector: list[float], limit: int = 5) -> list[dict]: + import numpy as np + + table = self._table() + if table is None: + raise VectorStoreError( + f"collection '{self.collection}' is empty — ingest first" + ) + q = np.asarray(vector, dtype="float32") + q = (q / np.linalg.norm(q)).tolist() + rows = table.search(q).limit(limit).to_list() + # Lance distances are squared L2; normalized squared L2 d + # satisfies cos = 1 - d/2 — matching every other target's + # cosine score. + return [{"score": 1.0 - r.get("_distance", 0.0) / 2.0, + "payload": _snippet_payload(json.loads(r["payload"]))} + for r in rows] + + +# ────────────────────────────────────────────────────────────────── +# FAISS (file) +# ────────────────────────────────────────────────────────────────── + +class FaissStore(BaseVectorStore): + """FAISS IndexIDMap2 over an inner-product index with L2-normalized + vectors (= cosine). Payloads live in a JSON sidecar keyed by the + same deterministic ids as the index.""" + + def __init__(self, data_dir: Path, collection: str, dim: int, + log: Callable[[str], None] = lambda _msg: None): + super().__init__(collection, dim, log) + faiss = self._require_module("faiss") + self._faiss = faiss + self._index_path = data_dir / f"{collection}.faiss" + self._meta_path = data_dir / f"{collection}.json" + self._index = None + self._meta: dict[str, dict] = {} + + def ensure_collection(self) -> None: + self._require_dimension() + if self._index is not None: + return + if self._index_path.exists(): + self._index = self._faiss.read_index(str(self._index_path)) + self._meta = json.loads(self._meta_path.read_text(encoding="utf-8")) + else: + self._index = self._faiss.IndexIDMap2( + self._faiss.IndexFlatIP(self.dim) + ) + self._persist() + + def _persist(self) -> None: + self._index_path.parent.mkdir(parents=True, exist_ok=True) + self._faiss.write_index(self._index, str(self._index_path)) + self._meta_path.write_text( + json.dumps(self._meta, ensure_ascii=False), encoding="utf-8" + ) + + @staticmethod + def _faiss_id(key: str, idx: int) -> int: + digest = hashlib.md5(f"{key}|{idx}".encode()).digest() + return int.from_bytes(digest[:8], "big") & (2**63 - 1) + + def replace_document(self, title: str, obsidian_rel_path: str, + chunks: list[Chunk]) -> int: + import numpy as np + + if not chunks: + return 0 + self.ensure_collection() + key = doc_key(title, obsidian_rel_path) + + stale = [pid for pid, p in self._meta.items() + if p.get("doc_key") == key] + if stale: + # Old chunk ids are (key, 0..count-1) by construction. + self._index.remove_ids(np.array( + [self._faiss_id(key, i) for i in range(len(stale))], + dtype="int64")) + for s in stale: + del self._meta[s] + + vectors = np.asarray(self._vectors(chunks), dtype="float32") + vectors /= np.linalg.norm(vectors, axis=1, keepdims=True) + ids = np.array([self._faiss_id(key, i) for i in range(len(chunks))], + dtype="int64") + self._index.add_with_ids(vectors, ids) + for i, chunk in enumerate(chunks): + pid = str(ids[i]) + self._meta[pid] = _payload(title, obsidian_rel_path, chunk, i) + self._persist() + return len(chunks) + + def search(self, vector: list[float], limit: int = 5) -> list[dict]: + import numpy as np + + self.ensure_collection() + q = np.asarray([vector], dtype="float32") + q /= np.linalg.norm(q) + scores, ids = self._index.search(q, limit) + return [{"score": float(s), "payload": _snippet_payload(self._meta[str(i)])} + for s, i in zip(scores[0], ids[0]) if i != -1] + + +# ────────────────────────────────────────────────────────────────── +# Milvus Lite (embedded) +# ────────────────────────────────────────────────────────────────── + +class MilvusStore(BaseVectorStore): + """Milvus Lite local database via pymilvus MilvusClient — one + flat cosine index, JSON payloads, VARCHAR primary ids.""" + + def __init__(self, data_dir: Path, collection: str, dim: int, + log: Callable[[str], None] = lambda _msg: None): + super().__init__(collection, dim, log) + self._require_module("pymilvus") + from pymilvus import MilvusClient + data_dir.mkdir(parents=True, exist_ok=True) + import time + + last_error: Exception | None = None + for attempt in range(3): # lock may linger one beat after close + try: + self._client = MilvusClient(str(data_dir / f"{collection}.db")) + break + except Exception as e: # noqa: BLE001 — retry, then report + last_error = e + time.sleep(2) + else: + raise VectorStoreError( + f"Milvus Lite unavailable: {last_error} — another process " + f"may hold the database lock, or run: " + f"pip install 'pymilvus[milvus_lite]'" + ) from last_error + + def ensure_collection(self) -> None: + self._require_dimension() + if self._client.has_collection(self.collection): + self._client.load_collection(self.collection) # no-op when loaded + return + from pymilvus import DataType + + schema = self._client.create_schema(auto_id=False) + schema.add_field("id", DataType.VARCHAR, max_length=64, is_primary=True) + schema.add_field("vector", DataType.FLOAT_VECTOR, dim=self.dim) + schema.add_field("doc_key", DataType.VARCHAR, max_length=64) + schema.add_field("payload", DataType.JSON) + index = self._client.prepare_index_params() + index.add_index(field_name="vector", index_type="FLAT", + metric_type="COSINE") + self._client.create_collection(self.collection, schema=schema, + index_params=index) + self._client.load_collection(self.collection) + self._log(f"Creating collection '{self.collection}' in Milvus Lite...") + + def replace_document(self, title: str, obsidian_rel_path: str, + chunks: list[Chunk]) -> int: + import uuid as _uuid + + if not chunks: + return 0 + self.ensure_collection() + key = doc_key(title, obsidian_rel_path) + self._client.delete(self.collection, + filter=f'doc_key == "{key}"') + + rows = [ + {"id": str(_uuid.UUID(bytes=hashlib.md5( + f"{key}|{i}".encode()).digest())), + "vector": v, + "doc_key": key, + "payload": _payload(title, obsidian_rel_path, c, i)} + for i, (c, v) in enumerate(zip(chunks, self._vectors(chunks))) + ] + for i in range(0, len(rows), UPSERT_BATCH): + self._client.insert(self.collection, rows[i:i + UPSERT_BATCH]) + # Lite has no Strong-consistency knob in this client version; + # flush makes delete+re-insert immediately visible to search. + self._client.flush(self.collection) + return len(rows) + + def search(self, vector: list[float], limit: int = 5) -> list[dict]: + self.ensure_collection() + result = self._client.search( + self.collection, data=[vector], limit=limit, + output_fields=["payload"], + search_params={"metric_type": "COSINE"}, + ) + return [{"score": float(hit["distance"]), + "payload": _snippet_payload(hit["entity"].get("payload", {}))} + for hit in result[0]] + + +# ────────────────────────────────────────────────────────────────── +# Weaviate (service) +# ────────────────────────────────────────────────────────────────── + +def _weaviate_name(name: str) -> str: + """Weaviate requires collection names to start uppercase with + [A-Za-z0-9_] bodies — map deterministically and keep the mapping + logged.""" + cleaned = re.sub(r"[^A-Za-z0-9_]", "_", name) + return (cleaned[:1].upper() + cleaned[1:]) or "Thicket" + + +class WeaviateStore(BaseVectorStore): + """Weaviate server (container or compose), HNSW cosine index. + Connects over HTTP (host/port) with the standard gRPC port 50051.""" + + GRPC_PORT = 50051 + + def __init__(self, host: str, port: int, collection: str, dim: int, + log: Callable[[str], None] = lambda _msg: None, **_ignored): + super().__init__(collection, dim, log) + self._require_module("weaviate") + import weaviate + if host == "localhost": + # The gRPC channel resolves localhost to ::1 first, which + # host-networked servers routinely refuse — pin IPv4. + host = "127.0.0.1" + try: + self._client = weaviate.connect_to_local( + host=host, port=port, grpc_port=self.GRPC_PORT, + ) + except Exception as e: + raise VectorStoreError( + f"cannot connect to Weaviate at {host}:{port} " + f"(gRPC {self.GRPC_PORT}): {e}" + ) from e + self._mapped = _weaviate_name(collection) + if self._mapped != collection: + self._log(f"Weaviate maps collection '{collection}' " + f"-> '{self._mapped}' (name rules).") + + def ensure_collection(self) -> None: + from weaviate.classes.config import ( + Configure, DataType, Property, Tokenization, VectorDistances, + ) + if not self._client.collections.exists(self._mapped): + self._log(f"Creating collection '{self._mapped}' in Weaviate...") + self._client.collections.create( + name=self._mapped, + vector_index_config=Configure.VectorIndex.hnsw( + distance_metric=VectorDistances.COSINE), + properties=[ + Property(name="doc_key", data_type=DataType.TEXT, + tokenization=Tokenization.FIELD), + Property(name="document_title", data_type=DataType.TEXT), + Property(name="obsidian_path", data_type=DataType.TEXT), + Property(name="section_header", data_type=DataType.TEXT), + Property(name="content", data_type=DataType.TEXT), + Property(name="chunk_index", data_type=DataType.INT), + ], + ) + + def replace_document(self, title: str, obsidian_rel_path: str, + chunks: list[Chunk]) -> int: + if not chunks: + return 0 + import uuid as _uuid + from weaviate.classes.query import Filter + + coll = self._client.collections.get(self._mapped) + key = doc_key(title, obsidian_rel_path) + coll.data.delete_many(where=Filter.by_property("doc_key").equal(key)) + + with coll.batch.fixed_size(batch_size=UPSERT_BATCH) as batch: + for i, (chunk, vector) in enumerate( + zip(chunks, self._vectors(chunks))): + batch.add_object( + uuid=_uuid.UUID(bytes=hashlib.md5( + f"{key}|{i}".encode()).digest()), + vector=vector, + properties=_payload(title, obsidian_rel_path, chunk, i), + ) + failed = coll.batch.failed_objects + if failed: + raise VectorStoreError( + f"Weaviate rejected {len(failed)} object(s): " + f"{failed[0].message if failed else '?'}" + ) + return len(chunks) + + def search(self, vector: list[float], limit: int = 5) -> list[dict]: + from weaviate.classes.query import MetadataQuery + + coll = self._client.collections.get(self._mapped) + response = coll.query.near_vector( + near_vector=vector, limit=limit, + return_metadata=MetadataQuery(distance=True), + ) + return [{"score": 1.0 - (obj.metadata.distance or 0.0), + "payload": _snippet_payload(dict(obj.properties))} + for obj in response.objects] + + +# ────────────────────────────────────────────────────────────────── +# pgvector (Postgres service) +# ────────────────────────────────────────────────────────────────── + +class PgVectorStore(BaseVectorStore): + """Postgres + pgvector: one table per collection, HNSW cosine + index, JSONB payloads. Connection comes from the standard libpq + environment (PGHOST / PGPORT / PGDATABASE / PGUSER / PGPASSWORD, + or a full PGDSN) — Unix convention, no extra config surface.""" + + def __init__(self, collection: str, dim: int, + log: Callable[[str], None] = lambda _msg: None, **_ignored): + super().__init__(collection, dim, log) + self._require_module("psycopg") + import os + + import psycopg + dsn = os.environ.get("PGDSN") + try: + self._conn = psycopg.connect(dsn or "", connect_timeout=5) + except Exception as e: + raise VectorStoreError( + f"cannot connect to Postgres ({dsn or 'libpq env'}): {e} — " + f"set PGHOST/PGUSER/PGPASSWORD/PGDATABASE or PGDSN" + ) from e + try: + from pgvector.psycopg import register_vector + register_vector(self._conn) + except ImportError as e: + raise VectorStoreError( + "pgvector package not installed — run: " + "pip install 'thicket[pgvector]'" + ) from e + + def _table_sql(self, template: str): + """Compose SQL with the collection name as a quoted identifier — + user-supplied names never interpolate as raw SQL.""" + from psycopg import sql + return sql.SQL(template).format(tbl=sql.Identifier(self.collection)) + + def ensure_collection(self) -> None: + self._require_dimension() + from psycopg import sql + + try: + with self._conn.cursor() as cur: + cur.execute("CREATE EXTENSION IF NOT EXISTS vector") + cur.execute(sql.SQL( + "CREATE TABLE IF NOT EXISTS {tbl} (" + "id TEXT PRIMARY KEY, doc_key TEXT NOT NULL, " + "embedding vector({dim}), payload JSONB)" + ).format(tbl=sql.Identifier(self.collection), + dim=sql.Literal(self.dim))) + cur.execute(sql.SQL( + "CREATE INDEX IF NOT EXISTS {idx} ON {tbl} USING hnsw " + "(embedding vector_cosine_ops)" + ).format(idx=sql.Identifier(f"ix_{self.collection}_cos"), + tbl=sql.Identifier(self.collection))) + cur.execute(sql.SQL( + "CREATE INDEX IF NOT EXISTS {idx} ON {tbl} (doc_key)" + ).format(idx=sql.Identifier(f"ix_{self.collection}_dk"), + tbl=sql.Identifier(self.collection))) + self._conn.commit() + except Exception as e: + raise VectorStoreError( + f"pgvector setup failed: {e} — the server needs the " + f"pgvector extension available (image: pgvector/pgvector)" + ) from e + + def replace_document(self, title: str, obsidian_rel_path: str, + chunks: list[Chunk]) -> int: + import uuid as _uuid + + if not chunks: + return 0 + key = doc_key(title, obsidian_rel_path) + insert = self._table_sql( + "INSERT INTO {tbl} VALUES (%s, %s, %s::vector, %s::jsonb)") + try: + with self._conn.cursor() as cur: + cur.execute(self._table_sql("DELETE FROM {tbl} WHERE doc_key = %s"), + (key,)) + cur.executemany(insert, [ + (str(_uuid.UUID(bytes=hashlib.md5( + f"{key}|{i}".encode()).digest())), + key, json.dumps(vector), + json.dumps(_payload(title, obsidian_rel_path, chunk, i))) + for i, (chunk, vector) + in enumerate(zip(chunks, self._vectors(chunks))) + ]) + self._conn.commit() + except Exception: + # Roll back so one bad document never poisons the shared + # connection for the rest of the queue. + self._conn.rollback() + raise + return len(chunks) + + def search(self, vector: list[float], limit: int = 5) -> list[dict]: + query = self._table_sql( + "SELECT payload, 1 - (embedding <=> %s::vector) AS score " + "FROM {tbl} ORDER BY embedding <=> %s::vector LIMIT %s") + qtext = json.dumps(vector) + with self._conn.cursor() as cur: + cur.execute(query, (qtext, qtext, limit)) + rows = cur.fetchall() + return [{"score": float(score), + "payload": _snippet_payload( + row if isinstance(row, dict) else json.loads(row))} + for row, score in rows] + + +# ────────────────────────────────────────────────────────────────── +# DuckDB (embedded, VSS step-down) +# ────────────────────────────────────────────────────────────────── + +class DuckStore(BaseVectorStore): + """DuckDB with FLOAT[dim] columns. Vectors are L2-normalized on + write and query; ranking therefore equals cosine. The vss HNSW + extension loads when available and steps down to an exact scan + otherwise (exact is fine at personal-knowledge scale).""" + + def __init__(self, data_dir: Path, collection: str, dim: int, + log: Callable[[str], None] = lambda _msg: None): + super().__init__(collection, dim, log) + self._require_module("duckdb") + import duckdb + data_dir.mkdir(parents=True, exist_ok=True) + self._db = duckdb.connect(str(data_dir / f"{collection}.duckdb")) + try: + self._db.execute("INSTALL vss") + self._db.execute("LOAD vss") + except Exception: + self._log("DuckDB vss extension unavailable — exact scan.") + + def ensure_collection(self) -> None: + self._require_dimension() + self._db.execute( + f'CREATE TABLE IF NOT EXISTS "{self.collection}" (' + f'doc_key VARCHAR, embedding FLOAT[{self.dim}], payload JSON)' + ) + + def _normalize(self, vectors): + import numpy as np + arr = np.asarray(vectors, dtype="float32") + return (arr / np.linalg.norm(arr, axis=-1, keepdims=True).clip(1e-12)).tolist() + + def replace_document(self, title: str, obsidian_rel_path: str, + chunks: list[Chunk]) -> int: + if not chunks: + return 0 + self.ensure_collection() + key = doc_key(title, obsidian_rel_path) + self._db.execute( + f'DELETE FROM "{self.collection}" WHERE doc_key = ?', [key]) + rows = [ + (key, vector, + json.dumps(_payload(title, obsidian_rel_path, chunk, i))) + for i, (chunk, vector) + in enumerate(zip(chunks, self._normalize(self._vectors(chunks)))) + ] + self._db.executemany( + f'INSERT INTO "{self.collection}" ' + f'VALUES (?, ?::FLOAT[{self.dim}], ?::JSON)', rows) + return len(chunks) + + def search(self, vector: list[float], limit: int = 5) -> list[dict]: + import numpy as np + + self.ensure_collection() + q = np.asarray(vector, dtype="float32") + q = (q / max(float(np.linalg.norm(q)), 1e-12)).tolist() + rows = self._db.execute( + f'SELECT payload, array_distance(embedding, ' + f'?::FLOAT[{self.dim}]) AS d FROM "{self.collection}" ' + f'ORDER BY d LIMIT ?', [q, limit]).fetchall() + # Normalized L2 distance d satisfies cos = 1 - d²/2. + return [{"score": 1.0 - float(d) ** 2 / 2.0, + "payload": _snippet_payload(json.loads(p))} + for p, d in rows] + + +# ────────────────────────────────────────────────────────────────── +# sqlite-vec (embedded) +# ────────────────────────────────────────────────────────────────── + +class SqliteVecStore(BaseVectorStore): + """sqlite-vec vec0 virtual table (float vectors, L2) with a + sidecar metadata table. Vectors L2-normalized — ranking equals + cosine; score conversion matches the other normalized-L2 targets.""" + + def __init__(self, data_dir: Path, collection: str, dim: int, + log: Callable[[str], None] = lambda _msg: None): + super().__init__(collection, dim, log) + sqlite_vec = self._require_module("sqlite_vec") + import sqlite3 + data_dir.mkdir(parents=True, exist_ok=True) + self._vec = sqlite_vec + self._db = sqlite3.connect(str(data_dir / f"{collection}.sqlite3")) + self._db.enable_load_extension(True) + sqlite_vec.load(self._db) + self._db.enable_load_extension(False) + + def ensure_collection(self) -> None: + self._require_dimension() + self._db.execute( + f'CREATE VIRTUAL TABLE IF NOT EXISTS "vec_{self.collection}" ' + f'USING vec0(embedding float[{self.dim}])') + self._db.execute( + f'CREATE TABLE IF NOT EXISTS "{self.collection}_meta" ' + f'(rowid INTEGER PRIMARY KEY, doc_key TEXT, payload TEXT)') + self._db.commit() + + def replace_document(self, title: str, obsidian_rel_path: str, + chunks: list[Chunk]) -> int: + if not chunks: + return 0 + self.ensure_collection() + key = doc_key(title, obsidian_rel_path) + + stale = self._db.execute( + f'SELECT rowid FROM "{self.collection}_meta" WHERE doc_key = ?', + (key,)).fetchall() + for (rowid,) in stale: + self._db.execute( + f'DELETE FROM "vec_{self.collection}" WHERE rowid = ?', + (rowid,)) + self._db.execute( + f'DELETE FROM "{self.collection}_meta" WHERE rowid = ?', + (rowid,)) + + for i, (chunk, vector) in enumerate( + zip(chunks, self._normalized(self._vectors(chunks)))): + cur = self._db.execute( + f'INSERT INTO "vec_{self.collection}" (embedding) ' + f'VALUES (?)', (self._vec.serialize_float32(vector),)) + self._db.execute( + f'INSERT INTO "{self.collection}_meta" VALUES (?, ?, ?)', + (cur.lastrowid, key, + json.dumps(_payload(title, obsidian_rel_path, chunk, i)))) + self._db.commit() + return len(chunks) + + def _normalized(self, vectors: list[list[float]]) -> list[list[float]]: + import numpy as np + arr = np.asarray(vectors, dtype="float32") + norms = np.linalg.norm(arr, axis=1, keepdims=True).clip(1e-12) + return (arr / norms).tolist() + + def search(self, vector: list[float], limit: int = 5) -> list[dict]: + import numpy as np + + self.ensure_collection() + q = np.asarray(vector, dtype="float32") + q = (q / max(float(np.linalg.norm(q)), 1e-12)).tolist() + rows = self._db.execute( + f'SELECT m.payload, v.distance ' + f'FROM "vec_{self.collection}" v ' + f'JOIN "{self.collection}_meta" m ON m.rowid = v.rowid ' + f'WHERE v.embedding MATCH ? AND k = ? ' + f'ORDER BY v.distance', + (self._vec.serialize_float32(q), limit)).fetchall() + return [{"score": 1.0 - float(d) ** 2 / 2.0, + "payload": _snippet_payload(json.loads(p))} + for p, d in rows] + + +# ────────────────────────────────────────────────────────────────── +# MariaDB (service, VECTOR columns) +# ────────────────────────────────────────────────────────────────── + +def _mariadb_ident(name: str) -> str: + """Table/database identifier: [A-Za-z0-9_] only — user-supplied + names never reach SQL as raw text.""" + cleaned = re.sub(r"[^A-Za-z0-9_]", "_", name) + return cleaned or "thicket" + + +class MariaDbStore(BaseVectorStore): + """MariaDB 11.7+ VECTOR(dim) columns with VEC_Distance_Cosine + ranking and a vector index. Connection comes from MARIADB_* env + vars (HOST/PORT/USER/PASSWORD/DATABASE, or UNIX_SOCKET). + + The vector index requires primary keys <= 256 bytes; ids are + 36-char UUID strings (144 bytes utf8mb4), and index creation + steps down to a logged exact scan on servers without vector + support.""" + + def __init__(self, collection: str, dim: int, + log: Callable[[str], None] = lambda _msg: None, **_ignored): + super().__init__(collection, dim, log) + self._require_module("pymysql") + import os + + import pymysql + self._pymysql = pymysql + self._db = _mariadb_ident(os.environ.get("MARIADB_DATABASE", "thicket")) + self._table = _mariadb_ident(collection) + if self._table != collection: + self._log(f"MariaDB maps collection '{collection}' " + f"-> table '{self._table}' (identifier rules).") + try: + self._conn = pymysql.connect( + host=os.environ.get("MARIADB_HOST", "127.0.0.1"), + port=int(os.environ.get("MARIADB_PORT", "3306")), + user=os.environ.get("MARIADB_USER", "root"), + password=os.environ.get("MARIADB_PASSWORD", ""), + unix_socket=os.environ.get("MARIADB_UNIX_SOCKET") or None, + connect_timeout=5, + ) + except Exception as e: + raise VectorStoreError( + f"cannot connect to MariaDB: {e} — set MARIADB_HOST / " + f"MARIADB_USER / MARIADB_PASSWORD (or MARIADB_UNIX_SOCKET)" + ) from e + with self._conn.cursor() as cur: + cur.execute(f"CREATE DATABASE IF NOT EXISTS `{self._db}`") + cur.execute(f"USE `{self._db}`") + self._conn.commit() + + def ensure_collection(self) -> None: + self._require_dimension() + with self._conn.cursor() as cur: + cur.execute( + f"CREATE TABLE IF NOT EXISTS `{self._db}`.`{self._table}` (" + f"id VARCHAR(36) PRIMARY KEY, " + f"doc_key VARCHAR(64) NOT NULL, " + f"embedding VECTOR({self.dim}) NOT NULL, " + f"payload JSON)") + cur.execute( + "SELECT COLUMN_TYPE FROM information_schema.COLUMNS " + "WHERE TABLE_SCHEMA = %s AND TABLE_NAME = %s " + "AND COLUMN_NAME = 'embedding'", + (self._db, self._table)) + row = cur.fetchone() + if row and f"vector({self.dim})" not in str(row[0]).lower(): + raise VectorStoreError( + f"table `{self._db}`.`{self._table}` was built at a " + f"different vector dimension ({row[0]}) than " + f"'{self.model_hint}' produces ({self.dim}) — pick a " + f"new collection name or drop the table") + try: + cur.execute( + f"ALTER TABLE `{self._db}`.`{self._table}` " + f"ADD VECTOR INDEX IF NOT EXISTS vix_{self._table} " + f"(embedding)") + except Exception: + self._log("MariaDB vector index unavailable — exact scan.") + cur.execute( + f"ALTER TABLE `{self._db}`.`{self._table}` " + f"ADD INDEX IF NOT EXISTS ix_{self._table}_dk (doc_key)") + self._conn.commit() + + def replace_document(self, title: str, obsidian_rel_path: str, + chunks: list[Chunk]) -> int: + import uuid as _uuid + + if not chunks: + return 0 + self.ensure_collection() + key = doc_key(title, obsidian_rel_path) + with self._conn.cursor() as cur: + cur.execute( + f"DELETE FROM `{self._db}`.`{self._table}` " + f"WHERE doc_key = %s", (key,)) + cur.executemany( + f"INSERT INTO `{self._db}`.`{self._table}` " + f"VALUES (%s, %s, VEC_FromText(%s), %s)", + [(str(_uuid.UUID(bytes=hashlib.md5( + f"{key}|{i}".encode()).digest())), + key, json.dumps(vector), + json.dumps(_payload(title, obsidian_rel_path, chunk, i))) + for i, (chunk, vector) + in enumerate(zip(chunks, self._vectors(chunks)))]) + self._conn.commit() + return len(chunks) + + def search(self, vector: list[float], limit: int = 5) -> list[dict]: + self.ensure_collection() + qtext = json.dumps(vector) + with self._conn.cursor() as cur: + cur.execute( + f"SELECT payload, " + f"1 - VEC_Distance_Cosine(embedding, VEC_FromText(%s)) " + f"FROM `{self._db}`.`{self._table}` " + f"ORDER BY VEC_Distance_Cosine(embedding, VEC_FromText(%s)) " + f"LIMIT %s", (qtext, qtext, limit)) + rows = cur.fetchall() + return [{"score": float(score), + "payload": _snippet_payload(json.loads(payload))} + for payload, score in rows] + + +# ────────────────────────────────────────────────────────────────── +# Registry +# ────────────────────────────────────────────────────────────────── + +@dataclass(frozen=True, slots=True) +class TargetSpec: + key: str + label: str # UI display label + modules: tuple[str, ...] # probe requirements + service: str | None # live service required: "qdrant" | "postgres" + factory: Callable[..., object] + + +def _qdrant_factory(*, host: str, port: int, collection: str, dim: int, + log: Callable[[str], None], data_dir: Path): + from .qdrant_store import QdrantStore + return QdrantStore(host=host, port=port, collection=collection, + dim=dim, log=log) + + +def _weaviate_factory(*, host: str, port: int, collection: str, dim: int, + log: Callable[[str], None], data_dir: Path): + return WeaviateStore(host=host, port=port, collection=collection, + dim=dim, log=log) + + +def _embedded_factory(store_cls): + def factory(*, collection: str, dim: int, + log: Callable[[str], None], data_dir: Path, **_unused): + return store_cls(data_dir=data_dir, collection=collection, + dim=dim, log=log) + return factory + + +TARGETS: dict[str, TargetSpec] = { + spec.key: spec for spec in ( + TargetSpec("qdrant", "Qdrant (service)", ("qdrant_client", "fastembed"), + "qdrant", _qdrant_factory), + TargetSpec("chroma", "Chroma (embedded)", ("chromadb",), None, + _embedded_factory(ChromaStore)), + TargetSpec("lancedb", "LanceDB (embedded)", ("lancedb",), None, + _embedded_factory(LanceStore)), + TargetSpec("faiss", "FAISS (file)", ("faiss",), None, + _embedded_factory(FaissStore)), + TargetSpec("milvus", "Milvus Lite (embedded)", ("pymilvus",), None, + _embedded_factory(MilvusStore)), + TargetSpec("weaviate", "Weaviate (service)", ("weaviate",), + "weaviate", _weaviate_factory), + TargetSpec("pgvector", "pgvector (Postgres)", ("psycopg", "pgvector"), + "postgres", _embedded_factory(PgVectorStore)), + TargetSpec("duckdb", "DuckDB (embedded)", ("duckdb",), None, + _embedded_factory(DuckStore)), + TargetSpec("sqlitevec", "sqlite-vec (embedded)", ("sqlite_vec",), None, + _embedded_factory(SqliteVecStore)), + TargetSpec("mariadb", "MariaDB (service)", ("pymysql",), + "mariadb", _embedded_factory(MariaDbStore)), + ) +} + + +def create_store(target: str, *, collection: str, dim: int, + host: str = "localhost", port: int = 6333, + data_dir: Path | None = None, + log: Callable[[str], None] = lambda _msg: None): + """Build the store for *target* — the single dispatch point every + caller (pipeline, retrieval, tests) shares.""" + spec = TARGETS.get(target) + if spec is None: + known = ", ".join(sorted(TARGETS)) + raise VectorStoreError(f"unknown vector target '{target}' — known: {known}") + return spec.factory(host=host, port=port, collection=collection, + dim=dim, log=log, data_dir=data_dir) diff --git a/thicket/widgets/__init__.py b/thicket/widgets/__init__.py new file mode 100644 index 0000000..f07b63a --- /dev/null +++ b/thicket/widgets/__init__.py @@ -0,0 +1,5 @@ +"""Custom widgets — self-contained, theme-agnostic building blocks.""" + +from .radio_knob import RadioKnob + +__all__ = ["RadioKnob"] diff --git a/thicket/widgets/radio_knob.py b/thicket/widgets/radio_knob.py new file mode 100755 index 0000000..50a875a --- /dev/null +++ b/thicket/widgets/radio_knob.py @@ -0,0 +1,282 @@ +"""RadioKnob widget — retro radio-style rotary knob. + +A self-contained PySide6 widget (arc range, tick marks, glowing +indicator dot). Has no internal package dependencies — only PySide6 +and ``math`` from the stdlib — so it can be imported standalone. +""" + +import math + +from PySide6.QtCore import Qt, Signal, QPointF, QRectF +from PySide6.QtGui import ( + QFont, QColor, QPainter, QPen, QBrush, + QRadialGradient, QFontMetrics, +) +from PySide6.QtWidgets import QWidget + +# ────────────────────────────────────────────── +# RADIO KNOB WIDGET (oldschool rotary control) +# ────────────────────────────────────────────── + +class RadioKnob(QWidget): + """ + A retro radio-style rotary knob widget. + Supports arc range, tick marks, and a glowing indicator dot. + + Rotation: 7 o'clock (min) to 5 o'clock (max) = 300 degrees. + """ + valueChanged = Signal(float) + + def __init__( + self, + parent=None, + min_val: float = 0.0, + max_val: float = 100.0, + default_val: float = 50.0, + label: str = "", + unit: str = "", + color: tuple = (42, 130, 218), + num_ticks: int = 17, + tick_labels: list[str] | None = None, + snap_ticks: bool = False, + compact: bool = False, + ): + super().__init__(parent) + self.min_val = min_val + self.max_val = max_val + self._value = default_val + self.label = label + self.unit = unit + self.color = QColor(*color) + self.num_ticks = num_ticks + self.tick_labels = tick_labels + self.snap_ticks = snap_ticks + self._dragging = False + self.compact = compact + + # Arc geometry: 300-degree sweep, centered at 12 o'clock + self._arc_start = 210.0 # degrees (7 o'clock) + self._arc_span = -300.0 # negative = clockwise + + # Scaling factor for compact mode (~70% of full size) + s = 0.70 if compact else 1.0 + self._s = s + self.setFixedSize(int(180 * s), int(210 * s)) + self.setCursor(Qt.CursorShape.PointingHandCursor) + + # --- Public API --- + + def value(self) -> float: + return self._value + + def setValue(self, v: float): + v = max(self.min_val, min(self.max_val, v)) + if self.snap_ticks: + v = self._snap(v) + if v != self._value: + self._value = v + self.update() + self.valueChanged.emit(v) + + def intValue(self) -> int: + return int(round(self._value)) + + def _snap(self, v: float) -> float: + """Snap to nearest tick.""" + step = (self.max_val - self.min_val) / max(1, self.num_ticks - 1) + return round((v - self.min_val) / step) * step + self.min_val + + def _val_to_angle(self, v: float) -> float: + """Map value to angle in degrees (matching the conical gradient).""" + ratio = (v - self.min_val) / (self.max_val - self.min_val) if self.max_val != self.min_val else 0 + return self._arc_start + ratio * self._arc_span # goes from 210 -> -90 + + def _angle_to_val(self, angle_deg: float) -> float: + """Map angle back to value.""" + # Normalize angle relative to arc start + ratio = (angle_deg - self._arc_start) / self._arc_span + ratio = max(0.0, min(1.0, ratio)) + v = self.min_val + ratio * (self.max_val - self.min_val) + if self.snap_ticks: + v = self._snap(v) + return v + + # --- Painting --- + + def paintEvent(self, event): + p = QPainter(self) + p.setRenderHint(QPainter.RenderHint.Antialiasing) + w, h = self.width(), self.height() + s = self._s # scale factor (0.7 for compact, 1.0 for full) + + cx = w / 2 + cy = h / 2 - 4 * s + outer_r = 70 * s + knob_r = 40 * s + arc_w = max(1, int(8 * s)) + tick_w = max(1, 1.5 * s) + bezel_pad = 6 * s + + # --- Outer bezel ring --- + bezel_grad = QRadialGradient(cx, cy, outer_r + bezel_pad) + bezel_grad.setColorAt(0.85, QColor(48, 48, 52)) + bezel_grad.setColorAt(1.0, QColor(26, 26, 30)) + p.setBrush(QBrush(bezel_grad)) + p.setPen(Qt.PenStyle.NoPen) + p.drawEllipse(QPointF(cx, cy), outer_r + bezel_pad, outer_r + bezel_pad) + + # --- Inactive arc (dark track) --- + p.setPen(QPen(QColor(50, 50, 56), arc_w, Qt.PenStyle.SolidLine, Qt.PenCapStyle.RoundCap)) + p.drawArc(QRectF(cx - outer_r, cy - outer_r, outer_r * 2, outer_r * 2), + int(self._arc_start * 16), int(self._arc_span * 16)) + + # --- Active arc (colored fill up to current value) --- + val_angle = self._val_to_angle(self._value) + active_span = val_angle - self._arc_start + if abs(active_span) > 0.5: + arc_color = QColor(self.color) + p.setPen(QPen(arc_color, arc_w, Qt.PenStyle.SolidLine, Qt.PenCapStyle.RoundCap)) + p.drawArc(QRectF(cx - outer_r, cy - outer_r, outer_r * 2, outer_r * 2), + int(self._arc_start * 16), int(active_span * 16)) + + # --- Tick marks --- + for i in range(self.num_ticks): + t = i / (self.num_ticks - 1) if self.num_ticks > 1 else 0 + tick_angle = self._val_to_angle(self.min_val + t * (self.max_val - self.min_val)) + tick_rad = tick_angle * math.pi / 180.0 + ox = cx + (outer_r + 12 * s) * (-1) * math.sin(tick_rad) + oy = cy + (outer_r + 12 * s) * (-1) * (-math.cos(tick_rad)) + ix_ = cx + (outer_r + 3 * s) * (-1) * math.sin(tick_rad) + iy_ = cy + (outer_r + 3 * s) * (-1) * (-math.cos(tick_rad)) + p.setPen(QPen(QColor(130, 130, 130), tick_w)) + p.drawLine(QPointF(ix_, iy_), QPointF(ox, oy)) + + # Tick labels (if provided) + if self.tick_labels: + p.setFont(QFont("Sans", max(5, int(7 * s)))) + p.setPen(QColor(160, 160, 160)) + step = max(1, self.num_ticks // len(self.tick_labels)) + label_idx = 0 + for i in range(0, self.num_ticks, step): + if label_idx >= len(self.tick_labels): + break + t = i / (self.num_ticks - 1) if self.num_ticks > 1 else 0 + tick_angle = self._val_to_angle(self.min_val + t * (self.max_val - self.min_val)) + tick_rad = tick_angle * math.pi / 180.0 + lx = cx + (outer_r + 24 * s) * (-1) * math.sin(tick_rad) + ly = cy + (outer_r + 24 * s) * (-1) * (-math.cos(tick_rad)) + txt = self.tick_labels[label_idx] + fm = QFontMetrics(p.font()) + tw = fm.horizontalAdvance(txt) + p.drawText(QPointF(lx - tw / 2, ly + 2 * s), txt) + label_idx += 1 + + # --- Knob body (dark brushed aluminum) --- + knob_grad = QRadialGradient(cx - 6 * s, cy - 6 * s, knob_r * 1.3) + knob_grad.setColorAt(0.0, QColor(72, 72, 78)) + knob_grad.setColorAt(0.5, QColor(50, 50, 55)) + knob_grad.setColorAt(1.0, QColor(34, 34, 38)) + p.setBrush(QBrush(knob_grad)) + p.setPen(QPen(QColor(26, 26, 30), max(1, 1.5 * s))) + p.drawEllipse(QPointF(cx, cy), knob_r, knob_r) + + # --- Inner shadow ring --- + inner_shadow = QRadialGradient(cx, cy, knob_r - 2) + inner_shadow.setColorAt(0.85, QColor(0, 0, 0, 0)) + inner_shadow.setColorAt(1.0, QColor(0, 0, 0, 60)) + p.setBrush(QBrush(inner_shadow)) + p.setPen(Qt.PenStyle.NoPen) + p.drawEllipse(QPointF(cx, cy), knob_r - 1, knob_r - 1) + + # --- Indicator line (pointer) --- + ptr_angle = self._val_to_angle(self._value) + ptr_rad = ptr_angle * 3.14159265 / 180.0 + ptr_len = knob_r - 8 * s + px = cx + ptr_len * (-1) * math.sin(ptr_rad) + py = cy + ptr_len * (-1) * (-math.cos(ptr_rad)) + p.setPen(QPen(QColor(255, 255, 255, 220), max(1, 2.5 * s), + Qt.PenStyle.SolidLine, Qt.PenCapStyle.RoundCap)) + p.drawLine(QPointF(cx, cy), QPointF(px, py)) + + # --- Center cap dot --- + cap_r = max(2, 5 * s) + cap_grad = QRadialGradient(cx, cy, cap_r) + cap_grad.setColorAt(0.0, QColor(60, 60, 65)) + cap_grad.setColorAt(1.0, QColor(30, 30, 34)) + p.setBrush(QBrush(cap_grad)) + p.setPen(Qt.PenStyle.NoPen) + p.drawEllipse(QPointF(cx, cy), cap_r, cap_r) + + # --- Glow dot at arc tip --- + glow_r = max(3, 10 * s) + glow_x = cx + outer_r * (-1) * math.sin(ptr_rad) + glow_y = cy + outer_r * (-1) * (-math.cos(ptr_rad)) + glow = QRadialGradient(glow_x, glow_y, glow_r * 1.2) + glow.setColorAt(0.0, QColor(self.color.red(), self.color.green(), self.color.blue(), 200)) + glow.setColorAt(1.0, QColor(self.color.red(), self.color.green(), self.color.blue(), 0)) + p.setBrush(QBrush(glow)) + p.setPen(Qt.PenStyle.NoPen) + p.drawEllipse(QPointF(glow_x, glow_y), glow_r, glow_r) + + p.end() + + # --- Label + value text below knob --- + p2 = QPainter(self) + p2.setRenderHint(QPainter.RenderHint.Antialiasing) + + # Value line (e.g. "32.0 CRF") + val_font_sz = max(6, int(13 * s)) + p2.setFont(QFont("Consolas", val_font_sz, QFont.Weight.Bold)) + val_color = QColor(self.color.red(), self.color.green(), self.color.blue()) + p2.setPen(val_color) + val_text = f"{self._value:.0f} {self.unit}" if self.unit else f"{self._value:.0f}" + p2.drawText(QRectF(0, h - 38 * s, w, 20 * s), Qt.AlignmentFlag.AlignCenter, val_text) + + # Label line (e.g. "Quality") + lbl_font_sz = max(5, int(9 * s)) + p2.setFont(QFont("Consolas", lbl_font_sz, QFont.Weight.Bold)) + p2.setPen(QColor(160, 160, 160)) + p2.drawText(QRectF(0, h - 18 * s, w, 16 * s), Qt.AlignmentFlag.AlignCenter, self.label) + p2.end() + + # --- Input handling --- + + def mousePressEvent(self, event): + if event.button() == Qt.MouseButton.LeftButton: + self._dragging = True + self._update_from_mouse(event.position()) + + def mouseMoveEvent(self, event): + if self._dragging: + self._update_from_mouse(event.position()) + + def mouseReleaseEvent(self, event): + if event.button() == Qt.MouseButton.LeftButton: + self._dragging = False + + def wheelEvent(self, event): + delta = event.angleDelta().y() + step = (self.max_val - self.min_val) / max(1, self.num_ticks - 1) + if delta > 0: + self.setValue(self._value + step) + elif delta < 0: + self.setValue(self._value - step) + + def _update_from_mouse(self, pos: QPointF): + cx = self.width() / 2 + cy = self.height() / 2 - 4 * self._s + dx = pos.x() - cx + dy = pos.y() - cy + angle = math.degrees(math.atan2(dx, -dy)) # 0=north, CW positive + if angle < 0: + angle += 360 + # Clamp to arc range: 210..510 (which is 210..360 and 0..150) + # Our arc: 210 degrees to -90 (=270) degrees clockwise + if angle < 210 and angle > 150: + # Dead zone at bottom (between 150 and 210) + # Push to nearest end + angle = 210 if abs(angle - 210) < abs(angle - 510) else 510 + if angle > 360: + angle -= 360 # normalize back to 0..360 + self.setValue(self._angle_to_val(angle)) +