301194963b
一些检查失败了
automerge / automerge (push) Has been cancelled
react-native-android-build-apk / AssembleRelease (push) Has been cancelled
Build macOS M1 / Main (push) Has been cancelled
Joplin Continuous Integration / Main (macos-15-intel) (push) Has been cancelled
Joplin Continuous Integration / Main (ubuntu-22.04) (push) Has been cancelled
Joplin Continuous Integration / Main (ubuntu-22.04-arm) (push) Has been cancelled
Joplin Continuous Integration / Main (windows-11-vs2026-arm) (push) Has been cancelled
Joplin Continuous Integration / Main (windows-2025) (push) Has been cancelled
Joplin Continuous Integration / ServerDockerImage (ubuntu-22.04) (push) Has been cancelled
Joplin Continuous Integration / ServerDockerImage (ubuntu-22.04-arm) (push) Has been cancelled
Joplin UI tests / Main (ubuntu-22.04) (push) Has been cancelled
Joplin UI tests / Main (windows-2025) (push) Has been cancelled
Joplin UI tests / Mobile (push) Has been cancelled
Close stale issues / ProcessStaleIssues (push) Has been cancelled
83 行
6.1 KiB
Markdown
83 行
6.1 KiB
Markdown
# AI embeddings
|
||
|
||
How Joplin builds and queries the local note-embeddings index. See [ai_primitives.md](ai_primitives.md) for how it fits with the other AI primitives.
|
||
|
||
## Overview
|
||
|
||
```
|
||
Notes ──► EmbeddingIndexer ──► embedding provider ──► NoteEmbedding
|
||
(background, 5 min) (local ONNX) (sqlite-vec)
|
||
▲
|
||
│
|
||
SearchService ◄── joplin.ai.search() │
|
||
│
|
||
AiIndexStatus ──► reads counts ───────┘
|
||
(settings panel)
|
||
```
|
||
|
||
- **EmbeddingIndexer** watches note changes, chunks each note, asks the active provider to embed the chunks, and persists them.
|
||
- **NoteEmbedding** is the storage model. Metadata sits in a regular table; vectors sit in a sqlite-vec virtual table. Joined by rowid.
|
||
- **LocalEmbeddingProvider** runs `multilingual-e5-small` via ONNX. The model (~140 MB) is downloaded from a GitHub release on first AI enable.
|
||
- **SearchService** backs `joplin.ai.search()`. Takes a text or note-id query and a scope, returns ranked chunks.
|
||
- **AiIndexStatus** is the settings panel that shows model + indexer state.
|
||
|
||
## Indexer behaviour
|
||
|
||
The indexer runs on a 5-minute timer (matching `OcrService`) and has two modes:
|
||
|
||
- **Initial scan.** On first enable (or after a model swap), walk the entire notes table one 100-note batch per tick. At the start of the scan the change-feed cursor is snapped to the current `lastChangeId` so edits made *during* the scan are picked up normally by the change feed when the scan finishes. The scan is complete when no indexable notes remain that aren't already in `note_embeddings_meta`; at that point `ai.embedding.initialScanDone` flips to true.
|
||
- **Change feed.** Once the scan completes, only the change-feed loop runs. Each tick drains `item_changes` past the durable cursor, collapses duplicates, processes deletes/creates/updates, then advances the cursor.
|
||
|
||
**Order of processing during the scan**: unspecified. The "not yet indexed" query has no `ORDER BY`, so SQLite returns rows in storage order (roughly insertion order). Determinism doesn't matter for the final state — every indexable note ends up embedded.
|
||
|
||
**Resume across restarts**: free because progress is the disk state, not a counter. After a restart the indexer re-enters scan mode (flag still false) and the `NOT EXISTS` query naturally skips notes that already have rows. Already-indexed notes from the previous session don't get re-processed.
|
||
|
||
**Failure handling during the scan**: per-note failures are logged, and the note is added to an in-memory skip set for the rest of the session — no retry loop, no log flood. The skip set is **not** persisted, so the next process restart retries the previously failed notes.
|
||
|
||
Once the scan completes, the indexer never sweeps the notes table again. A note that failed during the scan only gets re-tried if (a) the user edits it (the change feed picks it up), or (b) the model id changes (full re-scan). It will not be retried just because time passed.
|
||
|
||
Per note, the title is injected (doubled) into chunk 0. Without this, notes whose body is just an attachment link would never match title-anchored queries.
|
||
|
||
The first tick fires fire-and-forget at startup, so the model load doesn't block the splash screen. A model id change (e.g. switching providers) wipes the index, clears the scan flag, and rebuilds.
|
||
|
||
## Chunking
|
||
|
||
Fixed-size character windows sized to fit the model's 512-token context, with 10% overlap. Two profiles: a Latin-script default and a CJK profile (used when CJK is the dominant script — CJK tokenises ~3× denser).
|
||
|
||
## Search
|
||
|
||
- `query: { text }` runs the active provider's query-side encoding.
|
||
- `query: { noteId }` reuses the note's stored vectors as the query — no re-embedding pass.
|
||
- `scope`: `all` / `note` / `folder` / `tag`. Trashed and conflict notes are always excluded.
|
||
- `relevance`: `strict` / `normal` / `loose`. Maps internally to model-specific `(k, minScore)` so plugins survive model changes.
|
||
- Score is cosine similarity in `[0, 1]`, computed from the L2 distance sqlite-vec returns (vectors are L2-normalised, so `cos = 1 − d²/2`).
|
||
|
||
## Platform support
|
||
|
||
| Platform | Embeddings | Notes |
|
||
|-------------------------|:----------:|----------------------------------------|
|
||
| macOS Apple Silicon | ✓ | |
|
||
| macOS Intel | ✗ | `onnxruntime-node` ships no darwin-x64 |
|
||
| Linux x64 / arm64 | ✓ | |
|
||
| Windows x64 / arm64 | ✓ | |
|
||
| CLI / mobile / server | ✗ | No ONNX wired in; chat still works |
|
||
|
||
Unsupported platforms degrade cleanly: the provider isn't installed, the indexer stays paused, search throws a clear error.
|
||
|
||
## Performance, in rough numbers
|
||
|
||
- Model load: 2–15 s (one-time per process, runs in the background).
|
||
- Per-chunk inference: 30–200 ms depending on CPU.
|
||
- Search latency: dominated by the query-embedding pass; the vec MATCH itself is sub-millisecond on tens of thousands of chunks.
|
||
- Big vaults take a while: 100 notes per tick × 5-min interval, so a fresh 10k-note vault is ~8 hours of background indexing. Intentional pacing.
|
||
|
||
## Durable state
|
||
|
||
| Setting | Purpose |
|
||
|----------------------------------------|----------------------------------|
|
||
| `ai.enabled` | Top-level AI toggle |
|
||
| `ai.embedding.enabled` | Indexer kill switch (default on) |
|
||
| `ai.embedding.lastProcessedChangeId` | Cursor into `item_changes` |
|
||
| `ai.embedding.lastIndexedModelId` | Detects model swap → re-index |
|
||
| `ai.embedding.initialScanDone` | True once the full-vault scan finished |
|