From 5b9a806af13d38163156302097e405a7b145b6cc Mon Sep 17 00:00:00 2001 From: JSDurand Date: Thu, 19 Nov 2026 09:29:47 +0800 Subject: elel: notes engine (Rust) and Gnus-like Emacs frontend Tantivy full text index, link graph, inline query filters, Ollama embeddings with paragraph chunks, and an Elisp summary/article UI. Co-Authored-By: Claude Sonnet 5.5 --- README.org | 170 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 170 insertions(+) create mode 100644 README.org (limited to 'README.org') diff --git a/README.org b/README.org new file mode 100644 index 0000000..ae298ed --- /dev/null +++ b/README.org @@ -0,0 +1,170 @@ +#+TITLE: elel + +Emacs Lexical Embedding Link: a notes system whose essence is +searching. Notes are found by their words (index), their meaning +(semantics, second pass) and their relations (structure). + +* Layout +- =src/= :: the engine, a Rust CLI that only /reads/ note files. +- =elel.el= :: the Emacs front end, with a Gnus-like interface. + +* Files and notes +A file is a container of one or more notes. Each note carries its own +ID. Formats: +- =.org= :: each top-level heading is a note; metadata in the property + drawer (=:ID: :DATE: :KIND: :PARENT: :LINKS: :CONTEXT: :SUMMARY:=), + tags in the heading. +- =.txt= :: notes are separated by a form-feed line; each starts with + =Key: value= lines (=ID Title Date Kind Tags Parent Links Context + Summary=), a blank line, then the body. +A link in a body is written =[[note:ID][TITLE]]= in every format, and +=[[note::KIND:ID][TITLE]]= when it has a type (=KIND= is letters, digits, +=-= or =_=). The title is for the reader; the ID is how the engine finds +the note. An ID has no whitespace; a uuid-style ID is fine as it is, +e.g. =[[note:][T]]=. Typed links can also go in the =LINKS= +property, as =supports:ID=. In org, =note= is a registered link type, so +org follows (=C-c C-o=), completes (=C-c C-l=, which offers the title as +the description) and exports these links itself. + +* Searching +One query string. Filters are words of the form =key:value=; they are +always ANDed. The remaining words go to the full text index (ANDed; +="a phrase"=, =-word=, =a OR b= and =title:foo= work; Chinese is +matched by character pairs). +| Filter | Meaning | +|-------------------------+-------------------------------------------------| +| =tag:X= =-tag:X= | has / lacks the tag X | +| =tag:*= =-tag:*= | has some tag / has no tag | +| =tag:alg*= | has a tag that starts with =alg= | +| =kind:X= =kind:*= | the KIND is X / there is a KIND (=-kind:= too) | +| =file:X= =-file:X= | the path (relative to the root) contains X | +| =from:ID= | the notes that ID links to | +| =to:ID= | the notes that link to ID | +| =parent:ID= | the children of ID | +| =thread:ID= | the whole thread of ID | +| =date:2025= | the DATE starts with 2025 | +| =date:2025-03..2025-06= | a range; =date:..2024= and =date:2025..= work | +| =is:inbox= | no links, parent, children or tags | +| =like:X= =-like:X= | by meaning: X is a note ID or a text (see Embeddings) | +Quote a value with spaces: =file:"my notes"=. The empty query lists +every note. Without words, results are newest first; with words, by +score. + +* Engine +#+begin_src sh +cargo build --release +elel index [--no-embed] # scan ELEL_ROOT (default ~/notes), update the index and the embeddings +elel search [-n N] [--hybrid] -- QUERY +elel count -- QUERY... # the number of notes for each query +elel hood ID [--hops N] +elel get ID # the note, with its exact text in "raw" +elel check # missing/duplicate IDs, dangling links +elel tags [--like ID] # with --like: the tags that fit the note, best first +#+end_src +The index lives in ELEL_INDEX (default =~/.local/share/elel=) and can be +deleted at any time. + +* Emacs +#+begin_src emacs-lisp +(add-to-list 'load-path "~/.emacs.d/claude/elel") +(autoload 'elel "elel" nil t) +(setq elel-root "~/notes") +#+end_src +The interface follows Gnus. =M-x elel= opens the views buffer (the +group buffer); =RET= opens a view in the summary buffer, with the +article below it. =elel-views= is a list of =(NAME QUERY . PLIST)=; +=:limit N= caps a view and =:hidden t= keeps it out of sight until =L=. +Views for a tag or a file are just more entries, e.g. +=("Math" "tag:math")=. + +*Views buffer*: =RET= open, =L= list hidden views, =G= or =/= search, +=u= update the index, =g= redraw, =N= new note, =C= check. + +*Summary buffer*: the notes are threaded by PARENT, drawn as in Gnus +(see =elel-thread-*=). =RET= shows the note, =SPC= / =DEL= scroll it, +=n= / =p= next / previous note, =g= show the note again (as in Gnus; to +refresh the list, search again with =G=), =/= narrow the query, =s= +snippets, ==== make the summary fill the frame (=C-u ==== restores both), +=A= select the article window, =#= / =-= mark liked / disliked, =R= related, =S= suggestions, =a= add the whole thread of the note, =^= parent, =c= children, +=o= outgoing links, =b= backlinks, =H= neighbourhood, =e= edit, =r= / =N= +child / new note (finish with =C-c C-c=), =q= leave. + +*Article buffer*: the note as it is in its file, in the major mode that +=elel-major-mode-alist= picks (org-mode for =.org=), with the minor mode +=elel-article-mode= on top. =SPC= / =j= scroll (at the end, go to the next +note), =DEL= / =k= back, =n= / =p= next / previous note of the summary, +=TAB= / =S-TAB= and =RET= move to and follow links, =l= back, =^= =c= =o= +=b= =H= =a= as in the summary, =h= go to the summary, =q= close the +article, =e= edit, =r= child note, =g= reread. You may close the +summary window; =n= and =p= still work. =elel-summary-window-height= +and =elel-article-keep-summary= control the windows. + +=M-x elel-assign-ids= adds missing IDs. + +In the note files (=elel-link-mode=, on automatically under +=elel-root=): =C-c n l= inserts a link by picking a title (an active +region becomes the description), =C-c C-o= follows the link at point +(org's own behaviour elsewhere), =C-c n o= opens the note around +point. + +* Importing old notes +=elel-import.el= (Emacs Lisp; it only reads the sources and never +overwrites). Without an output directory, or with =dry-run=, it only +reports. +#+begin_src sh +# notes kept as Gnus drafts (Gcc nnmaildir+notes:GROUP): one GROUP.org +# per group, Message-IDs as IDs, In-Reply-To as PARENT +emacs --batch -L ~/.emacs.d/claude/elel -l elel-import \ + --eval '(elel-import-drafts "~/News/drafts/drafts" "~/notes")' +# an org-roam v1 directory: notes are collected into few files, each named +# by a tag chosen to cover many notes (see elel-import-roam-min-group) +emacs --batch -L ~/.emacs.d/claude/elel -l elel-import \ + --eval '(elel-import-roam "~/org/roam" "~/notes")' +#+end_src +Both are also commands (=M-x elel-import-drafts=, =M-x elel-import-roam=) +that ask before writing. + +* Embeddings +The "E" of elel. Install [[https://ollama.com][Ollama]] and run +=ollama pull qwen3-embedding:0.6b=. Then =elel index= also embeds the +notes (the first run takes a few minutes; later runs only embed what +changed). If Ollama is not running, indexing still works; the notes +stay pending and the next run catches up. =--no-embed= skips it. + +- *Units*: a note is cut into paragraph chunks (=src/chunk.rs=). A + paragraph that is only a formula joins its neighbour, small paragraphs + are merged, source blocks and =\begin..\end= stay whole, and the title is + put in front of each chunk. A note's own vector is the mean of its + chunks. +- *Storage*: =/vec/= holds =f16.bin= (exact vectors), + =bits.bin= (one sign bit per dimension, for a first Hamming pass when + there are more than 2000 chunks) and =meta.json=. All of it can be + rebuilt. Changing the model (=ELEL_MODEL=, default + =qwen3-embedding:0.6b=) re-embeds everything. =ELEL_OLLAMA= is the + server URL. +- *=like:X=*: notes close in meaning. X is a note ID (the whole note + is compared) or a text such as =like:"moduli of curves"= (each note + scores as its best chunk, which is shown as the snippet). Several + =like:= are merged by their mean; =-like:Y= pushes away from Y. + Other filters still apply: =like:ID tag:algebra date:2024..=. +- *Hybrid*: =search --hybrid= with words fuses the text ranking and the + meaning ranking by reciprocal rank fusion. In Emacs + =elel-hybrid-search= (default t) does this for every search; without + Ollama it falls back to the text search. +- *Tags as anchors*: =elel tags --like ID= scores each tag (the mean of + the other notes that carry it) against the note, to suggest tags. + +In the summary buffer, =#= marks a note as liked, =-= as disliked, and +=R= shows the notes closest to the marked ones (or to the note at point). +In the article, =R= shows the notes like this one. =S= (=elel-suggest=) +lists the notes that the note could link to (the ones it is already +related to are left out), with the best tags in the header; =i= there +inserts =[[note:ID][TITLE]]= where the suggestion was asked for. After a +new note is saved, the suggestions appear by themselves +(=elel-suggest-after-capture=), and =i= inserts at the point where you +typed =C-c C-c=. =elel-insert-link= lists the notes closest to the +current note first. + +* Not yet done +Anchors written as free phrases (instead of tags), a Gnus link +adapter, an optional daemon for faster calls. -- cgit v1.2.3-18-g5258