This article is a development log of seCall — a CLI tool that collects and searches AI agent conversation sessions. Since it was created as part of a vibe coding test project, I don't know and don't want to know the detailed code. I don't need to know. Knowing ruins it. Rather, I'm writing this to explain the technologies used in features applied after trial and error, and the reasons for their adoption.

The Problem: Too Many AI Conversations

Dozens of conversations pile up every day. Moments like "What did I say about BM25 last week?" happen daily. But Claude Code stores them as JSONL in ~/.claude, ChatGPT exports as ZIP, and Codex is stored somewhere under ~/.codex... They're scattered in different formats. And since conversations follow FIFO, past conversations quickly get pushed up and become hard to find (AI chatbots do talk a lot). Plus, once a session is interrupted, you can't even find which session contained the previous conversation. So I even recorded the last session in resume.md. 😇

What I want is simple: Collect everything in one place, search in Korean, and have it organized somewhere so I can review projects during free time.

Why SQLite?

Elasticsearch comes to mind first when thinking of search engines, and there are many strong open-source tools like ChromaDB for vector databases. But I thought it was overkill to run a JVM-based service on a personal tool. So I decided to keep it lightweight. You could use PostgreSQL for a personal diary, but that would be incredibly excessive.

Why I chose SQLite:

• Nothing to install. A single file is the entire database (sufficient for personal use)

• Metadata and search indexes in one place. Agent, project, date filtering and full-text search in a single query

• FTS5, a built-in full-text search engine. Even supports BM25 ranking

There's no reason not to use it.

What is FTS5?

Full-Text Search 5 — A full-text search extension built into SQLite.

While a simple LIKE '%search term%' has to scan the entire table, FTS5 pre-builds an inverted index. It records which words appear how many times in each document, and during search, it only looks up that index. Results come in milliseconds even from thousands of conversations.

And it also has BM25 ranking built-in. It considers how many times the search term appears in a document and how rare the word is among all documents, then sorts results by relevance. It's the same family of algorithm used by Google Search.

The Problem: FTS5 Doesn't Understand Korean

Here's where I hit a wall. Previously, when reimplementing QMD (a Python script + BM25 project that parses Claude Code sessions and organizes them in Obsidian vault — probably the most famous among similar projects) as a Gemini version, this was the part I gave up on. I abandoned it thinking whether I should really study tokenizers, but then our André Karpathy (OpenAI co-founder → Tesla AI chief) threw out a topic on wiki, so I decided to tackle it again. This is how the GitHub-starred (?) mega-hit project seCall was born

FTS5's default tokenizer splits words by whitespace and punctuation. This works for English, but not for Korean.

The problem is:

If "임베딩 모델을 최적화했습니다" is saved, FTS5 splits it English-style as "임베딩" "모델을" "최적화했습니다", and when you search for "모델", it won't match "모델을". Same with "최적화". Applying this to Korean (and CJK in general) makes it useless. So Korean requires separate morphological analysis (breaking words into meaningful units).

After morphological analysis, it's separated as "임베딩" "모델" "최적화", so now searching for "모델" works.

Solution: Pre-tokenize in Rust Before Storage

Fitting a Korean analyzer directly into FTS5 is cumbersome because you have to write C language extensions somehow. Instead, I used a more practical method:

1. When saving conversations: Korean morphological analysis in Rust → Join the analyzed words with spaces and store in FTS5

2. When searching: Process the search term through the same morphological analyzer, then query FTS5

Since both storage and search go through the same analyzer, FTS5 only needs to deal with cleanly organized words. This pattern is called Pre-Tokenization. It's a method of tokenizing ahead of time at the front of a search engine. (Applying layering, pipeline, and workflow concepts to objects this way improves vibe coding quality)

Korean Morphological Analyzer: Lindera

And in the layer where a real tokenizer is needed, I navigated the sea of open source. I chose Lindera as the morphological analyzer.

• MeCab-based Korean dictionary (ko-dic) included in the binary

• Works immediately without additional downloads or external services

• Rust native, so the build is done in one go

The tradeoff is that the binary size increases by ~80MB. But I thought this was acceptable for a personal project. As an alternative, I added Kiwi (a Korean-specialized analyzer) as an option. I later learned that Kiwi is said to have better analysis quality, which is why it became an option. There doesn't seem to be a significant difference in practice.

POS Filtering: Keep Only Meaningful Words

If you index every piece the morphological analyzer produces, search accuracy actually gets worse.

"임베딩 모델을 최적화했습니다"

All morphemes → 임베딩 / 모델 / 을 / 최적화 / 하 / 았 / 습니다

"을", "하", "았", "습니다" are useless for search. So I select and preserve only meaningful parts of speech:

• Nouns (모델, 최적화, 검색, SQLite)

• Verbs/Adjectives (구현하다, 빠르다)

• Foreign words (embedding, vector, BM25)

Particles, endings, and suffixes are all discarded. Single-character tokens also become noise and are removed.

Result: "임베딩" "모델" "최적화" — Only the essentials for search remain. To exclude noise as much as possible, I layered it into nouns, verbs/adjectives, and foreign words. Once you get to vector DB, chunk quality ultimately becomes context quality. You have to reduce noise as much as possible. (And the more recent trend is to put raw conversation text and RAG documents directly in the DB, then apply morphological separation, compression, or organization layers from there. The advantage is you can change the algorithm and reindex regardless of how much noise is generated)

Why I Didn't Use sqlite-vec

The original plan was to handle both FTS5 (text search) and sqlite-vec (vector search) entirely within SQLite. (Everyone's doing it!) But sqlite-vec gave me a C compilation error on my Mac. Since it's still in alpha (0.1), platform compatibility was unstable.

I ended up solving the vector side another way, and if interested, I'd like to explain how I implemented vector search without external services to find "semantically similar conversations" where text matching alone isn't enough.

The core was implementing Korean full-text search on top of a single SQLite database without external services. Using the Pre-Tokenization workaround, you can cleanly bypass the problem that FTS5 doesn't understand Korean.

This post is written without (no) vibe. You can read it with confidence. 😁 Real agentic development isn't about headfirst diving into bare ground. It's a two-person-three-legged race, a grand chaos collaboration between domain knowledge from human intelligence built through experience and the tireless accurate work of AI used without restraint.