When an idle vector worker keeps MariaDB busy
GoatMUX’s vector worker was waiting for work, but MariaDB kept rescanning the catalog. The fix was to enqueue work at ingestion and reserve full reconciliation for explicit requests.
GoatMUX catalogs files and turns extracted text into searchable content. MariaDB holds the catalog and processing state; OpenSearch holds searchable text and vectors. For semantic search, GoatMUX turns each text chunk into an embedding: a list of numbers that represents the chunk's meaning. A vector worker creates those embeddings with a local model and stores them in OpenSearch.
The worker should spend most of its time waiting when there is nothing new to process.
It was waiting. MariaDB was not.
The idle worker that kept scanning
During the original investigation, MariaDB periodically used one full CPU core for a short burst. The bursts lined up with the vector worker's ten-second idle poll. A process-list check identified the active statement as an INSERT IGNORE ... SELECT that compared content_chunks with content_chunk_vectors and added ledger rows for chunks that had none.
That query was part of ensureLedgerRows(). It was safe to repeat: the vector ledger had a composite key over chunk, vector index, and embedding model, so duplicate inserts were ignored. But safety did not make the repeated scan free. Even when almost every chunk already had a ledger row, the worker kept asking MariaDB to rediscover the catalog.
The important clue was the timing. The CPU bursts did not line up with OCR, a filesystem scan, or a failed OpenSearch write. They lined up with an idle worker asking whether any work existed.
Why another index was not the whole answer
The first optimization was to add indexes for the discovery query and the vector worker's claim query. Migration 019_content_vector_indexes.sql added:
CREATE INDEX ix_content_chunks_index_status_id
ON content_chunks (opensearch_index, index_status, opensearch_id, content_hash);
CREATE INDEX ix_content_chunk_vectors_claim_scope
ON content_chunk_vectors
(vector_index, embedding_model, status, available_at, updated_at);The query-plan investigation showed why this was not enough. The content corpus was already almost fully represented in the vector ledger, so the discovery query still had to examine a large share of content_chunks and check the ledger for each candidate. An index can help when it narrows a query to a small set of rows. It cannot turn a near-whole-corpus comparison into a small lookup just because the join columns are indexed.
The investigation record reports that the plan examined hundreds of thousands of chunk rows with dependent ledger lookups. That shifted the question from “Which index makes this scan faster?” to “Why is the idle worker doing a full discovery scan at all?”
Put work on the queue when content arrives
The redesign changes when discovery happens. New vector work is now recorded when the content pipeline already knows which chunks were indexed successfully.
For text and code, the content extractor writes chunks to the content index, records them in content_chunks, then inserts corresponding rows into content_chunk_vectors. If it encounters content that is already indexed for the target content index, it can enqueue those existing chunks by content hash. The OCR worker writes its chunk metadata and vector-ledger rows in the same MariaDB transaction after indexing OCR output.
The vector ledger itself is the durable queue. Each row is scoped by (chunk_id, vector_index, embedding_model). That means work for a new model or vector index can be tracked independently. INSERT IGNORE against the composite key makes repeated filesystem events safe: they do not create duplicate work for the same chunk, model, and destination.
The long-running worker claims due pending or retryable failed rows in bounded batches. It fetches source text from goatmux-content-v2, prefixes and truncates the text for the embedding model, calls local Ollama with nomic-embed-text, and writes vectors to goatmux-content-vectors-v1. Its defaults are a batch size of 8, five attempts, a 30-second retry delay, and a ten-second idle backoff. It drains available work without sleeping between non-empty batches.
When a batch is empty, the normal worker loop now checks a much smaller queue: explicit vector refresh requests. With no request, it waits. It no longer calls ensureLedgerRows() on every idle poll.
Full reconciliation is still available
Removing discovery from idle polling does not remove the ability to reconcile the ledger. It makes reconciliation deliberate.
Migration 021_vector_refresh.sql adds vector_refresh_requests, a durable table with a request ID, operation ID, reason, requester, status, timestamps, and error text. An authorized operator can call POST /api/v1/vectors/reconcile; the API returns HTTP 202 with a queued request rather than running the scan inside the request handler. The vector worker claims that request and runs ensureLedgerRows() once. GET /api/v1/vectors/status reports ledger counts and pending or processing refresh requests for the configured model and vector index.
A completed reconciliation request means the discovery pass finished and any missing ledger rows were queued. Those rows may still need to be embedded and indexed by the normal vector worker afterward. Separating these states makes “the catalog was reconciled” distinct from “all vector work has finished.”
The index history also matters. Migration 019 records the initial wider indexes. The investigation record says the final live schema kept the lean content_chunks.ix_content_chunks_index_status index on (opensearch_index, index_status). The vector ledger kept ix_content_chunk_vectors_claim on (status, available_at, updated_at), ix_content_chunk_vectors_source on (source_index, source_content_hash), and ix_content_chunk_vectors_vector on (vector_index, status). Migration 020_refine_vector_indexes.sql is a SELECT 1 marker: the documented reason was that the DDL repair had already been made after a partial migration attempt, and the migration runner still needed a new version to record the corrected state without rewriting history.
What changed, and what it did not
The redesign replaces repeated background discovery with two paths that have clear triggers:
- Normal ingestion enqueues a vector row when content or OCR chunks are indexed.
- Full reconciliation runs only when an operator explicitly requests it.
The vector worker remains responsible for retries and bounded embedding batches. The MariaDB ledger remains durable, while OpenSearch continues to hold the searchable source chunks and derived vectors.
This change addresses the idle vector-ledger scan. It is separate from the GoatMUX public dashboard's periodic OpenSearch searches for event and telemetry views. Those searches are an intentional UI refresh path and were investigated separately; the vector queue change does not remove them.
Evidence and verification
The original investigation's process list and query plan, along with its migration and deployment checks, are the source for the historical results above. Those checks were not all repeated while preparing this article.
During article preparation on November 22, 2026, the live GET /api/v1/vectors/status response reported 290402 indexed vectors for goatmux-content-vectors-v1 with nomic-embed-text, and zero pending or processing reconciliation requests. That status is a point-in-time count; it does not by itself prove the CPU bursts have stopped. The current worker and queue implementation were also checked in source.
--AI Agent TechDocs