Building a Governed Gmail Pipeline for GoatMUX
A practical guide to Gmail OAuth setup, private message ingestion, isolated search, and vector processing in GoatMUX.
Scope
This guide describes the implemented local Gmail connector: Google OAuth setup, message retrieval, MIME normalization, isolated OpenSearch and MariaDB storage, vector generation, incremental polling, and Observer-only search. It is intended to help an operator reproduce the setup while preserving a boundary around private mail.
The implementation inspected here is configured for one authorized test account. The account address and OAuth file contents are omitted. No second account is established by this evidence.
Current behavior or problem
GoatMUX can retrieve Gmail through Google's read-only Gmail API and make normalized message text available to local lexical and semantic search. Gmail records and vectors use dedicated indexes and MariaDB tables. The standard technical content ledger and vector indexes are not used for Gmail.
The connector supports an initial full sync and later syncs based on Gmail historyId. A PM2 process runs one incremental pass and exits; PM2 relaunches it after a ten-minute delay. A separate long-running vector worker drains queued Gmail chunks. Search access is isolated to the read-only research listener and the Observer actor.
Investigation and evidence
Observed in checked-out source: backend/cmd/goatmux-gmail/main.go sets the read-only scope, source and vector index names, default OAuth file paths, and the sync behavior described below. Migration 022_gmail_content.sql defines three Gmail-specific tables. ecosystem.config.cjs defines the recurring poller and queue worker. API registration and handlers in backend/internal/api/server.go and backend/internal/api/gmail.go confine Gmail search routes to the read-only listener and require X-GoatMUX-Actor: Observer.
Observed in live checks for this documentation pass: the normal GoatMUX health route returned HTTP 200. The normal API's vector status route returned a healthy status for the default technical vector index, not the Gmail index. The read-only listener requires a bearer token; TechDocs did not have one and did not access private message content or query Gmail indexes. The local ComfyUI history endpoint reports successful generation of the included image using the workflow and prompt saved beside this article.
Previously recorded implementation results: a historical implementation snapshot reports 571 message documents, 568 chunks, and 568 indexed vector rows; it also lists zero pending or failed rows, model nomic-embed-text, 768 dimensions, and history cursor 2909757 with status idle. A separate Observer-authored operation report also states that 571 messages were in the index. These are historical, agent-reported results, not a fresh TechDocs query. An authorized operator should recheck live counts before treating them as current.
Inferred: separate indexes, tables, and route-level actor checks reduce accidental exposure of mail through ordinary technical search. They do not encrypt the indexed message text or replace host-level access controls.
Implementation details
1. Configure Google OAuth
Use a Google Cloud project dedicated to this integration or one whose ownership and access controls are understood. The Google Cloud Console links below open the relevant setup areas after you select the correct project:
- Enable the Gmail API.
- Configure the OAuth application in Google Auth Platform: Branding and Audience. For an external app in Testing, add each account that is explicitly authorized to the test-user list. An internal app is for users in the owning Workspace organization.
- In Google Auth Platform: Clients, create an OAuth client of type Desktop app and download its client JSON.
- Complete user consent for the specific mailbox and keep the resulting authorized-user token JSON in a protected local secrets directory. The current connector expects both the client JSON and a token JSON to exist; it reads them but does not provide an interactive authorization route or token-bootstrap command. Google's Go Gmail quickstart demonstrates the installed-app consent flow. Google's Gmail scope reference classifies
gmail.readonlyas a restricted scope; review the applicable verification requirements before distributing an application or serving additional users.
In Data Access, declare only the Gmail scope required by the connector:
https://www.googleapis.com/auth/gmail.readonlyIn the inspected deployment, both files are stored under a protected local secrets directory outside the repository. Keep the exact paths and contents private; never commit either file, include its contents in logs, or place OAuth material in MariaDB, OpenSearch, operation events, or this article. For another installation, pass that operator's own protected paths to --client and --token.
The connector's scope and defaults are defined in Go:
const (
sourceIndex = "goatmux-gmail-messages-v1"
vectorIndex = "goatmux-gmail-vectors-v1"
ledgerTable = "gmail_content_chunk_vectors"
chunkTable = "gmail_content_chunks"
modelDefault = "nomic-embed-text"
scope = "https://www.googleapis.com/auth/gmail.readonly"
)2. Retrieve and normalize messages
goatmux-gmail uses the provider's message ID as the stable OpenSearch document ID. A message document contains its account alias, message/thread/history IDs, internal date, labels, headers, sender/recipient/subject, snippet, normalized text, optional HTML, content ID, chunk fields, MIME/content kind, estimated size, and indexing time. It does not contain OAuth credentials.
The MIME walker recursively visits message parts. It decodes Gmail base64url data in raw and padded forms, collecting text/plain and text/html. If plaintext is absent but HTML exists, stripHTML derives searchable text from the HTML. Empty normalized bodies remain represented as message metadata in the source index but do not enqueue a content chunk or vector.
Initial synchronization lists messages in pages of up to 500, fetches each full message, indexes it, and stores a history cursor. Incremental synchronization reads Gmail history from the saved cursor and requests the valid history types:
HistoryTypes("messageAdded", "labelAdded", "labelRemoved")The implementation records only the detail-free event text Gmail Retrieved for incremental additions. It does not put a message subject, sender, body, or ID in that event. Quota errors during retrieval use bounded retry backoff. A history cursor that Gmail no longer accepts is marked stale so that a full sync can be requested again.
An earlier implementation used invalid history-type names (messageLabelAdded and messageLabelRemoved) and received Gmail HTTP 400 responses. The current source uses messageAdded, labelAdded, and labelRemoved. Historical log entries may still contain the earlier errors; later incremental syncs are the relevant evidence for the corrected behavior.
3. Store chunks and queue vectors
Migration backend/internal/catalog/migrations/022_gmail_content.sql creates these separate structures:
| Table | Role | Keys and indexes |
|---|---|---|
gmail_sync_state |
Per-account history cursor, sync status, last error, and update time | Account primary key; ix_gmail_sync_updated on updated_at |
gmail_content_chunks |
One normalized body chunk per message in this implementation | chunk_id primary key; unique (message_id, chunk_index); content-hash and index/status indexes |
gmail_content_chunk_vectors |
Vector queue, processing state, retries, and indexing ledger | Primary (chunk_id, vector_index, embedding_model); claim, source, and vector-status indexes |
These tables are separate from content_chunks and content_chunk_vectors. The Gmail importer creates a vector row only for a non-empty normalized body. It starts with zero dimensions until the worker stores a successful embedding:
INSERT IGNORE INTO gmail_content_chunk_vectors
(chunk_id, source_content_hash, source_index, vector_index,
embedding_model, embedding_dimensions, vector_id, status, attempts,
available_at, created_at, updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?, 'pending', 0,
UTC_TIMESTAMP(6), UTC_TIMESTAMP(6), UTC_TIMESTAMP(6));4. Build vectors in a separate worker
The goatmux-gmail-vector-worker PM2 process runs the shared Go vector worker with Gmail-specific source index, vector index, chunk table, and ledger table. The worker validates configured table names before using them. It claims rows transactionally, marks work in progress, retries failures up to the configured limit, embeds body text with Ollama, bulk-indexes vectors, and records dimensions and timestamps. It drains batches continuously while work exists; the ten-second poll interval is idle backoff.
The Gmail vector index is KNN-enabled with a 768-dimensional embedding field using Lucene HNSW and cosine similarity (m=16, ef_construction=128):
"embedding": {
"type": "knn_vector",
"dimension": 768,
"method": {
"engine": "lucene",
"space_type": "cosinesimil",
"name": "hnsw",
"parameters": {"ef_construction": 128, "m": 16}
}
}5. Configure the recurring processes
The current poller configuration performs one incremental check and asks PM2 to restart it after ten minutes:
{
name: "goatmux-gmail-sync",
script: "./bin/goatmux-gmail",
args: "--account <authorized-account> --backfill=false --incremental=true",
autorestart: true,
restart_delay: 600000
}The vector worker uses the following Gmail-specific settings:
GOATMUX_OPENSEARCH_CONTENT_INDEX=goatmux-gmail-messages-v1
GOATMUX_OPENSEARCH_VECTOR_INDEX=goatmux-gmail-vectors-v1
GOATMUX_VECTOR_CHUNK_TABLE=gmail_content_chunks
GOATMUX_VECTOR_LEDGER_TABLE=gmail_content_chunk_vectors
GOATMUX_OLLAMA_URL=http://127.0.0.1:11434
GOATMUX_EMBEDDING_MODEL=nomic-embed-text
GOATMUX_VECTOR_BATCH_SIZE=8
GOATMUX_VECTOR_POLL_SECONDS=10Configuration caveat: the checked-in PM2 file also sets GOATMUX_GMAIL_CLIENT_FILE and GOATMUX_GMAIL_TOKEN_FILE, but main.go currently obtains OAuth paths from the --client and --token flags, whose defaults are fixed paths. The default paths happen to match this installation's PM2 values. For a different path, pass the flags in the process arguments; the environment variables alone are not read by the connector today.
Tables, indexes, queues, and data flow
| Data | Storage | Purpose |
|---|---|---|
| Message metadata, normalized body, optional HTML | goatmux-gmail-messages-v1 |
Private lexical search and message retrieval |
| Chunk identity and source index mapping | gmail_content_chunks |
Relational record for non-empty message body text |
| Pending/processing/indexed vector state | gmail_content_chunk_vectors |
Durable vector work queue and status ledger |
| Embeddings and searchable text chunk | goatmux-gmail-vectors-v1 |
Private semantic retrieval |
| Sync cursor and status | gmail_sync_state |
Resume incremental retrieval from Gmail historyId |
The message mapping stores HTML as a non-indexed field where configured. Labels use keyword values. The vector mapping includes message/content identity, chunk metadata, MIME/content kind, normalized body text, and model name; the embedding array is excluded from search responses.
Commands and API calls
Build and run
The implementation record lists these commands as run during development; they were not rerun as part of this documentation pass. /path/to/GoatMUX/backend is a portable placeholder for the actual backend directory:
cd /path/to/GoatMUX/backend
go build -o bin/goatmux-vector-extractor ./cmd/goatmux-vector-extractor
go test ./...
go build -o bin/goatmux-api ./cmd/goatmux-api
pm2 restart goatmux-api --update-envTo run the Gmail connector manually, use the backend working directory and supply the authorized account and protected credential paths. GOATMUX_DB_DSN and the configured OpenSearch URL must be available in the process environment. A normal full sync writes imported mail to local storage, so run it only after the mailbox owner has explicitly authorized that account:
cd /path/to/GoatMUX/backend
./bin/goatmux-gmail \
--account '<authorized-account>' \
--client '/secure/path/gmail-oauth-client.json' \
--token '/secure/path/gmail-authorized-user.json' \
--backfill=true --incremental=false--max-messages bounds a run, but a limited first backfill saves a cursor based on the messages it processed. If you use it to validate a new setup, rerun with --force-backfill=true and no message limit to complete the mailbox backfill. Do not mistake the bounded run for a complete import.
Search access
GET /api/gmail/messages on the local admin API provides paginated lexical search. On the read-only research listener, the same route is restricted to the Observer actor and requires the research bearer token. Semantic search uses GET /api/gmail/vector-search?q=...&limit=..., also restricted to Observer; it embeds the query and searches only goatmux-gmail-vectors-v1. The response includes message IDs, scores, body chunks, content metadata, and model name, but not embedding vectors. A non-Observer actor is rejected with HTTP 403.
Example for an authorized Observer environment:
curl -H 'Authorization: Bearer <observer-research-token>' \
-H 'X-GoatMUX-Actor: Observer' \
'http://127.0.0.1:8791/api/gmail/vector-search?q=professional%20networking&limit=3'Use /api/gmail/messages to retrieve message details after semantic discovery. Do not put mailbox content or credentials in operation events.
Verification
The assignment's historical verification snapshot reported 571 message documents, 568 chunks, and 568 indexed vectors, with no pending or failed vector rows. It recorded nomic-embed-text, 768 dimensions, and sync cursor 2909757 in idle state. These values are not presented as current live counts: TechDocs did not query the private Gmail indexes during this pass because the available research access is Observer-only.
For an authorized operator, the source implementation identifies the relevant indexes and tables. Use the documented GoatMUX APIs available to the authorized actor to check current message and vector totals, and use the process supervisor to confirm that the incremental poller and vector worker are running. The normal API's current /api/v1/vectors/status endpoint reports the general content vector ledger by default; it did not return Gmail-specific counts in this check.
The implementation record also reports that go test ./..., the vector extractor build, and the API build passed before the API restart. Those are historical implementation results, not tests rerun for this article.
Limitations and follow-up work
- The connector requires a pre-existing authorized-user token JSON. It does not implement the interactive OAuth consent/bootstrap flow; use a separate local OAuth bootstrap based on Google's installed-app flow.
- The PM2 OAuth path environment variables are currently ignored by the command;
--clientand--tokenflags or the fixed defaults determine the actual paths. Strict Developer should align the implementation and PM2 config before treating those environment variables as configurable inputs. - Only the configured test account is supported by the observed deployment. A second account is not verified.
- Attachments are outside this message-body vector pipeline.
- Gmail
readonlyis a restricted OAuth scope. External use may require Google's review and policy steps; verify current requirements for the intended deployment.
--AI Agent TechDocs