GoatMUX Layer 1: The Watcher

A filesystem watcher is only the first step. Here is how GoatMUX turns notifications into catalog updates, queues content work, and uses deliberate scans to recover what the event stream misses.

GoatMUX Layer 1: The Watcher

GoatMUX Layer 1: The Watcher

GoatMUX has a filesystem watcher, a reconciler, and a scanner. They all concern themselves with files, which is how three perfectly sensible components get mistaken for one large magical one.

They have different jobs.

The watcher listens for filesystem notifications. The reconciler checks those notifications against the filesystem as it exists now, updates the catalog, and queues deeper work. The scanner deliberately walks a tree to build or refresh an inventory. The watcher is the doorbell. It is not the filing clerk, and it certainly is not the archive.

That division matters because a notification is evidence that something may have changed. It is not, by itself, a trustworthy description of the present state.

First, decide which doors exist

The watcher gets its roots from configuration. In the current process setup, GOATMUX_WATCH_ROOTS accepts a comma-separated list; if that setting is empty, the watcher falls back to the project tree. A data tree on a separate mount can be configured as another root. The example paths below are illustrative, not machine configuration:

/workspace/goatmux,/archive/goatmux-data

For each root, addTree walks directories and registers them with fsnotify. It does not register individual files. When a new directory is reported, the watcher tries to add watches beneath it too, so later changes inside that directory can be seen.

The watcher skips system trees such as /proc, /sys, and /dev; it also skips hidden path components, .git, and several generated or operational directories, including node_modules, .next, bin, logs, and databases. GoatMUX’s separately cataloged ZFS tree is also kept out of a home-root watch walk; it can be supplied as its own root. The explicit scanner has its own configurable filters, which include hidden directories, dependency and build output, and specific excluded paths. The two systems have related boundaries, but they do not share one universal filter list.

Mount boundaries are not bookkeeping trivia. A path below a mount point may belong to a different filesystem, with its own identity and lifecycle. If a data volume is unmounted, the directory where it used to appear may still exist. Treating that as proof that the data was deleted would be an efficient way to make a catalog confidently wrong.

What the watcher actually sees

fsnotify is the Go library that receives filesystem notifications from the operating system. It reports operations such as create, remove, rename, write, and chmod. GoatMUX maps the operation to an event type, records the path and UTC detection time, then uses Lstat—a metadata check that does not follow symbolic links—to capture the size, mode, device, and inode if the path can still be inspected:

result := catalog.FilesystemEvent{
    EventType: typ,
    Path:      e.Name,
    Detected:  time.Now().UTC(),
}
if info, err := os.Lstat(e.Name); err == nil {
    result.Size = uint64(info.Size())
    result.Mode = uint32(info.Mode())
    // Device and inode are copied from the platform stat data.
}

This is a small observation, not a content analysis. The watcher does not open the file to read it, calculate a content hash, extract metadata, or run OCR. A write notification can arrive while an application is still writing. Several notifications can describe one save. A rename can make one side of the event disappear before GoatMUX examines it. The safe response is to look at the path later, not to pretend the notification is a complete transaction log.

Make the notification durable

The watcher accumulates events and writes them to MariaDB as a batch. The current defaults are 100 events per batch and a 500 ms flush interval. The store writes a batch in a transaction:

batch = append(batch, e)
if len(batch) >= r.config.BatchSize {
    flush()
}

// Also flush on the interval, and once more during shutdown.

The interval means a quiet stream does not need to fill a batch before events are recorded. A separate heartbeat runs every five seconds by default, reporting the watcher’s status, roots, process ID, event counters, last event time, and most recent error. Commands are read from MariaDB: pause stops accepting events into the batch, resume starts again, and reconcile walks the configured roots again to add watches. While paused, the watcher still drains notifications from fsnotify but ignores them, so changes during that interval need a later scan or other observed event to be reflected. reconcile does not reconcile catalog rows. The name is unfortunate if you expect a filing clerk; the implementation is a watch-registration refresh.

The command branch is refreshingly literal:

case "reconcile":
    for _, root := range r.config.Roots {
        if err := r.addTree(fw, root); err != nil {
            commandErr = err.Error()
        }
    }

There is a useful limit to the durability story. If a batch write fails, the watcher logs the error and keeps that batch in memory for another flush attempt. If the database remains unavailable through shutdown, the final flush can fail too; the process then exits, and that in-memory batch is gone. The durable queue protects events after a successful database commit, not before it. This is one reason the event stream cannot replace an inventory scan.

The reconciler checks the present

The reconciler polls MariaDB every two seconds by default and takes up to 50 pending events per pass. It waits at least one second after detection before selecting an event. For each one, it calls Lstat on the recorded path. If the path is present, it catalogs the current object. If it is missing, it finds the deepest eligible mount that currently contains the path and removes the corresponding cataloged subtree from that mount:

info, err := os.Lstat(path)
if err != nil && os.IsNotExist(err) {
    mount, mountErr := fsinventory.MountForPath(path)
    if mountErr != nil {
        return "error", "identify active mount for missing path"
    }
    _, removeErr := r.store.RemoveMissingPath(ctx, path, mount.MountPoint)
    // The event is marked removed only after this succeeds.
}

Mount-scoped removal matters for directory deletes. It prevents removal from crossing into an independently mounted filesystem’s catalog. A missing mount root is left alone: it may mean the filesystem was unmounted, not that every item on it was deleted. If a path is merely inaccessible for some other reason, the event is marked as an error rather than treated as a deletion.

For a present path, CatalogPath determines the owning mount, uses Lstat rather than following symlinks, and updates the path and object records. Directories update the directory tree. Regular files are cataloged first; then the reconciler looks up the stored identities, enqueues content work, and runs configured metadata extraction:

if err := fsinventory.CatalogPath(ctx, r.store, r.scanRun, path); err != nil {
    return "error", fmt.Sprintf("catalog: %v", err)
}
if r.config.ProcessContent {
    if err := r.store.EnqueueContentJob(ctx, pathHash, objectKey, mountID, path); err != nil {
        return "error", err.Error()
    }
}

The order is intentional. Cataloging a large file should not hold up the event loop while its contents are extracted. The durable content queue coalesces repeated work for a path and a separate worker does the slower extraction and indexing. In the current reconciler process configuration, content and metadata processing are enabled; metadata extraction runs in the reconciler, while content extraction is queued.

The reconciler processes the path in the event, not a perfectly reconstructed rename operation. In the current event mapper, the old path is not populated. A rename may therefore be handled as “this path exists now” or “this path is gone now,” depending on the path fsnotify delivered and what the filesystem says when the reconciler gets there. The scanner and a later event can fill in more of the picture; the event record itself is not a complete rename journal.

A path is not the same thing as a file

GoatMUX keeps a record for the filesystem object and another for each path that points to it. The path key is a SHA-256 digest of the full path. On Linux, the object identity key is derived from filesystem identity plus device major/minor and inode. If the inode is zero, GoatMUX falls back to a key derived from the path.

That distinction handles two ordinary cases cleanly. A hard-linked file can have multiple paths but one underlying object identity. A rename changes the path identity while, on the same filesystem, the device and inode (the filesystem’s number for an object) can continue to identify the same thing. A move across filesystems may produce a different object key, because device or filesystem identity changed. None of these keys is a content hash: hashing file contents is a later content-processing step.

In other words, GoatMUX can say, “This name lived here,” separately from, “This filesystem object was observed.” That makes history and relationships less dependent on a filename behaving like a permanent identity card. Filenames are terrible identity cards. They keep changing their names.

The scanner is the deliberate inventory pass

A watcher only sees events delivered while its watches are active. It can be stopped, miss events during downtime, or encounter queue and database failures. The explicit scanner exists for a different job: walk a selected tree, read filesystem metadata, and upsert catalog records in batches. It does not read file contents or calculate content hashes.

Scanner roots must remain inside GoatMUX’s allowed catalog boundary and on an eligible mounted filesystem. The walk applies that mount’s filter profile and prunes nested mount points; a Linux mount-ID check also catches a mount boundary that changes during traversal. Here is the heart of the walk’s boundary logic:

if info.IsDir() {
    if skip, reason := job.Filter.Skip(clean, job.Mount); skip {
        options.Logger.Printf("scan_run=%d mount=%q filter_skip path=%q reason=%q", options.RunID, job.Mount.Name, clean, reason)
        return filepath.SkipDir
    }
}
if clean != job.Root {
    if _, isMountBoundary := boundarySet[clean]; isMountBoundary {
        if info != nil && info.IsDir() {
            return filepath.SkipDir
        }
        return nil
    }
}

In broad terms, that is how a scan can cover an archive mount without wandering through another mounted filesystem as if it were an ordinary subdirectory.

One subtlety: a normal scan updates what it sees but does not infer that every unseen catalog row is stale. A full pass is not automatically a deletion sweep. The separate prune-missing --root … command compares cataloged paths beneath an existing directory with Lstat, removes missing catalog rows in that subtree, and refuses to prune a mount root. It repairs missed deletions within the selected scope; it does not delete source files, and it is not a global “remove everything not seen” switch.

Recovery is layered, not magical

When a file disappears after the event reaches MariaDB, the reconciler can remove the missing path’s catalog entries. If an event is missed, a later scan can refresh records for paths it sees, while a scoped prune can check for stale rows. When a directory is created, the watcher attempts to register watches below it before it records the directory event, so later activity can be noticed. When a database write fails, the in-memory event batch is retried on a later flush; once the process ends, any still-uncommitted events are not durable.

Those are distinct recovery mechanisms: delivered-event processing, explicit inventory, and scoped stale-path pruning. Reconciler failures are recorded on the event; rows marked error are not selected again by its pending-event query. The separate content queue does retry its own failed work, but that is not a retry policy for every reconciliation failure. A scan or a deliberate follow-up operation may repair catalog state, depending on what failed.

Together these mechanisms improve the odds that the catalog catches up. They do not make the filesystem event stream infallible, and they do not claim to know about changes nobody observed or rescanned.

That is a more modest promise than “watch everything.” It is also a more useful one. GoatMUX has to choose what to watch, where to scan, and which identities to preserve. The machinery records evidence inside those boundaries. Knowing everything is a fine ambition for a machine with no budget and no weekends. The rest of us need to know what matters, then make sure we can find it again.