A friend recently suggested I build an AI agent loaded with vendor documentation, troubleshooting guides, and technical information. He also asked what kind of hardware I use to run local language models.

Well, I've actually been building something along those lines for quite some time. Except I approached it a little differently.

I built the data infrastructure first. AI came later.

The result is Helix, an on-premises operational intelligence platform I developed for our telecommunications network.

And although AI has become an important part of it, the real machinery underneath is traditional database and software engineering.

Helix: The Architecture

At its core, Helix is a hybrid database environment built around MariaDB, OpenSearch, Redis, and a substantial amount of Go code.

Each component serves a different purpose.

MariaDB is the relational foundation.

The current system contains roughly 98 tables maintaining relationships among customers, equipment, services, addresses, network infrastructure, and other operational information.

Data comes from provisioning systems, customer management platforms, equipment APIs, and various network databases.

MariaDB maintains the structured relationships and operational state that connect all these otherwise independent systems.

OpenSearch is the analytical and knowledge platform.

It contains billions of operational records collected from network equipment, cable modems, optical networks, telephone switches, SIP trunks, system logs, alarms, and call detail records.

It also holds our entire TSC knowledge base, historical trouble tickets, and practically every piece of technical documentation I've been able to locate from our equipment vendors.

That information is indexed for traditional searches, full-text searches, and vector-based semantic retrieval.

This is an important distinction. I don't depend exclusively on vector searches.

Sometimes I want to find a specific telephone number, MAC address, equipment identifier, or error message. Other times I'm looking for a concept buried somewhere in hundreds of pages of documentation.

OpenSearch handles both, along with the time-series analytics needed to understand network behavior.

Go is the glue holding everything together.

I've written numerous Go services that communicate with equipment APIs, poll network devices, synchronize databases, parse enormous log files, and correlate records from systems that were never designed to communicate with one another.

These services continuously enrich the data using authoritative information from MariaDB and analytical information from OpenSearch.

Much of the real intelligence in Helix comes from this correlation.

Instead of forcing someone to manually connect customer records, equipment information, network statistics, and historical events, the software establishes those relationships ahead of time.

Redis provides fast temporary state and caching where needed.

The result is a continuously updated operational environment rather than a collection of disconnected dashboards.

Where AI Fits In

Codex sits on top of this infrastructure.

It can search the technical knowledge base, examine operational history, query the databases, inspect code, and investigate problems using information collected directly from our network.

It isn't simply answering questions based on documents.

It has access to structured relationships, historical telemetry, equipment information, and operational tools.

The important part is that the AI doesn't have to maintain all that knowledge internally.

The information lives in the databases.

The model retrieves what it needs, interprets the results, and helps with the investigation.

I use OpenAI's models extensively, but Helix isn't dependent on any particular language model.

I can change models without rebuilding the underlying data infrastructure.

The databases hold the knowledge. Go maintains the relationships. AI provides another way to interact with it.

The Hardware Running Helix

One misconception about systems like this is that they require an enormous GPU cluster.

Helix doesn't.

The core infrastructure runs on three refurbished Xeon servers under Proxmox.

Each server has:

• 28 CPU cores
• 512 GB RAM
• 16 TB NVMe storage
• 16 TB additional disk storage

The servers communicate over 10-gigabit networking, with a 10-gigabit fiber uplink.

The hardware investment was approximately $15,000.

That cluster handles the OpenSearch indexes, MariaDB environment, Redis, Go services, and other supporting applications.

We're processing billions of records, polling thousands of network devices, and continuously ingesting operational information.

Most of that work requires no AI whatsoever.

A database query doesn't need a language model. Neither does parsing a log file or polling a modem.

Conventional software handles those jobs faster and more efficiently.

AI comes into play when we need flexible investigation, interpretation, or natural-language interaction.

My Home Lab: A Separate Environment

My home laboratory uses a similar philosophy, but it is completely independent of Helix and contains no TSC proprietary information.

I've built a personal technical knowledge environment containing approximately 800 technical books, along with a large collection of documentation and reference material gathered from the internet.

That knowledge is searchable using OpenSearch, including full-text indexing and vector embeddings.

I use Codex and OpenAI models to help develop and expand the environment, but I also experiment with locally hosted language models.

My main workstation is a fairly substantial Linux machine.

The 256 GB of system RAM allows me to experiment with larger quantized models that don't fit entirely in GPU memory, although inference speed depends heavily on how much can be accelerated.

I've tested several model families, but Mistral Nemo 12B has been one of my regular choices.

Like Helix, the home environment keeps its knowledge separate from the language model itself.

I'm not retraining foundation models every time I add another technical book or document.

I index the material, generate embeddings, and retrieve the relevant information when needed.

OpenAI and Codex help me build and refine the software, while local models can use the same underlying knowledge without requiring every request to go through a hosted service.

The Point of All This

I didn't set out to build a chatbot that happened to know a few things about telecommunications.

I wanted an infrastructure capable of collecting information, maintaining relationships, and making that information useful.

MariaDB handles the structured facts.

OpenSearch handles the analytical history, full-text knowledge, and vector searches.

Go keeps the information flowing and connects the pieces.

Redis handles fast state and caching.

And AI sits above it all, using the information those systems provide.

The same philosophy applies to my home laboratory, just with different data and considerably different hardware.

The most important part of an AI system isn't necessarily the model. It's the quality, accessibility, and organization of the information the model can use.

Models will continue improving. Some will disappear, and better ones will replace them.

But the databases, documentation, relationships, and software I've built will still be there.

That's why I built the infrastructure first.

Bryan

That most told us could not be done.

For years, the BroadSoft XS logs occupied an awkward place in our network.

They contained some of the most valuable diagnostic information available from the platform, but they were so large, complex, and difficult to manage that nobody treated them as a practical operational data source.

The logs were there.

The information was there.

Actually turning the entire logset into something continuously searchable was another matter.

Even our vendor told us that processing the full XS logset was not realistically possible.

With the help of Codex, we did it anyway.

That does not mean the underlying problem was beyond my ability to solve without AI. At its core, parsing these files is not radically different from handling any other large multiline log format. The real difficulty was the scale, the number of distinct record types, the correlation between different log sets, and the amount of supporting pipeline work required to make the process continuous, reliable, and resumable.

I could have built this without Codex. The difference is that it likely would have taken weeks of development, testing, and refactoring instead of a handful of focused hours spread across a few days. Codex did not replace the engineering judgment. It accelerated the path from design decisions to working code.

Today, Helix continuously retrieves BroadSoft XS files, verifies that they are safe to process, parses their multiline records, extracts structured operational data, writes the results into multiple OpenSearch index families, updates endpoint state in MariaDB, compresses the original logs for retention, and remembers exactly where it left off.

The pipeline is not technically real time because it does not tail the active XS files while BroadSoft is still writing them.

Operationally, however, it is just-in-time.

Under normal conditions, data becomes searchable in Helix approximately one to two minutes after BroadSoft finishes writing the source file.

That is close enough to real time for active troubleshooting, without taking the risk of parsing incomplete records from a file that is still changing.

Why BroadSoft XS Logs Are So Difficult

The first problem is scale.

BroadSoft XS logs are large rolling text files containing a mixture of:

• SIP requests and responses
• Registration traffic
• Subscription activity
• XSI and EventNotification data
• Internal BroadSoft subsystem events
• XML payloads
• Session and call-leg identifiers
• Endpoint and enterprise information
• Diagnostic messages generated by different platform components

A single file may contain an enormous number of records, and those records do not all share one simple format.

Calling them “text logs” understates the problem.

They are closer to a serialized stream of loosely related protocol events, application events, XML fragments, SIP messages, internal state changes, and diagnostic context. All of that is interleaved inside rolling files that were designed primarily for vendor support and forensic inspection, not modern structured analytics.

The second problem is that the records are multiline.

You cannot reliably process an XS file by treating each line as an independent event.

A record begins with a recognizable header containing a timestamp and pipe-delimited fields. The lines that follow may include SIP headers, SDP, XML, BroadSoft-specific diagnostic data, or other payload content. That body continues until the next valid record header appears.

The parser therefore has to maintain state.

It must:

1. Read the file line by line.
2. Recognize the beginning of a legitimate XS record.
3. Accumulate every following line into the current record body.
4. Detect the next record boundary.
5. Flush the completed record.
6. Parse and classify the assembled content.
7. Repeat that process without losing offsets or corrupting multiline payloads.

A simple regular expression over individual lines is not enough.

A conventional log shipper configured around newline-delimited events is not enough.

Even after a record has been assembled correctly, the system still has to determine what that record represents.

Is it a SIP REGISTER?

A successful 200 OK response to a registration?

A SUBSCRIBE request?

An XSI EventNotification snapshot?

A call-processing event?

An internal BroadSoft message?

A SIP dialog fragment that must be correlated with another record through a shared callhalf, call leg, session identifier, or endpoint context?

The parser has to extract enough structure to make the data useful without destroying the original forensic payload.

That balance is important.

If we only stored the raw record, we would have a searchable pile of text but limited analytical value.

If we extracted only a handful of normalized fields, we could lose the obscure diagnostic detail needed during the worst incidents.

Helix keeps both.

Each parsed record can include:

• The event timestamp
• Source host
• Source path
• Source byte offset
• Parsed XS header fields
• Body classification
• SIP method
• SIP response status
• Event class
• Callhalf or call-leg identifiers
• Session keys
• Endpoint data
• Enterprise information
• XML content
• SIP forensic payloads
• BroadSoft internal payloads
• Search tokens derived from the record

The result is structured enough for analytics but detailed enough for forensic reconstruction.

The Logs Are Rolling Files, Not a Message Stream

Another major complication is how the files are produced.

BroadSoft writes rolling XS text files into a remote directory. The files are accessed over SFTP. They are not delivered through a streaming interface, message queue, Kafka topic, or API designed for incremental consumption.

That means the ingestion system has to infer when a file is complete.

Downloading a file simply because it exists is dangerous. BroadSoft may still be writing to it.

If the streamer grabs a file too early, several things can happen:

• The final record may be incomplete.
• A SIP message may be truncated.
• An XML payload may be cut in half.
• The same file may later appear with a different size.
• The system may incorrectly mark partial data as processed.
• Reprocessing logic may become much more complicated.

We deliberately chose not to tail the active remote file in place.

Instead, the streamer waits for a file to settle.

A file is considered stable only when:

1. It is older than the configured minimum age.
2. Its size matches what was observed during a previous poll.
3. Its modification time also matches the previous observation.

If either the size or modification time changes, the file remains unstable.

This introduces a small delay, but it protects the integrity of the data.

In practice, the delay is usually only about one to two minutes. Once BroadSoft closes or stops updating a file, the streamer confirms that it has stabilized, downloads it, and begins processing.

That is why I describe the system as just-in-time rather than real time.

It does not inspect bytes while BroadSoft is still writing them. It processes the completed file almost immediately afterward.

For actual troubleshooting, that distinction rarely matters. By the time an engineer begins investigating a failed call, the relevant events are generally already searchable in Helix.

The Original Workflow Was Not Sustainable

Before the streamer, processing XS logs was a manual workflow.

Someone had to:

1. Locate the relevant XS files.
2. Download them.
3. Run a separate processor.
4. Compress or archive the source files.
5. Track what had already been processed.
6. Avoid accidentally processing the same history again.

That could work for a targeted investigation, but it could not create a continuously available operational dataset.

The full pipeline needed to be automatic, resumable, and safe to leave running unattended.

It also needed to survive restarts.

A daemon that forgets its position every time it restarts is not an ingestion pipeline. It is a recurring data incident.

Giving the Streamer Durable Memory

The streamer uses a MariaDB table named broadsoft_xs_sftp_files as durable pipeline state.

For each remote file, it can track:

• Remote filename
• Remote path
• Remote size
• Remote modification time
• Fingerprint identity
• Whether it was selected as the startup anchor
• Stability state
• Download status
• Archive status
• Processing status
• Local incoming path
• Local archive path
• First-seen timestamp
• Last-seen timestamp
• Download timestamp
• Archive timestamp
• Processing timestamp
• Failure stage
• Last error

This table is the memory of the pipeline.

It allows the daemon to restart without guessing which files were completed, which files were only downloaded, which archives were successfully written, or which processing stage failed.

The streamer also maintains a watermark.

The watermark represents the newest remote file that the system has accepted as part of its processed history. A file can qualify if it has been marked either processed or intentionally skipped.

That distinction matters during the first deployment.

If the remote BroadSoft directory contains months or years of XS history, starting the daemon should not automatically unleash a historical ingestion avalanche.

On its first run, when no watermark exists, the streamer finds the newest stable remote file, marks it as the startup anchor, marks it as skipped, and begins processing files that arrive after that point.

Historical backfill can still be performed deliberately, but the production daemon starts at the current edge.

Following the Frontier Instead of Rewalking History

The remote XS directory may contain a large number of old files.

The streamer still has to perform an SFTP directory listing so it can discover new arrivals. That operation remains one of the unavoidable costs in the design.

The initial version of the streamer also synchronized every matching remote file into MariaDB during every polling cycle.

That worked, but it was wasteful.

Every cycle repeatedly touched files that were already far behind the watermark. The system performed SQL lookups and updates for history that could no longer affect the next processing decision.

The optimized design works on a frontier.

After listing and sorting the remote files, the daemon focuses only on:

• Files newer than the current watermark
• Or, during first-run initialization, a small slice of the newest remote files

This changes the problem from “reconcile the entire directory every twenty seconds” to “inspect the moving edge of the stream.”

That is a much smaller and more stable workload.

The database access pattern was also reduced.

The older behavior effectively performed:

1. A SELECT for one fingerprint.
2. An INSERT or UPDATE.
3. Another SELECT.

That sequence happened for each file during each synchronization pass.

The revised approach performs one batched read of the existing tracking rows for the active frontier and then one upsert for each relevant file.

The difference may sound small when looking at one file.

Across a large remote directory and a continuously running daemon, it is substantial.

Parse Once, Archive Once

One of the largest performance improvements came from changing how downloaded files were parsed and archived.

The old processing path looked roughly like this:

1. Download the plain-text XS file.
2. Compress the file into a .zst archive.
3. Open the compressed archive.
4. Decompress it.
5. Parse the decompressed content.

That meant the same data was being read and transformed multiple times immediately after download.

The new streamer opens the downloaded text file once.

That single byte stream is simultaneously:

• Fed into the XS parser
• Written through a Zstandard compression writer
• Stored as the retained .zst archive

The file is parsed and archived in one pass.

After successful parsing, indexing, database updates, and archive finalization, the temporary uncompressed download is deleted.

This eliminates an entire compress, reopen, decompress, and parse cycle.

It also aligns the archive with the exact byte stream that was processed.

Routing the Full Logset Into OpenSearch

Running the streamer with --fullprocess enables complete raw XS ingestion.

Each assembled record is classified and routed into one of several OpenSearch index families:

• helix-broadsoft-xs-registrations-* for SIP REGISTER
• helix-broadsoft-xs-subscriptions-* for SIP SUBSCRIBE
• helix-broadsoft-xs-sip-traffic-* for other SIP messages
• helix-broadsoft-xs-broadsoft-internal-* for internal or non-SIP records

This separation is important.

Dumping everything into one enormous index would simplify the first hour of development and complicate nearly everything afterward.

Different record families have different fields, query patterns, retention requirements, and operational purposes.

Registration analysis is not the same workload as reconstructing a SIP call.

Subscription-state research is not the same workload as searching internal BroadSoft events.

Routing records into purpose-built index families creates cleaner mappings and makes it easier to manage growth over time.

OpenSearch writes are performed through a bulk writer.

The default batch size was increased to 1,500 documents, reducing the number of HTTP requests and lowering per-document indexing overhead.

That value still has to be watched alongside cluster health, node pressure, shard behavior, and indexing latency. There is no universally perfect bulk size.

For our environment, however, the larger batch significantly improves throughput compared with the earlier smaller default.

Deterministic IDs Prevent Duplicate History

Every raw OpenSearch document receives a deterministic ID based on the remote source path and record offset.

That means a specific record in a specific XS file always resolves to the same document ID.

If a file is reprocessed, Helix overwrites the same logical documents instead of generating a second copy of every event.

This is a critical property for a resumable pipeline.

Without deterministic IDs, retrying a failed or uncertain ingestion could duplicate millions of records.

Deduplication after the fact would be expensive, unreliable, and unpleasant in all the ways only distributed data cleanup can be unpleasant.

With deterministic IDs, reprocessing becomes idempotent at the document level.

Extracting More Than Raw SIP Traffic

The streamer does not stop at indexing raw records.

It also creates higher-level operational state.

Subscription-State Snapshots

Records generated by the BroadSoft EventNotification subsystem may contain XSI subscription-state information.

The streamer normalizes these records and writes snapshots into:

helix-broadsoft-subscription-state

Snapshots are deduplicated using a hash derived from the subscription key and normalized payload.

If BroadSoft emits the same logical state repeatedly, Helix does not need to create a new document every time.

That makes the index useful as a state history rather than a warehouse of identical repeats.

Endpoint Registration State

The parser also correlates BroadSoft registration context with successful SIP registration responses.

It ties together:

• Registration candidate records
• Successful SIP 200 REGISTER responses
• Shared callhalf or session relationships
• Enterprise context
• Normalized endpoint identity

The resulting state is upserted into the MariaDB table:

bs_enterprise_endpoints

This gives Helix a durable SQL view of endpoint registration activity by enterprise and endpoint.

The XS logs are therefore serving two purposes at once.

They remain available as deep forensic evidence in OpenSearch, while also feeding normalized operational state into SQL.

Failure Handling Is Part of the Architecture

A pipeline that only works when everything succeeds is a demo.

The streamer tracks the major stages of each file independently:

• Download
• Archive
• Process

If a stage fails, the system records where it failed and stores the latest error.

That makes failures inspectable and recoverable.

An SFTP problem is different from a compression problem.

A malformed record is different from an OpenSearch bulk failure.

A MariaDB error is different from a local filesystem issue.

Keeping stage-specific state prevents every problem from collapsing into a generic “file failed” status.

It also allows us to resume work without losing the identity and history of the affected file.

What Codex Actually Contributed

It would be easy to summarize this project by saying that AI wrote a parser.

That would be inaccurate, and it would undersell both the engineering problem and the value of the tool.

Codex did not arrive with an understanding of our BroadSoft environment, Helix, our operational needs, our OpenSearch architecture, or the failure modes we had already encountered.

It did not make the architectural decisions on its own.

What it provided was leverage.

It helped us:

• Explore parser designs
• Refactor working but inefficient approaches
• Compare data-flow options
• Identify repeated I/O
• Move full processing into the streamer
• Develop the tracking and watermark logic
• Tighten SQL access patterns
• Build deterministic indexing behavior
• Iterate on failure handling
• Turn operational observations into code changes quickly

The expertise still mattered.

Someone had to understand why partial remote files were dangerous.

Someone had to recognize which BroadSoft records were related.

Someone had to decide what should live in OpenSearch, what should live in MariaDB, and what needed to remain in the forensic payload.

Someone had to evaluate whether the output was operationally correct.

AI did not replace that judgment.

It allowed that judgment to be applied across a much larger amount of code and a much faster sequence of experiments.

That is the part of AI-assisted engineering that is often missed.

The real advantage is not asking a model to build a mysterious black box.

It is using the model to shorten the distance between an engineering observation and a testable implementation.

The Result

The simplest mental model for the final system is this:

It watches the moving edge of the BroadSoft XS files, waits until a new file settles, downloads it once, parses it once, archives it once, writes the Helix outputs, and remembers where it left off.

The production service normally runs with options equivalent to:

/opt/helix/bin/xs-logs-streamer \
  --config /opt/helix/config/config.yml \
  --incoming-dir /opt/helix/bw-xs-logs/incoming \
  --archive-dir /opt/helix/bw-xs-logs/archive \
  --log-file /opt/helix/logs/xs-logs-streamer.log \
  --poll-interval 20s \
  --stable-min-age 2m \
  --fullprocess \
  --write

The logs are not processed byte-for-byte in real time.

They are processed as soon as they can be safely treated as complete.

In practice, that means Helix is usually only one or two minutes behind BroadSoft.

For us, that is the right tradeoff.

We gain the integrity of processing stable files while still having the data available quickly enough to investigate current incidents.

What was previously considered an unmanageable collection of enormous rolling text files is now a continuously updated operational data source.

We can search raw SIP exchanges.

We can reconstruct call flows.

We can inspect internal BroadSoft events.

We can analyze registrations and subscriptions.

We can track endpoint state.

We can preserve the original logs in compressed form.

We can restart the service without losing our place.

And we can do all of it without repeatedly grinding through the entire history of the remote directory.

Everyone said the full BroadSoft XS logset could not be processed.

The real answer was that it could not be processed effectively using the old assumptions.

Once we stopped treating it as a pile of files and started treating it as a stateful, moving ingestion frontier, the problem changed.

Codex helped us move through that change much faster.

But the breakthrough was not magic.

It was architecture.

Though my IT career I have worked everything from SS7 and SIP to DSL, GPON, CMTS, Linux, databases, and large scale distributed systems. I am currently building an internal platform we have named Helix. It is a system designed to correlate operational data across an entire telecom network, from signaling edges to physical access layers.

I am saying that up front for one reason. This perspective did not come from reading marketing decks or experimenting with chatbots. It came from years of being the person who can dig into the internals of any system and extract truth from the data, no matter how fractured it is.

Right now, everyone thinks they need AI.

They do not. At least not in the generalized, large language model sense that dominates headlines.

What most organizations actually need is a modern expert system.

And those are very different things.

A short history of expert systems and why they failed

Expert systems are not new. They peaked in popularity in the 1980s and early 1990s. The idea was simple and ambitious. Capture the knowledge of human experts as rules and let computers reason over them.

If condition A and condition B are true, then conclusion C follows.

In theory, this would preserve expertise, reduce dependency on a few senior people, and allow faster decisions. In practice, most expert systems failed.

Not because the idea was wrong, but because the assumptions were.

They assumed experts could fully explain how they think. They cannot. Most real expertise is situational and contextual. It emerges while solving the problem, not beforehand.

They assumed rules would remain stable. They do not. Networks change, software changes, vendors change, humans make mistakes, and edge cases become the norm.

They assumed reasoning could happen without deep, rich context. It cannot. The systems had thin inputs and brittle logic, so when reality deviated even slightly, the system collapsed.

As a result, expert systems became rigid, expensive, and fragile. They froze knowledge in time and failed to keep up with the environments they were meant to support.

If this all sounds familiar, it should. Expert systems were taken seriously enough in the 1980s to warrant mainstream coverage. There is a Computer Chronicles episode that walks through the promise and limitations of early expert systems, and it is worth watching with modern eyes. The ideas were not naive. The tooling and data simply were.

So the industry moved on.

What changed, and why the idea deserves resurrection

What we have now would have been unimaginable when those systems were built.

We have cheap storage, fast search, and massive compute. We can retain raw operational exhaust instead of summaries. Logs, events, metrics, state changes, and historical traces can all live side by side.

We also understand systems better now. We know that truth in complex environments is rarely found in a single signal. It emerges from correlation across layers.

Most importantly, we no longer need to pretend the computer is the expert.

That is the key shift.

A modern expert system does not try to replace human judgment. It preserves context, memory, and causality so that human experts can do what they do best.

Think less rule engine, more institutional memory.

What a modern expert system actually is

A modern expert system is not a chatbot. It does not guess. It does not hallucinate. It does not generate answers because answers are rarely the problem.

The problem is that the evidence is scattered.

A modern expert system collects, retains, and correlates evidence across domains. It understands time. It understands sequence. It understands that SS7 signaling, SIP dialogs, application server logic, access network behavior, and physical infrastructure failures are all part of the same story.

In the system I am building, Redis functions like short term memory. Relational databases store structured facts. Search engines preserve long term narrative history. That architecture is not accidental. It mirrors how humans actually think under pressure.

The system does not say, here is the answer.

It says, here is everything that happened, in order, across layers, with enough fidelity that the answer becomes obvious to someone who understands the domain.

That is an expert system.

Why most organizations think they want AI

AI sounds powerful. It sounds modern. It sounds like progress.

But when you look closely at most enterprise use cases, what people are really asking for is not intelligence. They are asking for visibility, correlation, and memory.

They want to know why something broke.
They want to know what changed.
They want to know whether this has happened before.
They want to know the blast radius.
They want to know what matters and what does not.

Those are not language problems. They are context problems.

Dropping a generalized LLM on top of fragmented, low quality, uncorrelated data does not solve that. It just adds another opaque layer between the operator and reality.

In many cases, AI is being used because it sounds cooler than expert system.

The uncomfortable truth

If you tune an expert system deeply to your environment, your network, your workflows, and your failure modes, it will outperform a generalized AI every time for operational decision making.

Not because it is smarter.

Because it is grounded.

It understands your topology, your vendors, your history, your weird edge cases, and your human processes. It does not need to reason abstractly about the world. It needs to remember what actually happened last Tuesday at 3:17 AM.

That is where expertise lives.

So what is Helix, really

Helix is a modern expert system for telecom operations.

It does not automate decisions. It removes fog.

It does not predict. It reconstructs.

It does not replace humans. It keeps them sane.

And this is the part that matters most to me personally. It reduces the need for heroics. It turns hard won experience into durable infrastructure instead of tribal knowledge locked inside a few burned out people.

Final thought

AI is not evil. Large language models are impressive tools when used for the right problems.

But most organizations chasing AI are skipping a step.

Before you ask a machine to think for you, you should make sure it can remember for you.

For many real world systems, especially infrastructure, networks, and operations, the future is not artificial intelligence.

It is modern expert systems, rebuilt with humility, context, and the power we finally have to make them work.

--Sometimes Thinking Ahead of the Curve Means Looking Behind the Curve
-Bryan

I just installed the Facebook app on a freshly reinstalled Windows 11 laptop. A 17 inch 2019 HP Omen, i7, 16GB of RAM, NVMe boot drive, GTX 1070M. It ran Linux for a long time after I finally got tired of Windows 10. But when Windows 11 came out and everyone else was loudly declaring they were done with Windows, I jumped back on. This machine has a permanent license, so there was no purchase, no friction, no ceremony. Everything snapped into place instantly. Then I installed the ChatGPT client. Same experience. Immediate. Responsive. It feels local, personal, owned.

And yet none of it is. These days this machine spends more time acting as a terminal than actually processing anything itself. Maybe an Excel spreadsheet here or there, though even that likely came from OneDrive. Mostly, it is just a fast window into responses generated somewhere else. Still a terminal.

Every bit of this is running somewhere else. In data centers most of us will never see, slicing millions of processing threads across networks that routinely move ten gigabits per second. Compared to what we grew up with, this is absurdly fast. Compared to the old days of time share, it barely resembles the same idea.

Back then, time share announced itself. You could feel it in your bones. Jobs went into queues. You waited. The machine reminded you that you were not alone. Downloading a one gigabyte file over a 128k connection was not just slow, it was a commitment. You planned around it. You hoped nothing failed. You knew other people were competing for the same limited resources.

The friction was the point. The slowness made the system visible.

Today, that same file crosses the wire in seconds. Often you do not even notice it happening. The wait is gone, and with it the cues that reminded you of the bargain you were making.

That is the real shift.

Cloud is time share, but it is fast enough to feel like ownership.

We are all sharing compute now, just at a scale that breaks intuition. Instead of a few users dividing a mainframe, we share planetary infrastructure. Millions of cores. Vast memory pools. Storage deep enough to feel endless. Latency low enough that your brain stops asking where things live.

Speed changes psychology.

When time share was slow, you understood the deal. You traded control for access. You accepted limits. You learned patience, or you learned how to work around the system. Either way, you knew where you stood.

Now the bargain hides itself. The interface is smooth. The response is instant. The abstraction is nearly perfect. It feels like the machine is yours even when it very clearly is not.

Facebook. ChatGPT. Streaming music. Cloud storage. AI copilots. All of it lives in someone else’s data center, running on someone else’s schedule, governed by someone else’s rules. You are no longer waiting for CPU cycles. You are sharing them so efficiently that the sharing disappears.

That is why this is harder to resist.

You can argue philosophy all day, but the practical reality is simple. The performance gap is too wide. The convenience is too real. The economics are too compelling. Nobody wants to go back to overnight downloads or waiting weeks for hardware to arrive.

So the question is not whether we can stop this. We cannot.

The real question is what we lose when speed erases awareness.

In the old time share world, you knew you were renting. In the personal computer era, you knew you owned the machine. Today the line is deliberately blurred. Ownership feels emotional instead of technical. Control feels implied instead of actual.

Into that blur step certifications, platforms, consoles, roles, and managed services. Knowledge gets re siloed under corporate umbrellas. You know your dashboard. You know your permissions. You know your slice. You do not know the whole.

That model scales. It is efficient. It also reshapes how engineers think.

When everything is abstracted and fast, understanding starts to feel optional. Curiosity becomes friction. Deep system knowledge gives way to interfaces that promise you do not need to know what is underneath.

Gen X sits in an odd place here. We remember slow networks and screaming modems. We remember when downloading a file was a decision with consequences. We also live comfortably in a world where gigabytes move without thought and AI responds in real time.

This is not an anti cloud argument. It is not anti progress. It is memory.

We remember when the machine showed its seams.

Cloud is time share perfected. Smoothed. Accelerated until resistance disappears. The cage is still there, but you do not feel the bars because everything moves at the speed of expectation.

So can we fight it. No, not really.

But we can remember what it means to understand systems instead of just using them. We can teach curiosity alongside convenience. We can remind people that behind every instant response is a real machine, burning power, moving data, and obeying physics.

Because the danger is not that we are sharing compute.

The danger is forgetting that we are sharing at all.

--Deep Thoughts Always
-Bryan

There is a strange idea that keeps threading itself through modern tech. If you want intelligence, you must go big. Big models, big GPUs, big training budgets, big vendors. The assumption is that complexity requires scale, and scale requires the cloud.

The truth is quieter. Intelligence is not about size. It is about structure. It is about how information is shaped, aligned, cross-referenced, and made coherent.

The more I work with operational data, the more obvious it becomes that the future is not in shipping every question to a giant generalized mind in the sky. The future lives in smaller systems that actually understand the environment they live in. Intelligence that is local, specialized, and built on top of everything your own infrastructure already knows.

The challenge is not training a model. The challenge is giving your data a nervous system.

The data itself already exists. Call logs, signaling traces, device telemetry, SNMP counters, tickets, weather spikes, human behavior patterns that repeat so consistently they might as well be physics. Every system emits signals. What is missing is a single place where those signals land, synchronize, and form a picture.

My view is that you get there by stacking three layers.

At the bottom sits the immutable record. Every log line, every heartbeat, every state change. Not cleaned, not simplified, not modeled. Raw fact. Time is the only structure. This is the layer that never lies.

Above it sits the structured layer. The place where raw sequences become meaningful objects. A storm of BroadSoft logs becomes a single callflow. A thousand modem telemetry packets become a behavioral window on a neighborhood. A loose heap of metrics becomes a timeline. This layer speaks the language of the domain.

Then above that sits the intelligence layer. Not AI in the marketing sense. AI in the literal sense. Local inference systems that understand your own infrastructure because they were built out of your own infrastructure. A knowledge graph that sees patterns across every subsystem. A rules engine that adapts because it has history. A set of micro models that are not trying to answer the entire world, only the piece of the world they sit inside.

When all three layers align, the system starts acting alive. You do not ask it what happened. It tells you. You do not dig for root cause. It hands you a timeline. You do not correlate weather, calls, power stability, ticket spikes, and fiber alarms. It already sees the relationships.

And none of this requires a trillion-parameter model. It requires coherence.

The large models will always be good at language, summarization, generic creativity. They are universal tools. But universality is not insight. Insight comes from context. Insight comes from systems that are shaped by their environment, not from systems that try to model every environment at once.

Local intelligence is not a smaller version of cloud AI. It is a different species entirely. One that listens more than it predicts. One that adapts locally instead of statistically. One that grows directly out of the signals your own network produces.

If the last decade was about scale, the next one will be about precision. Not giant models, but finely tuned minds built out of the data you already own. Minds small enough to run anywhere, smart enough to be useful, and structured enough to see reality instead of hallucinating patterns that are not there.

You do not need a thinking machine. You need a system that understands the story your infrastructure is already telling.

That is what real operational intelligence will become. Not magic. Not hype. Just coherence built at the right place in the stack.

-Think Bold
--Bryan

Before there were dashboards, pollers, or clusters, there was Linux. Not the polished, GUI-heavy version you get now with auto-updates and a software store. I am talking about the kind that barely booted, came on a stack of CDs, and expected you to know what you were doing.

My first Linux install was Red Hat 5 Manhattan, somewhere around 1998. I did it because I wanted to run my own CircleMUD server. That was before apt, before yum, before dnf, before anything held your hand. I think I broke that system a dozen times installing the wrong RPM versions, and it took about 200 recompiles (give or take) to finally get CircleMUD running. But I did it. And it ran for a couple of months, which at the time felt like eternity.

Then life moved on. For a few years, I did not touch Linux much. I was still working at a mom-and-pop computer shop, fixing Windows 98 machines and building beige towers with jumpers and ribbon cables. It was a good start, but it was small.

When I finally moved from that world into the world of ISPs around 2001, Linux came back with a vengeance. It was everywhere — on servers, routers, mail systems, DNS, everything that actually mattered. And it just worked. Once you got your hands around it, you realized it was the only layer that did not lie to you.

Why Linux Still Matters

Linux is the bedrock of everything I do. It is not about open source for ideology’s sake. It is about control. When I run a system, I want to know what it is doing, why it is doing it, and how to stop it when it should not be. Linux gives me that.

It is stable, predictable, and transparent. It does not reboot itself in the middle of your night to “apply important updates.” It does not phone home. It does not hide processes. It simply runs what you tell it to run, and when something breaks, you can trace the logic all the way down to the kernel if you have to.

That kind of honesty is rare. And once you learn it, it becomes part of how you think. You stop trusting black boxes. You stop accepting mystery as an answer. You start building systems that earn your trust instead of demanding it.

The Foundation of Everything Else

Every tool in my stack — OpenSearch, Grafana, Logstash, Go pollers, MariaDB — depends on Linux. It is the platform that makes all of them possible.

When I say I build systems that tell the truth, that truth starts here. Linux gives me the freedom to build from the ground up without waiting on a vendor. I can deploy, automate, containerize, or run bare metal depending on what the job needs. It adapts to my intent instead of the other way around.

The filesystem is clean. The process tree is honest. The logs are readable. And when I put it all together, I know that every event I track and every metric I visualize starts from a stable, trusted base.

What It Taught Me

Running Linux teaches patience and consequence. You learn to think before you type. You learn to read before you reboot. You learn that every fix comes with a trade-off.

I learned those lessons the hard way back on Red Hat 5, staring into the depths of dependency hell because I wanted to host a text-based game for a handful of friends. But those same lessons are what make me good at what I do now. Whether it is a network outage, a routing loop, or a corrupted index, the logic is the same — find the cause, fix the cause, and understand the cause.

Linux taught me to respect the cause.

Where It Stands Now

Today, I run Linux everywhere — on my workstation, my lab systems, my data pipelines, and my servers. I trust it because it has earned that trust over decades.

When people ask me why I still build everything on Linux instead of some polished proprietary system, the answer is simple: because I can see it. I can touch it. I can fix it.

It is the real foundation of the intelligence layer, the quiet constant that keeps the rest of the system honest.

Stay tuned for the next article, “Grafana – The EKG of the System,” where I will talk about how visualization becomes intuition when the data finally starts speaking in color and motion.

--Bryan

When I talk about OpenSearch, I am talking about the part of my system that remembers. It is not just a database. It is the memory engine that holds everything I have ever taught the system to see.

I started using Elastic long before OpenSearch existed, back when the interface still looked like a science experiment and every upgrade felt like walking across a moving bridge. I stuck with it because it gave me something most systems could not: a way to search and reason about data in near real time. When Amazon forked Elastic into OpenSearch, I followed that path, not because of branding, but because it preserved what I valued most — open access, transparency, and control.

Why I Use It

OpenSearch gives me a way to view my world over time. Every log line, every metric, every CDR, and every device event lands there, timestamped, structured, and searchable. When something goes wrong in the network, I can replay the story exactly as it happened.

It is the core of my tool belt because everything else in the stack orbits around it. Logstash feeds it. The Go pollers enrich and shape what they receive. Grafana visualizes its heartbeat. But OpenSearch is the one holding the truth.

It scales the way I think. I can work at the level of a single ONT or zoom out to see a month of network behavior in a single query. I can run ten indexes for ten different data types and still cross-correlate them in milliseconds. It is both a microscope and a telescope.

My History With It

I have been indexing telecom data since before most people in this industry knew what JSON was. I started with early modem telemetry, DSL metrics, and provisioning logs, all flattened into Perl scripts and dumped into MySQL tables.

Then I met Elasticsearch, and suddenly all of that could be searched, graphed, and cross-referenced. I could take a MAC address, trace it through a dozen systems, and find its full life story in seconds. Over the years, I built dashboards that told real stories: who was down, when it started, what triggered it, and what changed just before it happened.

Of course, it does help that I worked at Elastic as a support engineer for nearly 3 years right at the IPO threshold.

When Elasticsearch turned into a closed ecosystem, I moved to OpenSearch and never looked back. It was like getting my keys back to the engine room. Now I can build exactly what I need, how I need it, and at a scale that fits my environment, not someone else’s licensing model.

What I Have Already Pulled Off

With OpenSearch as the backbone, I have ingested and correlated data from systems that were never meant to speak to each other.

I have it tying together ONT logs, SS7 CDRs, DHCP churn, provisioning changes, and ticketing data. I can tell when a customer’s line went down, how long it stayed down, which optical path it followed, and what the DSLAM or PON port saw in the same moment.

I built the RFPhantom poller in Go to feed it DOCSIS metrics from thousands of modems in under half a minute. I built the ONT log enricher that cross-checks events against CMS and enriches them in real time without bringing the system to its knees. I built the indexes that let me track trends, not just incidents.

This means I can see the living network, not just snapshots. OpenSearch turned raw logs into a time-aware narrative engine.

Why It Works For Me

I like systems that can grow without collapsing. OpenSearch is built for that. It shatters into shards, distributes across nodes, merges segments, and keeps pace with whatever you throw at it. You can start with one node and grow to ten without rewriting a thing.

For me, that is critical. I do not have a hyperscale data center. I have commodity hardware, solid engineering, and a need to make every core count. OpenSearch lets me expand horizontally and tune performance as I go.

When I optimize mappings, set shard counts, and tune refresh intervals, I am not chasing benchmarks. I am making sure the data can breathe, merge cleanly, and scale into tomorrow’s load without stalling today’s queries. That is the kind of scalability that matters in the real world — the kind that keeps working while you are still building.

k-NN Will It Help?

The next frontier for me is k-NN search inside OpenSearch. It is one of the few features that actually moves the needle for what I am building.

Right now, I can find patterns by logic and by correlation. With k-NN, I can add similarity — the ability for the system to recognize shapes in the data that look like problems it has seen before.

It is not a prediction in the AI sense. It is memory at scale. When a certain combination of optical drift, temperature swing, or call failure cluster appears, k-NN can say, “I have seen this before,” and point me toward the last time it happened and what fixed it.

That is where this is heading. From correlation to recognition. From storytelling to intuition. From data that describes to data that remembers.

Closing Thoughts

OpenSearch is not just a tool I use; it's a way of life. It is the center of gravity for everything I have built. It gives shape to noise and history to signal.

When people ask what powers my systems, they expect to hear a long list of microservices or some glossy AI framework. What they get instead is OpenSearch — open, honest, and powerful enough to hold the mind of an entire network.

This is the foundation. Everything else I build, from middleware to the eventual interface, starts and ends here.

Up Next

Linux – The Real Foundation

--Stay tuned
-Bryan

I have spent most of my life surrounded by tools. Physical ones, digital ones, some I built myself, some I borrowed, and a few that should have been retired a decade ago but still work because I know exactly how to hit them with the right command. What I have learned over twenty-four years in telecom is that the right tools are not just utilities. They are extensions of how you think.

The stack I use today did not come from a vendor roadmap or a course syllabus. It came from years of building things that had to work, survive, and make sense under pressure. Every piece of it has a reason to exist. It is not about trends. It is about trust.

Linux – The Ground Beneath Everything

I run Linux because it does not lie to me. It does not hide what it is doing. It does not reboot itself for “my convenience.” It simply runs. You can pull it apart, see the logic, and understand the sequence of events that led to any state.

In a field where uptime is currency and mystery is failure, I cannot afford to run something that treats me like a guest in my own machine. Linux lets me work close to the metal, to feel the heartbeat of the system, not just the interface on top of it. It is the quiet hum that everything else in my world stands on.

OpenSearch – The Core of Memory

OpenSearch is not just my database or dashboard engine. It is the memory of my entire operation. I have worked with Elastic and OpenSearch since around 2016, and what I learned is that speed, scale, and clarity mean nothing if you cannot trust the shape of your data.

I use OpenSearch because I can. Because it is open, predictable, and does not decide what I am allowed to see. It indexes everything, speaks fluently in time, and does not forget. When I look at a dataset from months ago, it should look the same as it did then, with every field still connected to its lineage. That kind of honesty in data is rare.

I do not run it because it is trendy. I run it because I know it will still be telling the truth long after the marketing slides have faded.

MariaDB – The Anchor

Every system needs an anchor, a place for data that does not drift. OpenSearch may be my dynamic memory, but MariaDB holds the static truths. Customer records, service addresses, topology maps, and reference tables that do not change unless I change them.

There is comfort in SQL. You can reason about it, trace it, and know exactly what a query is doing before you run it. It is the calm voice in a storm of JSON. I use it when I need reliability, precision, and something that will never pretend to be smarter than it is.

Logstash – The Old Workhorse

Logstash is not the fastest tool in the shed, but it is still one of the most dependable. It takes chaos and turns it into structure. It can take the weirdest logs from the weirdest devices and make sense of them. I can write Grok filters in my sleep, and sometimes I do.

People have told me there are newer, sleeker options. Maybe. But I know how this one behaves under load, how it fails, and how to make it recover. It has been around long enough to earn its place. When I drop a new feed into the pipeline, I know exactly what will happen. There is something to be said for an old tool that still gets the job done.

Grafana – The EKG

Grafana is where the data breathes. It is the pulse monitor of the system. When something drifts, spikes, or simply feels off, Grafana is the first place I look.

I do not use it because of pretty graphs or corporate dashboards. I use it because it shows me the rhythm of the network in real time. It is how I see when a problem is just noise, or when the heartbeat is skipping for a reason.

Grafana does not reason. It reflects. It is not the mind of the system, it is its vital signs. And when I have built a good dashboard, I can see the health of the entire operation in a single glance.

Go – The Precision Tool

Go is where my logic lives. It is fast, clean, and predictable. It does not hide behind abstraction or surprise me with behavior I did not ask for. When I write pollers, enrichers, or middleware in Go, I know exactly what is happening down to the byte.

It gives me confidence. It compiles into small, self-contained binaries that I can drop anywhere. It is elegant in its simplicity. Go is not the flashiest language, but it is one of the most honest. It rewards thoughtfulness and punishes chaos, just like networks do.

Perl – The Swiss Army Chainsaw

Ah, Perl. The language that refuses to die, and honestly, I hope it never does. Perl is like that stubborn old technician who knows every backdoor, every undocumented command, and exactly which piece of equipment needs a kick instead of a reboot.

When a piece of logic refuses to pin down, I pull out Perl. Sometimes I just need to grab a text file by the throat and make it talk. Perl does that. It might not be pretty, but it is fast, effective, and completely loyal to the person wielding it.

I have built systems in it, scraped data from places it was never meant to leave, and stitched entire processes together with nothing more than a few regular expressions and a bad sense of humor. It is the tool I keep in the back of the drawer for emergencies, and it still saves the day more often than I would like to admit.

ZFS – The Memory That Does Not Forget

When you work with as much data as I do, backups are not optional, and corruption is not an acceptable risk. ZFS gives me the peace of mind that everything I have built can be rolled back, cloned, or snapshotted in seconds.

It is not just storage, it is assurance. Every bit of the intelligence layer depends on being able to trust that yesterday’s data is still valid today. ZFS gives me that trust. It is the kind of technology that quietly makes everything else possible.

React + Tailwind – The Face of the Machine (Coming Soon)

On the roadmap is a new interface built with React, Tailwind, and Vite. It is not running yet, but it will be built step by step through this series as the system grows.

The goal is not flash or animation. It is clarity and speed. The people who use my tools are operators, and they need information, not decoration. React will handle the interaction, Tailwind will keep the design uniform, and together they will give the intelligence layer a clean, fast face that can keep up with the human behind the keyboard.

Why These Tools Matter

Every tool in this kit earned its place the hard way. Not because it was fashionable, but because it worked when it mattered. They are not pieces of a tech stack; they are a philosophy of control, transparency, and resilience.

These are the core components I rely on most, but they are not all the tools I use. If I find something that does a job better, I will use it. If I need to write a new tool that fits my data more precisely, I will do that too. The point is not loyalty to any single language or platform. The point is using the right tool for the problem, and understanding why it works.

The real intelligence in operations does not come from machine learning or vendor dashboards. It comes from humans who understand how their tools think, who can trace a problem down to its root without being distracted by noise.

My toolbox is built for that kind of work. Each piece is a lens that helps me see the system more clearly. Together, they form the foundation of what I call The Intelligence Layer, a way of making sense of modern networks without surrendering to them.

Stay tuned for the next article, “OpenSearch – The Core of My Tool Belt,” where I will explore how this engine became the heart of everything I build, and why I trust it to hold the memory of my systems.

--If You're Not First, You're Last (Maybe)
-Bryan

I build systems that tell the truth. I pull signal from noise. From network logs and call records, to service data, optics, metrics, and all the other pieces that generally live in separate silos with separate logins, I bind them into one pane of glass. I write pollers, enrichers, and dashboards that translate the chaos of rural telecom into something you can reason about.

My work cuts across the full telecom stack... SS7, SIP, PON, DSL, DOCSIS, provisioning, and everything that ties them together. Operations do not live inside a single protocol, and neither do I. If it speaks, I can listen. Once I can listen, I can teach the system to listen too.

Why I’m Doing This

Because I got tired of seeing MBs and GBs of useful data just get stored static on a forgotten drive somewhere. Because I could see the stories in the data. I could see how to connect them together. I believe the real intelligence in our networks does not come from AI models or vendor dashboards, it comes from engineers who actually understand what they are looking at.

This blog is where I show how that understanding is built — through systems, through code, through iteration. It is where I document how I think, not just what I make.

The Lineage: From CodexMCP to Now

If you browse back through this blog or my LinkedIn posts, you will see a lot of time went into a project called CodexMCP. That project started as an experiment to emulate user churn inside a VM. It was going to be a way to simulate a living network ecosystem. But like most experiments, it grew into something more. Once I proved the mappings I created for the data and the tuning of OpenSearch were tight, the project started to shift a little.

Over time, CodexMCP stopped being a specific tool and became a way of thinking — about systems as living entities, about feedback loops, about understanding data as an organism that evolves when you observe it the right way.

What you are reading now is the evolution of that thinking. This is no longer about simulation. It is about working directly with real, live network data and using it to build operational intelligence at scale. The idea remains the same... learn, listen, adapt.

The Moving Target

There is no “end result.” Every time you think you have caught up, the landscape shifts. In this field, you do not just maintain, you chase. Step away for a few months, and you will spend a year catching up.

So this is not a static documentation project. It is a record of motion — of staying sharp, staying curious, and turning daily operations into a continuous feedback loop of learning.

Where It’s Going

Over time this space will evolve into a full technical atlas:

• How dashboards become decision systems
• How clean data enables intelligence
• How small even small networks can operate with the precision of hyperscalers
• How to think about your own data as a living system

There is no finish line. The work never ends, and that is exactly the point.

Stay tuned for the next article, My Toolbox, where I will discuss the open source tools I use to build these systems, and how I create specialized views to troubleshoot specific problems.

--Here We Go
-Bryan

I keep seeing a wave of new commercials lately. Cisco. GitLab. Dell. IBM. Each one promising “AI-powered insight,” each one showing the same thing — a human dashboard with all kinds of nifty visuals and tools on it. Somewhere along timeline you might see an LLM chat window.

But look closely and you’ll see what’s really going on. None of them are selling artificial intelligence. They’re selling augmented intelligence, the illusion that you can buy understanding as a service.

You can’t.

Understanding still comes from people. The machine only amplifies it.

The truth is, you can build everything they’re showing on one commodity server. Ten thousand dollars of hardware. No contracts. No GPU cloud bills. No vendor markup on your own data.

All it takes is time, cooperation, and a few people who actually understand what the data is saying.

The Mirage of Turnkey Intelligence

Every marketing campaign tells the same story: plug in our product, and your data will finally “talk” to you. But that story skips the part where your data leaves your house.

Once your metrics, logs, and telemetry live in a vendor’s cloud, the balance of power shifts. You stop being the operator and become the consumer of your own infrastructure. You end up renting insight from the very systems you built.

You do not need to do that.

Open source has already given us the full toolkit.
OpenSearch, Grafana, React, Go, Python, and local models like Mistral or Llama run perfectly well on a standard multi-core system. If you want a bit of a boost add even a consumer grade GPU card, python will use it. You can poll your devices, enrich the results, and visualize them in real time.

That’s not science fiction. That’s Tuesday afternoon if you’re given the freedom to work.

Build Locally, Think Globally

When you own your pipelines, you control the story your data tells. You can experiment without asking permission. You can enrich a dataset three different ways and see which one correlates to the truth on the ground.

This is how the next generation of NOC engineers needs to be trained. Not as ticket closers. Not as vendor whisperers. As listeners.

Teach them to listen to the data. Teach them to question why a value spiked, what it means, and what system whispered that change into being. That is how you grow a NOC that can outthink its vendors instead of waiting on them.

The Edge Cases Still Matter

You can automate 99% of the work. I encourage it. Let the systems handle repetition. But that last one percent , that’s where the real story hides.

A model can summarize thousands of modem metrics, but it still takes a human to know that a BIP counter rising on an ONT might mean water in a splice tray two miles away. The model doesn’t have intuition. It doesn’t know what Ohio humidity does to fiber. It doesn’t smell burnt dust on a card slot. You do.

That’s why we’re not replaceable. We’re the validation layer between data and reality.

Don’t Sell Your Soul for Convenience

This isn’t a call to boycott vendors. Use them. Learn from them. But don’t surrender your autonomy for ease. The moment someone tells you that you have to “buy your data back,” the game is over.

Keep your raw data. Keep your enrichment logic. Build your own indexes. If you don’t have the time, fight for it. If you don’t have the cooperation, demand it. Because every bit of data you own outright is one more step toward independence.

Where We’re Going

The future of operations isn’t vendor dashboards or pre-trained models. It’s a generation of engineers who treat data as story of the system, not as a wall of text.

The smartest systems will not replace the human. They will serve as instruments we learn to play. The ones who master that harmony, human intuition backed by open tools, will run circles around those who rented their understanding.

Take back your data. Build your own intelligence.
Because the real revolution isn’t artificial. It’s human.

--Is It Live or Is It Memorex
-Bryan

Most people look at a dashboard and see a collection of panels. A few charts, some graphs, maybe a live table or a map. Two dimensions. Something you look at.

But that is not how I see it.

When I am deep inside a network, watching ONT logs, SS7 call traces, DSL syncs, optical levels, and customer endpoints all move together, it stops feeling like a dashboard. It feels like a rotatable cube. A living model of everything happening in that system, across space, across time, across cause and effect.

Each data point is not just a number. It is an event, a relationship, a moment. It has neighbors, context, and gravity. And when you can pivot through those relationships quickly enough, the entire thing becomes spatial. You are no longer scrolling through metrics. You are navigating an environment.

Click a link, filter a subset, merge a field, aggregate a pattern. Each one is another rotation of that cube. You are not drilling down. You are moving through dimensions.

I build dashboards that move like that. They are not about making things pretty. They are about moving through information as naturally as you would move through a room. They are instruments of function, motion, and truth.

The Fourth Dimension

Dimension one is the static metric, the literal number.
Dimension two is the relationship between those metrics, the visual plane where comparisons live.
Dimension three is time, the motion between states.
Dimension four is correlation, the invisible architecture connecting everything.

Once you start working in that fourth dimension, old dashboards start to look primitive. A five-minute poll is not visibility. It is a shadow of what already happened. Real awareness happens when you can see cause and effect unfolding as it occurs.

That is the shift. From static monitoring to living systems.

Do We Really Stop at Four?

When I say four dimensions, I am simplifying. In truth, it does not stop there. The deeper you get into real data, the more dimensions begin to emerge.

Every metric has dependencies, patterns, behavioral fingerprints, and human context. If you start counting the ways those interact, you are no longer in four dimensions. You are in something closer to multi-dimensional cognition, where each new connection spawns another axis of understanding.

A network, when visualized fully, is not a cube at all. It is a cloud of overlapping realities. Physical signal paths, logical routes, service policies, customer behaviors, historical anomalies, predictive probabilities. Each layer exists simultaneously. The human brain collapses it into something we can visualize, but the data itself does not live in two or three or four dimensions. It lives in something larger than we can picture, and we navigate it instinctively when the tools stop slowing us down.

That is what I am really building when I say a dashboard. It is not an interface. It is an extension of perception. A way for the human mind to move through more dimensions of reality than the senses allow.

Rethinking Data

We are entering a time where data will no longer sit still. It will move, interact, and respond. The question is not how we will visualize it, but how we will think within it.

If your tools let you manipulate thousands of data points instantly, if you can pivot between context and correlation without losing your sense of place, you are not just analyzing. You are experiencing. You are interacting with a living model of your own environment.

That is not analytics. That is dimensional intelligence.

When I rotate that cube, I can watch a packet’s entire story unfold. From ONT to PON, from softswitch to trunk, through a route, into a call trace, all connected, all alive. It is not reporting. It is cognition.

The Road Ahead

At some point, every serious operator, engineer, and analyst will have to leave the comfort of the 2D dashboard. The networks are too complex, the data too fast, and the decisions too important.

We do not need more charts. We need systems that think spatially. Tools that allow a human to experience data in motion.

Four dimensions are only the start. Once the interface becomes as fast as thought, the data stops being a product and starts being a landscape.

The fourth dimension is not science fiction. It is what happens when you stop looking at your data and start living inside it.

--Always Thinking
-Bryan

Intro:
Ever tried to make a * code like *98 route through DNIS on an Adtran TA and hit a brick wall? I did—and after scouring the internet for an answer, I came up empty. So I tore it apart myself, and here’s the straight truth.

The Problem:

• Subscriber dials *98 expecting voicemail.
• DNIS entry exists, but the call dies.
• Nothing in the docs screams why.

The Reason:

• Adtran treats * codes as feature access codes, not digits to feed into DNIS.
• With Spre Map Local, the box eats the *98 before DNIS ever sees it.
• That’s why your DNIS entries look fine but never trigger.

The Fix:

• Flip Spre Map from Local to Network.
• Create a DNIS entry translating *98 → full 10-digit voicemail number.
• Now *98 passes through, DNIS rewrites it, and the call completes.

The Caveat:
When you move Spre Map to Network, the Adtran’s built-in star code map stops working. That means internal features tied to those codes will no longer function. If you rely on the TA’s native *xx features, you need to replicate or reassign them before making the switch. Otherwise, you’ll fix one problem and create another.

Takeaway:
Star codes will never work with DNIS when Spre Map is set to Local. If you want *98 (or any other *xx code) to hit DNIS, you must set Spre Map to Network. Simple once you know it—but nobody spells it out.

On LinkedIn my posts may be long, but I try to keep them within the groove of technical and professional. Here, my mind is a little more free — well, maybe a lot more free.

Take BroadSoft logs for example. They’re like a Java error dump that got drunk and fell off a cliff. Multiline chaos, overlapping, nested in different streams, but each one still begins with a line you can anchor to. If you know that, you can split it, parse it, tame it with the least number of grok rules possible. It’s messy, but not meaningless. That’s the trick. The drunk gibberish, if you stare at it long enough, actually has a rhythm.

And that’s where my head goes every time I’m staring at these logs, or tickets, or SNMP traps. What does any of this really mean? Why does it matter? Because data, at its core, is nothing but the fossil record of human intent colliding with systems. A phone call made, a packet dropped, a technician rolling a truck — each one leaving a trace, an imprint. By itself, each trace is noise. Together, they form a story.

CodexMCP was never just software. It was an attempt at translation. To take all those fossils — logs, traps, notes, metrics — and read them not as disconnected events, but as the mythology of a network. Codex was, is, and might always be a language exercise. Not in the sense of English versus German, but in the sense of decoding the grammar of reality itself: what happened, why, and how the system experienced it.

Most people see data as numbers or lines on a screen. I see it as memory. Collective memory. A network remembers its own life through the messages it emits, just as we remember ours through journals, scars, photographs. A bounced ONT isn’t just a device glitching — it’s a neuron misfiring in the brain of the telco. A series of SIP cancels isn’t just call setup noise — it’s a stutter in the voice of communication itself.

The absurdity, of course, is that I spend hours and years building pipelines to make this visible, while the world at large shrugs. To most, it’s just IT work. To me, it’s almost sacred. Because beneath the noise is a chance to glimpse order. A chance to pull back the curtain and say: this is not random. These patterns carry meaning. They are the fingerprints of both failure and resilience.

Will CodexMCP ever be finished? Probably not. Because it’s not really a tool — it’s a way of thinking. It began in my lab, evolved in my head, and now leaks out through every system I touch. And maybe that’s the point. It’s not about an end state. It’s about the act of seeing, of refusing to let noise remain noise.

So yes, BroadSoft logs look like a Java dump that fell off a cliff. But maybe cliffs are where language begins. Maybe out of those broken lines, those multiline errors, the real stories emerge — the stories that tell us not just what the network is doing, but what we are doing with it, to it, through it. And maybe, just maybe, those stories outlast the network itself.

-Quite a Ride, Try it Sometime
--Bryan

Earlier this week, I was working a support case for a rural fiber customer. They live far out in the country—clean spectrum, no neighboring SSIDs, 1G fiber connection running through a Gigaspire router. Normally a stable setup. But suddenly their Wi-Fi fell apart. Devices dropped off. Speeds tanked. Phones and TVs struggled to stay online.

I pulled up the spectrum analytics and right there—loud and clear—was a rogue SSID broadcasting as AROMA-013-240709. It was consuming 85% of the 2.4GHz airtime. Not bandwidth. Airtime. That means the medium itself was congested. Devices couldn’t talk. Even though the router stayed up and signal levels looked fine, the channel was gridlocked.

The device responsible had a MAC that fingerprinted to an ESP32 chip. Likely some cheap IoT device. At that point, I didn’t know what it was, just that it was hammering the air. I called the customer and explained something on their network was blasting the spectrum. They asked me to disable SSID broadcast and reset the Wi-Fi password.

The Smoking Gun

I told the customer I would figure out what is going on and reach out to them in about an hour. After lunch, I rechecked the spectrum. Airtime usage had fallen back to a normal 22%. The network was calm. That told me everything I needed to know—the problem device hadn’t reconnected after the password reset. It was boxed out.

I called the customer back. The wife answered. I explained that something on their network had gone haywire and asked if they had an internet-connected aromatherapy diffuser. She said yes, explaining that she could control it from her phone. That was the moment it all clicked.

This device had lost its connection and decided to flip into pairing mode, broadcasting its own SSID—fully open, 40MHz wide, and right on channel 1, which is exactly where their Gigaspire router operates by design.

Why This Is Bad Design

• 2.4GHz only has three non-overlapping channels: 1, 6, and 11
• A 40MHz AP on channel 1 bleeds into channels 2–5
• The diffuser was essentially stepping sideways into traffic, blocking 5 of 11 channels
• On a clean spectrum, this obliterates throughput

And let’s be clear: 85% airtime usage is the same as 100% in real terms. Once you hit that threshold, everything else is queuing up, backing off, retrying. From the end user’s view? Nothing works.

Why 2.4GHz Still Matters

Even with 5GHz and 6GHz widely available, 2.4GHz is still critical:

• It has longer range and better wall penetration
• Many IoT devices only support 2.4GHz
• It’s often used for onboarding and failover
• Devices like phones and smart hubs still fall back to it in edge cases

That means 2.4GHz remains fragile, and it’s under constant attack from low-cost, poorly-behaved devices flooding the market.

The Real Problem

• ESP32-based devices are everywhere. They’re cheap, programmable, and easy to mass produce
• Vendors prioritize low BOM cost and fast pairing over responsible RF behavior
• Firmware often defaults to open AP mode when a device loses Wi-Fi
• Users have no idea these devices are misbehaving

This diffuser joined the network, blew through DHCP, failed, and responded by opening its own AP, likely to reach a phone app. But in doing so, it jammed the entire spectrum.

Who Supports This?

Here’s the kicker: we do. As ISPs, we spend as much time supporting third-party garbage like this as we do maintaining our own infrastructure. It’s not our hardware, not our firmware, not our failure—but it’s still our support queue.

And who else is going to track down a rogue ESP32 aromatherapy diffuser at 1 a.m. causing spectral collapse? No one. That’s not in the user manual. That’s not in the Alexa app. That’s not what the Best Buy Geek Squad is trained to handle.

But we are.

Takeaways

• IoT vendors: Stop shipping devices that broadcast open APs when they fail
• ISPs: Start treating spectrum as a shared battlefield, not just a signal graph
• Consumers: Understand that not all "smart" means smart for your network

This wasn’t an attack. It wasn’t malware. It was just bad firmware doing exactly what it was told.

And it nearly took down a household.

In a different setting, it could’ve taken down dozens.

The air is the wire now. And we better start defending it like one.

--Be One Step Ahead
-Bryan