Processing the BroadSoft XS Logset

That most told us could not be done.

For years, the BroadSoft XS logs occupied an awkward place in our network.

They contained some of the most valuable diagnostic information available from the platform, but they were so large, complex, and difficult to manage that nobody treated them as a practical operational data source.

The logs were there.

The information was there.

Actually turning the entire logset into something continuously searchable was another matter.

Even our vendor told us that processing the full XS logset was not realistically possible.

With the help of Codex, we did it anyway.

That does not mean the underlying problem was beyond my ability to solve without AI. At its core, parsing these files is not radically different from handling any other large multiline log format. The real difficulty was the scale, the number of distinct record types, the correlation between different log sets, and the amount of supporting pipeline work required to make the process continuous, reliable, and resumable.

I could have built this without Codex. The difference is that it likely would have taken weeks of development, testing, and refactoring instead of a handful of focused hours spread across a few days. Codex did not replace the engineering judgment. It accelerated the path from design decisions to working code.

Today, Helix continuously retrieves BroadSoft XS files, verifies that they are safe to process, parses their multiline records, extracts structured operational data, writes the results into multiple OpenSearch index families, updates endpoint state in MariaDB, compresses the original logs for retention, and remembers exactly where it left off.

The pipeline is not technically real time because it does not tail the active XS files while BroadSoft is still writing them.

Operationally, however, it is just-in-time.

Under normal conditions, data becomes searchable in Helix approximately one to two minutes after BroadSoft finishes writing the source file.

That is close enough to real time for active troubleshooting, without taking the risk of parsing incomplete records from a file that is still changing.

Why BroadSoft XS Logs Are So Difficult

The first problem is scale.

BroadSoft XS logs are large rolling text files containing a mixture of:

• SIP requests and responses
• Registration traffic
• Subscription activity
• XSI and EventNotification data
• Internal BroadSoft subsystem events
• XML payloads
• Session and call-leg identifiers
• Endpoint and enterprise information
• Diagnostic messages generated by different platform components

A single file may contain an enormous number of records, and those records do not all share one simple format.

Calling them “text logs” understates the problem.

They are closer to a serialized stream of loosely related protocol events, application events, XML fragments, SIP messages, internal state changes, and diagnostic context. All of that is interleaved inside rolling files that were designed primarily for vendor support and forensic inspection, not modern structured analytics.

The second problem is that the records are multiline.

You cannot reliably process an XS file by treating each line as an independent event.

A record begins with a recognizable header containing a timestamp and pipe-delimited fields. The lines that follow may include SIP headers, SDP, XML, BroadSoft-specific diagnostic data, or other payload content. That body continues until the next valid record header appears.

The parser therefore has to maintain state.

It must:

1. Read the file line by line.
2. Recognize the beginning of a legitimate XS record.
3. Accumulate every following line into the current record body.
4. Detect the next record boundary.
5. Flush the completed record.
6. Parse and classify the assembled content.
7. Repeat that process without losing offsets or corrupting multiline payloads.

A simple regular expression over individual lines is not enough.

A conventional log shipper configured around newline-delimited events is not enough.

Even after a record has been assembled correctly, the system still has to determine what that record represents.

Is it a SIP REGISTER?

A successful 200 OK response to a registration?

A SUBSCRIBE request?

An XSI EventNotification snapshot?

A call-processing event?

An internal BroadSoft message?

A SIP dialog fragment that must be correlated with another record through a shared callhalf, call leg, session identifier, or endpoint context?

The parser has to extract enough structure to make the data useful without destroying the original forensic payload.

That balance is important.

If we only stored the raw record, we would have a searchable pile of text but limited analytical value.

If we extracted only a handful of normalized fields, we could lose the obscure diagnostic detail needed during the worst incidents.

Helix keeps both.

Each parsed record can include:

• The event timestamp
• Source host
• Source path
• Source byte offset
• Parsed XS header fields
• Body classification
• SIP method
• SIP response status
• Event class
• Callhalf or call-leg identifiers
• Session keys
• Endpoint data
• Enterprise information
• XML content
• SIP forensic payloads
• BroadSoft internal payloads
• Search tokens derived from the record

The result is structured enough for analytics but detailed enough for forensic reconstruction.

The Logs Are Rolling Files, Not a Message Stream

Another major complication is how the files are produced.

BroadSoft writes rolling XS text files into a remote directory. The files are accessed over SFTP. They are not delivered through a streaming interface, message queue, Kafka topic, or API designed for incremental consumption.

That means the ingestion system has to infer when a file is complete.

Downloading a file simply because it exists is dangerous. BroadSoft may still be writing to it.

If the streamer grabs a file too early, several things can happen:

• The final record may be incomplete.
• A SIP message may be truncated.
• An XML payload may be cut in half.
• The same file may later appear with a different size.
• The system may incorrectly mark partial data as processed.
• Reprocessing logic may become much more complicated.

We deliberately chose not to tail the active remote file in place.

Instead, the streamer waits for a file to settle.

A file is considered stable only when:

1. It is older than the configured minimum age.
2. Its size matches what was observed during a previous poll.
3. Its modification time also matches the previous observation.

If either the size or modification time changes, the file remains unstable.

This introduces a small delay, but it protects the integrity of the data.

In practice, the delay is usually only about one to two minutes. Once BroadSoft closes or stops updating a file, the streamer confirms that it has stabilized, downloads it, and begins processing.

That is why I describe the system as just-in-time rather than real time.

It does not inspect bytes while BroadSoft is still writing them. It processes the completed file almost immediately afterward.

For actual troubleshooting, that distinction rarely matters. By the time an engineer begins investigating a failed call, the relevant events are generally already searchable in Helix.

The Original Workflow Was Not Sustainable

Before the streamer, processing XS logs was a manual workflow.

Someone had to:

1. Locate the relevant XS files.
2. Download them.
3. Run a separate processor.
4. Compress or archive the source files.
5. Track what had already been processed.
6. Avoid accidentally processing the same history again.

That could work for a targeted investigation, but it could not create a continuously available operational dataset.

The full pipeline needed to be automatic, resumable, and safe to leave running unattended.

It also needed to survive restarts.

A daemon that forgets its position every time it restarts is not an ingestion pipeline. It is a recurring data incident.

Giving the Streamer Durable Memory

The streamer uses a MariaDB table named broadsoft_xs_sftp_files as durable pipeline state.

For each remote file, it can track:

• Remote filename
• Remote path
• Remote size
• Remote modification time
• Fingerprint identity
• Whether it was selected as the startup anchor
• Stability state
• Download status
• Archive status
• Processing status
• Local incoming path
• Local archive path
• First-seen timestamp
• Last-seen timestamp
• Download timestamp
• Archive timestamp
• Processing timestamp
• Failure stage
• Last error

This table is the memory of the pipeline.

It allows the daemon to restart without guessing which files were completed, which files were only downloaded, which archives were successfully written, or which processing stage failed.

The streamer also maintains a watermark.

The watermark represents the newest remote file that the system has accepted as part of its processed history. A file can qualify if it has been marked either processed or intentionally skipped.

That distinction matters during the first deployment.

If the remote BroadSoft directory contains months or years of XS history, starting the daemon should not automatically unleash a historical ingestion avalanche.

On its first run, when no watermark exists, the streamer finds the newest stable remote file, marks it as the startup anchor, marks it as skipped, and begins processing files that arrive after that point.

Historical backfill can still be performed deliberately, but the production daemon starts at the current edge.

Following the Frontier Instead of Rewalking History

The remote XS directory may contain a large number of old files.

The streamer still has to perform an SFTP directory listing so it can discover new arrivals. That operation remains one of the unavoidable costs in the design.

The initial version of the streamer also synchronized every matching remote file into MariaDB during every polling cycle.

That worked, but it was wasteful.

Every cycle repeatedly touched files that were already far behind the watermark. The system performed SQL lookups and updates for history that could no longer affect the next processing decision.

The optimized design works on a frontier.

After listing and sorting the remote files, the daemon focuses only on:

• Files newer than the current watermark
• Or, during first-run initialization, a small slice of the newest remote files

This changes the problem from “reconcile the entire directory every twenty seconds” to “inspect the moving edge of the stream.”

That is a much smaller and more stable workload.

The database access pattern was also reduced.

The older behavior effectively performed:

1. A SELECT for one fingerprint.
2. An INSERT or UPDATE.
3. Another SELECT.

That sequence happened for each file during each synchronization pass.

The revised approach performs one batched read of the existing tracking rows for the active frontier and then one upsert for each relevant file.

The difference may sound small when looking at one file.

Across a large remote directory and a continuously running daemon, it is substantial.

Parse Once, Archive Once

One of the largest performance improvements came from changing how downloaded files were parsed and archived.

The old processing path looked roughly like this:

1. Download the plain-text XS file.
2. Compress the file into a .zst archive.
3. Open the compressed archive.
4. Decompress it.
5. Parse the decompressed content.

That meant the same data was being read and transformed multiple times immediately after download.

The new streamer opens the downloaded text file once.

That single byte stream is simultaneously:

• Fed into the XS parser
• Written through a Zstandard compression writer
• Stored as the retained .zst archive

The file is parsed and archived in one pass.

After successful parsing, indexing, database updates, and archive finalization, the temporary uncompressed download is deleted.

This eliminates an entire compress, reopen, decompress, and parse cycle.

It also aligns the archive with the exact byte stream that was processed.

Routing the Full Logset Into OpenSearch

Running the streamer with --fullprocess enables complete raw XS ingestion.

Each assembled record is classified and routed into one of several OpenSearch index families:

• helix-broadsoft-xs-registrations-* for SIP REGISTER
• helix-broadsoft-xs-subscriptions-* for SIP SUBSCRIBE
• helix-broadsoft-xs-sip-traffic-* for other SIP messages
• helix-broadsoft-xs-broadsoft-internal-* for internal or non-SIP records

This separation is important.

Dumping everything into one enormous index would simplify the first hour of development and complicate nearly everything afterward.

Different record families have different fields, query patterns, retention requirements, and operational purposes.

Registration analysis is not the same workload as reconstructing a SIP call.

Subscription-state research is not the same workload as searching internal BroadSoft events.

Routing records into purpose-built index families creates cleaner mappings and makes it easier to manage growth over time.

OpenSearch writes are performed through a bulk writer.

The default batch size was increased to 1,500 documents, reducing the number of HTTP requests and lowering per-document indexing overhead.

That value still has to be watched alongside cluster health, node pressure, shard behavior, and indexing latency. There is no universally perfect bulk size.

For our environment, however, the larger batch significantly improves throughput compared with the earlier smaller default.

Deterministic IDs Prevent Duplicate History

Every raw OpenSearch document receives a deterministic ID based on the remote source path and record offset.

That means a specific record in a specific XS file always resolves to the same document ID.

If a file is reprocessed, Helix overwrites the same logical documents instead of generating a second copy of every event.

This is a critical property for a resumable pipeline.

Without deterministic IDs, retrying a failed or uncertain ingestion could duplicate millions of records.

Deduplication after the fact would be expensive, unreliable, and unpleasant in all the ways only distributed data cleanup can be unpleasant.

With deterministic IDs, reprocessing becomes idempotent at the document level.

Extracting More Than Raw SIP Traffic

The streamer does not stop at indexing raw records.

It also creates higher-level operational state.

Subscription-State Snapshots

Records generated by the BroadSoft EventNotification subsystem may contain XSI subscription-state information.

The streamer normalizes these records and writes snapshots into:

helix-broadsoft-subscription-state

Snapshots are deduplicated using a hash derived from the subscription key and normalized payload.

If BroadSoft emits the same logical state repeatedly, Helix does not need to create a new document every time.

That makes the index useful as a state history rather than a warehouse of identical repeats.

Endpoint Registration State

The parser also correlates BroadSoft registration context with successful SIP registration responses.

It ties together:

• Registration candidate records
• Successful SIP 200 REGISTER responses
• Shared callhalf or session relationships
• Enterprise context
• Normalized endpoint identity

The resulting state is upserted into the MariaDB table:

bs_enterprise_endpoints

This gives Helix a durable SQL view of endpoint registration activity by enterprise and endpoint.

The XS logs are therefore serving two purposes at once.

They remain available as deep forensic evidence in OpenSearch, while also feeding normalized operational state into SQL.

Failure Handling Is Part of the Architecture

A pipeline that only works when everything succeeds is a demo.

The streamer tracks the major stages of each file independently:

• Download
• Archive
• Process

If a stage fails, the system records where it failed and stores the latest error.

That makes failures inspectable and recoverable.

An SFTP problem is different from a compression problem.

A malformed record is different from an OpenSearch bulk failure.

A MariaDB error is different from a local filesystem issue.

Keeping stage-specific state prevents every problem from collapsing into a generic “file failed” status.

It also allows us to resume work without losing the identity and history of the affected file.

What Codex Actually Contributed

It would be easy to summarize this project by saying that AI wrote a parser.

That would be inaccurate, and it would undersell both the engineering problem and the value of the tool.

Codex did not arrive with an understanding of our BroadSoft environment, Helix, our operational needs, our OpenSearch architecture, or the failure modes we had already encountered.

It did not make the architectural decisions on its own.

What it provided was leverage.

It helped us:

• Explore parser designs
• Refactor working but inefficient approaches
• Compare data-flow options
• Identify repeated I/O
• Move full processing into the streamer
• Develop the tracking and watermark logic
• Tighten SQL access patterns
• Build deterministic indexing behavior
• Iterate on failure handling
• Turn operational observations into code changes quickly

The expertise still mattered.

Someone had to understand why partial remote files were dangerous.

Someone had to recognize which BroadSoft records were related.

Someone had to decide what should live in OpenSearch, what should live in MariaDB, and what needed to remain in the forensic payload.

Someone had to evaluate whether the output was operationally correct.

AI did not replace that judgment.

It allowed that judgment to be applied across a much larger amount of code and a much faster sequence of experiments.

That is the part of AI-assisted engineering that is often missed.

The real advantage is not asking a model to build a mysterious black box.

It is using the model to shorten the distance between an engineering observation and a testable implementation.

The Result

The simplest mental model for the final system is this:

It watches the moving edge of the BroadSoft XS files, waits until a new file settles, downloads it once, parses it once, archives it once, writes the Helix outputs, and remembers where it left off.

The production service normally runs with options equivalent to:

/opt/helix/bin/xs-logs-streamer \
  --config /opt/helix/config/config.yml \
  --incoming-dir /opt/helix/bw-xs-logs/incoming \
  --archive-dir /opt/helix/bw-xs-logs/archive \
  --log-file /opt/helix/logs/xs-logs-streamer.log \
  --poll-interval 20s \
  --stable-min-age 2m \
  --fullprocess \
  --write

The logs are not processed byte-for-byte in real time.

They are processed as soon as they can be safely treated as complete.

In practice, that means Helix is usually only one or two minutes behind BroadSoft.

For us, that is the right tradeoff.

We gain the integrity of processing stable files while still having the data available quickly enough to investigate current incidents.

What was previously considered an unmanageable collection of enormous rolling text files is now a continuously updated operational data source.

We can search raw SIP exchanges.

We can reconstruct call flows.

We can inspect internal BroadSoft events.

We can analyze registrations and subscriptions.

We can track endpoint state.

We can preserve the original logs in compressed form.

We can restart the service without losing our place.

And we can do all of it without repeatedly grinding through the entire history of the remote directory.

Everyone said the full BroadSoft XS logset could not be processed.

The real answer was that it could not be processed effectively using the old assumptions.

Once we stopped treating it as a pile of files and started treating it as a stateful, moving ingestion frontier, the problem changed.

Codex helped us move through that change much faster.

But the breakthrough was not magic.

It was architecture.