BroadWorks XS Logs: Why “Index Everything” Is the Wrong Answer
BroadWorks XS logs are where many call truths live, but they arrive as noise at scale.
On a busy system, roughly six hundred thousand XS log lines are generated every one to five minutes. At that volume, blindly indexing everything is not just inefficient, it actively works against understanding. Most log lines have no value on their own. Their meaning only emerges when they are correlated into a call.
The first decision was to treat logs as transient material, not permanent records.
XS log files are pulled from the BroadWorks server using rsync and land in a staging directory in their uncompressed form. This staging area exists purely for processing efficiency. Once the file arrives, it is split into ten smaller chunks, each around sixty thousand lines. Those chunks are immediately compressed using zstd and moved into an archive directory. What began as roughly ten three megabyte files becomes ten compressed files of roughly three hundred forty kilobytes each. Storage becomes cheap, predictable, and fast to read later.
Nothing else happens until there is a reason to ask a question.
When a deep call flow search is needed, the input is intentionally narrow: a date range and an identifier, typically a phone number or a MAC address. The system first resolves that identifier to the correct enterprise or tenant context. That single step prevents unnecessary scanning and keeps the problem bounded.
From there, the archived chunks for the selected time window are processed in parallel. Each compressed chunk is decompressed on the fly and handled by Go routines. Throughput is limited only by disk I/O and the number of CPU cores dedicated to the task. There is no single threaded parser crawling through massive files. The work is sliced, fanned out, and consumed concurrently.
As each chunk is processed, individual call halves and call segments are extracted and written into Redis. Redis is not a cache in the traditional sense. It is short term working memory. Every fragment of every call that matches the query lives there temporarily. As more data flows in, Redis fills in the missing pieces until full call paths begin to take shape, including all legs, transitions, and termination points.
This call processing includes all SIP transactions as well as internal BroadWorks system messages. Those internal messages often never surface in traditional troubleshooting views, but they are critical for understanding how and why a call behaved the way it did. Having both the SIP signaling and the internal system events in the same assembled call context dramatically reduces guesswork during investigations.
Only after processing completes is anything written to OpenSearch. What gets indexed is not raw log data. It is correlated call information assembled from start to end, structured specifically for search, visualization, and aggregation. OpenSearch never sees the flood. It only sees the what you tell it you need.
Each call is represented by a single structured document. That document contains a main record holding all call level metadata, identifiers, enterprise context, ingress and egress details, timing, and final disposition. Attached to that record is an events array. The events array holds JSON objects representing every XS log line that participated in that call, grouped by call half and ordered in sequence.
This single document is what is ingested into OpenSearch and becomes the unit of search, analysis, and visualization.
The size of the events array varies widely. Some call flows are simple and contain five or six events. Others may contain hundreds. Calls that traverse multiple auto attendants, hunt groups, transfers, and retries can easily reach two hundred distinct log events before termination. The structure holds regardless of complexity.
The important part is that OpenSearch is indexing completed call narratives, not fragments. Queries operate on whole calls, not orphaned log lines. Correlation happens once, upstream, instead of being re invented at search time.
This allows a single call to be traced end to end across the entire system without grepping files, manually stitching timelines, or guessing. You can see where a call entered, how it moved, where it branched, and why it failed, using the same tools you would expect from any modern search interface.
Most off the shelf dashboard and log visualization tools do not know what to do with data in this format. They are built around flat events, not structured call narratives. A single document that contains both call level metadata and a variable length array of ordered event objects breaks most assumptions those tools make.
Because of that, a custom search interface is used. The same interface that initiates the log pull and correlation process is where the results are explored once processing completes. There is no handoff between tools and no second step where context is lost. You ask the question, wait for the data to assemble, and then immediately start searching and tracing within the same view.
That interface also controls the lifecycle of the research itself. Once the investigation is complete, the results can be dumped and discarded, or retained for future reference. The raw archives remain unchanged, and only the correlated output lives on if it is actually useful.
This keeps the system honest. Data exists to answer questions, not to accumulate indefinitely.
--Parse You Logs Efficiently
-Bryan