Meet the Harbor group, a name we invented. They are made up, a composite of a few engineering teams we have worked alongside at Cotocus, but their week is real enough that you will probably recognise your own. The Harbor group looked after the platform for a mid-sized software company, and on a damp Monday their security manager asked a question that had no quick answer: “If someone had copied our customer export last spring, could we prove it?”
That question is the doorway into SecDataOps, the discipline that the Ops Atlas describes as engineering and governing security telemetry so that evidence is complete, trustworthy and affordable. The Atlas’s own entry, the SecDataOps page in the Ops Atlas, also says the term is young and vendor-driven, so we will treat the story as an illustration, not a prescription.
Monday: the question nobody could answer
The Harbor group’s first instinct was to search. They opened the SIEM, typed a date range from spring and found nothing, which could mean either “nothing happened” or “nothing was kept”. Within an hour they had their first useful insight: they could not tell the two apart. The retention on the main store was the vendor default, ninety days, and nobody remembered choosing it.
The group’s lead, who had once sat through a talk on dwell times, said what the Atlas says more formally: ninety days is a default, not a standard, and attackers who linger for months turn it into a blindfold. The Atlas backs this with figures from a 2026 casework review of the previous year’s investigations, where the typical intruder was found after about two weeks, yet espionage-type intrusions ran around four months and one campaign roughly 400 days.
Tuesday: drawing the supply chain on a whiteboard
Instead of buying anything, they drew the chain. Producers on the left, then collection, then the pipeline, then storage tiers, then consumers on the right. The Atlas gives that picture a name, the evidence supply chain, and drawing it exposed something odd: every box had a person who sort of looked after it, and none had a person who was accountable for it. The firewall team owned the firewall but not its log forwarding. The platform team owned forwarding but not the format. The analysts owned the rules but not what fed them.
So they gave each box a single owner and made a coverage register, a plain table that gave every source a grade from zero to five for machine coverage, populated fields, speed, uniformity and how much history survived. The identity provider scored a four. Endpoint logs scored a two. Cloud object storage scored a one, because, as they discovered, the object-level events were opt-in and had never been switched on.
Wednesday: the contract that changed the tone
The contract template fit on a single page. It asked who owns the source, what format it uses, which fields cannot be empty, how fast it must arrive, where it is stored, for how long, what test event proves life, what it costs and what is still missing. The conversation it provoked was the useful part. When the cloud team wrote “freshness” they said “about an hour”. The security manager said “five minutes”. They settled on a number each side could live with and wrote it down. The sample targets in the Atlas gave them a reference point without dictating the answer.
The contract also had a line for “known gaps”, and that was where honest facts went. One entry noted that a warehouse’s access-history view arrives hours late and omits failed queries. Writing it down turned an accident waiting to happen into an accepted, documented limit.
Thursday: a parser bites back
On Thursday, an engineer wrote a quick test for the firewall parser and found that malformed destination ports were being coerced to zero. Dashboards had shown a persistent trickle of denied traffic to port 0, and the team had written it off as scanners. It was the parser quietly turning garbage into data, the exact trap the Atlas warns about in one common pipeline tool.
They fixed it by anchoring the pattern and routing misfits to a quarantine stream. They added fixtures, sample records stored beside the parser, and wired the tests into continuous integration. That afternoon, a vendor changed a field name in a software update and the build failed in minutes instead of the dashboards drifting for weeks.
Friday: the money conversation
On Friday, finance joined. The raw firewall flows ran at roughly 400 gigabytes a day, all kept hot. The Atlas’s worked example became their template: keep summaries of about 8 gigabytes a day in the fast tier, ship the raw copy to a low-cost lake held for 400 days, and the combined bill drops to roughly 164 where the old way cost about 400, even though raw history now reaches back more than a year. Real numbers differ, but the shape of the argument won over finance.
Before touching the live stream they followed the cutting playbook. They listed the detection rules that used the stream, classified its fields, chose summarising as the lever and replayed a week of raw data. Two rules failed in replay because a summary had dropped a field they needed. They put the field back. Had they skipped the replay, they would have found that out during an incident.
The following Monday: the silence test
A week on, the Harbor group did something that felt faintly mischievous: they switched off a lab source and started a stopwatch. Nothing alerted for several hours. That became the first line of the next quarter’s plan. They added source-health monitors that compared each stream’s volume with its contract, with a page after fifteen minutes of nothing, and they ran the test again. The second time, the page came in under twenty minutes.
The detail that surprised them most
Of everything they learned, the one that made the room go quiet was about timestamps. Two systems in the chain recorded time in different zones, and one recorded only when a record arrived, not when the event happened. Reconstructing a sequence of actions across those sources had been guesswork. The Atlas’s advice is to record the moment of the event and the moment of processing separately, in UTC ISO 8601 form down to the millisecond. It sounds fussy until you are the person assembling an incident timeline at midnight and every source disagrees about the hour.
They also found out, almost by accident, that replays after an outage were being counted twice. Without a stable unique identifier on each event, deduplication was impossible, and the dashboards had been inflated every time a collector restarted. A small change to carry a stable identifier through the pipeline cleaned up the numbers and, for the first time, made an event count something they could trust.
What the story shows
- The question that exposes a gap is usually simple and slightly embarrassing.
- Owners and contracts turn shared worries into specific tasks.
- Replay beats hope when you change a stream.
- Keeping raw data in a cheap tier gives you room to correct mistakes later.
- Silence is a signal only when someone is measuring it.
On the Atlas’s five-rung ladder the Harbor group moved from the first rung to the third for its most important sources within a quarter, taking the registers and monitors first, contracts and fixtures next, then re-tiering and a baseline report.
Where to read further
If you are a data or machine-learning team wondering how this connects to your own pipelines, our roadmap for data scientists and engineers building ML delivery covers the neighbouring territory, and our article on shortening IT problem resolution with modern AIOps tooling shows how operational data gets used once it is trustworthy. Those are different disciplines, but they share one dependency: good data arriving reliably.
As for the Harbor group, their security manager got an answer to the spring question, eventually. Not a perfect one, since the old logs were gone, but the follow-up question changed to “how long would it take us to know?”, and for the first time they had numbers.