Last spring our platform team — three engineers, one very tired rotation schedule, and a shared chat room that had become the de facto office — decided to try ChatOps properly: operational work run through the channel itself, a bot announcing events and performing only the operations an explicit list permits, so everything stays visible, permissioned, and auditable. We had read the theory, quoted the detailed ChatOps reference at the Ops Atlas at each other, and still nearly made every classic mistake in the first fortnight. What follows is the path we actually took, arranged into four weeks, written down while the details are fresh. Your tooling will differ; the order of operations, we suspect, will not.
Week Zero: Choosing the First ChatOps Command
The first decision was the most important and the least technical: what should the bot do first? The confident answer — “deployments, obviously” — is the one the history books gently mock. GitHub, the practice’s birthplace, famously ran deploys as a chat queue with CI guards and locks, moving hundreds of releases a week through the room; but they had years of foundation under that confidence, and by 2021 even they had relocated deploy state into a dedicated state machine behind its own interface, keeping the channel as the trigger rather than the memory. A team’s first command should be incapable of waking anyone at night. Ours was a read: a backup status lookup that answered with the last successful run per database. Zero write access, zero blast radius, and it made the daily “did the backup run?” thread in our team channel obsolete on day one.
Week One: Writing It Down Before Building It
Before any code, we wrote a one-page contract for the command: its name, what it accepts, who may run it, where it may run, and what its reply must contain. The exercise felt bureaucratic for about twenty minutes, and then it caught two problems — the name we had chosen collided with an existing tool’s flag, and the reply format would have dumped connection strings into a room with an intern in it. The contract also became our first allow-list entry, which is the heart of the whole discipline: the bot knows a fixed menu, and typed prose never approaches a shell. Commands get written, reviewed, and added deliberately, the way pull requests are. Everything else simply is not a command.
The contract habit scales up as the list grows, too. Each new command brings its own line: what it does, who may call it, which room it answers in, what evidence it leaves behind. Read-only commands sign the shortest contracts; anything that writes state signs a longer one, with a named approver and a rollback sentence. Two months in, the document doubles as onboarding material — a new engineer can read every capability the room has, in plain language, over one coffee. That is a paperwork return nobody expects and everybody keeps.
Week Two: The Gateway Gets a Memory
The second week was plumbing with consequences. Our gateway had to do four things every time a message arrived: verify the request genuinely came from the platform, work out which human was behind it, check that human’s permissions, and log the outcome somewhere sturdier than scrollback. The verification part is unglamorous but non-negotiable — Slack hands you a signature to check and a short window to acknowledge within, and skipping that check means trusting any payload that claims to be Slack. Identity was the part we underestimated. The room shows friendly names and faces; the gateway must ignore all of that and resolve the sender’s permanent platform identifier against our own records, because display names are costumes anyone can wear. We gave the bot its own scoped credentials, separate from any human’s, wrote its token into a proper secret store, and set the audit log to an append-only file the chat platform could not touch. By Friday the command ran end to end — verified, authorised, executed, logged — and had cost us exactly one embarrassing bug involving a timezone.
Week Three: Opening the Channel to the Team
With one safe command working, we invited the wider team into the room and made our first cultural rule: operational talk stays in the channel, not in direct messages. DMs feel efficient and quietly destroy the point of the exercise — half the team stops seeing what is happening, and the record forks into private histories nobody can audit. The backup lookup became habitual within days, mostly because answering “did the backup run?” in a shared room teaches everyone the answer without interrupting anyone. We also let the bot announce a handful of pre-existing events, curated hard, using three rules we stole from better-run rooms:
- every notification names the thing affected in its first four words;
- every notification implies a next step, even if the step is “no action needed”;
- anything that never once prompted action in a fortnight gets silenced without ceremony.
The room stayed quiet enough that people kept it open, which we came to see as the real adoption metric.
A note on what we deliberately postponed. Pipelines, restarts, and anything with write access stayed out of scope for the whole month, despite a certain amount of hallway pressure to “automate something real”. The temptation is understandable and the timing is wrong: write commands amplify whatever identity and audit work you have actually finished, not what you plan to finish. Earning the right to act takes the full four weeks; spending that right carelessly takes about a minute.
Week Four: Numbers Before Applause
The fourth week was honestly the most fun, because it was the week we stopped guessing. We counted command runs and successes, refusals and their reasons, and notifications per person per shift. The refusal log earned its keep immediately: a teammate had tried the command from the wrong room, which was not a discipline problem but a layout problem, fixed by making the command work sensibly in both places. The notification count explained a fatigue nobody had articulated — our alert stream had the classic ungrouped shape, hundreds of messages a week multiplied across the team into something like thirteen hours of collective attention a month, and pruning it felt less like tuning and more like opening a window. Anyone who has fought that battle will recognise the terrain from practical approaches to cutting IT alert fatigue; the channel simply makes the arithmetic visible enough to act on.
What Our First ChatOps Command Taught Us
Four lessons survived contact with reality. First, small commands earn trust faster than impressive ones — the backup lookup changed how the team worked, and it could not have broken anything. Second, the contract is the product: the one-page spec caught more problems than the code review did. Third, identity is the whole security story; signatures and tokens matter, but the discipline of resolving permanent identifiers is what keeps a friendly-looking room from becoming a spoofing opportunity. Fourth, fail loudly and report changes — a lesson the incident-tooling world learned long ago, where a paging command announces its own failure and falls back rather than shrugging, and status replies describe what shifted rather than re-dumping state. Incident rooms now automate their own lifecycle end to end — declared, channel created, people paged, timeline kept, archive closed — which is the direction we are heading next, with the pipeline work and the AI-operations reading our team already follows in structured AIOps training for modern IT teams and the faster diagnosis paths described in speeding up problem resolution with modern AIOps platforms.
If you are about to attempt the same month, our honest advice is to steal the sequence and not the specifics. Pick a command that cannot bite. Write its contract first. Give the gateway a memory for identities and a filing cabinet for receipts. Open the room, then defend its quietness fiercely. Measure before you celebrate. The practice is older than it looks and gentler than its reputation, and a single well-made command will teach you more about it than a quarter of slide decks — we know, because we sat through the decks, and the backup lookup still taught us more.