Custom detection rules (authoring, validation, backtesting)

CoPilot Searches ships a large shared catalog of detection rules, but every environment has detections that only make sense locally: a naming convention, an internal application, a tuning exception. The Detection Rule Editor lets you author those rules in CoPilot, check them, prove them against your own data, and publish them to a repository you control — where they merge back into the same catalog you already browse. This page covers the custom-rule half of CoPilot Searches. For browsing, executing and provisioning the shared catalog, see CoPilot Searches.

The mental model

Three ideas explain almost every behaviour in this feature. 1. Rules live in git, not in a CoPilot database. There is no rules table. A rule is a YAML file in a GitHub repository, and CoPilot reads those files into an in-memory cache. Two sources feed that cache:

Source	Repository	Access	Who sees it
Catalog	`socfortress/CoPilot-Search-Queries`	Read-only	Everyone
Custom	Your own repo, one per customer	Read and write	Everyone (see Guardrails)

Both are pulled the same way and merged into one catalog, each rule tagged with its provenance. That is why a published rule shows up beside catalog rules with a Custom badge rather than in a separate list, and why the grid’s Source filter (catalog / custom) exists. The consequence worth internalising: git is the source of truth. Your rules are versioned, reviewable, diffable, and portable to another CoPilot deployment by pointing it at the same repository. Equally, deleting a rule means deleting the file — there is nothing to delete in CoPilot. 2. The only CoPilot-side state is a pointer, not rule content. Per customer, CoPilot stores {repo, branch, token, enabled} — nothing else. It lives in MinIO (bucket copilot-searches, key custom-repos/<customer_code>.json), so enabling this feature required no database schema change. 3. Custom rules are Graylog-only. The editor targets rules with a top-level graylog.query string and an optional aggregation block. An OpenSearch DSL search: block or a parameters: block is rejected outright. This is not an arbitrary restriction: validation and backtesting run entirely through Graylog’s Search API, so a rule the editor cannot execute against Graylog is a rule it cannot verify for you.

Catalog rules that carry a search: block still work as they always have — they are simply not editable in this editor.

Anatomy of a rule

Keys have a canonical order. It is enforced as a warning (except aggregation, which must come after graylog — that one is an error), and it exists so that every rule in the catalog reads the same way.

Key	Required	Type	Notes
`name`	Yes	string	Human-readable rule name
`id`	Yes	string	A UUID — generate a fresh one per rule
`version`	Yes	integer ≥ 1	Bump when you change a published rule
`schema_version`	Yes	quoted string	`"1.0"` — unquoted `1.0` parses as a number and is rejected
`date`	Recommended	quoted ISO date	`"2026-08-26"`
`author`	Recommended	string
`description`	Yes	folded string	What it detects and why it matters
`data_source`	Recommended	list of strings	Quote any entry containing `:`
`how_to_implement`	Optional	folded string	What must be collected or enabled
`known_false_positives`	Optional	folded string	Benign triggers and how to tune them
`response`	Recommended	object	`risk_score` (0–100), `severity` (`low`/`medium`/`high`/`critical`)
`tags`	Recommended	object	`asset_type`, `mitre_attack_id[]`, `custom_tags[]`, `product[]`, `security_domain`
`graylog`	Yes	object	Only `query` — any other key is an error
`aggregation`	Optional	object	Threshold rules; must come after `graylog`

“Required” means publishing fails without it. “Recommended” means you get a warning but can still publish.

A simple match rule

name: Audit Policy Changed
id: 7b1f0a3e-2c44-4a1f-9f3b-6c1d2e8a5b90
version: 1
schema_version: "1.0"
date: "2026-08-26"
author: SOCFortress LLC
description: >
  Detects modification of the Windows audit policy, a common precursor to
  anti-forensic activity.
data_source:
  - Windows Security Event Log
how_to_implement: >
  Requires the Security channel to be collected and audit policy change
  auditing to be enabled.
known_false_positives: >
  Legitimate GPO rollouts will trigger this. Tune by excluding your
  configuration management service accounts.
response:
  risk_score: 50
  severity: medium
tags:
  asset_type: Endpoint
  mitre_attack_id:
    - T1562.002
  custom_tags:
    - audit
  product:
    - Wazuh
  security_domain: endpoint
graylog:
  query: data_win_system_eventID:"4719"

A threshold rule

Add the aggregation block when the signal is a rate, not a single event — “30 failed logons for one user in 10 minutes” rather than “a failed logon”.

graylog:
  query: data_win_system_eventID:"4625"
aggregation:
  enabled: true
  function: count            # count | distinct_count
  field: null                # required only when function is distinct_count
  group_by:
    - data_win_eventdata_targetUserName
  window: 10m
  threshold: 30
  condition: ">"             # one of  >  >=  <  <=  ==

Field	Meaning
`enabled`	`false` keeps the block as documentation without activating it
`function`	`count` (how many events) or `distinct_count` (how many unique values of `field`)
`field`	Must be `null`/omitted for `count`; required for `distinct_count`
`group_by`	Evaluate the threshold per value of these fields — usually a user, host, or source IP
`window`	Sliding window: `30s`, `10m`, `1h`, `1d`
`execute_every`	Optional — how often the check runs once provisioned
`threshold` / `condition`	Integer ≥ 1, compared with `>`, `>=`, `<`, `<=`, `==`

Before you start

Create the repository. A GitHub repository you control. Rules must live under detections/ and end in .yaml or .yml — CoPilot scans the repo tree and ignores everything else. Nested folders are fine (detections/custom/windows/my-rule.yaml).
Create a GitHub token. A PAT with Contents: read and write on that repository. A fine-grained token scoped to the single repo is the right choice — the token is used only to read rules and commit rule files.
Register the repo in CoPilot. Go to Agents → CoPilot Searches → Custom repos and add: Customer, Repository (owner/name), Branch, and the token. Press Test first — it dry-runs the pull and reports how many detection YAMLs it found, so a wrong branch or a bad token surfaces immediately instead of as an empty catalog later.
Confirm the customer’s Graylog stream. Backtesting resolves the customer’s Graylog stream from their customer metadata. It must be a real Graylog stream id (a 24-character hex ObjectId); a placeholder value is rejected with a clear message. Everything else works without this — only backtesting needs it.

Creating a rule

Agents → CoPilot Searches → Create rule opens the editor at /copilot-searches/editor.

Start from a template. The template menu offers a simple match rule and a rule with aggregation. A fresh editor loads the simple rule with the aggregation block present but enabled: false, so the shape is discoverable without being active. Each template generates a fresh UUID and today’s date for you.
Write the rule. Required keys are highlighted and cannot be deleted — name, id, version, schema_version, description, graylog, query. Their values stay freely editable; attempting to remove the key itself is rejected with a toast. Everything else, including the whole aggregation block, is yours to change or delete. The right-hand panel has a Graylog syntax tab: a searchable cheat-sheet of Graylog/Lucene query syntax (fields, _exists_, boolean operators, wildcards, ranges, regex, escaping) with click-to-copy examples.
Read the validation panel. Validation runs automatically as you type (debounced), splitting findings into Errors and Warnings. Click any finding to jump to the offending line, which flashes briefly.
Backtest against real data. Press Backtest, pick a customer and a look-back window. See Backtesting below.
Publish. Publish is enabled only once validation is error-free. See Publishing.

Your work is auto-saved to the browser’s local storage as you type and restored when you come back. Loading a template deliberately starts fresh.

Validation logic

Validation runs in four layers. The first three are pure text analysis and run live in the editor; the fourth needs a customer’s real data and therefore runs inside the backtest.

Layer	What it checks	Where it runs
L1 — Structure	Required keys, types, value ranges, enums	Editor, live
L2 — Reference integrity	Whether the rule’s parts agree with each other	Editor, live
L3 — Query parse	Whether the Graylog query is well-formed	Editor, live
L4 — Field existence	Whether the fields exist in this tenant’s data	Backtest

Errors block publishing. Warnings never do — L2 in particular is deliberately advisory, because an enrichment pipeline can create fields the linter cannot see.

L1 — Structure

Errors: a missing required key; a wrong type; schema_version that is not a quoted string; a graylog block containing anything other than query; a forbidden search: or parameters: block; aggregation placed before graylog; field set when function: count; field missing when function: distinct_count. Warnings: an id that is not a UUID; top-level keys out of canonical order; a window that doesn’t look like 10m; long text fields not using a folded scalar (>); a data_source entry containing : that isn’t quoted; a missing recommended key (author, date, data_source, response, tags).

L2 — Reference integrity

All warnings. These catch rules that are structurally perfect but internally inconsistent:

response.message interpolates $some_field$ , but the query and aggregation never use that field — it would render blank in the alert.
A risk_objects or threat_objects entry points at a field the rule never touches, so the object arrives empty.
A mitre_attack_id that isn’t shaped like T1059 or T1059.001.
A risk_score far outside the usual band for its severity (roughly: low 1–40, medium 25–75, high 55–95, critical 75–100). The bands are wide on purpose — only clear mismatches are flagged.
A date that isn’t a quoted ISO date.

L3 — Query parse

A lightweight parse of graylog.query that catches what Graylog would reject at runtime: an unbalanced double-quote, unbalanced parentheses, an unclosed /regex/, or a query starting or ending on a dangling AND / OR / NOT. Quote-aware, so parentheses inside a quoted phrase don’t produce false alarms. One performance advisory (warning only): a leading .* in a regex scans everything — anchor it or use a literal where you can.

L4 — Field existence

Runs during the backtest, where a customer and stream are in scope. Every field used in the query, group_by, or aggregation field is compared against the fields that actually appear in that tenant’s stream, and anything missing is reported as a warning on the results. This is the check that catches the typo you cannot see — eventlD with a lowercase L instead of eventID is a perfectly valid query that will simply never match. It is skipped when the stream has fewer than five known fields, since an empty or brand-new stream would flag everything.

Backtesting: what it actually does

A backtest answers “what would this rule have done?” against one customer’s real Graylog data, before the rule goes anywhere near production. The flow: resolve the customer’s Graylog stream → run graylog.query over the chosen window through Graylog’s Search API → pull the matching events once → compute everything locally. What you get back:

Total hits over the window.
A sparkline of matches per time bucket, so a steady trickle is visually distinguishable from one spike.
Sample events — click any row to open a full-event inspector with a field filter, click-to-copy values, collapsible internal gl2_* fields, and Copy JSON.
Top values per field, which is how you tell “one noisy host” from “genuine spread across the fleet”.
Missing fields, the L4 check above.
For threshold rules: how many alerts the rule would have raised, the top offenders, and a threshold sensitivity row showing how that alert count changes at neighbouring thresholds. This is the fastest way to pick a threshold that isn’t a guess.

Bounds and honesty about them:

Bound	Value
Look-back window	5 minutes to 30 days (default 7 days)
Events fetched for analysis	10,000
Sample events shown	20

When a rule matches more than the fetch cap, the result says so explicitly and the total is labelled as a lower bound — a truncated result is never presented as a complete one. The threshold simulation uses the same count / distinct_count semantics that CoPilot uses when it provisions the rule to Graylog, so the simulated alert count reflects what the deployed rule will actually do rather than an approximation.

Backtesting is entirely read-only: it queries Graylog and writes nothing. Run it as often as you like.

Publishing

Publish commits the rule to the customer’s own repository via the GitHub Contents API — a direct commit to the configured branch, using the write token from that customer’s repo pointer. Four things happen before a commit is made:

The rule must be lint-valid. Any error blocks the publish, with the findings returned so you can fix them.
The path is constrained to detections/**.yaml (or .yml) — no .., no backslashes, no other locations in the repo. A write token handed to CoPilot must not become a way to overwrite a workflow file or a README. The default path is detections/custom/<slugified-name>.yaml.
The rule id must be unique across the shared catalog and every custom repo. If the id is already taken, you are told exactly where — the catalog, or another customer’s file — and asked to generate a new UUID. This matters because the cache resolves collisions by keeping the first rule it loaded (the catalog always wins), so a duplicate id would mean your rule silently never appears.
Same file, same rule = update. Re-publishing to the path a rule already occupies is an update, not a collision. That is how you edit a published rule.

Afterwards, the rule appears in the catalog alongside everything else with a Custom badge, filterable via Source → custom. Note: the catalog cache refreshes on a 30-minute cycle, so a freshly published rule is not visible instantly. Use Refresh on the CoPilot Searches page to pull it immediately.

Updating and deleting

There is no in-app editor for an already-published rule — this was a deliberate product decision, not an oversight.

To update: publish to the same path, and bump version.
To delete: remove the file from your repository and refresh the catalog. CoPilot holds no copy to delete.

Guardrails and things to know

Custom rules are visible to every authenticated user, not just the customer they belong to. Rule content is shared across tenants even though the repo pointer is per-customer. Treat rule text as non-sensitive: put tuning logic in it, not customer secrets or internal hostnames you would not want another tenant to read.
Write tokens are never returned by the API. Reads report only whether a token is set. When editing a repo pointer, leaving the token field blank keeps the stored token.
The shared catalog wins id collisions. If a custom rule and a catalog rule share an id, the catalog rule loads and yours is skipped — which is exactly what the publish-time collision check prevents.
Rule authoring, validation, backtesting and publishing are admin/analyst actions. Browsing the catalog is open to customer users.
A repo that fails to pull is visible, not silent. The Custom repos panel shows a per-repo status chip with the last refresh outcome, the rule count, and the error if there was one.

Troubleshooting

Symptom	Cause	Fix
”Customer has no Graylog stream configured” on backtest	No stream in the customer’s metadata	Set the customer’s Graylog stream
”placeholder/invalid Graylog stream”	The stored value isn’t a 24-hex Graylog stream id	Replace it with the real stream id
”has no write token” on publish	The repo pointer was saved without a PAT	Add a token with Contents: read and write
”Rule id … is already used”	Duplicate UUID	Generate a new UUID, or publish to that rule’s existing path to update it
”path must be a YAML file under detections/“	Path outside the allowed area	Use `detections/…/name.yaml`
Published rule doesn’t appear	30-minute cache TTL	Press Refresh on CoPilot Searches
Custom repo loads 0 rules	Wrong branch, rules outside `detections/`, or a token without read access	Use Test in Custom repos — it reports the count and the error
Backtest returns 0 hits but the query looks right	A field name that doesn’t exist in that stream	Check the missing fields warning on the backtest result
Rule fires far more than expected in the backtest	Threshold too low, or `group_by` too broad	Use the threshold sensitivity row to choose a better value

Setup checklist

A GitHub repository with a detections/ folder
A PAT with Contents: read and write on that repository
The repo registered under Custom repos for the customer, with Test passing
The customer’s Graylog stream set (required for backtesting)
One rule authored, validated clean, backtested, and published
The published rule visible in the catalog with a Custom badge after a refresh

CoPilot Searches — browsing, executing and provisioning the shared catalog
Shared rule catalog: https://github.com/socfortress/CoPilot-Search-Queries
Graylog search query language

​The mental model

​Anatomy of a rule

​A simple match rule

​A threshold rule

​Before you start

​Creating a rule

​Validation logic

​L1 — Structure

​L2 — Reference integrity

​L3 — Query parse

​L4 — Field existence

​Backtesting: what it actually does

​Publishing

​Updating and deleting

​Guardrails and things to know

​Troubleshooting

​Setup checklist

​Related resources