AI Analyst — Analyst review workflow

Documentation Index

Fetch the complete documentation index at: https://docs.socfortress.co/llms.txt

Use this file to discover all available pages before exploring further.

Every report Talon produces is a draft. The review workflow lets a SOC analyst grade it, correct what’s wrong, teach the agent with a lesson, and — if needed — replay the investigation with a different template. Feedback is aggregated per customer so you can see which templates are reliable and which need tuning. This page is for operators (analysts reviewing reports). For the architecture and deployment guide, see AI Analyst (Talon).

---

​

Why review matters

AI reports are fast, consistent, and cheap — but they’re not infallible. Without a feedback loop you can’t tell:

• Whether the agent picked the right investigation template
• Whether IOC verdicts match reality (was that hash really malicious?)
• Whether the severity call was appropriate for your environment
• What recurring patterns the agent should treat as benign (and stop paging you about)

The review workflow turns every investigation into a training signal. Lessons you capture land in MemPalace — the agent’s persistent memory — and are surfaced on the next investigation for that customer.

---

​

Where it lives in the UI

Reviews and feedback live across two places:

---

​

Reviewing a report

​

Open the report

1. Incident Management → Alerts → open any alert that has an AI investigation
2. Click the AI Analyst tab (pulses if a report exists)
3. Inside that tab, click Review

You’ll see a rubric if this is the first review, or your previous grades pre-filled if you’ve reviewed this report before — submitting again updates the existing review (one review per analyst per report).

​

The rubric

Leave any axis blank if you don’t have a confident opinion — averages ignore nulls.

​

IOC corrections

Below the rubric you’ll see the IOCs the agent extracted, each with its VirusTotal verdict. For every IOC you can mark:

• Verdict correct ✓ — agent’s verdict matches reality
• Verdict wrong ✗ — explain in the note field (e.g. “this IP is our jumphost, not malicious”)

IOC-level accuracy rolls up into the feedback dashboard separately from the overall rubric — useful for spotting when the agent trusts VirusTotal too much or too little for your environment.

​

Submit

Click Submit review (or Update review if you’re editing). The review persists immediately — no pending state.

---

​

Teach the palace

The Teach the palace section under the rubric lets you add a lesson to the agent’s persistent memory. Lessons get retrieved automatically at the start of every investigation for that customer, so the next time the agent sees a similar pattern it already has your context.

​

When to add a lesson

• After a false positive — “Host X runs nightly backups at 02:00 UTC; Sysmon 1 on robocopy during that window is benign”
• After confirming threat intel — “APT group Y targets customer; any outbound to IP range Z should be escalated”
• Asset context — “DC-01 is the primary domain controller; any unsigned binary execution there is critical”
• Environment specifics — “This customer uses piHole at 192.168.1.53; DNS traffic to that IP is expected”

​

Lesson types (rooms)

Lessons are filed into one of four “rooms” so the agent can retrieve them by context: Pick the room that matches how you’d want to retrieve the lesson later.

​

Durable vs one-off

One-off lessons are swept automatically after their TTL — CoPilot tracks the expiry and tells MemPalace to forget them. Keep the palace clean so retrieval stays relevant.

​

Similar-lessons preview

As you type a lesson, a debounced search runs against the palace and shows up to 5 already-stored lessons that overlap your draft. Use it to:

• Avoid duplicating an existing lesson
• See what the agent already “knows” about this pattern
• Phrase the new lesson consistently with prior ones

​

Submit the lesson

Click Queue lesson. The lesson is persisted to CoPilot’s database with status=pending. A background drainer (APScheduler) picks it up within ~30 seconds, POSTs to Talon, and flips the row to status=ingested with a drawer_id handle. After that, the agent will retrieve it on the next investigation for that customer.

---

​

Replay with a different template

If the agent picked the wrong template — or you want to try a different one — click Replay on the Review tab.

1. The modal lists all templates currently deployed in Talon’s groups/copilot/prompts/ directory
2. Pick a template (e.g. sysmon_event_1.txt, windows_defender.txt)
3. Click Replay

Talon spins up a brand-new investigation job for the same alert with your chosen template forced. The original report is untouched — CoPilot now has two (or more) reports for the alert, and the Compare tab lets you view them side-by-side. Good use cases:

• Agent ran the generic template when a specific one would’ve been better
• You want to see how a different template frames the same raw evidence
• A/B test a newly tuned template against the previous one

---

​

Palace consolidation

Over time the palace accumulates lessons. Some expire, some duplicate each other, some get stale. The Consolidate lessons button (Feedback tab → top right) opens a point-in-time digest for the selected customer: Click Copy markdown to paste the digest into a ticket, a team channel, or your own knowledge base — useful for monthly palace reviews. This is read-only — you can’t edit lessons from the drawer. To remove a lesson, either wait for the one-off TTL or mark the row manually via the database / API.

---

​

Feedback dashboard

AI Analyst page → Feedback tab. Pick a customer and see:

​

Tiles

• Total reviews — how much feedback you have for this customer
• Thumbs up % — overall sentiment
• IOC verdict accuracy % — of all IOC corrections submitted, how often did the agent agree with the analyst
• Avg rating (overall) — composite of instructions / artifacts / severity (nulls excluded)

​

Template choice distribution

Stacked bar — correct / partial / wrong. If the “wrong” bar is non-trivial, your agent is mis-selecting templates. Candidates for fixing:

• The template detection logic (rule.groups matching in Talon)
• Adding a more specific template for the miss case

​

Per-template performance

Table showing per-template counts and averages. Use it to spot:

• Templates with consistently low instructions ratings → the template itself may be wrong
• Templates with high template_choice=wrong for a rule type → detection rule is mis-classified
• Templates with low IOC accuracy → the template’s enrichment steps may be flawed

​

Recent reviews

Last 10 reviews with drill-in. Click any one to open a drawer with the full rubric, IOC corrections, and free-text fields.

---

​

Typical workflows

​

Fast triage (10 seconds)

1. Open alert → AI Analyst tab → skim report
2. If report matches reality → thumbs up, submit
3. If obviously wrong → thumbs down, one-line in “Missing steps”, submit

​

False-positive capture (30 seconds)

1. Confirm the alert is benign (e.g. scheduled job, known-good process)
2. Review tab → thumbs down → template choice correct (template was right, signal was noise)
3. Teach the palace → room false_positives, durable, describe the benign pattern with enough detail that the agent would recognize it next time
4. Queue lesson → submit review

​

Template tuning (2 minutes)

1. Report picked a bad template → review it, template choice wrong
2. Note in “Suggested edits” what template should have been used
3. Click Replay → select the correct template → submit
4. Compare tab → confirm the new report is better
5. Report the mis-selection pattern to whoever maintains Talon’s template detection

​

Monthly palace review (10 minutes)

1. Feedback tab → pick customer → Consolidate lessons
2. Review Expiring soon — promote anything still valid from one-off to a fresh durable lesson
3. Review Near-duplicates — pick the better-worded lesson, manually delete the other
4. Copy markdown → paste into your team’s wiki for the customer

---

​

Safety & guardrails

• Reviews are analyst-scoped — one review per analyst per report, updates overwrite. Multiple analysts can each leave their own review.
• Lessons are customer-scoped — a lesson queued for customer 00001 is only retrieved on investigations for that customer.
• One-off lessons auto-expire — use them for temporary context so the palace stays clean.
• Replays don’t mutate the original — every replay is a new job/report; the original stays for comparison.
• Palace consolidation is read-only — you can’t accidentally delete the palace from the UI.
• Don’t put secrets in lessons — lesson text is sent to Talon and embedded by MemPalace / ChromaDB. Treat it as you would a SIEM comment.

---

​

Troubleshooting

---

​

Video context

• AI analyst (alert-context + exclusion-rule assistance): https://www.youtube.com/watch?v=-2srPC-Dw-0