Evidence-Grounded LLM Copilot for SOC Alert Triage with Controlled Abstention
Evidence-Grounded LLM Copilot for SOC Alert Triage with Controlled Abstention
Description
Details
Context and Problem Statement
SOC analysts face alert overload and high false-positive rates. LLMs can assist with event correlation and reporting, but fluent unsupported explanations can create dangerous overconfidence.
Research Question
How can an LLM be constrained to generate a triage hypothesis only when it is traceable to observed log events, and how should abstention quality be measured?
Proposed Approach
Use RAG over normalized logs, mandatory event-ID citation, MITRE ATT&CK mapping, and a calibrated decision layer that chooses among recommendation, request for additional evidence, and human escalation. Adversarial tests include deceptive logs and prompt injection in attacker-controlled fields.
Expected Contribution
A benchmark jointly measuring triage accuracy, evidence traceability, hallucination, and abstention.
Expected Prototype
A monitoring-stack copilot generating source-backed summaries, confidence levels, reversible action recommendations, and an audit trail.
Datasets
CICIDS2017, CSE-CIC-IDS2018, UNSW-NB15, and controlled testbed logs.
Challenges
Sensitive logs, prompt injection, schema drift, expert evaluation cost, and over-automation risk.
Research Question
Innovation
Expected Deliverable
Technologies
Required Skills
- Information Security and Log Analysis
- Natural Language Processing and RAG
- Systems Administration and Containerization
- Experimental Evaluation Design
Datasets
- CICIDS2017 / CSE-CIC-IDS2018
- UNSW-NB15
- Logs generated in a controlled testbed