> ## Documentation Index
> Fetch the complete documentation index at: https://docs.socfortress.co/llms.txt
> Use this file to discover all available pages before exploring further.

# ARCHITECTURE

# SOCFortress CoPilot Architecture

## Overview

CoPilot is a “single pane of glass” security operations platform. It centralizes data from Wazuh, Graylog, Velociraptor, Grafana, InfluxDB, and other tools, provides alert/case management, and adds automation such as scheduled collectors, active response, and report generation. The backend is a FastAPI service with a MySQL database and MinIO object storage. The UI is a Vue 3 SPA, with an optional customer-facing portal.

Key entry points:

* Backend runtime: `backend/copilot.py`
* Frontend app: `frontend/src`
* Customer portal app: `customer-portal/src`

## Tech Stack

### Backend

* FastAPI, Starlette, Uvicorn: API server and routing. `backend/copilot.py`
* SQLModel + SQLAlchemy: ORM and DB access. `backend/app/db`
* Alembic: DB migrations. `backend/alembic`
* APScheduler: background job scheduling. `backend/app/schedulers/scheduler.py`
* MySQL: primary database. `docker-compose.yml`, `backend/app/db/db_session.py`
* MinIO: object storage for case/report artifacts, sysmon configs, Velociraptor artifacts. `backend/app/data_store`
* Loguru: logging. `backend` modules
* Requests/HTTPX: integrations with external systems. `backend/app/connectors/*/utils`, `backend/app/integrations/*`

### Frontend

* Vue 3 + TypeScript + Vite: SPA. `frontend/package.json`, `frontend/src`
* Pinia: state management. `frontend/src/stores`
* Naive UI + Tailwind CSS: UI components and styling. `frontend/package.json`, `frontend/tailwind.config.js`
* Axios: API client. `frontend/src/api`
* Cypress + Vitest: tests. `frontend/cypress`, `frontend/vitest.config.ts`

### Customer Portal (Optional)

* Vue 3 + Vite + Pinia + Naive UI. `customer-portal/package.json`, `customer-portal/src`

## Directory Layout & Responsibilities

### Root

* `docker-compose.yml`: default deployment stack (backend, frontend, MySQL, MinIO, MCP, Nuclei).
* `.env.example`: required runtime configuration for connectors and services.
* `build-dockers.sh`: build utility script.

### Backend

* `backend/copilot.py`: application entrypoint, API router mount, startup/shutdown hooks.
* `backend/settings.py`: local env loading (not the primary runtime env in Docker).
* `backend/app/routers/*`: API route modules for each feature/integration.
* `backend/app/auth/*`: authentication, user/role management.
* `backend/app/db/*`: sessions, migrations, bootstrapping, and data seeding.
* `backend/app/schedulers/*`: APScheduler instance, scheduled jobs, job metadata models.
* `backend/app/connectors/*`: connectors for platform services (Wazuh, Graylog, Grafana, etc.).
* `backend/app/integrations/*`: third‑party integrations and per‑customer integration config.
* `backend/app/network_connectors/*`: “network connectors” with customer‑scoped auth keys and configs.
* `backend/app/customer_provisioning/*`: provisioning workflows for external services (Graylog, Grafana, Portainer, Wazuh manager).
* `backend/app/stack_provisioning/graylog/*`: content packs, pipelines, streams, and input templates for Graylog.
* `backend/app/agents/*`: Wazuh and Velociraptor agent management, SCA, vulnerabilities.
* `backend/app/incidents/*`: incident management (cases, alerts, tags, reports).
* `backend/app/active_response/*`: automation scripts and invoke endpoints.
* `backend/app/data_store/*`: MinIO storage (cases, templates, artifacts).
* `backend/app/threat_intel/*`: EPSS, VirusTotal, SOCFortress threat intel.
* `backend/app/integrations/copilot_mcp/*`: local and cloud MCP queries.

### Frontend

* `frontend/src/router/index.ts`: primary navigation and feature routes.
* `frontend/src/api/endpoints/*`: typed API clients for each backend domain.
* `frontend/src/components/*`: feature UI modules (alerts, cases, agents, connectors, integrations, etc.).
* `frontend/src/views/*`: route views; corresponds closely to backend feature areas.
* `frontend/.env.example`: API base URL and UI behavior.

### Customer Portal

* `customer-portal/src/router/index.ts`: simple login + alerts/cases/agents views.
* `customer-portal/src/views/*`: customer‑limited UI.

## Core Runtime Flows

### Startup

1. `backend/copilot.py` loads env vars and initializes FastAPI.
2. On startup event:
   * Creates MySQL database and user if needed. `backend/app/db/db_setup.py`
   * Applies Alembic migrations. `backend/app/db/db_setup.py`
   * Creates MinIO buckets. `backend/app/data_store/data_store_setup.py`
   * Seeds connectors, roles, available integrations, available network connectors. `backend/app/db/db_setup.py`, `backend/app/db/db_populate.py`
   * Ensures admin and scheduler users. `backend/app/db/db_setup.py`
   * Initializes APScheduler and schedules enabled jobs. `backend/app/schedulers/scheduler.py`
3. Static mount: `scoutsuite-report` directory for cloud security assessment outputs. `backend/copilot.py`

### Auth & Authorization

* JWT auth with OAuth2 password flow. `backend/app/auth/utils.py`
* Roles/scopes: `admin`, `analyst`, `scheduler`, `customer_user`.
* Routes use `Security(AuthHandler().get_current_user, scopes=[...])` or `require_any_scope(...)`.

### API Routing

* `FastAPI` app mounts a single APIRouter at `/api`.
* Each domain lives in `backend/app/routers/*.py` and delegates to feature modules in `backend/app/<domain>/*`.
* Frontend API clients match route structure. `frontend/src/api/endpoints/*`

### Background Jobs

* APScheduler runs inside the FastAPI app.
* Job metadata is stored in MySQL and loaded on startup. `backend/app/schedulers/scheduler.py`
* Jobs invoke integration collectors and internal maintenance:
  * Agent sync (Wazuh + Velociraptor)
  * Wazuh index resize
  * Alert creation collection
  * Snapshot schedule execution
  * Integration collectors (Duo, Darktrace, Cato, Huntress, Carbon Black, etc.)
* Job definitions live in `backend/app/schedulers/services/*`.

### Integrations & Connectors

* Connectors are core service connections (Wazuh, Graylog, Grafana, etc.) stored in `Connectors` DB table.
* Connectors are seeded from env vars on startup. `backend/app/db/db_populate.py`
* Each connector implements a verify function in `backend/app/connectors/<service>/utils/universal.py` and is mapped in `backend/app/connectors/services.py`.
* Integration settings are customer‑scoped and stored in `CustomerIntegrations` / `CustomerIntegrationsMeta` tables.

### Data Storage

* MySQL for all operational data (users, integrations, connectors, alerts/cases, scheduler metadata).
* MinIO for case artifacts, report templates, sysmon configs, Velociraptor artifacts. `backend/app/data_store`
* Local filesystem: `scoutsuite-report` directory for report outputs.

## Major User‑Facing Features (Admin UI)

* Overview dashboard and system health.
* Connectors management (test/verify/update).
* Wazuh management: rules, groups, Sysmon config, MITRE browsing.
* Graylog management, metrics, pipelines, streams, inputs.
* Alerts and SIEM views, MITRE/Atomic Red Team views.
* Incident management: sources, alerts, cases, tags, comments, reports, data store.
* Agents (Wazuh + Velociraptor), SCA, vulnerabilities, data store.
* Artifacts and file collection.
* Active response actions.
* External services: third‑party integrations and network connectors.
* Reporting (Grafana dashboards, case reports, vuln/SCA reports).
* Scheduler management.
* Cloud security assessment (ScoutSuite).
* Web vulnerability assessment (Nuclei).
* GitHub Audit.
* Customer portal branding/settings.
* License management.

## Customer Portal Features

* Login and role‑restricted access for `customer_user`.
* Views for alerts, cases, case details, and agents.
* Separate SPA served by `copilot-customer-portal` container.

## Integrations & Open Source Services

### Wazuh

* Connector: `backend/app/connectors/wazuh_manager/*` and `backend/app/connectors/wazuh_indexer/*`
* Auth: API token cached in memory with TTL; requests via `requests`.
* Wazuh manager routes: `backend/app/connectors/wazuh_manager/routes/*` and `backend/app/routers/wazuh_manager.py`.
* Wazuh indexer routes: `backend/app/connectors/wazuh_indexer/routes/*`.

### Graylog

* Connector: `backend/app/connectors/graylog/*`
* Event shipper: GELF TCP to Graylog input. `backend/app/connectors/event_shipper/*`, `backend/app/integrations/utils/event_shipper.py`
* Graylog provisioning: content packs, pipelines, streams, inputs. `backend/app/stack_provisioning/graylog/*`
* Graylog API management used in routes and service modules.

### Grafana

* Connector: `backend/app/connectors/grafana/*`
* Reporting endpoints for orgs/dashboards/panels and iframe generation. `backend/app/routers/grafana.py`, `frontend/src/components/reportCreation`

### Velociraptor

* Connector: `backend/app/connectors/velociraptor/*`
* Agent management + artifacts; artifacts also stored in MinIO. `backend/app/agents/velociraptor/*`, `backend/app/data_store`

### Shuffle

* Connector: `backend/app/connectors/shuffle/*`
* Endpoints for Shuffle metadata. `backend/app/routers/shuffle.py`

### InfluxDB

* Connector: `backend/app/connectors/influxdb/*`
* Healthcheck and monitoring endpoints. `backend/app/routers/influxdb.py`, `frontend/src/components/healthcheck`

### Portainer

* Connector: `backend/app/connectors/portainer/*`
* Customer provisioning workflows can call Portainer. `backend/app/customer_provisioning/services/portainer.py`

### Nuclei

* Integration: `backend/app/integrations/nuclei/*` and `backend/app/routers/nuclei.py`
* Container present in `docker-compose.yml` as `copilot-nuclei-module`.

### ScoutSuite

* Integration: `backend/app/integrations/scoutsuite/*` and `backend/app/routers/scoutsuite.py`
* Outputs served from `scoutsuite-report` static mount.

### Threat Intel

* VirusTotal, EPSS, SOCFortress. `backend/app/threat_intel/*`

### MCP (CoPilot AI)

* Local MCP service container configured in Docker.
* Backend routes call `backend/app/integrations/copilot_mcp/services/copilot_mcp.py` to query local OpenSearch/MySQL/Wazuh/Velociraptor or cloud threat intel endpoints.

## Configuration Management & Secrets

* `.env` provides connector URLs, API keys, DB creds, MinIO creds, MCP settings. `.env.example` documents expected values.
* Frontend uses `VITE_API_URL` to target the backend.
* TLS handled by `copilot-frontend` container; TLS cert/key paths are configurable in `docker-compose.yml` and documented in README.

## Deployment (Docker Compose)

* `copilot-backend`: FastAPI service on port 5000.
* `copilot-frontend`: Vue app with TLS, ports 80/443.
* `copilot-mysql`: MySQL 8.
* `copilot-minio`: object storage.
* `copilot-nuclei-module`: web vulnerability scanner module.
* `copilot-mcp`: MCP service for AI queries.
* Optional `copilot-customer-portal` for customer‑facing UI.

## Extension Points

### Add a New Connector

1. Add connector metadata in `backend/app/db/db_populate.py`.
2. Add verification logic in `backend/app/connectors/<new_service>/utils/universal.py`.
3. Map the connector name in `backend/app/connectors/services.py`.
4. Provide routes in `backend/app/connectors/<new_service>/routes` and `backend/app/routers/<new_service>.py` as needed.
5. Add frontend UI and API client in `frontend/src/components` and `frontend/src/api/endpoints`.

### Add a New Integration (Per‑Customer)

1. Add integration metadata in `backend/app/db/db_populate.py`.
2. Add models/schema in `backend/app/integrations/models` and `backend/app/integrations/schema`.
3. Add integration service/routes in `backend/app/integrations/<integration_name>`.
4. Expose route in `backend/app/routers/<integration_name>.py`.
5. Add scheduled collection jobs if needed: `backend/app/schedulers/services/*` and `backend/app/schedulers/scheduler.py`.

### Add a New Scheduler Job

1. Implement job function in `backend/app/schedulers/services`.
2. Add to `known_jobs` in `backend/app/schedulers/scheduler.py`.
3. Add function mapping in `get_function_by_name`.
4. Expose job control in `backend/app/schedulers/routes/scheduler.py`.

### Add a New Customer Portal Feature

1. Add backend route with `customer_user` role scope.
2. Add UI in `customer-portal/src/views` and wire in `customer-portal/src/router/index.ts`.