feat: initial EDR system — agent + backend + frontend + ensemble ML
This commit is contained in:
kilyabin
@@ -0,0 +1,222 @@
|
||||
# EDR Monitor — Behavioral Malware Detection System
|
||||
|
||||
A university diploma project demonstrating real-time **Endpoint Detection & Response (EDR)**
|
||||
using behavioral analysis and machine learning on a Linux host.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Linux Host / VM │
|
||||
│ │
|
||||
│ ┌──────────────┐ HTTP POST /api/metrics ┌──────────────┐ │
|
||||
│ │ │ ─────────────────────────▶ │ │ │
|
||||
│ │ agent/ │ │ backend/ │ │
|
||||
│ │ (psutil + │ │ (FastAPI + │ │
|
||||
│ │ watchdog + │ │ SQLite + │ │
|
||||
│ │ IsoForest) │ │ WebSocket) │ │
|
||||
│ │ │ │ │ │
|
||||
│ └──────────────┘ └──────┬───────┘ │
|
||||
│ │ │ │
|
||||
│ Filesystem watcher WS /ws │ │
|
||||
│ /etc /tmp /var/log │ │
|
||||
│ ┌───────▼───────┐ │
|
||||
│ │ frontend/ │ │
|
||||
│ │ (React + │ │
|
||||
│ │ Recharts + │ │
|
||||
│ │ shadcn/ui) │ │
|
||||
│ │ :3000 │ │
|
||||
│ └───────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Components:**
|
||||
|
||||
| Directory | Tech | Purpose |
|
||||
|-----------|------|---------|
|
||||
| `agent/` | Python 3.11, psutil, watchdog, scikit-learn | Collects process metrics, runs ML inference |
|
||||
| `backend/` | FastAPI, aiosqlite, uvicorn | REST API, SQLite storage, WebSocket broadcast |
|
||||
| `frontend/`| React 18, Vite, shadcn/ui, Recharts | Real-time SOC dashboard |
|
||||
|
||||
---
|
||||
|
||||
## Quick Start (local, no Docker)
|
||||
|
||||
### 1. Backend
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
python -m venv .venv && source .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
uvicorn main:app --reload --port 8000
|
||||
```
|
||||
|
||||
### 2. Agent
|
||||
|
||||
```bash
|
||||
cd agent
|
||||
python -m venv .venv && source .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Train the ML model (runs once, saves data/model.pkl)
|
||||
python train.py
|
||||
|
||||
# Start the agent
|
||||
python main.py
|
||||
```
|
||||
|
||||
> **Note:** The agent reads process metadata — run with sufficient privileges
|
||||
> (or as root) to see all processes and network connections.
|
||||
|
||||
### 3. Frontend
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npm install
|
||||
npm run dev # http://localhost:3000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Start (Docker Compose)
|
||||
|
||||
```bash
|
||||
docker compose up --build
|
||||
```
|
||||
|
||||
Services:
|
||||
- Backend → http://localhost:8000 (API docs: http://localhost:8000/docs)
|
||||
- Dashboard → http://localhost:3000
|
||||
|
||||
---
|
||||
|
||||
## ML Model
|
||||
|
||||
The agent uses **Isolation Forest** (scikit-learn) for unsupervised anomaly detection.
|
||||
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| `cpu_percent` | Process CPU utilisation |
|
||||
| `memory_mb` | RSS memory in megabytes |
|
||||
| `open_files` | Number of open file descriptors |
|
||||
| `connections` | Number of open TCP connections |
|
||||
|
||||
Training data priority:
|
||||
1. `data/UNSW_NB15_training-set.csv` — real network intrusion dataset (download separately)
|
||||
2. Synthetic normal-behaviour data (fallback, auto-generated if CSV absent)
|
||||
|
||||
Contamination factor: **5%** (5 % of training data treated as outliers).
|
||||
|
||||
---
|
||||
|
||||
## REST API
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| `GET` | `/api/stats` | Dashboard statistics |
|
||||
| `GET` | `/api/metrics?limit=200` | Recent process metrics |
|
||||
| `GET` | `/api/alerts?limit=50` | Security alerts |
|
||||
| `GET` | `/api/file-events?limit=100` | Filesystem events |
|
||||
| `POST` | `/api/metrics` | Ingest batch from agent |
|
||||
| `WS` | `/ws` | Real-time alert stream |
|
||||
|
||||
Interactive docs: http://localhost:8000/docs
|
||||
|
||||
---
|
||||
|
||||
## Simulating Anomalies (demo)
|
||||
|
||||
### CPU spike (Linux)
|
||||
```bash
|
||||
# Requires stress-ng
|
||||
stress-ng --cpu 4 --timeout 30s
|
||||
|
||||
# Alternative — pure Python
|
||||
python3 -c "
|
||||
import multiprocessing, time
|
||||
def spin():
|
||||
while True: pass
|
||||
procs = [multiprocessing.Process(target=spin) for _ in range(4)]
|
||||
[p.start() for p in procs]
|
||||
time.sleep(30)
|
||||
[p.terminate() for p in procs]
|
||||
"
|
||||
```
|
||||
|
||||
### Memory spike
|
||||
```bash
|
||||
python3 -c "
|
||||
x = bytearray(2 * 1024 * 1024 * 1024) # allocate 2 GB
|
||||
import time; time.sleep(30)
|
||||
"
|
||||
```
|
||||
|
||||
### Filesystem trigger (write to /tmp)
|
||||
```bash
|
||||
for i in $(seq 1 20); do echo "test" > /tmp/edr_test_$i.txt; sleep 1; done
|
||||
```
|
||||
|
||||
Within ~10 seconds the dashboard will show the anomalous process highlighted in red
|
||||
and a toast notification will appear bottom-right.
|
||||
|
||||
---
|
||||
|
||||
## Screenshots
|
||||
|
||||
> *Add screenshots here after first run*
|
||||
|
||||
| Overview | Processes | Alerts |
|
||||
|----------|-----------|--------|
|
||||
|  |  |  |
|
||||
|
||||
---
|
||||
|
||||
## Dataset
|
||||
|
||||
The system optionally uses the **UNSW-NB15** network intrusion dataset for training.
|
||||
|
||||
Download: https://research.unsw.edu.au/projects/unsw-nb15-dataset
|
||||
|
||||
Place the file at: `data/UNSW_NB15_training-set.csv`
|
||||
|
||||
Then retrain:
|
||||
```bash
|
||||
cd agent && python train.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
dipl-edr/
|
||||
├── agent/
|
||||
│ ├── main.py # monitoring loop + fs watcher
|
||||
│ ├── model.py # IsolationForest wrapper
|
||||
│ ├── train.py # standalone training script
|
||||
│ └── requirements.txt
|
||||
├── backend/
|
||||
│ ├── main.py # FastAPI app + WebSocket
|
||||
│ └── requirements.txt
|
||||
├── frontend/
|
||||
│ ├── src/
|
||||
│ │ ├── App.tsx # root layout + WebSocket client
|
||||
│ │ ├── components/
|
||||
│ │ │ ├── Overview.tsx # stats cards + chart
|
||||
│ │ │ ├── Processes.tsx # process table
|
||||
│ │ │ ├── Alerts.tsx # alerts table
|
||||
│ │ │ ├── FileEvents.tsx # fs events table
|
||||
│ │ │ ├── SystemChart.tsx # Recharts line chart
|
||||
│ │ │ └── ui/ # shadcn-style primitives
|
||||
│ │ ├── hooks/use-toast.ts
|
||||
│ │ └── lib/
|
||||
│ │ ├── api.ts # typed fetch helpers
|
||||
│ │ └── utils.ts
|
||||
│ ├── package.json
|
||||
│ └── vite.config.ts
|
||||
├── data/ # .gitignored — holds model.pkl + edr.db
|
||||
├── docker-compose.yml
|
||||
└── README.md
|
||||
```
|
||||
Reference in New Issue
Block a user