kilyabin
223 lines
7.1 KiB
Markdown
223 lines
7.1 KiB
Markdown
# EDR Monitor — Behavioral Malware Detection System
|
|
|
|
A university diploma project demonstrating real-time **Endpoint Detection & Response (EDR)**
|
|
using behavioral analysis and machine learning on a Linux host.
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Linux Host / VM │
|
|
│ │
|
|
│ ┌──────────────┐ HTTP POST /api/metrics ┌──────────────┐ │
|
|
│ │ │ ─────────────────────────▶ │ │ │
|
|
│ │ agent/ │ │ backend/ │ │
|
|
│ │ (psutil + │ │ (FastAPI + │ │
|
|
│ │ watchdog + │ │ SQLite + │ │
|
|
│ │ IsoForest) │ │ WebSocket) │ │
|
|
│ │ │ │ │ │
|
|
│ └──────────────┘ └──────┬───────┘ │
|
|
│ │ │ │
|
|
│ Filesystem watcher WS /ws │ │
|
|
│ /etc /tmp /var/log │ │
|
|
│ ┌───────▼───────┐ │
|
|
│ │ frontend/ │ │
|
|
│ │ (React + │ │
|
|
│ │ Recharts + │ │
|
|
│ │ shadcn/ui) │ │
|
|
│ │ :3000 │ │
|
|
│ └───────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
**Components:**
|
|
|
|
| Directory | Tech | Purpose |
|
|
|-----------|------|---------|
|
|
| `agent/` | Python 3.11, psutil, watchdog, scikit-learn | Collects process metrics, runs ML inference |
|
|
| `backend/` | FastAPI, aiosqlite, uvicorn | REST API, SQLite storage, WebSocket broadcast |
|
|
| `frontend/`| React 18, Vite, shadcn/ui, Recharts | Real-time SOC dashboard |
|
|
|
|
---
|
|
|
|
## Quick Start (local, no Docker)
|
|
|
|
### 1. Backend
|
|
|
|
```bash
|
|
cd backend
|
|
python -m venv .venv && source .venv/bin/activate
|
|
pip install -r requirements.txt
|
|
uvicorn main:app --reload --port 8000
|
|
```
|
|
|
|
### 2. Agent
|
|
|
|
```bash
|
|
cd agent
|
|
python -m venv .venv && source .venv/bin/activate
|
|
pip install -r requirements.txt
|
|
|
|
# Train the ML model (runs once, saves data/model.pkl)
|
|
python train.py
|
|
|
|
# Start the agent
|
|
python main.py
|
|
```
|
|
|
|
> **Note:** The agent reads process metadata — run with sufficient privileges
|
|
> (or as root) to see all processes and network connections.
|
|
|
|
### 3. Frontend
|
|
|
|
```bash
|
|
cd frontend
|
|
npm install
|
|
npm run dev # http://localhost:3000
|
|
```
|
|
|
|
---
|
|
|
|
## Quick Start (Docker Compose)
|
|
|
|
```bash
|
|
docker compose up --build
|
|
```
|
|
|
|
Services:
|
|
- Backend → http://localhost:8000 (API docs: http://localhost:8000/docs)
|
|
- Dashboard → http://localhost:3000
|
|
|
|
---
|
|
|
|
## ML Model
|
|
|
|
The agent uses **Isolation Forest** (scikit-learn) for unsupervised anomaly detection.
|
|
|
|
| Feature | Description |
|
|
|---------|-------------|
|
|
| `cpu_percent` | Process CPU utilisation |
|
|
| `memory_mb` | RSS memory in megabytes |
|
|
| `open_files` | Number of open file descriptors |
|
|
| `connections` | Number of open TCP connections |
|
|
|
|
Training data priority:
|
|
1. `data/UNSW_NB15_training-set.csv` — real network intrusion dataset (download separately)
|
|
2. Synthetic normal-behaviour data (fallback, auto-generated if CSV absent)
|
|
|
|
Contamination factor: **5%** (5 % of training data treated as outliers).
|
|
|
|
---
|
|
|
|
## REST API
|
|
|
|
| Method | Endpoint | Description |
|
|
|--------|----------|-------------|
|
|
| `GET` | `/api/stats` | Dashboard statistics |
|
|
| `GET` | `/api/metrics?limit=200` | Recent process metrics |
|
|
| `GET` | `/api/alerts?limit=50` | Security alerts |
|
|
| `GET` | `/api/file-events?limit=100` | Filesystem events |
|
|
| `POST` | `/api/metrics` | Ingest batch from agent |
|
|
| `WS` | `/ws` | Real-time alert stream |
|
|
|
|
Interactive docs: http://localhost:8000/docs
|
|
|
|
---
|
|
|
|
## Simulating Anomalies (demo)
|
|
|
|
### CPU spike (Linux)
|
|
```bash
|
|
# Requires stress-ng
|
|
stress-ng --cpu 4 --timeout 30s
|
|
|
|
# Alternative — pure Python
|
|
python3 -c "
|
|
import multiprocessing, time
|
|
def spin():
|
|
while True: pass
|
|
procs = [multiprocessing.Process(target=spin) for _ in range(4)]
|
|
[p.start() for p in procs]
|
|
time.sleep(30)
|
|
[p.terminate() for p in procs]
|
|
"
|
|
```
|
|
|
|
### Memory spike
|
|
```bash
|
|
python3 -c "
|
|
x = bytearray(2 * 1024 * 1024 * 1024) # allocate 2 GB
|
|
import time; time.sleep(30)
|
|
"
|
|
```
|
|
|
|
### Filesystem trigger (write to /tmp)
|
|
```bash
|
|
for i in $(seq 1 20); do echo "test" > /tmp/edr_test_$i.txt; sleep 1; done
|
|
```
|
|
|
|
Within ~10 seconds the dashboard will show the anomalous process highlighted in red
|
|
and a toast notification will appear bottom-right.
|
|
|
|
---
|
|
|
|
## Screenshots
|
|
|
|
> *Add screenshots here after first run*
|
|
|
|
| Overview | Processes | Alerts |
|
|
|----------|-----------|--------|
|
|
|  |  |  |
|
|
|
|
---
|
|
|
|
## Dataset
|
|
|
|
The system optionally uses the **UNSW-NB15** network intrusion dataset for training.
|
|
|
|
Download: https://research.unsw.edu.au/projects/unsw-nb15-dataset
|
|
|
|
Place the file at: `data/UNSW_NB15_training-set.csv`
|
|
|
|
Then retrain:
|
|
```bash
|
|
cd agent && python train.py
|
|
```
|
|
|
|
---
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
dipl-edr/
|
|
├── agent/
|
|
│ ├── main.py # monitoring loop + fs watcher
|
|
│ ├── model.py # IsolationForest wrapper
|
|
│ ├── train.py # standalone training script
|
|
│ └── requirements.txt
|
|
├── backend/
|
|
│ ├── main.py # FastAPI app + WebSocket
|
|
│ └── requirements.txt
|
|
├── frontend/
|
|
│ ├── src/
|
|
│ │ ├── App.tsx # root layout + WebSocket client
|
|
│ │ ├── components/
|
|
│ │ │ ├── Overview.tsx # stats cards + chart
|
|
│ │ │ ├── Processes.tsx # process table
|
|
│ │ │ ├── Alerts.tsx # alerts table
|
|
│ │ │ ├── FileEvents.tsx # fs events table
|
|
│ │ │ ├── SystemChart.tsx # Recharts line chart
|
|
│ │ │ └── ui/ # shadcn-style primitives
|
|
│ │ ├── hooks/use-toast.ts
|
|
│ │ └── lib/
|
|
│ │ ├── api.ts # typed fetch helpers
|
|
│ │ └── utils.ts
|
|
│ ├── package.json
|
|
│ └── vite.config.ts
|
|
├── data/ # .gitignored — holds model.pkl + edr.db
|
|
├── docker-compose.yml
|
|
└── README.md
|
|
```
|