anti
48f486ac97
- Core Telegram monitoring pipeline (scraper, processor, notifier, downloaders) - Textual TUI frontend with thread-safe event bus - SQLite persistence, severity scoring, dedup cache - Fixed ULP parser: handles https:// truncation, port+path URLs, semicolon separator - Test suite: 88 tests across scorer, cache, database, processor
147 lines
4.1 KiB
Markdown
147 lines
4.1 KiB
Markdown
# ULP Credential Monitor
|
|
|
|
A Telegram-based credential exposure monitor for threat intelligence teams.
|
|
Watches channels for combo/stealer log files and alerts you when your
|
|
organization's credentials appear in them.
|
|
|
|
---
|
|
|
|
## How it works
|
|
|
|
```
|
|
User session (Telethon)
|
|
└─ watches N channels
|
|
└─ detects file attachments (.txt, .zip, .7z, .rar)
|
|
└─ downloads → extracts → searches line by line
|
|
└─ hit? → writes to data/ + sends bot alert
|
|
└─ no hit? → deletes file, moves on
|
|
```
|
|
|
|
---
|
|
|
|
## Project structure
|
|
|
|
```
|
|
ulp_monitor/
|
|
├── main.py Entry point
|
|
├── config.py All settings (keywords, channels, paths)
|
|
│
|
|
├── core/ Telegram I/O pipeline
|
|
│ ├── scraper.py Live listener + backfill
|
|
│ ├── tdl_downloader.py Fast downloads via tdl (Go MTProto)
|
|
│ ├── bot_downloader.py Inline button / bot-dispatched file flows
|
|
│ ├── processor.py Archive extraction + line-by-line search
|
|
│ └── notifier.py hits.txt / hits.csv writer + bot alerts
|
|
│
|
|
├── utils/ Pure logic — no Telegram dependencies
|
|
│ ├── scorer.py Hit severity scoring
|
|
│ ├── cache.py Seen-file deduplication
|
|
│ └── database.py SQLite persistence layer
|
|
│
|
|
├── tui/ Textual TUI frontend
|
|
│ ├── app.py MonitorApp + all Screen classes
|
|
│ └── events.py Thread-safe event bus (bot thread → TUI)
|
|
│
|
|
└── data/ Runtime-generated (gitignored)
|
|
├── hits.db SQLite database
|
|
├── hits.txt Human-readable hit log
|
|
├── hits.csv CSV hit log (importable into Excel / pandas)
|
|
├── dedup.json Deduplication hashes
|
|
├── cache.json Seen file-ID cache
|
|
└── logs/monitor.log
|
|
```
|
|
|
|
---
|
|
|
|
## Setup
|
|
|
|
### 1. Get Telegram API credentials
|
|
- Go to https://my.telegram.org → *API development tools*
|
|
- Create an app → note your `api_id` and `api_hash`
|
|
|
|
### 2. Create a bot
|
|
- Message [@BotFather](https://t.me/BotFather) → `/newbot`
|
|
- Start a chat with your new bot before running
|
|
|
|
### 3. Get your chat ID
|
|
- Message [@userinfobot](https://t.me/userinfobot)
|
|
|
|
### 4. Configure
|
|
|
|
```bash
|
|
cp .env.example .env
|
|
# fill in API_ID, API_HASH, BOT_TOKEN, NOTIFY_CHAT_ID
|
|
```
|
|
|
|
Open `config.py` and set:
|
|
|
|
- **`TARGET_KEYWORDS`** — your org's domains and email patterns.
|
|
Keywords with `@` (e.g. `r"@myorg\.cl"`) are **employee email domains** → CRITICAL.
|
|
Keywords without `@` are plain domain matches → LOW baseline.
|
|
- **`WATCHED_CHANNELS`** — channel usernames or numeric IDs
|
|
- **`BACKFILL_LIMIT`** — past messages to scan per channel on startup
|
|
|
|
### 5. Install dependencies
|
|
|
|
```bash
|
|
pip install -r requirements.txt
|
|
# rarfile needs the unrar binary:
|
|
# Ubuntu/Debian: sudo apt install unrar
|
|
# macOS: brew install rar
|
|
```
|
|
|
|
### 5a. Install tdl (strongly recommended)
|
|
|
|
```bash
|
|
curl -sSL https://raw.githubusercontent.com/iyear/tdl/main/scripts/install.sh | bash
|
|
tdl login -n monitor_session
|
|
```
|
|
|
|
### 6. First run — complete Telegram auth
|
|
|
|
```bash
|
|
python main.py --no-tui
|
|
# follow the phone + 2FA prompts once
|
|
```
|
|
|
|
### 7. Run
|
|
|
|
```bash
|
|
python main.py # TUI mode (recommended)
|
|
python main.py --no-tui # plain CLI
|
|
```
|
|
|
|
---
|
|
|
|
## TUI keybindings
|
|
|
|
| Key | Action |
|
|
|-----|--------|
|
|
| `s` | Search hits database |
|
|
| `h` | Browse hits by severity |
|
|
| `k` | Edit keyword patterns live |
|
|
| `c` | Clear logs |
|
|
| `r` | Refresh stats |
|
|
| `q` | Quit |
|
|
|
|
---
|
|
|
|
## Output
|
|
|
|
| File | Description |
|
|
|------|-------------|
|
|
| `data/hits.db` | SQLite — all hits with scores, severity, dedup flag |
|
|
| `data/hits.txt` | Human-readable grouped log |
|
|
| `data/hits.csv` | CSV — easy to pull into Excel / pandas |
|
|
| `data/logs/monitor.log` | Full run log |
|
|
|
|
Telegram alerts fire for CRITICAL / HIGH / MEDIUM only. LOW is stored silently.
|
|
|
|
---
|
|
|
|
## Notes
|
|
|
|
- **Session files are sensitive** — equivalent to a logged-in account. Gitignored, never share.
|
|
- **Flood limits** — `FloodWaitError` is handled automatically.
|
|
- **Private channels** — your user account must already be a member.
|