Initial commit: ULPgrammer

- Core Telegram monitoring pipeline (scraper, processor, notifier, downloaders) - Textual TUI frontend with thread-safe event bus - SQLite persistence, severity scoring, dedup cache - Fixed ULP parser: handles https:// truncation, port+path URLs, semicolon separator - Test suite: 88 tests across scorer, cache, database, processor
2026-04-02 01:58:49 -03:00
commit 48f486ac97
41 changed files with 5270 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,146 @@
+# ULP Credential Monitor
+
+A Telegram-based credential exposure monitor for threat intelligence teams.
+Watches channels for combo/stealer log files and alerts you when your
+organization's credentials appear in them.
+
+---
+
+## How it works
+
+```
+User session (Telethon)
+  └─ watches N channels
+       └─ detects file attachments (.txt, .zip, .7z, .rar)
+            └─ downloads → extracts → searches line by line
+                 └─ hit? → writes to data/ + sends bot alert
+                 └─ no hit? → deletes file, moves on
+```
+
+---
+
+## Project structure
+
+```
+ulp_monitor/
+├── main.py           Entry point
+├── config.py         All settings (keywords, channels, paths)
+│
+├── core/             Telegram I/O pipeline
+│   ├── scraper.py        Live listener + backfill
+│   ├── tdl_downloader.py Fast downloads via tdl (Go MTProto)
+│   ├── bot_downloader.py Inline button / bot-dispatched file flows
+│   ├── processor.py      Archive extraction + line-by-line search
+│   └── notifier.py       hits.txt / hits.csv writer + bot alerts
+│
+├── utils/            Pure logic — no Telegram dependencies
+│   ├── scorer.py         Hit severity scoring
+│   ├── cache.py          Seen-file deduplication
+│   └── database.py       SQLite persistence layer
+│
+├── tui/              Textual TUI frontend
+│   ├── app.py            MonitorApp + all Screen classes
+│   └── events.py         Thread-safe event bus (bot thread → TUI)
+│
+└── data/             Runtime-generated (gitignored)
+    ├── hits.db           SQLite database
+    ├── hits.txt          Human-readable hit log
+    ├── hits.csv          CSV hit log (importable into Excel / pandas)
+    ├── dedup.json        Deduplication hashes
+    ├── cache.json        Seen file-ID cache
+    └── logs/monitor.log
+```
+
+---
+
+## Setup
+
+### 1. Get Telegram API credentials
+- Go to https://my.telegram.org → *API development tools*
+- Create an app → note your `api_id` and `api_hash`
+
+### 2. Create a bot
+- Message [@BotFather](https://t.me/BotFather) → `/newbot`
+- Start a chat with your new bot before running
+
+### 3. Get your chat ID
+- Message [@userinfobot](https://t.me/userinfobot)
+
+### 4. Configure
+
+```bash
+cp .env.example .env
+# fill in API_ID, API_HASH, BOT_TOKEN, NOTIFY_CHAT_ID
+```
+
+Open `config.py` and set:
+
+- **`TARGET_KEYWORDS`** — your org's domains and email patterns.
+  Keywords with `@` (e.g. `r"@myorg\.cl"`) are **employee email domains** → CRITICAL.
+  Keywords without `@` are plain domain matches → LOW baseline.
+- **`WATCHED_CHANNELS`** — channel usernames or numeric IDs
+- **`BACKFILL_LIMIT`** — past messages to scan per channel on startup
+
+### 5. Install dependencies
+
+```bash
+pip install -r requirements.txt
+# rarfile needs the unrar binary:
+# Ubuntu/Debian: sudo apt install unrar
+# macOS:         brew install rar
+```
+
+### 5a. Install tdl (strongly recommended)
+
+```bash
+curl -sSL https://raw.githubusercontent.com/iyear/tdl/main/scripts/install.sh | bash
+tdl login -n monitor_session
+```
+
+### 6. First run — complete Telegram auth
+
+```bash
+python main.py --no-tui
+# follow the phone + 2FA prompts once
+```
+
+### 7. Run
+
+```bash
+python main.py          # TUI mode (recommended)
+python main.py --no-tui # plain CLI
+```
+
+---
+
+## TUI keybindings
+
+| Key | Action |
+|-----|--------|
+| `s` | Search hits database |
+| `h` | Browse hits by severity |
+| `k` | Edit keyword patterns live |
+| `c` | Clear logs |
+| `r` | Refresh stats |
+| `q` | Quit |
+
+---
+
+## Output
+
+| File | Description |
+|------|-------------|
+| `data/hits.db`  | SQLite — all hits with scores, severity, dedup flag |
+| `data/hits.txt` | Human-readable grouped log |
+| `data/hits.csv` | CSV — easy to pull into Excel / pandas |
+| `data/logs/monitor.log` | Full run log |
+
+Telegram alerts fire for CRITICAL / HIGH / MEDIUM only. LOW is stored silently.
+
+---
+
+## Notes
+
+- **Session files are sensitive** — equivalent to a logged-in account. Gitignored, never share.
+- **Flood limits** — `FloodWaitError` is handled automatically.
+- **Private channels** — your user account must already be a member.