Threat intelligence · GitOps · AWS
Threat Intel Pipeline
Every hour this service collects known-bad IP addresses, phishing URLs and actively exploited vulnerabilities from three public feeds, merges them into one deduplicated database, and answers one question over an authenticated API: is this indicator known to be malicious?
- 3 open feeds
- Hourly ingest
- API-key auth
- Immutable backups
- $0 on the AWS free tier
How the data flows
From public feeds to an answer, all inside one small Kubernetes cluster.
- 1
Feeds
abuse.ch Feodo Tracker (botnet C2 IPs), OpenPhish (phishing URLs), CISA KEV (exploited CVEs).
- 2
Ingest job
A CronJob fetches each feed over HTTPS only, with size caps and strict parsing. Hostile or malformed values are dropped.
- 3
Normalise
Every IP, URL and CVE becomes one indicator schema, deduplicated across feeds and runs. Stale entries age out.
- 4
SQLite store
Kept on its own encrypted EBS volume, so the data survives the server being rebuilt.
- 5
Query API
FastAPI with a read-only database connection, behind Traefik TLS and per-client rate limits.
- 6
You
Look up an indicator, search by type or source, or get totals. Every data endpoint needs an API key.
How code reaches production
GitOps: Git is the only way in. CI never holds cluster credentials.
- 1
git push
A change under
app/lands onmain. - 2
GitHub Actions
Runs the tests, scans the image with Trivy, and fails on known vulnerabilities.
- 3
GHCR
Pushes the image and its SBOM to the GitHub Container Registry.
- 4
Deploy commit
CI commits the new image tag and digest to the prod overlay. That commit is the release.
- 5
Argo CD
Inside the cluster, it notices the commit and syncs, pruning drift and self-healing manual changes.
- 6
k3s
Pulls the image by digest and rolls out the new version.
Architecture
API
Send your key in the X-API-Key header. A browser can't add it by clicking a link, so use curl or an API client.
| Endpoint | Returns | Key |
|---|---|---|
GET /lookup/{value} | The match for an IP, URL or CVE, or 404 if no feed knows it | Required |
GET /indicators?type=&q=&source=&limit= | Search by type (ip, url, cve), text, or source feed | Required |
GET /stats | Totals by type and the last run of each feed | Required |
GET /version | The deployed build | Required |
GET /healthz, /readyz | Health probes that reveal nothing | Open |
curl -H "X-API-Key: $KEY" https://shaheer-ti.duckdns.org/lookup/CVE-2024-3400Security, layer by layer
Fail-closed auth
Only SHA-256 hashes of keys are stored, compared in constant time. With no key configured, the API locks rather than opens.
Hardened pods
Non-root, read-only filesystems, Pod Security restricted, and default-deny NetworkPolicies.
No open admin ports
No SSH. The Kubernetes API is reached through an IAM-authenticated SSM tunnel.
Least-privilege IAM
The server can append backups and read one token. It can't list, read or delete backups.
Supply chain
SHA-pinned Actions, hash-locked dependencies, Trivy gates, and images deployed by digest.
Immutable backups
Nightly, integrity-checked snapshots in S3 Object Lock. Nobody can delete them for 7 days, not even with the AWS keys.