[Opportunity Crawler] - I Built a Bot That Scrapes Cybersecurity Jobs and Sends Them to My Telegram Every Day
Hey everyone,
So this started from a genuinely frustrating moment. I found out about APCSIP — a government cybersecurity internship program — after the application deadline had already passed. Not by a few days. By weeks. I just hadn’t been checking the right places at the right time.
That got me thinking: how many other opportunities am I missing just because I’m not actively monitoring 10 different portals every day? LinkedIn, NCS, CDAC, NCIIPC, CERT-In, Reddit netsec hiring threads, HackerEarth — nobody has time to check all of these manually every morning.
So I built a bot to do it for me.
This post covers what it does, how it works technically, and how you can set up your own version in about 30 minutes.
What It Does
Every day at 9 PM, a GitHub Actions cron job runs and:
- Scrapes 11 sources for cybersecurity jobs, internships, CTF competitions, and government programs
- Hard-filters obvious noise — senior roles, non-India locations, physical security jobs, AI annotation jobs disguised as “Security Analyst” roles
- Keyword scores everything against 67 cybersecurity-specific terms
- Sends surviving items to Groq’s free LLM API, which scores each one 1–10 against my resume profile
- Deduplicates against the last 10 days of sent items
- Fires off Telegram messages sorted by score — best matches first, Hyderabad roles prioritized within same score
The result: every evening I get a clean batch of actually relevant opportunities I haven’t seen before. No noise, no repeats, no manual work.
The Pipeline
1
2
3
4
5
6
7
8
GitHub Actions (cron 9PM IST)
→ scraper.py scrapes all 11 sources (~260 raw items/day)
→ filter.py 3-stage pipeline:
Stage 1: hard_exclude() instant, no API (~100 items dropped)
Stage 2: keyword_score() 67 keywords, min 1 to pass (~28 miss)
Stage 3: ai_score_groq() Groq LLM, 1-10, threshold ≥7 (~30 filtered)
→ dedup.py seen_urls.json, 10-day retention
→ notifier.py Telegram Bot API, sorted by AI score
Typical run: 260 raw → 94 hard-excluded → 28 keyword-miss → 30 AI-filtered → ~108 sent to Telegram.
Sources
| Source | What I scrape |
|---|---|
| SOC L1, VAPT, security analyst, intern roles — India only | |
| NCS (National Career Service) | Government job portal |
| Google News via DuckDuckGo | Cybersecurity internship announcements |
| CERT-In / DSCI | Government cyber programs and challenges |
| CDAC / MeitY / NCIIPC / AICTE / NICSI | Govt portals for training and intern programs |
| HackerEarth | CTF competitions and hackathons |
| GitHub API | Security research programs |
| Reddit netsec / cybersec | Hiring megathreads |
| RSS feeds | InfoSec Jobs, HackerNews |
Permanently disabled: Indeed (403 forever), Internshala (fake listings), Unstop (too noisy), TimeJobs (connection always fails).
Stage 1 — Hard Exclude (No API, Instant)
This runs before anything hits an API. It’s pure pattern matching and kills about 40% of raw items immediately.
Things that get hard-excluded:
- Senior/mid-level titles — “Senior Security Analyst”, “Security Manager”, “SOC Analyst L2/L3”, “IT Security Analyst II”
- Non-India locations — checks the
locationfield, job title, and URL slug (because LinkedIn sometimes leaves the location field empty but encodes it in the URL like/soc-analyst-l1-al-khobar-saudi-national-at-...) - Blocked companies — Alignerr (AI annotation jobs dressed up as “Security Operations Analyst”), ScoutIT (posts 8 identical listings with different IDs), Pinkerton (physical security firm)
- Wrong field — fleet operations analyst, fraud analyst, geopolitical risk analyst, marketing intern
- Bad URLs — Reddit hiring threads, LinkedIn posts/articles, Medium blogs, Facebook
1
2
3
4
# Example: catching non-India LinkedIn roles even when location field is empty
url_lower = opp.get("url", "").lower()
if any(m.replace(" ", "-") in url_lower for m in _NON_INDIA_MARKERS):
return True, f"non-India location in URL"
This caught a “SOC Analyst L1 Al Khobar Saudi National” role that kept slipping through because the scraper was setting location='India' (generic fallback) while the actual location was encoded in the URL slug.
Stage 2 — Keyword Scoring
67 keywords covering everything from "soc analyst" to "threat hunting" to "apcsip". Each keyword has a weight. Items with total score below 1 are dropped. Intentionally low threshold — let the AI decide, not keyword matching.
The keywords are split into single words ("splunk", "vapt", "forensics") and phrases ("security operations center", "incident response"). Single words are critical because titles like “SOC Analyst” would get zero keyword score if you only match on "cybersecurity intern".
Stage 3 — AI Scoring with Groq
This is where the real filtering happens. Each surviving item gets sent to a Groq LLM with the job details and my full resume_profile.yaml:
1
2
3
4
5
6
7
8
9
10
11
12
Score this opportunity for this candidate from 1-10.
Candidate: B.Tech CSE Cybersecurity, JNTUH, 3rd year.
Skills: Splunk, Burp Suite, Nmap, Python, Wireshark, Metasploit.
Experience: Web VAPT internship (1 month). SOC home lab.
Seeking: Entry-level/intern only. India only.
Opportunity:
Title: {title}
Company: {company}
Description: {description}
Respond with JSON only: {"score": N, "reason": "..."}
Score ≥ 7 gets sent. Score ≤ 6 is dropped.
Why this works better than just keywords:
A “Security Analyst” role at a company that does AI training data annotation keeps passing keyword filters — the title is perfect. But Groq sees the description mentioning “annotate security scenarios for AI model training” and scores it 2/10. Dropped.
A “CD&E-Cybersecurity-SOC L1 Support - Associate 2 Bangalore” at PwC looks weird in a keyword filter (what even is CD&E?) but Groq understands it’s a SOC L1 fresher role at a Big 4 and scores it 9/10.
Model chain (Groq free tier, round-robin):
1
2
3
4
5
GROQ_MODEL_CHAIN = [
"qwen/qwen3.6-27b",
"openai/gpt-oss-20b",
"openai/gpt-oss-120b",
]
Round-robin spreads the load across all three models. If one hits a rate limit or connection drops, it automatically falls to the next. The free tier gives 1,000 requests/day per model = 3,000 total daily calls. A full run uses ~120 calls. Plenty of headroom.
One annoying thing I debugged: reasoning models (qwen3.6, gpt-oss) output a <think>...</think> chain before the JSON. You have to strip it before parsing:
1
2
3
think_end = content.find("</think>")
if think_end != -1:
content = content[think_end + 8:].strip()
Deduplication
Every sent URL goes into seen_urls.json with a timestamp and the AI score:
1
2
3
4
"https://in.linkedin.com/jobs/view/soc-analyst-at-airtel-digital-4430231018": {
"t": "2026-07-25T08:20:08",
"s": 9
}
Entries older than 10 days are pruned on each run. This means a listing re-surfaces after 10 days if it’s still active — which is intentional, since you may have missed it or the situation changed.
In GitHub Actions, this file is committed back to the repo after each run by the bot:
1
2
3
4
5
6
7
- name: Commit seen_urls.json
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add seen_urls.json debug_ai_scores.json debug_excluded.json || true
git diff --cached --quiet || git commit -m "chore: update seen_urls [skip ci]"
git push
The [skip ci] tag in the commit message prevents this commit from triggering another workflow run.
What a Telegram Message Looks Like
Each notification is a separate Telegram message with the job title, company, location, source, AI score, and a direct link. They arrive sorted — score 10 first, then 9, then 8, and within the same score, Hyderabad roles come before other cities.
So the first message I see every evening is always the highest-scored, most locally relevant opportunity from that day’s run.
Bugs I Had to Fix
A few interesting ones from building this:
1. Groq model deprecations causing silent failures
Models llama-4-scout-17b and qwen3-32b were deprecated on June 17, 2026 and started returning 404. The exception handler was catching these as generic errors and returning a fallback score of 7 — so everything was passing AI filtering. Fixed by tracking dead models per run and skipping them immediately on subsequent calls.
2. gpt-oss-120b returning 400 on response_format: json_object
Some Groq models just don’t support the JSON response format parameter. Solution: remove the parameter entirely and parse the JSON manually from the text response.
3. SSL connection drops burning AI budget
[SSL: UNEXPECTED_EOF_WHILE_READING] errors on Groq’s side were hitting the generic exception handler and returning fallback score=7 instead of trying the next model. Each failed call still counted against the 120-call budget, so 4-5 SSL drops early in a run meant 30+ items never got AI-scored. Fixed by detecting SSL errors specifically and routing them to the next model in chain.
4. Saudi Arabia roles slipping through India filter
The LinkedIn scraper sometimes sets location='India' (generic fallback) even when the actual job is in Saudi Arabia. The location filter only checked the location field, so it passed. Fixed by also scanning the URL slug — LinkedIn encodes the actual location in the URL like /soc-analyst-l1-al-khobar-saudi-national-at-...
Setting Up Your Own
The repo is public and designed to be forked. You need three things:
1. Groq API key — free at console.groq.com. 1,000 req/day per model, no card required.
2. Telegram bot — create one via @BotFather, get the token and your chat ID.
3. Edit resume_profile.yaml — replace my details with yours. This is what the AI scores against. Change your skills, target roles, and experience and the scoring adapts automatically.
Then fork the repo, add three GitHub Secrets (GROQ_API_KEY, TELEGRAM_BOT_TOKEN, TELEGRAM_CHAT_ID), and trigger a manual run from the Actions tab.
Full setup guide in the README: github.com/arvdch/opportunity-crawler
Cost
Everything runs free:
| Service | Usage | Cost |
|---|---|---|
| GitHub Actions | ~18 min/day = ~540 min/month | Free (2,000 min/month limit) |
| Groq API | ~120 calls/day = ~3,600/month | Free (3,000/day limit × 3 models) |
| Telegram Bot API | unlimited messages | Free forever |
| Total | ₹0/month |
What’s Next
A few things I want to add:
- Naukri.com scraper — high volume Indian source for fresher roles, not scraped yet
- Content-hash dedup — catch identical job descriptions posted under different IDs (some companies do this aggressively)
- Telegram digest mode — group all notifications into one message instead of 100 individual ones
- Score shown in notification — right now you’d have to check
debug_ai_scores.jsonto see the score; should just show it in the message
The filtering is in a pretty good state now. Running it for a few weeks to see what leaks through before adding new sources.
Repo: github.com/arvdch/opportunity-crawler
If you set it up and find something broken or want to add a new source, open an issue or PR.
— Aravind