Detection Reality Check

If your goal is to detect government traffic in logs or crawlers inside tools such as:

• UA-INTEL Console
• HTTPXRay
• Kandi

or other common forensic utilities, user-agent strings are only one clue in the machine.

Most government, defense, contractor, and public-sector systems use one or more of the following:

• Standard browsers like Chrome, Edge, Firefox, Safari
• Contractor vulnerability scanners
• Research crawlers and indexing agents
• Custom scripts with self-identifying names
• Normal browser strings used as camouflage

Stronger detection comes from combining:

• user-agent keywords
• reverse DNS
• IP ownership / ASN ownership
• request patterns and crawl behavior
• TLS and client fingerprinting
• header consistency and timing analysis

Treat everything below as a practical heuristic field guide, not divine law chiseled into basalt.

Government / Contractor Security Scanner Identifiers

These are often more useful than agency names because a lot of public-sector and defense traffic is generated by contractor-operated or commercial scanning stacks.

Nessus
Tenable
OpenVAS
Greenbone
Qualys
Acunetix
Nikto
Burp
BurpSuite
AppScan
Rapid7
InsightVM
Nexpose
Nmap
Nmap Scripting Engine
Masscan
ZGrab
ZMap
Shodan
Censys
WhatWeb
Wget
curl
python-requests
Go-http-client
aiohttp
httpx
http-client
libwww-perl
sqlmap
w3af
Arachni
DirBuster
Gobuster
ffuf
feroxbuster
wpscan
joomscan
skipfish
arachni
netsparker
invicti
arachni-vuln-scanner
securityscanner
vulnerabilityscanner
compliancescanner
auditscanner
assetdiscovery
reconscanner
siteinspector
contentdiscovery
webinspector

Public Sector / Research / Archive Bots

GovResearchBot
PublicSectorCrawler
GovDataHarvester
GovStatsBot
GovArchiveCrawler
GovTransparencyBot
GovOpenDataCrawler
GovIndexBot
GovRecordsBot
GovDigitalArchive
GovAnalyticsBot
GovComplianceBot
GovProbe
GovReconBot
GovDataCollector
GovInspector
GovWebCrawler
GovWebInspector
FederalAuditBot
FederalComplianceBot
FederalCrawler
ResearchBot
ArchiveBot
ArchiveCrawler
IndexBot
DataCollector
OpenDataCrawler
RecordsCrawler
CensusBot
GeoSurveyBot
SurveyBot
MappingBot
TelemetryCollector
DataHarvester

Regex Detection Pack

This is a practical catch-net for UA analysis. It is intentionally broad and should be used for classification, scoring, or tagging, not instant conviction.

(?i)\b(
gov(bot|crawler|scanner|monitor|audit|probe|research|archive|index|security|webcrawler|webinspector|analytics|records|transparency|opendata|networkscanner|vulnscanner)|
usgov|
usgovernment|
federal(bot|gov|auditbot|compliancebot|crawler)?|
publicsector(crawler)?|
agency(bot)?|
dhs|fbi|cia|nsa|irs|usgs|noaa|nasa|epa|fema|gsa|opm|uscis|ice|cbp|tsa|atf|dea|doj|treasury|justice|state|energy|census|nist|uspto|secretservice|
dod|army|navy|air ?force|marines|marinecorps|coastguard|spaceforce|cybercom|darpa|dia|
nessus|tenable|openvas|greenbone|qualys|acunetix|nikto|burp|burpsuite|appscan|rapid7|insightvm|nexpose|
nmap|masscan|zgrab|zmap|shodan|censys|whatweb|sqlmap|w3af|arachni|dirbuster|gobuster|ffuf|feroxbuster|wpscan|joomscan|skipfish|
securityscanner|vulnerabilityscanner|compliancescanner|auditscanner|assetdiscovery|reconscanner|siteinspector|contentdiscovery|
archivebot|researchbot|indexbot|datacollector|dataharvester|surveybot|mappingbot|telemetrycollector|
army\.mil|navy\.mil|af\.mil|marines\.mil|spaceforce\.mil|defense\.gov|dhs\.gov|justice\.gov|state\.gov|fbi\.gov|cia\.gov|nsa\.gov|nasa\.gov|noaa\.gov|usgs\.gov
)\b

Python Classifier Example

This is a lightweight classifier you can fold into something like UA-INTEL Console, a Flask endpoint, a crawler analysis pipeline, or a log triage utility.

#!/usr/bin/env python3
# ============================================================================
# UA Classification Utility
# ============================================================================
# Author: K0NxT3D
# Purpose:
#   Tag suspicious, government-related, military, scanner, crawler, or
#   research-related user-agent strings using heuristic keyword matching.
# ============================================================================

import re
from typing import Dict, List

PATTERNS = {
    "government": re.compile(
        r"(?i)\\b("
        r"gov(bot|crawler|scanner|monitor|audit|probe|research|archive|index|security|webcrawler|webinspector|analytics|records|transparency|opendata|networkscanner|vulnscanner)|"
        r"usgov|usgovernment|federal(bot|gov|auditbot|compliancebot|crawler)?|publicsector(crawler)?|agency(bot)?|"
        r"dhs|fbi|cia|nsa|irs|usgs|noaa|nasa|epa|fema|gsa|opm|uscis|ice|cbp|tsa|atf|dea|doj|treasury|justice|state|energy|census|nist|uspto|secretservice|"
        r"army\\.mil|navy\\.mil|af\\.mil|marines\\.mil|spaceforce\\.mil|defense\\.gov|dhs\\.gov|justice\\.gov|state\\.gov|fbi\\.gov|cia\\.gov|nsa\\.gov|nasa\\.gov|noaa\\.gov|usgs\\.gov"
        r")\\b"
    ),
    "military": re.compile(
        r"(?i)\\b("
        r"dod|army|navy|air ?force|marines|marinecorps|coastguard|spaceforce|cybercom|darpa|dia|"
        r"army\\.mil|navy\\.mil|af\\.mil|marines\\.mil|spaceforce\\.mil|defense\\.gov"
        r")\\b"
    ),
    "scanner": re.compile(
        r"(?i)\\b("
        r"nessus|tenable|openvas|greenbone|qualys|acunetix|nikto|burp|burpsuite|appscan|rapid7|insightvm|nexpose|"
        r"nmap|masscan|zgrab|zmap|shodan|censys|whatweb|sqlmap|w3af|arachni|dirbuster|gobuster|ffuf|feroxbuster|wpscan|joomscan|skipfish|"
        r"securityscanner|vulnerabilityscanner|compliancescanner|auditscanner|assetdiscovery|reconscanner|siteinspector|contentdiscovery"
        r")\\b"
    ),
    "crawler": re.compile(
        r"(?i)\\b("
        r"bot|crawler|spider|harvester|indexer|indexbot|archivebot|archivecrawler|researchbot|datacollector|dataharvester|surveybot|mappingbot|telemetrycollector"
        r")\\b"
    ),
    "script_client": re.compile(
        r"(?i)\\b("
        r"curl|wget|python-requests|aiohttp|httpx|go-http-client|libwww-perl|java/|okhttp|powershell|urllib"
        r")\\b"
    ),
    "normal_browser": re.compile(
        r"(?i)(mozilla/5\\.0|chrome/\\d|safari/\\d|firefox/\\d|edg/\\d|opr/\\d)"
    ),
}

def classify_user_agent(user_agent: str) -> Dict[str, object]:
    ua = (user_agent or "").strip()

    tags: List[str] = []
    score = 0

    if not ua:
        return {
            "user_agent": ua,
            "tags": ["empty"],
            "score": 0,
            "summary": "Empty user-agent string"
        }

    if PATTERNS["government"].search(ua):
        tags.append("Government")
        score += 50

    if PATTERNS["military"].search(ua):
        tags.append("Military")
        score += 35

    if PATTERNS["scanner"].search(ua):
        tags.append("Security Scanner")
        score += 30

    if PATTERNS["crawler"].search(ua):
        tags.append("Crawler / Bot")
        score += 20

    if PATTERNS["script_client"].search(ua):
        tags.append("Script Client")
        score += 10

    if PATTERNS["normal_browser"].search(ua) and not tags:
        tags.append("Normal Browser")
        score += 1

    if PATTERNS["normal_browser"].search(ua) and tags and "Normal Browser" not in tags:
        tags.append("Browser-like / Possible Spoof")
        score += 5

    if not tags:
        tags.append("Unknown / Unclassified")

    if score >= 70:
        summary = "High-confidence suspicious or institutional traffic"
    elif score >= 40:
        summary = "Moderate-confidence automated or institutional traffic"
    elif score >= 15:
        summary = "Likely automated traffic"
    else:
        summary = "Low-confidence / normal-looking traffic"

    return {
        "user_agent": ua,
        "tags": tags,
        "score": score,
        "summary": summary
    }

if __name__ == "__main__":
    test_samples = [
        "Mozilla/5.0 (compatible; GovBot/1.0; +https://www.usa.gov)",
        "Mozilla/5.0 (compatible; Qualys; +https://www.qualys.com)",
        "python-requests/2.31.0",
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/121.0.0.0 Safari/537.36",
        "Mozilla/5.0 (compatible; FederalAuditBot/2.1)",
        "zgrab/0.x",
    ]

    for sample in test_samples:
        print(classify_user_agent(sample))

Extended Bot / Crawler / Scanner Database

This block is intentionally broad and useful as a seed-list for regex packs, keyword files, or JSON pattern sets.

GovBot
GovCrawler
GovScanner
GovMonitor
GovSecurityScanner
GovAuditBot
GovDataCollector
GovResearchBot
GovIndexBot
GovComplianceBot
GovVulnScanner
GovSiteAudit
GovWebInspector
GovWebCrawler
GovArchiveBot
GovAnalyticsBot
GovNetworkScanner
GovProbe
GovReconBot
GovRecordsBot
GovTransparencyBot
GovOpenDataCrawler
GovDigitalArchive
FederalAuditBot
FederalComplianceBot
FederalCrawler
FederalGovBot
FederalResearchBot
PublicSectorCrawler
PublicRecordsBot
ResearchBot
ArchiveBot
ArchiveCrawler
IndexBot
Indexer
DataCollector
DataHarvester
SurveyBot
MappingBot
TelemetryCollector
GeoSurveyBot
GovHarvester
GovInspector
GovTelemetry
GovAssetScanner
GovSurfaceMapper
GovReconCrawler
GovIntelBot
GovArchiveScanner
GovEvidenceCollector
GovForensicsBot
USGovBot
USGovWebCrawler
DHSBot
DHSScanner
DOJBot
FBIBot
TreasuryBot
IRSBot
USGSBot
NOAABot
NASABot
Nessus
Tenable
OpenVAS
Greenbone
Qualys
Acunetix
Nikto
Burp
BurpSuite
AppScan
Rapid7
InsightVM
Nexpose
Nmap
Nmap Scripting Engine
Masscan
ZGrab
ZMap
Shodan
Censys
WhatWeb
sqlmap
w3af
Arachni
DirBuster
Gobuster
ffuf
feroxbuster
wpscan
joomscan
skipfish
Wget
curl
python-requests
aiohttp
httpx
Go-http-client
libwww-perl
okhttp
urllib
PowerShell
Java
bot
crawler
spider
harvester
scanner
inspector
recon
audit
compliance
telemetry
archive
records
mapping
survey
datacollector
dataharvester
webcrawler
webinspector
siteinspector
assetdiscovery
contentdiscovery
reconscanner
securityscanner
vulnerabilityscanner
compliancescanner
auditscanner

Operational Notes

• User-agent alone is not enough. Plenty of serious traffic wears a fake mustache and a normal Chrome string.
• Contractor scanners matter. A government hit may look like Qualys, Nessus, Rapid7, or a generic client.
• Use scoring, not absolutes. UA signals should feed a confidence model, not a guillotine.
• Correlate with ASN and reverse DNS. That is where the mask often slips.
• Track behavior. Rate, depth, path selection, robots behavior, and timing rhythm reveal the creature.

Best use-case: feed this into a classifier and tag logs as Government, Military, Security Scanner, Crawler / Bot, Browser-like / Possible Spoof, or Unknown.