The threat-intel data system#

glitched setup loads every source into a local Postgres database and builds the warehouse with dbt once at the end. Re-run it to refresh: every loader is idempotent and only fetches what's new. Options: --skip-nvd (NVD takes about a minute without an API key) and --papers (research papers; classification needs the [papers] extra and ANTHROPIC_API_KEY).

Start from a snapshot instead of downloading everything: someone with a loaded database exports one, and you set up from it in seconds.

glitched snapshot export intel.zip                 # ATLAS, EMB3D, NVD (~0.6 MB)
glitched setup --snapshot intel.zip                # import + build, no downloads
glitched setup --snapshot intel.zip --refresh      # ...then fetch anything newer

A snapshot holds only the raw data (every release, plus NVD's incremental cursors); dbt rebuilds everything else on import, so it always matches your version of Glitched. Your curated CVE notes and research papers are left out unless you add --include research / --include papers (paper abstracts carry their sites' terms; check before sharing).

Each source can also be loaded on its own (each runs its own dbt build):

python -m glitched.pipelines.mitre      # MITRE ATLAS   - AI/ML adversary tactics & techniques
python -m glitched.pipelines.emb3d      # MITRE EMB3D   - embedded-device threats, mitigations, properties
python -m glitched.pipelines.research   # research CVEs - curated robotics CVEs ($GLITCHED_RESEARCH_DIR)
python -m glitched.pipelines.nvd        # NVD CVEs      - CVEs mentioning ROS, Unitree, Universal Robots, LeRobot, MQTT, ...
python -m glitched.pipelines.papers     # papers        - robot / embodied-AI security research, classified by Claude

NVD: - What it searches: NVD for each keyword marked nvd_search in src/glitched/pipelines/transform/seeds/platform_keywords.csv. Keep that list to specific platforms; a generic word like "firmware" matches thousands of CVEs. - What it keeps: only CVEs where the keyword appears as a whole word, because NVD's own search also matches inside words ("ros" finds "roster"). - Repeat runs fetch only CVEs modified since the last one. - Speed: without an API key a run takes about a minute (NVD allows 5 requests per 30 seconds). Request a free key and set NVD_API_KEY in your environment or a .env file to go about 10x faster.

Research papers: - Sources: - Hugging Face Papers searches; Papers with Code now redirects there. Edit SEARCH_QUERIES and PINNED_PAPERS in src/glitched/pipelines/extract/research_papers.py. - arXiv searches, scoped to the Robotics and Security categories plus attacks on vision-language-action models. Edit ARXIV_QUERIES in src/glitched/pipelines/extract/arXiv.py. Each run fetches only papers submitted since the last one. A new search starts with its 50 newest papers rather than the full backlog (thousands). - NVIDIA Research publications in the Security, Resilience and Safety, Robotics and Physical AI areas. From Robotics and Physical AI, only titles that mention a security term are kept. Edit RESEARCH_AREAS in src/glitched/pipelines/extract/nvidia_research_papers.py. Each publication page is fetched only once. - A paper found on several sites is one row, keyed by its arXiv id. NVIDIA papers with no arXiv id are keyed nvidia:<slug>. - What Claude extracts (src/glitched/pipelines/classify/papers.py), in terms the assessment agent already uses: - whether the paper is really about robot or device security - which ATLAS techniques and EMB3D threats it covers, and which platforms it targets - concrete checks, each tagged passive, active or manual_review like the agent's own tests - Cost: classification calls the Claude API (ANTHROPIC_API_KEY), about $0.03 per paper, and only new papers are classified. Try it cheaply first:

python -m glitched.pipelines.papers --classify-limit 5   # classify at most 5 new papers
python -m glitched.pipelines.papers --skip-classify      # fetch and load only

• Results: the exposure report includes relevant papers. In code, queries.get_paper_checks(tags, property_ids, ml) returns the checks that apply to a device, each citing its paper.

How the pipelines work (ELT)#

All four loaders run the same three steps, with each step in its own folder:

Releases and history. - Raw keeps every release (EMB3D 2.0.1, ATLAS 2026.09, and a content hash for the CVE notes). Each raw schema's _loads table logs every load, and the latest load decides which release is current. - Pinned releases. EMB3D and ATLAS are pinned: change EMB3D_VERSION in src/glitched/pipelines/extract/emb3d.py or ATLAS_RELEASE in src/glitched/pipelines/extract/atlas.py to upgrade. The run logs a notice when a newer ATLAS release exists. - Nothing is deleted. Something a new release drops stays in its public table with is_current = false, because your assessments may still point at it. last_seen_release says when it was last present, and the app only reads current rows. - History. history.<table>_history has one row per version of each threat, technique, CVE and so on, valid from dbt_valid_from to dbt_valid_to.

Platform keywords (ROS, Unitree, BLE, …) live in src/glitched/pipelines/transform/seeds/platform_keywords.csv. dbt uses them to tag CVEs, and the app's onboarding reads the same file.

dbt runs through uv in its own Python 3.12 environment, so nothing extra goes into .venv. Set DBT_COMMAND to use a dbt you've installed yourself. To work on the models directly:

cd src/glitched/pipelines/transform
uvx --python 3.12 --from dbt-core~=1.12 --with dbt-postgres~=1.11 dbt build --profiles-dir .
uvx --python 3.12 --from dbt-core~=1.12 --with dbt-postgres~=1.11 dbt docs generate --profiles-dir .  # data dictionary

Tests: - Python: python -m unittest discover -s tests -t . covers the source contracts, new-field warnings and raw DDL. Database tests run when GLITCHED_TEST_DATABASE_URL (a server; default localhost) or GLITCHED_TEST_INTEL_URL (a database built with glitched setup --snapshot, used as a template) is set. - dbt: dbt build runs the data tests, unit tests for CVE-id parsing and keyword tagging, and per-table checks that current rows match staging.