Pipeline Execution Guide

Authoritative steps for download → ETL → SHACL → Neo4j load.

Prerequisites

Set Neo4j connection credentials as environment variables before running any pipeline script:

# Windows PowerShell
$env:NEO4J_URI      = "bolt://localhost:7687"
$env:NEO4J_USER     = "neo4j"
$env:NEO4J_PASSWORD = "yourpassword"

# Linux / macOS
export NEO4J_URI=bolt://localhost:7687
export NEO4J_USER=neo4j
export NEO4J_PASSWORD=yourpassword

Raw data must already be downloaded to data/raw/. If not, run the downloader scripts first (see scripts/downloaders/).

ETL Pipeline Orchestrator

scripts/run_all_etl.py runs all 11 ETL steps in strict causal-chain order and exits 1 on the first failure.

# Run the full pipeline (recommended on first load or after re-downloading)
python scripts/run_all_etl.py

# Run with a non-default data directory
python scripts/run_all_etl.py --raw-data-dir D:\data\raw

# Preview what would run without connecting to Neo4j
python scripts/run_all_etl.py --dry-run

# Re-run a single step (skip constraints if already applied)
python scripts/run_all_etl.py --step cve --skip-constraints

# Verbose debug output
python scripts/run_all_etl.py --log-level DEBUG

Parameters

Step Registry

Steps execute in this exact order. Each step requires all prior steps to have succeeded (hard stop on first failure).

Validation

SHACL + Post-Load Checks

scripts/validation/validate_all_standards.py runs pySHACL against OWL ontology files and optionally runs Cypher post-load integrity checks.

# Validate all SHACL shapes against OWL ontology files
python scripts/validation/validate_all_standards.py

# Validate a single standard only
python scripts/validation/validate_all_standards.py --standard attck

# Run post-load Cypher integrity checks (requires Neo4j connection)
python scripts/validation/validate_all_standards.py --post-load

# Skip SHACL; run post-load checks only
python scripts/validation/validate_all_standards.py --skip-shacl --post-load

Post-Load Integrity Checks

The --post-load mode runs four Cypher assertions. Each must return 0 to pass:

Graph Statistics

scripts/utilities/extract_neo4j_stats.py queries node/relationship counts and freshness dates.

# Pretty-printed table (requires rich library)
python scripts/utilities/extract_neo4j_stats.py --pretty

# JSON output (for CI/CD parsing)
python scripts/utilities/extract_neo4j_stats.py --json

# Exit 1 if any label count falls below the expected minimum
python scripts/utilities/extract_neo4j_stats.py --check-minimums

Expected Minimums

Smoke Test (Log4Shell)

After a full pipeline run, verify the end-to-end causal chain in Neo4j Browser:

MATCH (pc:PlatformConfiguration)<-[:AFFECTS]-(v:Vulnerability {cveId:'CVE-2021-44228'})
-[:CAUSED_BY]->(w:Weakness)
-[:DEMONSTRATED_BY]->(ap:AttackPattern)
-[:IMPLEMENTS]->(t:Technique)
RETURN v.cveId, w.cweId, ap.capecId, t.attackId

This must return at least one row. A zero-result response indicates a failure in one of the CPE, CVE, CWE, CAPEC, or ATT&CK steps.

Loader Mapping Note (Systems Standards Namespaces)

Use this mapping during ETL/Neo4j load so graph artifacts stay aligned with standards OWLs under standards/.

Core causality terms remain unchanged (kgcs:caused_by, kgcs:exploited_by, kgcs:implemented_as).

OWL Imports Checklist (Systems-Focused Loader)

Load modules in this order for systems-agent-aligned graph generation:

1. core/core-ontology-v1.0.owl — authoritative frozen backbone (kgcs:).
2. standards/cpe/cpe-ontology-v1.0.owl — platform/config identifiers and status fields (cpe:).
3. standards/cve/cve-ontology-v1.0.owl — vulnerability identifiers/metadata and compatibility predicates (cve:).
4. standards/cwe/cwe-ontology-v1.0.owl — weakness identifiers and abstraction fields (cwe:).
5. standards/cvss/cvss-ontology-v1.0.owl — score class alias, score predicates, and CVSS fields (cvss:).
6. extensions/asset-extension-v1.0.owl — organizational asset context layered on top of standards/core (asset:).

Guardrails:

• Do not modify core/core-ontology-v1.0.owl.
• Keep standards semantics in standards/; keep organizational context in extensions/.
• Ensure loaded graph edges/properties follow the mapping table above.