Custom security graph collectors

Built-in collectors cover AWS, Azure, GCP, Kubernetes, and the demo graph. Anything else enters through an external executable. In-process Go plugins are not supported. The contract is one JSON object on stdout, so the collector can be written in any language.

Plugins are trusted code. They run as the user who invokes om, inherit that environment and working directory, and should stay read-only toward the systems they inventory. om does not sandbox the process.

Run

go build -o example-collector ./examples/collector
./om scan plugin -- ./example-collector
./om scan plugin --timeout 5m -- ./my-collector --region us-east-1

--timeout defaults to 10 minutes and cancels the process. Arguments after -- belong to the plugin. Flags before -- belong to om. Exit 0 means the batch was produced. om validates it, then upserts. A non-zero exit fails the scan and includes stderr when the plugin wrote any.

Stdout is capped at 32 MiB. Larger output is rejected and the process is killed.
Stderr is capped at 64 KiB and is the place for logs. Do not print the batch there.
Stdout must be exactly one JSON object. A second value, or log lines mixed into stdout, fails decode.

A successful plugin scan prints the same line as a cloud scan: Ingested N nodes and M edges from plugin <basename>. It does not run CSPM rules. Follow it with om rules run.

Stdout document

Field names match sdk/collector. Schema version is the constant SchemaVersion = 1. The JSON itself has no version field.

{
  "nodes": [
    {
      "id": "internet:global",
      "type": "Internet",
      "name": "Internet",
      "provider": "plugin"
    },
    {
      "id": "plugin:host:edge-1",
      "type": "Workload",
      "name": "edge-1",
      "provider": "plugin",
      "account_id": "lab",
      "properties": { "public_ip": true }
    },
    {
      "id": "plugin:bucket:logs",
      "type": "Datastore",
      "name": "logs",
      "provider": "plugin",
      "account_id": "lab",
      "properties": { "public_access": true, "service": "s3" }
    }
  ],
  "edges": [
    {
      "source_id": "internet:global",
      "target_id": "plugin:host:edge-1",
      "type": "REACHABLE"
    },
    {
      "source_id": "plugin:host:edge-1",
      "target_id": "plugin:bucket:logs",
      "type": "CAN_ACCESS"
    }
  ]
}

Validation, after empty edge ids are filled:

Node id, type, and name are required. Duplicate node ids fail.
Types must be Internet, Network, Workload, Identity, Datastore, Finding, or Control.
Edge types must be REACHABLE, ASSUMES, CAN_ACCESS, AFFECTS, or VIOLATES.
An empty edge id becomes source|target|type. Duplicate edge ids fail.
Both endpoints must be nodes in the same batch. An edge to a node already in Postgres, but missing from this document, is rejected.

POST /v1/ingest does not run this validation. Prefer om scan plugin for external collectors. Include internet:global when the batch should show up in attack path queries. Set public_access, admin_access, and the other keys on Schema when you want pack rules to match. On a datastore, set sensitivity to a non-empty string to mark a crown jewel. internet-to-sensitive-datastore keeps only those paths. Built-in collectors copy that value from a tag or label named sensitivity or data-class; a plugin sets the property itself. On a workload, packages, image, and images are what om enrich cve matches. A package entry is a CPE 2.3 name or a versioned package URL. See CVE enrichment.

Go SDK

Implement collector.Collector (Collect(ctx) (Batch, error)) and call collector.Run from main. Run writes the batch to stdout and exits non-zero on error. examples/collector is the reference: an internet-reachable host with a package URL, an image ref, and CAN_ACCESS to a public datastore. Use collector.InternetNodeID (internet:global) and the type constants rather than string literals.

Upserts keep the last batch’s properties for a given id. A plugin that omits internet:global on a later run does not delete the internet node another scan wrote. Resources you stop emitting stay in the graph. Delete them yourself, or scope them with account_id and remove that account the way om scan demo does for its sample accounts.

The design record is ADR 004.