SaaS Collectors

Every other integration in this section is push: you configure the vendor, or a shipper you run, to send events to nano. Collectors invert that. nano holds your API credentials and pulls events from the vendor on a schedule, so there is nothing to deploy and nothing to keep running on your side.

Use a collector when the vendor has an event-export API but no practical way to push to you.

Concepts

Enabling a stream also creates a log source for it, by resolving the parser whose match values claim the stream's source type and importing it. That is what makes a collector's data visible in the surface where you already look for feeds. If no parser claims the source type, the connection says so — the events still arrive and are stored, they are just unparsed until you import one.

Deleting a connection keeps its log sources. By then they may carry parser edits or config of your own, so they are unlinked rather than destroyed; they simply stop receiving events.

The instance/integration split matters in practice: an MSSP connecting six customer tenants of the same product installs the integration once and creates six instances, each with its own token and its own cursors.

Getting started

1. Add the catalog repository. Go to Settings → Marketplace → Repositories and add https://github.com/nano-rs/nano-integrations with the content path integrations. The catalog populates within a few minutes.
2. Install the integration from the marketplace. Installing makes it available to configure; it does not start collecting.
3. Connect a tenant. Open the integration in the marketplace and go to its Connections tab, then Connect tenant. Supply the credentials and config the manifest asks for, select your streams, and enable it. You can add as many connections as you have tenants.
4. Watch it drain. Each enabled stream becomes a log source, so collection shows up in Ingestion → Log Sources alongside every other feed. The Connections tab shows per-stream event counts and staleness.

Collection starts from now by default. Set a backfill start only if you specifically want history — most iterator APIs will happily replay their entire retention window on first poll, which is rarely what you want.

Netskope

The reference collector. Uses the REST API v2 Data Export iterator endpoints.

Create an API token

In the Netskope admin console, go to Settings → Administration → Administrators → Service Account and create a REST API v2 token.

Grant it read access to each endpoint whose stream you intend to enable:

/api/v2/events/dataexport/events/alert
/api/v2/events/dataexport/events/application
/api/v2/events/dataexport/events/network
/api/v2/events/dataexport/events/audit
/api/v2/events/dataexport/events/page
/api/v2/events/dataexport/events/connection
/api/v2/events/dataexport/events/incident
/api/v2/events/dataexport/events/infrastructure
/api/v2/events/dataexport/events/endpoint

The token is scoped per endpoint. A missing grant returns 403 for that stream only — the run is marked partial and the other streams keep working, so check the per-stream error rather than assuming the whole integration is broken.

Configure the instance

Streams

alert, application, network and audit are enabled by default. page and connection are very high volume — enable them deliberately, and size your retention accordingly.

Each stream lands under its own source type (netskope_alert, netskope_network, …), all handled by the netskope parser in the parsers repository.

Iterator names are exclusive

Netskope tracks read position server-side, keyed by the index query parameter. Two consumers sharing an index name steal each other's events — each request advances the shared position, so each consumer sees only the pages the other did not take.

If you already run Netskope Cloud Exchange or another SIEM against the same tenant, give nano its own iterator name. Concurrent use of the same name returns HTTP 409 and can lose data outright; nano surfaces that as a stream error rather than retrying into it.

How delivery works

Collectors are at-least-once. A batch is shipped to the ingest pipeline, and only once nano confirms it does the collector commit the cursor. If the process dies in between, the next run re-fetches from the last committed cursor and re-delivers some events.

Duplicates are the deliberate trade. The alternative — committing first — turns any crash into silent data loss, which for an iterator API is unrecoverable because the vendor has already moved on.

Pivot fields are preserved so duplicates are recognisable: Netskope's _id, transaction_id and connection_id land in ext.

Monitoring

The per-stream staleness figure on the Connections tab is the number that matters. Iterator APIs do not queue indefinitely — undelivered events are dropped after a retention window (7 days on some Netskope streams). A stalled stream is therefore data loss in progress, not a backlog that will catch up.

nano warns at 3 hours and flags an error at 24 hours, well inside the shortest vendor window.

Run outcomes:

A run that ends because its time budget expired is not a failure — a busy stream legitimately needs more than one poll to catch up.

Limitations

Read these before committing to a collector for a high-volume source.

Throughput is bounded by the vendor, then by the sandbox

Netskope serves at most 10,000 records per request and rate-limits to 4 requests/second per endpoint, and its responses carry a wait_time the collector honours. That is the binding constraint for Netskope long before nano is.

One run at a time per instance

Runs are single-flight, enforced by a database lease. This is required — concurrent consumption of one cursor loses events on iterator APIs — but it means a single instance cannot be scaled out across pods. To increase parallelism, split streams across multiple instances with distinct iterator names.

Downtime has a hard deadline

Because vendors drop undelivered events, a collector that is down longer than the vendor's retention window loses that data permanently. There is no backfill for it. Alert on stream staleness.

Run budget

Each run has a wall-clock budget (default 10 minutes, maximum 60). When it expires the collector is asked to stop, gets 30 seconds to finish and checkpoint, and is then killed. Progress up to the last checkpoint is kept, and the next run resumes from there.

No air-gapped support

Collectors exist to reach the internet. In air-gapped mode the collector scheduler does not start, alongside the other egress jobs. Use a push transport instead.

No three-legged OAuth

Supported auth: none, bearer token, API key header, HTTP basic, and OAuth2 client credentials. Authorization-code flows with user consent and refresh tokens are not supported, which rules out a small number of user-scoped SaaS APIs. Most security-log APIs use static tokens or client credentials.

Egress is per-integration, not arbitrary

A collector can only reach hosts its manifest declares — either exact hostnames or a domain suffix (.goskope.com) that your tenant hostname must satisfy. A tenant hostname that does not match any declared suffix is rejected when you save the instance, not at runtime. This is deliberate: the config field would otherwise be a way to point nano's credentialed HTTP client anywhere.

Writing your own collector

Collectors are TypeScript, run in nano's Deno sandbox, and live in a Git repository you control — the same model as custom enrichments. Point the marketplace at your own repo to keep private integrations private.

See the nano-integrations repository for the manifest schema and the collector contract.

The essential shape:

async function collect(ctx: CollectorContext): Promise<void> {
  for (const stream of ctx.streams) {
    let cursor = ctx.cursors[stream];
    while (!ctx.shouldStop()) {
      const { events, nextCursor, drained } = await fetchPage(ctx, stream, cursor);
      if (events.length > 0) {
        await ctx.emit(stream, events);          // resolves once nano has them
        await ctx.checkpoint(stream, nextCursor); // only after emit resolves
      }
      cursor = nextCursor;
      if (drained) break;
    }
  }
}
export { collect };

Emit raw vendor events. Field mapping belongs in the parsers repository, so that parsers stay testable against real vendor samples.