SaaS Collectors
Pull SaaS audit events on a schedule with pull feeds or Marketplace collectors
Use a SaaS collector to pull audit events from a vendor API on a schedule. You grant vendor access and choose the streams; nano runs the collector and delivers its events into the local Vector, where each one is parsed like any other event.
Use a collector when the vendor has an event-export API but no practical way to push to you.
nano has two collector runtimes:
| Pull feeds | Marketplace collectors | |
|---|---|---|
| Edition | Open core and Self-run | Self-run (not in the open-source edition) |
| Where | Ingestion → Log Sources → Pull feeds | Marketplace → integration → Connections |
| What runs | A reviewed connector spec bundled with nano, pinned by sha256, run by the nano-connector-runner service | An integration from a catalog repository: a declarative spec or a TypeScript collector in nano's WebAssembly sandbox |
| Connectors | Okta System Log, Google Workspace, Slack access logs, Netskope; identity feeds for Okta users and the Entra directory | The official catalog (nano-rs/nano-integrations), or your own fork |
Pull feeds
A pull feed runs one bundled connector on a schedule and delivers its events to the local Vector over HEC. The cursor moves only after Vector acknowledged the events.
| Connector | Credential | Config | Default schedule |
|---|---|---|---|
| Okta System Log | API_TOKEN (SSWS) | OKTA_DOMAIN | every 2 minutes |
| Google Workspace | Service-account JSON key | ADMIN_EMAIL, streams | every 5 minutes |
| Slack access logs | USER_TOKEN | TEAM_ID (optional) | hourly |
| Netskope | API_TOKEN | TENANT_HOST, STREAMS, INDEX_NAME (optional) | every 5 minutes |
| Okta users (identity) | API_TOKEN (SSWS) | OKTA_DOMAIN | |
| Entra directory (identity) | CLIENT_SECRET | DIRECTORY_ID, CLIENT_ID | hourly |
- Publish the parser for each log source the connector emits into under Ingestion → Log Sources. A feed won't start until every source it emits into has an enabled, deployed parser, so it never pulls into Vector's drop path.
- Add a feed. Open Pull feeds, choose the connector, name the feed, and enter its config. Bind an existing credential, or enter the vendor secret with the feed; it is stored encrypted in the credentials store and never shown again. You can override the schedule with a cron expression.
- Watch it run. The list shows each feed's next run, last status and last error. Open a feed to see its runs (events, pages, errors) and per-stream checkpoints.
You can pause, resume, and remove a feed, rotate its secret from its row, and re-anchor one stream's checkpoint. When a nano upgrade bundles a newer connector spec, the feed shows that an update is available; adopting it resets the feed's cursors.
Identity feeds (Okta users, Entra directory) emit into no log source: they sync users and group memberships into nano's user directory. Removing one tombstones the users it synced.
Starting a feed (create or resume) needs log_sources:deploy; binding a credential needs credentials:use, and entering a new one also credentials:create.
Marketplace collectors
Concepts
| Term | What it is |
|---|---|
| Integration | The collector itself — code plus a manifest, installed from a catalog repository. One per vendor. |
| Instance | One configured connection to one vendor tenant. Holds credentials, config, stream selection, and cursors. You can have several per integration. |
| Stream | An independently toggleable feed within an integration (Netskope alerts vs. page events). Each maps to its own source, so each is parsed and routed separately and stored with its own metadata.log_source. |
| Cursor | Where a stream last got to. Opaque — the collector decides what it means. |
Enabling a stream also creates a log source for it, by resolving the parser whose match values claim the stream's source type and importing it. Check that source under Ingestion → Log Sources. If no parser claims the source type, the connection says so — the events still arrive and are stored, they remain unparsed until you import one.
Deleting a connection keeps its log sources and any parser or configuration edits you made. nano unlinks those sources from the connection, and they stop receiving its events.
The instance/integration split matters in practice: an MSSP connecting six customer tenants of the same product installs the integration once and creates six instances, each with its own token and its own cursors.
Getting started
- Find the integration. nano syncs the official catalog (
nano-rs/nano-integrations) by default, so collectors appear in the Marketplace with no setup. Running your own fork instead? See Writing your own collector. - Install the integration. Installing makes it available to configure; it does not start collecting.
- Connect a tenant. Open the integration in the marketplace and go to its Connections tab, then Connect tenant. Supply the credentials and config the manifest asks for, select your streams, and enable it. You can add as many connections as you have tenants.
- Verify collection. Each enabled stream becomes a log source, so collection shows up in Ingestion → Log Sources alongside every other feed. The Connections tab shows per-stream event counts and staleness.
Start position
Every collector starts point-forward. A new feed or connection starts at current activity, and older API history is not collected. An existing cursor is always resumed, never reset. There is no backfill start; if you need history, contact nano support. See what happens when you enable a pull source.
Netskope
Configure Netskope collection with the REST API v2 Data Export iterator endpoints.
Create an API token
In the Netskope admin console, go to Settings → Administration → Administrators → Service Account and create a REST API v2 token.
Grant it read access to each endpoint whose stream you intend to enable:
/api/v2/events/dataexport/events/alert
/api/v2/events/dataexport/events/application
/api/v2/events/dataexport/events/network
/api/v2/events/dataexport/events/audit
/api/v2/events/dataexport/events/page
/api/v2/events/dataexport/events/connection
/api/v2/events/dataexport/events/incident
/api/v2/events/dataexport/events/infrastructure
/api/v2/events/dataexport/events/endpointThe token is scoped per endpoint. A missing grant returns 403 for that stream only — the run is marked partial and the other streams keep working, so check the per-stream error rather than assuming the whole integration is broken.
Configure the feed or instance
| Field | Value |
|---|---|
Tenant hostname (TENANT_HOST) | Your Netskope tenant host, e.g. acme.goskope.com. A full URL is accepted and reduced to the host. |
API token (API_TOKEN) | The service-account token above. Encrypted at rest; never returned by the API. |
Iterator name (INDEX_NAME) | Defaults to nano. Change it only if another tool consumes the same tenant — see below. |
Streams
alert, application, network and audit are enabled by default. page and connection are very high volume — enable them deliberately, and size your retention accordingly.
Each stream lands under its own source type (netskope_alert, netskope_network, …), all handled by the netskope parser in the parsers repository.
Iterator names are exclusive
Netskope tracks read position server-side, keyed by the index query parameter. Two consumers sharing an index name steal each other's events — each request advances the shared position, so each consumer sees only the pages the other did not take.
If you already run Netskope Cloud Exchange or another SIEM against the same tenant, give nano its own iterator name. Concurrent use of the same name returns HTTP 409 and can lose data outright; nano surfaces that as a stream error rather than retrying into it.
How delivery works
Collectors are at-least-once. A batch is shipped to the ingest pipeline, and only once nano confirms it does the collector commit the cursor. If the process dies in between, the next run re-fetches from the last committed cursor and re-delivers some events.
Checkpointing after delivery avoids silent loss from a crash before ingestion. Iterator APIs may already have advanced the vendor's read position, so committing first can leave data unrecoverable.
Pivot fields are preserved so duplicates are recognizable: Netskope's _id, transaction_id and connection_id are kept on the event, under unmapped where OCSF has no home for them.
Collector monitoring
Monitor each feed's last success on Pull feeds, and each Marketplace stream's staleness on the Connections tab. Iterator APIs do not queue indefinitely — undelivered events are dropped after a retention window (7 days on some Netskope streams). A stalled stream is therefore data loss in progress, not a backlog that will catch up.
nano warns at 3 hours and flags an error at 24 hours, well inside the shortest vendor window.
Run outcomes:
| Status | Meaning |
|---|---|
success | Every enabled stream drained without error. |
partial | Some events collected, at least one stream failed. Usually a missing token scope on one endpoint. |
failed | Nothing collected. Bad credentials, unreachable host, or invalid config. |
running | A run holds the instance's lease. |
A run that ends because its time budget expired is not a failure — a busy stream legitimately needs more than one poll to catch up.
Collector limitations
Check these limits before using a collector for a high-volume source.
Throughput is bounded by the vendor, then by the sandbox
Netskope serves at most 10,000 records per request and rate-limits to 4 requests/second per endpoint, and its responses carry a wait_time the collector honors. That is the binding constraint for Netskope long before nano is.
One run at a time per instance
Runs are single-flight, enforced by a database lease. This is required — concurrent consumption of one cursor loses events on iterator APIs — but it means a single instance cannot be scaled out across pods. To increase parallelism, split streams across multiple instances with distinct iterator names.
Downtime has a hard deadline
Because vendors drop undelivered events, a collector that is down longer than the vendor's retention window loses that data permanently. There is no backfill for it. Alert on stream staleness.
Run budget
Each run has a wall-clock budget (Marketplace default 10 minutes, maximum 60; pull feeds use the spec's budget, for example 10 minutes for Okta System Log). When it expires the collector is asked to stop, gets 30 seconds to finish and checkpoint, and is then killed. Progress up to the last checkpoint is kept, and the next run resumes from there.
No air-gapped support
Collectors exist to reach the internet. In air-gapped mode the Marketplace collector scheduler does not start, alongside the other egress jobs, and pull feeds can't reach their vendor. Use a push transport instead.
No three-legged OAuth
Supported authentication depends on the collector runtime and spec. Options include bearer tokens, API keys, HTTP basic, OAuth2 client credentials, and Google service-account JWTs. Authorization-code flows with user consent and refresh tokens are not supported, which rules out a small number of user-scoped SaaS APIs. Most security-log APIs use static tokens or client credentials.
Egress is per-integration, not arbitrary
A collector can only reach hosts its manifest declares — either exact hostnames or a domain suffix (.goskope.com) that your tenant hostname must satisfy. A tenant hostname that does not match any declared suffix is rejected when you save the instance, not at runtime. This is deliberate: the config field would otherwise be a way to point nano's credentialed HTTP client anywhere.
Writing your own collector
Custom collectors are TypeScript, run in nano's embedded WebAssembly sandbox (the same rules as custom enrichments: https only, credentials only in request headers), and live in a Git repository you control, the same model as custom enrichments. First-party declarative connectors use a YAML spec whose engine handles authentication, paging, retries, and checkpoints. Custom collectors need the Marketplace runtime, which isn't in the open-source edition. Point the marketplace at your own repo to keep private integrations private.
See the nano-integrations repository for the manifest schema and the collector contract.
The essential shape:
async function collect(ctx: CollectorContext): Promise<void> {
for (const stream of ctx.streams) {
let cursor = ctx.cursors[stream];
while (!ctx.shouldStop()) {
const { events, nextCursor, drained } = await fetchPage(ctx, stream, cursor);
if (events.length > 0) {
await ctx.emit(stream, events); // resolves once nano has them
await ctx.checkpoint(stream, nextCursor); // only after emit resolves
}
cursor = nextCursor;
if (drained) break;
}
}
}
export { collect };Emit raw vendor events. Field mapping belongs in the parsers repository, so that parsers stay testable against real vendor samples.
Troubleshoot
Need help with these checks? Contact support with the deployment, request ID and time with timezone.
One stream returns 403
Check the vendor token's read grant for that stream's endpoint. A missing Netskope grant can leave a run partial while other streams continue.
Every request fails
If Netskope IP allowlisting is enabled, add your deployment's egress addresses to the Custom IP list. Check credentials, host reachability, and the per-stream error.
Staleness keeps increasing
Check run outcomes and fix the stalled stream before the vendor's retention window expires. Use distinct iterator names for separate consumers; sharing a name can lose events.