inputlookup

Fetch data from external URLs during search queries for real-time enrichment or data retrieval.

Description

The inputlookup command fetches data from external HTTP/HTTPS endpoints and either returns it as search results or joins it with existing results. This enables real-time enrichment from external APIs, threat intelligence feeds, and data sources.

The command supports two modes:

• Data source mode: Fetch URL and return parsed data as search results
• Enrichment mode: Join URL results with existing search results using a key field

Syntax

| inputlookup url="<url>" [format=json|csv] [key=<field>] [timeout=<seconds>] [max_rows=<count>] [cache_ttl=<seconds>]

Required Arguments

url Syntax: url="<url>" Description: URL to fetch. Can include {field} placeholders for enrichment mode (e.g., https://api.example.com/{ip}/json).

Optional Arguments

format Syntax: format=json|csv Default: json Description: Expected response format. JSON responses can be arrays or objects with data/results/items keys.

key Syntax: key=<field> Description: Field to join on. Enables enrichment mode when specified. The field value is substituted into {field} placeholders in the URL.

timeout Syntax: timeout=<seconds> Default: 30 Range: 1-60 Description: Request timeout in seconds.

max_rows Syntax: max_rows=<count> Default: 10000 Range: 1-100000 Description: Maximum rows to return from the URL response.

cache_ttl Syntax: cache_ttl=<seconds> Default: 300 Range: 0-3600 Description: Cache TTL in seconds. Set to 0 to disable caching.

Examples

Data source mode: Fetch threat feed

| inputlookup url="https://feeds.example.com/iocs.csv" format=csv
| table ip, category, threat_level

Fetches a threat intelligence feed and displays as search results.

Data source mode: Fetch JSON data

| inputlookup url="https://api.example.com/blocklist.json" format=json
| where risk_score > 80

Fetches JSON data and filters high-risk entries.

Enrichment mode: IP geolocation

* | inputlookup url="https://ipinfo.io/{src_ip}/json" key=src_ip format=json
| table src_ip, inputlookup_city, inputlookup_country

Enriches source IPs with geolocation data from ipinfo.io.

Enrichment mode: Domain reputation

* | inputlookup url="https://api.domaincheck.com/v1/{domain}?key=YOUR_KEY" key=domain format=json
| where inputlookup_risk > 0.7

Enriches domains with reputation scores.

Custom timeout for slow APIs

* | inputlookup url="https://slow-api.example.com/{hash}" key=file_hash format=json timeout=60

Uses extended timeout for slow external APIs.

Disable caching for real-time data

* | inputlookup url="https://api.example.com/live/{ip}" key=src_ip format=json cache_ttl=0

Disables caching for real-time lookups.

CSV threat feed enrichment

* | inputlookup url="https://feeds.example.com/malware/{file_hash}.csv" key=file_hash format=csv
| where inputlookup_malware_family != ""

Enriches file hashes with malware family information.

Chain with other commands

* | inputlookup url="https://api.ipinfo.io/{src_ip}/json" key=src_ip format=json
| eval geo_info = inputlookup_city . ", " . inputlookup_country
| table src_ip, geo_info, action

Uses enriched fields in calculations.

Multiple enrichments

* | inputlookup url="https://api.ipinfo.io/{src_ip}/json" key=src_ip format=json
| rename inputlookup_city as src_city, inputlookup_country as src_country
| inputlookup url="https://api.ipinfo.io/{dest_ip}/json" key=dest_ip format=json
| rename inputlookup_city as dest_city, inputlookup_country as dest_country
| table src_ip, src_city, dest_ip, dest_city

Performs multiple enrichments with field renaming.

Security

The inputlookup command includes SSRF (Server-Side Request Forgery) protection:

• Blocked addresses: Private IPs (10.x, 172.16-31.x, 192.168.x), localhost (127.x), link-local (169.254.x)
• Blocked endpoints: Cloud metadata endpoints (169.254.169.254, metadata.google.internal)
• Scheme validation: Only http:// and https:// are allowed (HTTPS recommended)
• Redirect validation: Each redirect target is validated
• Response limits: Maximum 10MB response size

Field Naming

Fields from inputlookup are prefixed with inputlookup_ to distinguish them from original fields:

Use the rename command to change field names after enrichment.

Caching

Responses are cached to improve performance and reduce API calls:

• Default TTL: 300 seconds (5 minutes)
• Cache key: Full URL including substituted values
• Set cache_ttl=0 to disable caching

Rate Limiting

The service applies rate limiting to prevent abuse:

• Default: 60 requests per minute
• Requests exceeding the limit return an error

Error Handling

When enrichment fails for a specific key:

• The original row is preserved unchanged
• A warning is logged
• Processing continues for other rows

Failed data source fetches return an error.

Performance Tips

1. Use caching: Keep default cache_ttl=300 for repeated lookups
2. Limit results first: Filter with where before inputlookup to reduce API calls
3. Batch if possible: Some APIs support batch lookups which are more efficient
4. Use appropriate timeouts: Set realistic timeouts based on API response times

Known Limitations

Commands that work after inputlookup

Most commands work with inputlookup_* fields in post-processing:

Commands that don't work after inputlookup

These commands require database operations and cannot be applied after inputlookup enrichment:

Workaround: Apply these commands before inputlookup:

* | prevalence enrich=true | inputlookup url="https://api.example.com/{src_ip}" key=src_ip
| table src_ip, hash_prevalence, inputlookup_country

Comparison with lookup

Use lookup for static reference data. Use inputlookup for dynamic external APIs.

Related Commands

• lookup - Enrich with internal lookup tables
• eval - Calculate fields using enriched data
• rename - Rename enriched fields
• table - Display enriched fields