nano SIEM
Enrichments

Enrichment Troubleshooting

Enrichment Troubleshooting

This guide helps you diagnose and resolve common issues with nano's enrichment system across all sources: IPInfo Lite, ThreatFox, TOR Exit Nodes, and custom enrichments.

Quick Diagnostics

Check Enrichment Status

  1. Navigate to Marketplace and click on the enrichment
  2. Verify Source Status:
    • Enabled: Source is active
    • Last Sync: Recent timestamp (within 24-48 hours)
    • Record Count: Should show expected counts per source
    • Status: "Success" (not "Failed" or "In Progress")

Expected record counts:

SourceExpected Records
IPInfo Lite~400,000+ IP ranges
ThreatFoxVaries (thousands of IOCs)
TOR Exit Nodes~1,000-2,000 exit nodes

Test IP Lookup

Use the API to test enrichment functionality:

# Test with a known public IP (Google DNS)
curl -H "Authorization: Bearer $TOKEN" \
  http://your-nanosiem/api/enrichments/lookup/8.8.8.8

# Expected response:
{
  "ip": "8.8.8.8",
  "found": true,
  "country": "United States",
  "country_code": "US",
  "asn": "AS15169",
  "as_name": "Google LLC"
}

IPInfo Lite Issues

"Source not configured" Error

Symptoms:

  • Sync fails with "IPinfo Lite URL not configured"
  • No download URL visible in settings

Solutions:

Check URL Configuration:

curl -H "Authorization: Bearer $TOKEN" \
  http://your-nanosiem/api/enrichments/sources

Reconfigure URL:

  1. Visit IPinfo.io free database page
  2. Copy the download link for "IP to Country ASN (CSV)"
  3. Paste into nano Marketplace → IPInfo Lite → Configure
  4. Click "Save Configuration"

Verify URL Format:

✅ Correct: https://ipinfo.io/data/free/country_asn.csv.gz?token=abc123...
❌ Wrong: https://ipinfo.io/data/free/country_asn.csv (missing token)
❌ Wrong: ipinfo.io/data/... (missing https://)

Sync Failures

Symptoms:

  • Status shows "Failed"
  • Last sync timestamp is old
  • Error messages in sync history

Common Causes & Solutions:

Network Connectivity:

# Test from server
curl -I "https://ipinfo.io/data/free/country_asn.csv.gz?token=YOUR_TOKEN"

# Should return: HTTP/2 200
# If not, check firewall/proxy settings

Invalid Token:

  • IPinfo tokens can expire or be revoked
  • Get a fresh download URL from IPinfo.io
  • Update configuration with new URL

Disk Space:

# Check available space
df -h /var/lib/postgresql/

# IPinfo Lite needs ~100MB for download + processing

Database Issues:

# Check database connectivity
psql -h localhost -U nanosiem -d nanosiem -c "SELECT 1;"

Missing Enrichments in Logs

Symptoms:

  • Logs contain IP addresses but no geo fields
  • Search for enriched_src_country returns no results

Diagnostic Steps:

  1. Verify Enrichment is Enabled: Check Marketplace → IPInfo Lite status
  2. Check Private IP Ranges: Private IPs are not enriched:
    • 10.x.x.x, 192.168.x.x, 172.16-31.x.x, 127.x.x.x
  3. Test Manual Lookup: 
    curl -H "Authorization: Bearer $TOKEN" \
  http://your-nanosiem/api/enrichments/lookup/YOUR_LOG_IP
  4. Verify Enrichment Timing: Logs ingested before enrichment was enabled won't have fields

Sync Stuck "In Progress"

Symptoms:

  • Status shows "In Progress" for >30 minutes
  • Cannot trigger new sync

Solutions:

# The scheduler will auto-reset after 30 minutes
# Or manually reset via API:
curl -X POST -H "Authorization: Bearer $TOKEN" \
  http://your-nanosiem/api/enrichments/ipinfo/sync

ThreatFox Issues

No IOC Matches

Symptoms:

  • ThreatFox is enabled and synced but no IOC fields appear in logs

Solutions:

  1. Verify Sync: Check that ThreatFox has synced recently and has records loaded
  2. Check TTL: IOCs older than the configured TTL are removed. Ensure TTL is appropriate (default: 7 days)
  3. Generate Test Traffic: Send test logs with known malicious IPs from ThreatFox's recent feed
  4. Verify Fields: Search for ioc_dest_ip_threat_type != "" to check if any IOCs have matched

Sync Rate Limiting

Symptoms:

  • Sync fails with HTTP 429 or rate limit errors

Solutions:

  • Register for a free API key at threatfox.abuse.ch
  • Enter the API key in ThreatFox settings
  • Increase sync interval to reduce request frequency

Stale IOC Data

Symptoms:

  • IOC data seems outdated or missing recent indicators

Solutions:

  • Reduce sync interval (e.g., every 6 hours instead of daily)
  • Trigger a manual sync
  • Check that auto-sync is enabled

TOR Exit Nodes Issues

No Anonymizer Matches

Symptoms:

  • TOR enrichment is enabled but ioc_src_ip_threat_type = "anonymizer" returns no results

Solutions:

  1. Verify Sync: Check that TOR Exit Nodes has synced and shows ~1,000+ records
  2. Check TTL: TOR exit node IPs change frequently. Ensure TTL is set to 1 day or less
  3. Test with Known Exit Node: Use the Tor Project's relay search to find a current exit node IP and test
  4. Verify Traffic: Ensure your logs actually contain traffic from TOR exit nodes

API Connectivity

Symptoms:

  • Sync fails with connection errors

Solutions:

# Test connectivity to Tor Project API
curl -I "https://onionoo.torproject.org/details?flag=Exit"

# Should return: HTTP/2 200
  • Ensure onionoo.torproject.org is accessible from your server
  • Check firewall rules for outbound HTTPS

Custom Enrichment Issues

Sandbox Errors

Symptoms:

  • Custom enrichment fails with "sandbox error" or "execution timeout"

Solutions:

  1. Check Allowed Domains: Ensure all external domains your code accesses are in the allowed list
  2. Execution Timeout: Code must complete within 60 seconds. Optimize API calls or reduce data volume
  3. Memory Limits: Reduce the amount of data processed in a single run
  4. Check Code Syntax: Use the validation feature to test before deploying

Validation Failures

Symptoms:

  • Enrichment code fails validation

Solutions:

  1. Check Output Format: Ensure your enrich function returns { records: [...] }
  2. IOC Field Placement: IOC fields (threat_type, confidence, malware) must be at the top level, not nested in data
  3. Key Required: Every record must have a key field
  4. Test Locally: Debug your code logic before deploying

Correct:

{ "key": "1.2.3.4", "threat_type": "malware", "confidence": 90, "malware": "Emotet" }

Wrong:

{ "key": "1.2.3.4", "data": { "threat_type": "malware" } }

No Data After Successful Run

Symptoms:

  • Run history shows success but no enrichment data in logs

Solutions:

  1. Check Record Count: Verify the run returned records (check Run History)
  2. Dictionary Refresh: ClickHouse dictionaries refresh every 1-5 minutes. Wait and check again
  3. Verify Key Format: Ensure keys match the format in your logs (e.g., IP addresses must match exactly)
  4. Check IOC Toggle: If creating threat intelligence data, ensure the IOC toggle is enabled

General Issues

Performance Issues

Symptoms:

  • Slow log ingestion
  • High database CPU usage
  • Enrichment timeouts

Solutions:

Check Database Performance:

SELECT query, calls, mean_exec_time
FROM pg_stat_statements
WHERE query LIKE '%enrichment%'
ORDER BY mean_exec_time DESC;

Verify Indexes:

SELECT indexname, tablename
FROM pg_indexes
WHERE tablename = 'ip_enrichments'
  AND indexname LIKE '%network%';

Monitor Connection Pool:

SELECT count(*) as active_connections
FROM pg_stat_activity
WHERE state = 'active';

Advanced Database Diagnostics

Check Enrichment Data:

-- Verify data exists
SELECT COUNT(*) FROM ip_enrichments;
SELECT COUNT(*) FROM enrichment_sources WHERE enabled = true;

-- Test lookup function
SELECT * FROM lookup_ip_enrichment('8.8.8.8');

-- Check table sizes
SELECT
  schemaname,
  tablename,
  pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) as size
FROM pg_tables
WHERE tablename LIKE '%enrichment%';

Log Analysis

# Check nano logs for enrichment errors
tail -f /var/log/nanosiem/app.log | grep -i enrichment

# Common error patterns to look for:
# - "Download error"
# - "Parse error"
# - "Repository error"
# - "Source not configured"
# - "Sandbox error"
# - "Validation failed"

Monitoring & Prevention

Automated Monitoring

Set up alerts for:

  • Sync Failures: Alert if last_sync_status = 'failed'
  • Stale Data: Alert if last_sync_at > 48 hours ago
  • Low Coverage: Alert if enrichment rate drops below threshold

Example monitoring query:

SELECT
  id,
  name,
  enabled,
  last_sync_at,
  last_sync_status,
  CASE
    WHEN last_sync_at < NOW() - INTERVAL '48 hours' THEN 'STALE'
    WHEN last_sync_status = 'failed' THEN 'FAILED'
    WHEN NOT enabled THEN 'DISABLED'
    ELSE 'OK'
  END as health_status
FROM enrichment_sources;

Best Practices

  1. Regular Monitoring: Check enrichment status weekly
  2. Backup Configuration: Save URLs and API keys securely
  3. Performance Tuning: Schedule syncs during low-traffic periods
  4. Security: Protect tokens and API keys; use HTTPS for all downloads
  5. Test After Changes: Always verify enrichment after configuration changes

Getting Help

If issues persist:

  1. Collect Information: Enrichment source status, error messages, system resources
  2. Check Documentation: Review the relevant source setup page
  3. Review Logs: Check system logs for error messages
  4. Community Support: Search existing issues on GitHub

Enrichment Architecture

Previous Page

Data Ingestion

Next Page

Edit this page on GitHub
On this page

On this page

Enrichment Troubleshooting
Quick Diagnostics
Check Enrichment Status
Test IP Lookup
IPInfo Lite Issues
"Source not configured" Error
Sync Failures
Missing Enrichments in Logs
Sync Stuck "In Progress"
ThreatFox Issues
No IOC Matches
Sync Rate Limiting
Stale IOC Data
TOR Exit Nodes Issues
No Anonymizer Matches
API Connectivity
Custom Enrichment Issues
Sandbox Errors
Validation Failures
No Data After Successful Run
General Issues
Performance Issues
Advanced Database Diagnostics
Log Analysis
Monitoring & Prevention
Automated Monitoring
Best Practices
Getting Help