Self-Host Your Own Search Engine and Ditch the API Bills

Stop paying for search APIs. Run SearXNG on Ubuntu 24, lock it down with ACLs, and wire it into your AI agent in minutes.

---

Why Bother?

Most AI agents — OpenClaw, AutoGPT, custom LangChain setups, whatever you're running — need web search to be useful. The default path is always some API key: Brave Search, Serper, SerpAPI. They're fine until they're not. Rate limits hit, free tiers disappear, and suddenly your morning briefing breaks because you burned through your monthly quota.

SearXNG is a self-hosted meta-search engine. It queries Google, DuckDuckGo, Brave, Startpage, and dozens of others simultaneously, then returns clean JSON. No API key. No rate limits. You control it.

This guide walks you through:

1. Installing SearXNG on Ubuntu 24

2. Locking down port 8080 with ACLs (since we're running it without auth)

3. Wiring it into your AI agent via a simple shell script

4. OpenClaw-specific integration as a concrete example

---

Prerequisites

• Ubuntu 24.04 server (local or VPS)

• A non-root user with `sudo`

• Your AI agent running on the same LAN or a known IP (needed for ACL rules)

• Basic comfort with the terminal

---

Part 1 — Install SearXNG

SearXNG's recommended install method on Ubuntu is via their official script, which sets up a Python virtualenv, nginx, and a systemd service.

1.1 Install dependencies

sudo apt update && sudo apt install -y \

python3-dev python3-babel python3-venv \

uwsgi uwsgi-plugin-python3 \

git build-essential libxslt-dev zlib1g-dev libffi-dev libssl-dev \

shellcheck

1.2 Run the SearXNG install script

cd /tmp

git clone https://github.com/searxng/searxng.git

cd searxng

sudo -H ./utils/searxng.sh install all

This creates a `searxng` system user, installs into `/usr/local/searxng/`, and registers a systemd service.

1.3 Configure it to listen on port 8080

Edit the uwsgi config:

sudo nano /etc/searxng/uwsgi.ini

Make sure the socket line points to HTTP on 8080:

http = :8080

Then restart:

sudo systemctl restart searxng

1.4 Verify it's running

curl "http://localhost:8080/search?q=test&format=json" | head -c 500

You should see a JSON blob with a `results` array. If you get an error about `format=json` being disabled, see the note below.

> Note: SearXNG disables JSON output by default. To enable it, edit `/etc/searxng/settings.yml` and find the `search:` section. Set `formats` to include `json`:

> ```yaml

> search:

> formats:

> - html

> - json

> ```

> Then `sudo systemctl restart searxng`.

---

Part 2 — Lock Down Port 8080 with UFW ACLs

Running SearXNG without authentication is intentional here — it simplifies agent integration. But that means you must restrict who can reach port 8080, or you're running an open proxy for anyone on the internet.

2.1 Enable UFW if not already active

sudo ufw status

sudo ufw enable # if inactive

2.2 Deny public access to port 8080

sudo ufw deny 8080

This blocks all external traffic to port 8080 by default.

2.3 Allow only trusted sources

Allow your AI agent's host. Replace `YOUR_AGENT_IP` with the actual IP:

sudo ufw allow from YOUR_AGENT_IP to any port 8080

If the agent runs on the same machine:

sudo ufw allow from 127.0.0.1 to any port 8080

If it's on the same LAN subnet (e.g., `192.168.1.0/24`):

sudo ufw allow from 192.168.1.0/24 to any port 8080

2.4 Verify the rules

sudo ufw status numbered

You want to see something like:

[ 1] 8080 DENY IN Anywhere

[ 2] 8080 ALLOW IN 192.168.1.0/24

> Rule ordering matters in UFW. More specific ALLOW rules should come before the broad DENY. If your DENY is listed first and applied first, adjust with `sudo ufw insert 1 allow from YOUR_IP to any port 8080`.

---

Part 3 — The Search Helper Script

This is the glue layer. A simple shell script that your AI agent can call with a query and get back results. Agents love this pattern — it's a tool they can invoke without needing to know anything about SearXNG's API.

3.1 Create the script

mkdir -p ~/.agent/scripts

nano ~/.agent/scripts/search.sh

Paste this:

#!/usr/bin/env bash

search.sh — Query local SearXNG and return results

Usage: ./search.sh "your query" [limit]

SEARXNG_URL="http://localhost:8080"

QUERY="${1:-}"

LIMIT="${2:-5}"

if [[ -z "$QUERY" ]]; then

echo "Usage: $0 \"query\" [limit]" >&2

exit 1

fi

ENCODED=$(python3 -c "import urllib.parse; print(urllib.parse.quote('$QUERY'))")

curl -s "${SEARXNG_URL}/search?q=${ENCODED}&format=json&pageno=1" \

| python3 -c "

import json, sys

data = json.load(sys.stdin)

results = data.get('results', [])[:${LIMIT}]

for i, r in enumerate(results, 1):

print(f\"[{i}] {r.get('title', 'No title')}\")

print(f\" URL: {r.get('url', '')}\")

print(f\" {r.get('content', '')[:200]}\")

print()

"

Make it executable:

chmod +x ~/.agent/scripts/search.sh

3.2 Test it

~/.agent/scripts/search.sh "self-hosted privacy tools" 3

Expected output:

[1] Privacy Guides — Tools

URL: https://www.privacyguides.org/tools/

Recommended privacy-respecting tools and services...

[2] Awesome Self-Hosted — GitHub

URL: https://github.com/awesome-selfhosted/awesome-selfhosted

A list of Free Software network services you can host yourself...

[3] ...

---

Part 4 — OpenClaw Integration

> This section is OpenClaw-specific. If you're using a different agent framework, skip to Part 5 for the generic pattern.

OpenClaw uses a tools system where the agent discovers and calls shell scripts. Once your `search.sh` is in place, the integration is straightforward.

4.1 Register the search script location

Place (or symlink) the script where OpenClaw expects tools:

mkdir -p ~/.openclaw/workspace/scripts

cp ~/.agent/scripts/search.sh ~/.openclaw/workspace/scripts/search.sh

4.2 Document it in TOOLS.md

OpenClaw reads `TOOLS.md` to understand what tools are available. Add an entry:

Web Search (SearXNG)

Script: `scripts/search.sh`

Usage: `./search.sh "query" [limit]`

Backend: Local SearXNG at http://localhost:8080

Returns: Numbered list of results with title, URL, and snippet

Examples:

• `./search.sh "bitcoin price" 3`

• `./search.sh "ubuntu 24 kernel update" 5`

No API key required. No rate limits.

4.3 Tell the agent in MEMORY.md

Add a note so OpenClaw remembers this across sessions:

Infrastructure

• **SearXNG**: Running at http://localhost:8080, no auth, port restricted to localhost via UFW

• **Web search**: Use scripts/search.sh — do NOT attempt Brave API (no key configured)

4.4 Cron / briefing usage

If you're using OpenClaw's cron system for a morning briefing, your cron instruction can now include search steps like:

1. Run: ./search.sh "cybersecurity news today" 3

2. Run: ./search.sh "self-hosting news" 3

3. Include top headlines in the briefing

---

Part 5 — Generic Agent Integration Pattern

If you're not using OpenClaw, the same approach works for any agent that can call shell commands or HTTP endpoints.

Shell-calling agents (LangChain, AutoGPT, etc.)

Register `search.sh` as a tool with a description like:

Name: web_search

Description: Search the web using local SearXNG. Input: search query string. Output: numbered list of results with title, URL, and snippet.

Command: /home/youruser/.agent/scripts/search.sh "{query}" 5

HTTP-native agents

If your agent prefers to make HTTP calls directly, just point it at SearXNG's API:

GET http://localhost:8080/search?q={encoded_query}&format=json&pageno=1

Parse `response.results[]` — each item has `title`, `url`, `content`, and `engines` (which sources returned it).

Environment variable pattern

For portability, export the URL so scripts and agents can find it without hardcoding:

In ~/.bashrc or your agent's environment

export SEARXNG_URL="http://localhost:8080"

Then update `search.sh` to use `${SEARXNG_URL:-http://localhost:8080}`.

---

Troubleshooting

| Problem | Likely cause | Fix |

|---|---|---|

| `curl` returns HTML instead of JSON | JSON format disabled | Enable `json` in `settings.yml` → restart |

| Connection refused from agent host | UFW blocking it | Check `ufw status`, add ALLOW rule for agent IP |

| Empty `results` array | SearXNG engines failing | Visit `http://localhost:8080/preferences` and check engine status |

| Slow responses | Too many engines enabled | Disable slow/unreliable engines in `settings.yml` |

| Script returns garbled output | Special chars in query | Ensure URL encoding in the script (the `python3 urllib.parse` line handles this) |

---

What You've Built

• A self-hosted meta-search engine aggregating 10+ sources simultaneously

• No API keys, no rate limits, no third-party dependency for search

• A firewall-restricted endpoint that's only accessible to your agent(s)

• A reusable shell tool your agent can call for any search task

The total monthly cost: whatever your server already costs. The search itself is free.

---

*Tested on Ubuntu 24.04 with SearXNG (latest), UFW 0.36, and OpenClaw. The generic patterns apply to any agent framework that supports tool/function calling.*