• Cost of maintaining AI search with Claude models is high
• Cost of performing evaluation with Claude models is higher
• Need of sovereignty. We cannot keep relying forever on Claude models to do things.

We have some cheap GPUs that we can use to run decent local models:

• NVIDIA L4
• NVIDIA RTX 3090

Both have a VRAM of 24 Gb but different architectures. See comparison at:

• https://advisor.forwardcompute.ai/compare/l4-vs-rtx-3090

Based on the specs, we came up with a selection of models that we can run.

For early comparison of models on NVIDIA L4, see:

• https://llmrun.dev/gpu/l4

• Download and unpack ollama binary from github:

wget https://github.com/ollama/ollama/releases/download/v0.32.6/ollama-linux-amd64.tar.zst
tar -xf ollama-linux-amd64.tar.zst

• Setting up path to ollama

Binary can be accessed from `bin` directory. We run ollama directly by specifying explicitly path or add the path to ollama binary to search paths.

export PATH=$PATH:/path/to/ollama/bin

• Serving local models

Ollama needs to start a server which listens to requests and manage them. We start it with:

OLLAMA_KEEP_ALIVE=-1 ollama serve

• Running local model with ollama

We need to pass model identifier so that ollama pulls the right model. This is usually from the hugging face page of the model. Example:

• https://huggingface.co/unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF

We can get the command to run the model with ollama from the same page. Select ollama in drop-down menu `Use this model` on the page. Command looks like:

ollama run hf.co/unsloth/Qwen3-30B-A3B-Thinking-2507-GGUF:Q4_K_M

Running this command starts a session where user can ask directly questions and get answers generated by the local model.

In our case, we want to use the model in our systems.

We start by setting up our codebase:

git clone https://github.com/genenetwork/gn-ai.git
cd aisearch
guix install poetry -p ~/opt/poetry
python3 -m venv --without-pip .venv
source .venv/bin/activate
poetry install

We need poetry to build the project and its dependencies.

We also need to build corpus and setup chroma DB for the document RAG and memory with:

python scripts/fetch_metadata.py
chroma run --path <path/to/corpus/chroma>
chroma run --path <path/to/corpus/chroma/mem0_chroma> --port 8001

We might also need to start a redis server for answer caching:

guix install redis -p ~/opt/redis
redis-server redis/data/redis.config

Finally, we pass the model identifier in the configuration file having the environment variables used by the codebase.

Configuration file could look like:

MODEL_TYPE=0
MODEL_NAME="hf.co/unsloth/Qwen3-30B-A3B-Thinking-2507-GGUF:Q4_K_M"
PORT=11434

JUDGE_MODEL="anthropic/claude-haiku-4-5"
API_KEY=<api-key>
SPARQL_ENDPOINT="https://rdf.genenetwork.org/sparql"
MEMORY_MODEL=<value>
SECRET_KEY=<value>
AUTH_SERVER_URL=<value>
SEED=<value>
N_ITERATIONS=<value>

CORPUS_PATH=<path/to/corpus>
DB_PATH=<path/to/corpus/chroma>
DATASET_PATH=<path/to/dataset>
OUTPUT_PATH=<path/to/output/evaluation>

For more details, see:

• https://github.com/genenetwork/gn-ai

It's time to run our scripts for

a. search with:

python scripts/rag_search.py <search-query>

b. evaluation with:

python scripts/dspy_evaluate.py