| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
| Name | Name | Last commit date | ||
|---|---|---|---|---|
Batch-process queries against local GGUF models or OpenAI-compatible endpoints.
# Install with uv (recommended)
uv sync
# The CUDA-accelerated llama-cpp-python wheel is automatically selected via pyproject.tomlIf the pre-built CUDA wheels fail with glibc errors, build from source:
# Load CUDA toolkit module (adjust for your HPC)
module load cuda/12.4.1
# Build llama-cpp-python from source with CUDA
CC=gcc CXX=g++ CMAKE_ARGS="-DGGML_CUDA=on" \
uv pip install --reinstall --no-binary llama-cpp-python \
--index-strategy unsafe-best-match 'llama-cpp-python>=0.3.0'# Activate env
source /ssu/gassu/GAU_tools/local_llm_cli/.venv/bin/activate
# Search for models on HuggingFace
llm-cli models search "llama gguf"
# Download a model
llm-cli models download "bartowski/Llama-3.2-1B-Instruct-GGUF" --filename "Llama-3.2-1B-Instruct-Q4_K_M.gguf"
# List local models
llm-cli models list
# Run batch queries
llm-cli query --input queries.json --output results.json --model Llama-3.2-1B-Instruct-Q4_K_MWhat is Python? Explain quantum computing. What is 2+2?
bio_1 What is DNA? bio_2 What is RNA? bio_3 Explain transcription.
["What is Python?", "Explain quantum computing.", "What is 2+2?"]{
"bio_1": "What is DNA?",
"bio_2": "What is RNA?",
"bio_3": "Explain transcription."
}[
{
"request_id": "bio_1",
"query": "What is DNA?",
"response": "DNA is a molecule that carries genetic information...",
"model": "Llama-3.2-1B-Instruct-Q4_K_M"
}
]request_id query response model bio_1 What is DNA? DNA is a molecule... Llama-3.2-1B-Instruct-Q4_K_M
| Backend | Flag | Use case | Throughput |
|---|---|---|---|
| local | --backend local | Quick tests, small batches | Sequential, good per-query |
| local-server | --backend local-server | Large batches (100+ queries) | Continuous batching, max GPU |
| openai | --backend openai | Cloud APIs, remote endpoints | Async parallel |
Loads the model directly and processes queries sequentially. Good for small batches and testing.
Auto-starts a llama.cpp server with continuous batching enabled, then queries it concurrently via the OpenAI-compatible API. This maximizes GPU utilization by sharing KV-cache across parallel requests.
llm-cli query \
--input queries.json \
--output results.json \
--model Llama-3.2-1B-Instruct-Q4_K_M \
--backend local-server \
--concurrency 8Connects to any OpenAI-compatible endpoint (OpenAI, Azure, Ollama, vLLM, LM Studio, etc.).
# First configure
llm-cli config set openai_base_url "https://api.openai.com/v1"
llm-cli config set openai_api_key "sk-..."
llm-cli config set openai_model "gpt-4o-mini"
# Then query
llm-cli query --input queries.json --output results.json --backend openai --model gpt-4o-miniConfiguration is stored in ~/.config/local-llm-cli/config.toml.
# View current config
llm-cli config show
# Set defaults
llm-cli config set default_model "Llama-3.2-1B-Instruct-Q4_K_M"
llm-cli config set default_backend "local-server"
llm-cli config set default_system_prompt "You are a helpful assistant."
llm-cli config set n_ctx 8192| Key | Default | Description |
|---|---|---|
| model_dir | ~/.local/share/local-llm-cli/models | Local model storage (also overridable with --model-dir) |
| knowledge_dir | ~/.local/share/local-llm-cli/knowledge | Knowledge base storage (also overridable with --knowledge-dir) |
| default_model | (not set) | Default model name |
| default_backend | local | local / local-server / openai |
| openai_base_url | (not set) | OpenAI API base URL |
| openai_api_key | (not set) | OpenAI API key |
| openai_model | (not set) | Model name for OpenAI backend |
| default_system_prompt | (not set) | System prompt for all queries |
| temperature | 0.7 | Sampling temperature |
| max_tokens | 512 | Max tokens per response |
| n_gpu_layers | -1 | GPU layers (-1 = all) |
| flash_attn | true | Enable Flash Attention |
| n_batch | 512 | Batch size for prompt eval |
| n_ctx | 4096 | Context window size |
Models are stored in ~/.local/share/local-llm-cli/models by default. You can change this permanently or per-command:
# Set permanently in config
llm-cli config set model_dir /path/to/my/models
# Override per-command with --model-dir
llm-cli models list --model-dir /data/shared_models
llm-cli models download "bartowski/Llama-3.2-1B-Instruct-GGUF" --model-dir /data/shared_models
llm-cli query -i queries.json -m my-model --model-dir /data/shared_modelsYou can augment queries with knowledge from local files (ontologies, reference data, etc.) using Retrieval-Augmented Generation. This requires the knowledge dependency group:
uv sync --group knowledge| Format | Extension | How it's chunked |
|---|---|---|
| OBO ontology | .obo | One chunk per [Term] stanza |
| TSV | .tsv | One chunk per row (header labels included) |
| JSON dict | .json | One chunk per key-value pair |
| JSON list | .json | One chunk per list item |
| Plain text | .txt | Overlapping word-based chunks (500 words, 50 overlap) |
| Markdown | .md | Overlapping word-based chunks (500 words, 50 overlap) |
| Text extracted per page, then overlapping word-based chunks | ||
| HTML | .html, .htm | Visible text extracted (tags stripped), then overlapping word-based chunks |
# From an OBO ontology file
llm-cli knowledge create efo_ontology --input efo.obo
# From a TSV reference file
llm-cli knowledge create gene_annotations --input genes.tsv
# With a custom knowledge directory
llm-cli knowledge create efo_ontology --input efo.obo --knowledge-dir /data/shared_kb# List all knowledge bases
llm-cli knowledge list
# Show details
llm-cli knowledge info efo_ontology
# Remove
llm-cli knowledge remove efo_ontology# Use --knowledge to augment each query with relevant context
llm-cli query \
--input queries.json \
--output results.json \
--model Llama-3.2-3B-Instruct-Q4_K_M \
--backend local-server \
--knowledge efo_ontology \
--top-k 10For each query, the top-k most relevant chunks from the knowledge base are retrieved and prepended to the prompt as reference data. The model then uses this context to answer.
When your knowledge bases contain independent information and any chunk from any KB may be relevant, use --knowledge multiple times. Results from all KBs are merged and re-ranked by similarity score:
llm-cli query \
--input queries.json \
--output results.json \
--model Llama-3.2-3B-Instruct-Q4_K_M \
--backend local-server \
--knowledge efo_ontology \
--knowledge mondo_ontology \
--top-k 10When you need information from one KB to make sense of another (e.g., look up a trait code in a table, then match the description against an ontology), use --knowledge-chain. KBs are queried sequentially — the retrieved text from KB[N] becomes the query for KB[N+1]:
# Trait code → trait description (from traits KB) → ontology term (from efo KB)
llm-cli query \
--input queries.json \
--output results.json \
--model Llama-3.2-3B-Instruct-Q4_K_M \
--backend local-server \
--knowledge-chain traits \
--knowledge-chain efo_ontology \
--top-k 5The augmented prompt shows labeled sections so the model can reason about the chain:
--- Reference data from traits --- T001 | description: high blood pressure --- Reference data from efo_ontology --- ID: EFO:0000537 Name: hypertension Definition: "Elevated blood pressure" ... --- End reference data --- Find the best ontology term for trait code T001
Note: --knowledge and --knowledge-chain are mutually exclusive.
# Set permanently in config
llm-cli config set knowledge_dir /path/to/my/knowledge
# Override per-command
llm-cli knowledge list --knowledge-dir /data/shared_kb
llm-cli query -i queries.json -m my-model --knowledge efo_ontology --knowledge-dir /data/shared_kbThe default settings are tuned for maximum GPU throughput:
For maximum throughput on large batches, use --backend local-server with --concurrency set to match your GPU memory headroom.
llm-cli query Run batch queries against an LLM model llm-cli models list List locally downloaded models llm-cli models search Search HuggingFace for GGUF models llm-cli models download Download a GGUF model from HuggingFace llm-cli models info Show details about a local model llm-cli models remove Remove a locally downloaded model llm-cli config show Show current configuration llm-cli config set Set a configuration value llm-cli knowledge create Create a knowledge base from a source file llm-cli knowledge list List available knowledge bases llm-cli knowledge info Show details about a knowledge base llm-cli knowledge remove Remove a knowledge base
# Install with dev dependencies
uv sync --group dev
# Run tests
uv run pytest tests/| Back | FazBrowse Home | New Git URL |