Skip to content
Sign in
  • Browse all toolsFilter by category and pricing
  • Browse categoriesWriting, code, image, audio and more
  • Compare toolsPut up to three side by side
  • Read the blogGuides and comparisons

RunNburn

Memory-aware GGUF inference with CPU and GPU offloading and an OpenAI-compatible server.

RunNburn is a memory-aware Rust inference runtime that runs quantized GGUF language models through a CLI, programming interfaces, and an OpenAI-compatible server. RunNburn targets personal, single-owner inference on consumer hardware and is available as an open-source project under the Apache-2.0 license, with source builds and official binaries.

Updated 9 Nov 20266 sections

RunNburn website screenshot

What is RunNburn?

RunNburn is an open-source Rust offloading runtime for quantized GGUF language models that are too large for fast memory. RunNburn targets personal inference on consumer hardware, keeping model weights file-backed rather than expanding them into oversized resident copies. Model architecture determines execution semantics, while available memory and backend capabilities determine placement.

RunNburn focuses on model capacity, correctness, generation latency, continuation reuse, and predictable memory behavior for an active generation. RunNburn does not silently requantize weights or change pretrained routing choices to fit a model into available hardware. Smaller machines may run more slowly under the same product approach.

What RunNburn can do

RunNburn combines direct GGUF loading with memory-aware execution and several ways to access inference. RunNburn coordinates host residency, sparse-expert pages, staging buffers, and accelerator caches around detected or supplied budgets. Supported architecture paths include dense attention, GatedDeltaNet, Mamba-style recurrence, and sparse mixture-of-experts models, although exact coverage varies.

  • RunNburn executes file-backed GGUF weights without generated model sidecars.
  • RunNburn provides streaming interactive chat with retained conversation history.
  • RunNburn exposes OpenAI-compatible Responses and Chat Completions endpoints.
  • RunNburn supports tools and structured output through implemented server endpoints.
  • RunNburn accepts image input for supported Qwen vision models.
  • RunNburn reuses cached sequence state for supported continuations.

How RunNburn works

RunNburn can be built with Rust and Cargo, using a GGUF model supported by the selected runtime path. RunNburn defaults to CPU execution, while accelerated builds can enable CUDA or Metal with CPU fallback retained. Users supply a model path and prompt, omit the prompt for an interactive session, or select the chat or server command.

RunNburn server processes each load a GGUF model and serialize inference through a bounded worker queue. OpenAI clients can connect to the local server address. Non-loopback binding requires a bearer key, and HTTPS requires a reverse proxy because the built-in server provides HTTP rather than TLS.

Who RunNburn is for

RunNburn serves developers and individual owners who want a personal inference server on consumer hardware with constrained RAM or VRAM. RunNburn also exposes a Rust API and Android C ABI for application integration. The stated optimization target is personal generation rather than shared, multi-tenant serving or requests-per-second throughput.

  • RunNburn supports local command-line conversations with quantized models.
  • RunNburn supports personal applications using its OpenAI-compatible API.
  • RunNburn supports Android integration through a shared library and public C header.

RunNburn pricing and access

RunNburn is an open-source project distributed through a public GitHub repository under the Apache-2.0 license. RunNburn provides source-build instructions and official release binaries. Source builds require Rust with Cargo and a supported GGUF model; accelerated and Android builds have additional toolchain requirements described in the repository.

RunNburn strengths and limitations

RunNburn offers direct GGUF execution, configurable memory budgets, and consistent loading policies across its CLI, Rust API, Android C ABI, and server. RunNburn remains under active development, and backend availability does not guarantee that every model operation runs on an accelerator. Unsupported operations may fall back or fail explicitly, depending on the execution contract.

  • RunNburn memory budgets control engine residency rather than total operating-system RSS.
  • RunNburn server compatibility does not cover the complete OpenAI API.
  • RunNburn stores conversations in process-local memory with possible cache eviction.
  • RunNburn treats Vulkan, OpenCL, and MediaTek paths as experimental.
  • RunNburn does not target continuous batching, tenant isolation, or distributed serving.

RunNburn FAQ

What is RunNburn?

RunNburn is an open-source Rust runtime for quantized GGUF language models on memory-constrained hardware. RunNburn keeps weights file-backed, manages host and accelerator memory, and exposes inference through a CLI, Rust API, Android C ABI, and OpenAI-compatible server.

Is RunNburn open source?

RunNburn is an open-source project available from a public GitHub repository under the Apache-2.0 license. RunNburn provides source-build instructions and official release binaries, with Rust and Cargo required for the documented source-build workflow.

Does RunNburn support CUDA, Metal, and Android?

RunNburn supports model-aware NVIDIA CUDA and Apple Metal acceleration alongside default CPU execution. RunNburn also supports Android ARM64 CPU execution through ARM NEON and an Android C ABI. Accelerated operator coverage varies by model, and mobile Vulkan requires explicit opt-in.

Does RunNburn have an OpenAI-compatible API?

RunNburn provides an OpenAI-compatible HTTP server with Responses, Chat Completions, model listing, and in-memory conversation endpoints. RunNburn supports streaming, tools, structured output, and supported multimodal input, but the documented compatibility surface is not a complete implementation of the OpenAI API.

Does RunNburn strictly limit total RAM usage?

RunNburn memory budgets control engine-owned host residency and file-backed sparse-expert page caches, not total operating-system RSS. RunNburn mapped weights, KV cache, temporary buffers, runtime libraries, and other process memory can still contribute to reported memory usage.

From the community

No reviews yet

Be the first to review this tool. What you write helps the next person choose.

More tools like RunNburn

ds4

www.github.com

Open-source local LLM inference for Metal, CUDA and ROCm, with a coding agent and HTTP server.

Free

TurboFieldfare

www.github.com

Run Gemma 4 26B-A4B locally on Apple Silicon Macs using about 2 GB of RAM.

Free

lithos-metal

www.github.com

Open-source LLM inference and local serving for Apple silicon Macs using Metal megakernels and DSpark decoding.

Free

llama-swap

www.github.com

On-demand model switching for local AI servers with OpenAI and Anthropic API compatibility.

Free