Make your hardware think, see, listen and speak.
The entire AI pipeline — language, vision and voice — running on your own silicon, from a MacBook to the Pi inside a robot. Pure Rust, one 8 MB download, nothing ever leaves the device.
curl -fsSL https://sapient.openhorizon.so/install | shRun in your terminal — installs to ~/.local/bin.
Runs the models you already use
One pure-Rust engine. A CLI for your terminal, a crate for your codebase, an SDK for your phone.
{}For your terminal
The Sapient CLI & server.
Streaming chat with live Markdown, Whisper transcription, Kokoro speech, a live voice mode — and a drop-in OpenAI-compatible /v1 server. One line to install, one line to run.
Chat · See · Transcribe · Speak · Converse — No daemon · AGPL-3.0 or commercial
Install in 30 secondsFor your codebase
The Rust crate.
Embed inference directly in your app with sapient-generate — the same engine, as a git dependency.
Real sessions, real outputs.
Every line in this terminal is a genuine, unedited output from our benchmark runs — a medical radiograph read by MedGemma, Gemma 3 answering on a bare CPU, a spoken conversation you can interrupt. Nothing staged, nothing cloud.
Reproduce any of it: docs/BENCHMARKS.md
All the engines to run a model anywhere.
Pluggable compute backends paired with hand-written forward engines for each model architecture — the same alias runs on a CPU, an Apple GPU, or a discrete card with the same one-line command.
fig. 01 — one pipeline, every backend
01 · Runs everywhere
StableCPU
Within ~1.3–1.6× of llama.cpp decode after the v0.5.1 int8 kernel ladder — row-interleaved weight repacking, SDOT/SMMLA dot products, Flash-Edge attention. GGUF K-quants load via mmap; 1B-class chat is interactive on a Raspberry Pi 5.
02 · Apple Silicon GPU
StableApple Metal
MlxForwardEngine runs the whole forward pass as one MLX lazy graph — every activation stays on the GPU, one eval() per token. Up to 9.4× faster decode than CPU, with a 21 ms time-to-first-token on 0.5B.
03 · Cross-platform GPU
Stablewgpu
Portable WGSL compute shaders over Vulkan, DX12 & Metal — Intel Arc, AMD Radeon and Nvidia. Q4_K/Q6_K/Q8_0 weights stay quantized on the GPU and dequantize in-shader, so VRAM ≈ the GGUF file size — verified on a Jetson with zero CUDA. The default Hybrid build pairs it with the CPU engine: it times both on your machine at first load and uses the faster one for each phase.
04 · Forward engine
LayerNorm + partial RoPE · SwiGLU path for Phi-3Phi engine
Powers Phi-1 · Phi-1.5 · Phi-2 · Phi-3.5 mini
05 · Forward engine
RMSNorm + RoPE + grouped-query attentionLlama engine
Powers Llama 3.2 · Qwen2.5 · SmolLM2 · TinyLlama · Mistral · DeepSeek-R1 · Mixtral & GLM-4.5-Air (sparse MoE)
06 · Speech-to-text
Pure-Rust log-mel front-end + encoder/decoderWhisper engine
Powers whisper-tiny · base · small · medium · large-v3-turbo — sapient transcribe
07 · Vision-language
SigLIP towers + SmolLM2 / Gemma3 backbonesVision engines
Powers smolvlm-256m · smolvlm-500m · gemma-3-4b · medgemma-4b (medical imaging) — sapient see
08 · Text-to-speech
StyleTTS2 + ISTFTNet · Llama → SNAC codecSpeech engines
Powers kokoro-82m (~2× real-time on CPU, 54 voices) · orpheus-3b — sapient speak
We don't think benchmarks tell the full story.
But we run them head-to-head anyway — against llama.cpp, Ollama and mlx-lm on the same hardware, model and quantization. Sapient runs within 10–20% of llama.cpp on Apple Metal and beats Ollama by 1.5× on Llama-1B — with the lowest time-to-first-token of the three.
- Metal: 90.6 tok/s on Llama-1B — lowest TTFT (52 ms warm), 1.5× Ollama
- CPU within ~1.3–1.6× of llama.cpp (was 1.8–3.8× at v0.5.0)
- 4.2× faster server TTFT than Ollama (14 ms vs 59 ms)
- Raspberry Pi 5: Llama-1B at 11.6 tok/s — 8.9× across two releases
- Jetson AGX Thor (v0.6.4): Qwen-1.5B at 62 tok/s on CPU; Hybrid reads a 450-token prompt in 2.6 s vs 4.7 s
Apple M4 · 4-bit GGUF · Sapient v0.5.3 · same file, same session vs llama.cpp b9860 / Ollama 0.12.6 · 2026-07-09
llama-3.2-1b-q44-bit · Apple M4 · v0.5.3 head-to-head (2026-07-09), same GGUF file, same session vs llama.cpp b9860 — SAPIENT keeps the lowest TTFT and is 1.5× Ollama (whose default 1b tag ships Q8_0).
Where local wins.
Four markets where the cloud can't follow — every command below runs today.
Robotics
A voice in every robot
The streaming loop runs on the Pi inside the chassis: it hears while you speak, answers out loud, and you can interrupt it mid-sentence. No uplink.
$ sapient converse llama-3.2-1b --speakHealthcare
Medical imaging, air-gapped
MedGemma reads radiographs on a laptop that never touches a network — the entire scan-to-narrative path stays inside the clinic.
$ sapient see xray.png --model medgemma-4bProduct teams
A drop-in local backend
An OpenAI-compatible /v1 server with multi-model residency. Point your existing SDK at localhost and ship the private version of your feature.
$ sapient serve --port 8080Field & industry
Edge boxes that just run
Thermal-aware sustained decode on passive hardware, models bigger than RAM via mmap, one self-contained binary. Built for the cabinet, not the rack.
$ sapient chat qwen2.5-1.5bHow it works.
Four commands, one binary. Install, pull, run, update — 30 seconds to a model streaming in your terminal.
01 / Install
One command, any platform.
The endpoint detects your shell and serves the right script.
curl -fsSL https://sapient.openhorizon.so/install | sh02 / Pull
Grab a curated model.
Curated aliases resolve to upstream Hugging Face repos.
sapient pull qwen2.5-0.5b03 / Run
Chat, speak, or serve.
Streaming chat, Whisper transcription, Kokoro speech — or an OpenAI-compatible /v1 server.
sapient chat qwen2.5-0.5b04 / Update
Stay current in place.
The CLI updates itself whenever a new release ships.
sapient updateA CLI, a server, a library, and mobile SDKs. One engine underneath.
CLI
Command line
Streaming chat with live Markdown rendering — plus on-device voice (a streaming converse you can interrupt mid-sentence) and on-device vision: ask questions about any image, including medical imaging with MedGemma.
sapient chat phi-2 sapient see photo.jpg -p "What's here?" sapient converse qwen2.5-1.5b --speak
Server
OpenAI-compatible server
A drop-in /v1 server — chat (text and images, as base64 data URIs), completions and audio transcriptions — with lazy loading, multi-model LRU residency and speculative decoding. Point the OpenAI SDK or LangChain at localhost.
sapient serve --port 8080
curl localhost:8080/v1/chat/completions \
-d '{"model":"smolvlm-256m",
"messages":[{"role":"user","content":[
{"type":"text","text":"What is this?"},
{"type":"image_url","image_url":{
"url":"data:image/png;base64,…"}}]}]}'Crate
Rust library
Embed inference directly with the sapient-generate crate, pulled straight from GitHub — streaming, chat templates and custom sampling.
use sapient_generate::Pipeline;
let p = Pipeline::from_pretrained(
"phi-2").await?;
println!("{}", p.generate(
"The key to good software is").await?);iOS · Android · RN
Mobile SDKs
The same engine inside your app — Swift, Kotlin and React Native, generated from one Rust FFI crate. GPU by default (Metal on iOS, Vulkan on Android) and the engine sheds decode threads as the phone heats up.
// Swift — add the package by URL, then: let session = try LlmSession.load( model: "qwen2.5-0.5b", options: GenerationOptions(maxTokens: 256)) let reply = try session.chat( userMessage: "Hi!")
One command.
Any platform.
The installer pulls the Hybrid build: CPU and GPU engines in one binary. At first load it times both on your machine and uses the faster one for each phase; with no usable GPU it runs on the CPU. Force a choice with SAPIENT_VARIANT=cpu|gpu|hybrid. Raspberry Pi and other non-Jetson ARM boards get the CPU build. Prefer a tarball? Grab a binary below.
curl -fsSL https://sapient.openhorizon.so/install | shInstalls to ~/.local/bin. If sapient isn't found afterward, run: export PATH="$HOME/.local/bin:$PATH"
Direct binary download
A curated model registry.
49 curated aliases (a selection is listed here), grouped into text generation, vision, speech-to-text and text-to-speech. Every alias resolves to an upstream Hugging Face repo and downloads on first use — Safetensors (F16/BF16/F32) or -q4 GGUF.
Gated models require sapient login.
sapient modelsphi-2Defaultphi-1.5phi-1qwen2.5-0.5bSmallestqwen2.5-1.5bqwen2.5-3bqwen2.5-7b-q4qwen2.5-coder-3bCodeqwen2.5-coder-7bCodedeepseek-r1-1.5bReasoningphi-3.5-minismollm2-135msmollm2-360msmollm2-1.7btinyllama-1.1bllama-3.2-1bllama-3.2-3bmistral-7bmixtral-8x7b-q4MoE · 32 GB+glm-4.5-air-q4MoE · 96 GB+whisper-tinySTTwhisper-baseSTTwhisper-smallSTTwhisper-mediumSTTwhisper-large-v3-turboSTTkokoro-82mTTSorpheus-3bTTSsmolvlm-256mVisionsmolvlm-500mVisiongemma-3-270mgemma-3-1bgemma-3-4bVisionmedgemma-4bMedicalOne PR per phase. Ship gradually, never a big bang.
01 / Shipped
- Hybrid build — CPU + GPU engines in one binary; it measures your machine at first load and uses the faster one per phase, and it is the default install (v0.6.4)
- Three builds per platform — Hybrid, CPU only, GPU only — including Jetson-class ARM Linux with Vulkan (v0.6.4)
- Mobile SDKs — Swift, Kotlin & React Native running the engine on-device, GPU by default, engine-level thermal governance (v0.6.0)
- Sparse MoE — Mixtral 47B & GLM-4.5-Air 106B on a Jetson Thor, pure Rust, zero CUDA (v0.5.3)
- Vision over HTTP — OpenAI-compatible image inputs in /v1/chat/completions (v0.5.3)
- Vision — sapient see: SmolVLM, Gemma3, and MedGemma medical imaging, fully offline (v0.5.2)
- Streaming voice loop — STT while you talk, replies you can interrupt; ~2.4 s perceived latency (v0.5.2)
- Fully-quantized wgpu — VRAM ≈ GGUF file size on any GPU vendor (v0.5.0)
- CPU int8 kernel ladder — llama.cpp CPU decode gap cut from 1.8–3.8× to ~1.3–1.6× (v0.5.1)
02 / In progress
- Sub-second voice replies (Kokoro decoder acceleration — RTF already 0.66 → 0.48)
- Server-ARM decode scaling (Jetson Thor: 37 → 62 tok/s in v0.6.4; llama.cpp is at 84)
- Prompt reading shared between CPU and GPU
- Lower peak RAM on Metal (MLX-Q4 embeddings)
- Intel Arc, AMD & Windows benchmark datapoints
03 / Planned
- Python bindings over the same UniFFI layer
- Continuous batching & paged KV in the server
- Faster vision towers (blocked W8A8 GEMM) — MedGemma first token < 15 s
Run a model on your machine in under a minute.
curl -fsSL https://sapient.openhorizon.so/install | shmacOS · Linux · Windows — no Python, no Docker, no CUDA