TensorSharp

TensorSharp

English | 中文

Native .NET LLM inference engine for GGUF models, including autoregressive LLMs and DiffusionGemma-style text-diffusion models. TensorSharp provides a console application, a web-based chatbot interface, and Ollama/OpenAI-compatible HTTP APIs for programmatic access.

Quick Start

Zero to a streaming reply in about 30 seconds (after the model download).

1. Prerequisites — .NET 10 SDK, git, and (optionally) a GPU toolchain: NVIDIA → CUDA Toolkit 12.x; Apple Silicon → Xcode command-line tools (Metal is built in). Full list in Prerequisites.

2. Clone & build — the native GGML library is compiled automatically on the first build.

git clone https://github.com/zhongkaifu/TensorSharp.git
cd TensorSharp
dotnet build TensorSharp.slnx -c Release

3. Download a model — a small, well-tested starting point is Gemma-4-E4B (Q8_0) from ggml-org/gemma-4-E4B-it-GGUF. More options in Verified Models.

4. Run it — choose the --backend for your hardware (see Pick a Backend):

# One-shot generation
echo "Explain mixture-of-experts in one sentence." > prompt.txt ./TensorSharp.Cli --model gemma-4-E4B-it-Q8_0.gguf --input prompt.txt --backend ggml_metal # macOS
./TensorSharp.Cli --model gemma-4-E4B-it-Q8_0.gguf --input prompt.txt --backend ggml_cuda # Windows/Linux + NVIDIA
./TensorSharp.Cli --model gemma-4-E4B-it-Q8_0.gguf --input prompt.txt --backend cpu # portable / debugging # Interactive chat (REPL)
./TensorSharp.Cli --model gemma-4-E4B-it-Q8_0.gguf -i --backend ggml_metal

Prefer a browser UI plus HTTP APIs? Start the server instead:

./TensorSharp.Server --model gemma-4-E4B-it-Q8_0.gguf --backend ggml_metal
# open http://localhost:5000 — also serves Ollama- and OpenAI-compatible endpoints

The CLI binary lands in TensorSharp.Cli/bin/... and the server in TensorSharp.Server/bin/... after the build. Full options: CLI usage · Server usage.

Pick a Backend

Not sure which backend to use? Start here. Every backend falls back to CPU for any op it does not yet implement, so output stays correct on all of them.

See Compute Backends for the full description of every backend and what each one accelerates.

Verified Models

These architectures are implemented and exercised by the test/benchmark matrix. Pick a quantization that fits your hardware (e.g. Q4_K_M for low memory, Q8_0 for higher quality). More sizes and multimodal projector files are in Model Downloads.

Highlights

• Trades wins with llama.cpp — from pure .NET — head-to-head on identical GGUF files and the same GPU, TensorSharp matches or beats the hand-tuned C++ llama.cpp on the workloads that matter: the Gemma 4 26B-A4B MoE prefills 1.32× faster and lands first tokens 1.30× sooner (geomean; up to 1.70× / 1.65× per scenario), Gemma 4 12B wins or ties every decode scenario (geomean 1.17×), streamed tool-call turns decode up to 2.37× faster, and structured-output (JSON) decode on Gemma 4 E4B streams 7.7× faster (405 vs 52 tok/s). → Head-to-head vs llama.cpp
• Continuous batching & paged KV cache — vLLM-style paged KV pool with block-hash prefix sharing and an iteration-level scheduler, on by default in the server. → deep dive
• MTP / NextN speculative decoding — multi-token-prediction draft heads accelerate solo decode on Qwen 3.6 (NextN block embedded in the trunk GGUF) and Gemma 4 (separate gemma4-assistant draft GGUF). The draft proposes several tokens per step and the trunk verifies them in one batched forward, with the request's own sampler driving both. Opt in with --mtp-spec (+ --mtp-draft-model for Gemma 4). → Speculative decoding
• DiffusionGemma text diffusion — block-wise EntropyBound denoising over a Gemma-4-derived MoE backbone, with CLI generation flags and a Web UI denoising preview stream. → DiffusionGemma card
• Qwen-Image-Edit image editing — prompt + input image → edited image, driving the 60-block MMDiT diffusion transformer with a Qwen-Image VAE and a Qwen2.5-VL-7B text encoder. CUDA-graph-captured whole-DiT forward, FlowMatch-Euler true-CFG denoise, and live denoising previews in the Web UI. → Qwen-Image-Edit card
• Multimodal — image / video / audio inputs (Gemma 4); image inputs for Gemma 3, Qwen 3.5-family, Mistral 3, and Nemotron-H Omni. → Multimodal Support
• Tool calling / function calling — multi-turn tool calls across all three API styles, with architecture-agnostic output parsing. → Tool Calling
• Thinking / reasoning mode — structured chain-of-thought for Qwen 3, Qwen 3.5/3.6-family, Gemma 4, GPT OSS, and Nemotron-H. → Thinking Mode
• Ollama- & OpenAI-compatible APIs — drop-in endpoints for existing tooling, plus a browser chat UI. → HTTP APIs
• Native quantized compute — Q4_K_M / Q8_0 / MXFP4 / IQ2_XXS and more run in matmul without dequantizing to FP32.

Everything below is detailed reference. New here? The five sections above are all you need to get running.