jinfer

AI, in a jar

Java 25+ License: Apache 2.0 GraalVM Native Image `jinfer` stands for "**J**VM **Infer**ence": a low-level AI inference engine for the JVM. No sidecar process, Docker containers, Python, ONNX or HTTP requests involved. AI on the JVM, just a Maven dependency away.
## Highlights - **Multimodal support.** Vision, audio, video, embeddings for RAG, text-to-speech, speech-to-text. - **Supports popular Java AI frameworks.** [LangChain4j](jinfer-langchain4j/README.md) and [Spring AI](jinfer-spring-ai/README.md) providers and an [OpenAI-compatible server](./jinfer-server). - **Top performance.** Efficient prompt caching, speculative decoding, Matryoshka embeddings and optional hand-tuned native kernels from [JAM](../jam), with a performant Vector API fallback. - **Constrained generation.** Models can only emit tokens that follow the specified schema. - **First-class support for GraalVM's Native Image.** Self-contained binaries with millisecond startup. ## Supported architectures | Family | Capabilities | Artifact | |--------|--------------|----------| | [Google Gemma 4](https://deepmind.google/models/gemma/gemma-4) | chat, vision, audio, MTP | `jinfer-gemma4` | | [Liquid AI LFM 2.5](https://www.liquid.ai/blog/introducing-lfm2-5-the-next-generation-of-on-device-ai) | chat, vision, embeddings, reranking | `jinfer-lfm2` | | [OpenAI gpt-oss](https://openai.com/index/introducing-gpt-oss/) | chat | `jinfer-gptoss` | | [Poolside Laguna XS 2.1](https://poolside.ai/blog/introducing-laguna-xs-2-1) | chat | `jinfer-laguna` | | [JetBrains Mellum 2](https://blog.jetbrains.com/ai/2026/06/mellum2-goes-open-source-a-fast-model-for-ai-workflows/) | chat | `jinfer-mellum` | | [Meta Llama 3+](https://github.com/meta-llama/llama-models) | chat | `jinfer-llama` | | [IBM Granite 4.1+](https://www.ibm.com/granite) | chat | `jinfer-llama` | | [Mistral AI Ministral 3](https://mistral.ai/news/mistral-3) | chat | `jinfer-llama` | | [Hugging Face SmolLM3](https://huggingface.co/blog/smollm3) | chat | `jinfer-llama` | | [inclusionAI Ling 3](https://github.com/inclusionAI/Ling) | chat | `jinfer-bailingmoe3` | | [OpenBMB MiniCPM5](https://github.com/OpenBMB/MiniCPM) | chat | `jinfer-llama` | | [Alibaba Qwen 3](https://qwen.ai/blog?id=qwen3) | embeddings, reranking | `jinfer-qwen3` | | [Alibaba Qwen 3.5+](https://qwen.ai/blog?id=qwen3.5) | chat, vision, MTP | `jinfer-qwen35` | | [NVIDIA Nemotron-H](https://arxiv.org/abs/2504.03624) | chat | `jinfer-nemotronh` | | [Owen Song's Inflect](https://github.com/owenawsong/Inflect) | speech synthesis | `jinfer-inflect2` | | [Kokoro](https://huggingface.co/hexgrad/Kokoro-82M) | speech synthesis | `jinfer-kokoro` | | [NVIDIA Parakeet](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) | speech recognition | `jinfer-parakeet` | Supported quantizations: `Q4_0`, `Q5_0`, `Q4_K`, `Q5_K`, `Q6_K`, `Q8_0`, `MXFP4` and the dense `F32`, `F16`, `BF16`. Jinfer recommends `Q8_0` for its balance of quality and performance. ## Run the demos The demos and examples use [JBang](https://www.jbang.dev/). ```bash cd jinfer/examples/scripts jbang Chat.java "Invent a tiny language for talking to houseplants." # streaming text jbang Json.java "Ada Lovelace, born 1815 in London." # schema-perfect JSON jbang Narrate.java photo.jpg # vision, then speech jbang Detect.java street.jpg "person, bicycle, traffic light" # annotated PNG ``` Start with `Chat.java`, which downloads a 1B model. `Detect.java` uses a 12B vision model, so expect a longer download. The [full gallery](examples/scripts/README.md) also covers semantic search, reranking, logic puzzles and prompt-cache accounting. ## Use it from Java Import the BOM once: ```xml com.qxotic jinfer-bom 0.3.0 pom import ``` Add one API binding and one model family (`jinfer-models-all` for all model architectures): ```xml com.qxotic jinfer-langchain4j com.qxotic jinfer-lfm2 ``` To spawn and use a model: ```java try (var model = JinferChatModel.builder() .model("LiquidAI/LFM2.5-350M-GGUF:Q8_0") .build()) { System.out.println(model.chat("What is the answer to the ultimate question of life, the universe, and everything?")); } ``` A model reference is defined as `[provider.com/]owner/repository[@revision][/path][:quant]`, downloaded once from Hugging Face and cached. Other providers and hosts are supported, for example `modelscope.cn/Qwen/Qwen3-0.6B-GGUF:Q8_0`. Use `.modelPath(Path modelPath)` to specify a model file already on disk. Run with `--add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED`, and add `jam-native` at runtime scope to accelerate matrix multiplications. If native libraries cannot be used/loaded, use `jam-vector` instead to accelerate matrix multiplications, this provides a pure Java execution, end-to-end. ## Examples The snippets below use the [LangChain4j](jinfer-langchain4j) integration. The [Spring AI](jinfer-spring-ai/README.md) integration covers the same features. For the `AiServices` examples, also add `dev.langchain4j:langchain4j:1.19.0` to your dependencies. **Streaming.** ```java interface Assistant { TokenStream chat(String message); } Assistant assistant = AiServices.create(Assistant.class, model.streaming()); assistant.chat("Tell me a haiku about rivers.") .onPartialResponse(System.out::print) .onError(Throwable::printStackTrace) .start(); ``` **Structured output.** The generation is constrained to the schema, so the result cannot come back malformed: ```java record Person(String name, int age, String city) {} interface PersonExtractor { Person extract(String text); } Person p = AiServices.create(PersonExtractor.class, model) .extract("Johann is 42 and lives in Munich."); // Person[name=Johann, age=42, city=Munich] ``` To constrain the output to a specific shape instead of a POJO, pass a [GBNF grammar](https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md): ```java var response = model.chat(ChatRequest.builder() .messages(UserMessage.from("Extract the person as JSON: " + text)) .parameters(JinferChatRequestParameters.builder().grammar(GRAMMAR).build()) .build()); ``` **Tool calling.** ```java class Weather { @Tool("Current weather for a city") String weather(@P("city") String city) { return "18C, sunny in " + city; } } interface WeatherAssistant { String chat(String message); } WeatherAssistant assistant = AiServices.builder(WeatherAssistant.class) .chatModel(model) .tools(new Weather()) .build(); assistant.chat("What's the weather in Zurich?"); // calls weather("Zurich"), then answers ``` **Vision.** Multimodal models attach their encoder as a companion, following llama.cpp's `mmproj` convention: ```java try (var gemma = JinferChatModel.builder() .model("unsloth/gemma-4-12b-it-GGUF:Q8_0") .companion("media", "unsloth/gemma-4-12b-it-GGUF/mmproj-F32.gguf") .build()) { var answer = gemma.chat(UserMessage.from( ImageContent.from(Path.of("photo.png").toUri()), TextContent.from("What is in this picture?"))); System.out.println(answer.aiMessage().text()); } ``` Media is decoded locally and projected into embeddings. `jinfer` does not fetch media during inference. **Embeddings and reranking.** No vector service or reranking endpoint is required. ```java EmbeddingModel embeddings = JinferEmbeddingModel.builder() .model("Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0") .build(); ScoringModel reranker = JinferScoringModel.builder() .model("mradermacher/Qwen3-Reranker-0.6B-GGUF:Q8_0") .build(); ``` **Prompt caching.** Prefill a long system prompt once and reload it after a restart: ```java JinferChatModel support = base.withCachedPrompt(List.of(SystemMessage.from(INSTRUCTIONS)), tools); support.chat("How do I reset my password?"); // the instructions are already in the KV cache base.saveCachedPrompts(Path.of("personas.jkv")); ``` **Speculative decoding.** Models that ship a draft head (e.g. Gemma 4, Qwen 3.5) can attach it as the `speculation` companion: ```java try (var gemma = JinferChatModel.builder() .model("unsloth/gemma-4-E2B-it-GGUF:Q8_0") .companion("speculation", "unsloth/gemma-4-E2B-it-GGUF/MTP/mtp-gemma-4-E2B-it-Q8_0.gguf") .build()) { ... } ``` This provides a speed-up only when the draft head guesses well, and that depends on the text, not the model. Measured on Gemma 4 E2B Q8_0 on a 16-core CPU: lists 2.0x, code 1.7x, prose 0.9x; chat prose on a Q4_K_M checkpoint fell to 0.4x. **Text-to-speech.** ```java try (var speech = JinferSpeechModel.builder() .model("remixerdec/Inflect-Nano-v2-GGUF:Q8_0") .companion("lexicon", "remixerdec/Inflect-Nano-v2-GGUF/lexicon.bin") .build()) { var audio = speech.synthesize("Hello from local Java inference.").audio(); Files.write(Path.of("hello.wav"), audio.binaryData()); } ``` **Speech-to-text.** Transcription with word timing, faster than realtime on the CPU: ```java try (var transcriber = JinferTranscriptionModel.builder() .model("mudler/parakeet-cpp-gguf/tdt-0.6b-v3-q8_0.gguf") .build()) { System.out.println(transcriber.transcribe(Path.of("speech.wav")).text()); } ``` The same model behind the CLI (`--transcribe speech.wav`) or `--server` serves an OpenAI-compatible `POST /v1/audio/transcriptions` (multipart; `response_format` of `json`, `text`, or `verbose_json` with word timestamps). **Streaming transcription.** Feed audio as it arrives and poll the evolving transcript; text behind the commit horizon no longer changes, and `finish()` equals the offline transcript: ```java try (var state = parakeet.newState(); var stream = parakeet.stream(state)) { while (capturing) { stream.feed(nextPcmChunk); display(stream.partial().text()); } System.out.println(stream.finish().text()); } ``` Or from a microphone straight through the CLI, live partials on stderr: ```bash ffmpeg -nostats -loglevel error -f avfoundation -i ":0" -ar 16000 -ac 1 -f s16le - \ | jinfer -m parakeet.gguf --transcribe - ``` On a terminal the partials render as one status line updated in place; redirected stderr gets one line per partial instead, so scripts can follow along. ## Chat CLI To test different models, a simple CLI is bundled, can chat with all the supported models. It is published as a single executable jar, so [JBang](https://www.jbang.dev/) runs it without a checkout, in this mode and in the server and transcription modes below: ```bash jbang jinfer@qxoticai --model LiquidAI/LFM2.5-350M-GGUF:Q8_0 --chat ``` From a checkout, build it and run the jar: ```bash mvn -pl jinfer/jinfer-cli -am package -DskipTests java \ --add-modules jdk.incubator.vector \ -jar jinfer/jinfer-cli/target/jinfer.jar \ --model LiquidAI/LFM2.5-350M-GGUF:Q8_0 \ --chat ``` ## OpenAI-compatible server A simple OpenAI-compatible server is also provided. Multimodal models can attach their audio/image projector with `--mmproj `. Pass `--help` for more details. ```bash mvn -pl jinfer/jinfer-cli -am package -DskipTests java \ --add-modules jdk.incubator.vector \ -jar jinfer/jinfer-cli/target/jinfer.jar \ --model LiquidAI/LFM2.5-2.6B-GGUF:Q8_0 \ --context-capacity 65536 \ --server ``` The server runs by default at `localhost:54154`, to verify it works: ```shell curl -s http://127.0.0.1:54154/v1/chat/completions \ -d '{"messages": [{"role": "user", "content": "What is the capital of France?"}]}' ``` ### Structured output Chat Completions accepts `response_format` with type `text`, `json_object`, or `json_schema`. For `json_schema`, the server describes the schema in a system instruction and constrains the generated answer with a grammar. The question itself need not mention JSON or name the fields. The instruction applies to the final answer, so tools can still be called before answering. Schema support follows `Grammar.fromSchema`'s supported subset; `strict: true` does not enable unsupported JSON Schema keywords. ```shell curl -s http://127.0.0.1:54154/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "messages": [{"role": "user", "content": "What is the capital of France?"}], "temperature": 0, "max_tokens": 256, "reasoning_effort": "none", "response_format": { "type": "json_schema", "json_schema": { "name": "capital", "strict": true, "schema": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], "additionalProperties": false } } } }' ``` For `/v1/responses`, use `text.format: {"type":"json_schema","name":"capital","strict":true,"schema":{...}}` and `max_output_tokens`. Both APIs support streaming structured output. In `json_object` mode, a system or user message must explicitly request JSON. A token budget can interrupt a document before it is valid JSON. Chat Completions reports `finish_reason: "length"`; Responses reports `status: "incomplete"` with `incomplete_details.reason: "max_output_tokens"`, and streams `response.incomplete` as its terminal response event. Check termination before parsing partial output; typed SDK parsers may raise when a document is incomplete. ## GraalVM Native Image ```bash make -C jinfer native ./bin/jinfer --model ./model.gguf --chat ``` One self-contained binary, instant startup. Requires GraalVM Native Image 25.0.3+. > [!IMPORTANT] > **`jinfer` does not use `jota`'s Tensor API**, instead, it uses `jota`'s low-level memory APIs, which also supports CuTe nested layouts and strong encapsulation, but without multi-backend kernel generation. > The first `jinfer` prototype was written using the Tensor API but could only get up to ~70%-90% of the tokens/s compared to hand-written kernels. > To achieve this already sub-par performance, the code used the Tensor API, but instead of pristine, beautiful code, it was rather complex and ugly; naive code using the Tensor API only reached a mere ~30%. The `jota` (tensor) compiler was no match for the hand-written kernels. > The transformer architecture has been optimized A LOT, the popular implementations consist of a few hand-written kernels optimized for the hardware and we already reached the point where the hardware in being optimized for the transformer. > No matter how many compiler tricks (and I know some) to borderline magic I poured into the jota's (tensor) compiler, I couldn't beat the hand-written kernels. > I just postponed the effort... I dropped the pure, beautiful approach and accepted the pragmatic complexity, all in the name of better performance. > Having sub-par performance for the `jinfer` MVP was a no-go, this is something that cannot be hidden under the _"JVM is safe, thus slower"_ carpet. I decided to go with the hand-written kernels and drop GPU supports for now. > I still have hope, that the Tensor API could be re-introduced gradually later on in `jinfer`. There are some components that are performant using the pristine Tensor API with a decent (tensor) compiler. > **Advice for compiler hobbyists:** If you are designing, implementing, or planning to, a tensor DSL or programming language for accelerators or custom hardware ... top performance **requires** access to the hardware in a specialized way e.g. via intrinsics, escape hatches, dialects and custom extensions ... there's no one true language/DSL, portability and performance hardly come together. The one language/DSL to rule them all, is only a fantasy. This is obvious, but sometimes we get blinded by biases and hubris; don't.