Perplexity AI open-sourced Lily, a Rust and Metal inference engine built to run a single model, Qwen3.6-35B-A3B, on a single hardware family, Apple Silicon, the company confirmed on September 2, 2026. The release, first detailed by MarkTechPost, positions Lily as the local half of Hybrid Compute, the split-execution mode inside the Perplexity Computer Mac app that divides work between on-device processing and Perplexity’s cloud. Perplexity says Lily beats Apple’s own MLX-LM framework by 1.23x on prefill throughput and 1.35x on decode throughput while running the exact same model, according to benchmarks the company published alongside the code.
The move matters beyond one product feature. It is a rare case of a well-funded AI company shipping a purpose-built, hand-optimized inference runtime instead of leaning on general-purpose frameworks like PyTorch or MLX, then giving the code away for free. For engineers who build local-first AI tools, and for the wider Apple Silicon inference ecosystem that already includes llama.cpp, Ollama, and LM Studio, Lily’s arrival on September 2, 2026 raises a direct question: is narrow, single-model specialization now the fastest way to squeeze more tokens per second out of a MacBook, and does it threaten to fragment the local-inference toolchain that developers currently rely on.
What Perplexity Actually Shipped on September 2, 2026
Lily is not a general-purpose inference server. According to Perplexity’s own blog post on optimizing on-device inference for Apple Silicon, the company built Lily “specifically for Apple silicon and Qwen3.6-35B-A3B, with separate optimizations for prefill and decode.” That single sentence explains most of what makes Lily unusual. Rather than building a runtime that can load any checkpoint in any format, Perplexity’s engineers wrote a system that assumes exactly one model architecture and exactly one chip family, and then hand-tuned every layer of the stack around that assumption.
The architecture, per Perplexity’s own description, runs as a single process with three components working together. A Rust runtime loads the Qwen3.6-35B-A3B checkpoint and manages session state and the generation loop. An OpenAI-compatible chat-completions API accepts requests and streams tokens back to callers, which means existing tools built against the OpenAI API format can talk to Lily with minimal changes. And custom Metal kernels, written by hand rather than generated by a general compiler, execute the Qwen-specific tensor operations directly on the Apple GPU. Perplexity states plainly that neither PyTorch nor MLX sits anywhere in that execution path.
That last detail is the crux of the engineering story. MLX, Apple’s own open-source array framework, is general-purpose by design: it supports a wide range of model architectures and lets developers write flexible Python or Swift code that Apple’s compiler and runtime translate into Metal operations. Lily throws that flexibility away in exchange for speed. Because the team knows in advance exactly which tensor shapes, which mixture-of-experts routing pattern, and which memory layout Qwen3.6-35B-A3B will produce, they could write Metal kernels tuned to that one case rather than kernels that have to handle arbitrary models reasonably well.
The Benchmark Numbers Perplexity Published
Perplexity ran its benchmarks on an M5 Max MacBook Pro, testing across ten prompt lengths and ten decode contexts to compare Lily against MLX-LM running the identical Qwen3.6-35B-A3B checkpoint. The headline results: Lily averaged 1.23x higher prefill throughput, the speed at which the model processes an incoming prompt before it starts generating, and 1.35x higher decode throughput, the speed at which it generates new tokens one at a time. Perplexity says output quality stayed “effectively unchanged” between the two engines despite the speed gain.
Independent coverage from AlphaSignal, which republished and analyzed the benchmark data, adds a quality-parity detail that Perplexity’s own materials touch on more briefly: Lily’s outputs showed roughly 0.04% higher perplexity than MLX-LM’s, alongside 96.35% identical top-token selection between the two engines. In plain terms, Lily is producing almost the same words in almost the same order as MLX-LM, just faster, rather than trading accuracy for speed. That distinction matters to any engineer evaluating whether a specialized runtime is safe to swap in for a general one in a production pipeline.
Community benchmark data collected independently of Perplexity, covering Qwen3.6-35B-A3B running under llama.cpp with multi-token prediction enabled, shows the model hitting approximately 105 tokens per second at short context and around 97 tokens per second at an 8,000-token context, using roughly 45GB of unified memory for a Q8 quantization. Perplexity has not published a direct, official Lily-versus-llama.cpp comparison, so that llama.cpp figure should be read as separate community data on the same model rather than a head-to-head benchmark against Lily specifically.
| Metric | Lily (Perplexity) | MLX-LM (Apple) |
|---|---|---|
| Test hardware | M5 Max MacBook Pro | M5 Max MacBook Pro |
| Model under test | Qwen3.6-35B-A3B | Qwen3.6-35B-A3B |
| Prefill throughput vs. baseline | 1.23x faster | Baseline |
| Decode throughput vs. baseline | 1.35x faster | Baseline |
| Top-token agreement | 96.35% identical to MLX-LM | Reference output |
| Perplexity delta | +0.04% vs. MLX-LM | Reference output |
| Execution path | Rust + hand-written Metal kernels, no PyTorch/MLX | MLX array framework |
| Model scope | Single model, single chip family | General-purpose, many models |
Source: Perplexity’s own benchmark disclosure via its official announcement post and blog, cross-referenced with independent analysis from AlphaSignal.
What Qwen3.6-35B-A3B Is, and Why Perplexity Picked It
Qwen3.6-35B-A3B comes from Alibaba’s Qwen team, according to the model family’s own GitHub repository. The name encodes the basic shape: 35 billion total parameters, built as a mixture-of-experts (MoE) architecture rather than a dense model, joining a crowded field of open MoE releases that already includes rivals covered in our Gemma 4 vs Phi-4 Mini vs Qwen3.5 size comparison. In an MoE design, the full parameter count is not active for every token generated; the model routes each token through a subset of its “expert” sub-networks rather than running every parameter every time, which is part of why MoE models can offer strong capability without the full inference cost of an equivalently sized dense model. Qwen’s own release notes confirm the model’s availability on Hugging Face and ModelScope but do not spell out an exact figure for active parameters per token, so any specific number beyond “a subset of the 35B total” would be unconfirmed.
For Perplexity’s purposes, Qwen3.6-35B-A3B is the model that runs locally inside Hybrid Compute, the split-execution mode in Perplexity Computer. The company’s own blog frames the engineering problem directly: on-device compute risks becoming a bottleneck for complex tasks if the local model can’t keep pace with what users expect from cloud-backed AI tools. Lily exists to remove that bottleneck by making the local half of the split as fast as the hand-tuned engineering effort allows, so that Hybrid Compute’s local leg doesn’t feel like the weak link next to Perplexity’s cloud infrastructure.
Why Rust and Metal, Specifically
The language and framework choice is not incidental. Rust gives Perplexity’s team a systems-level runtime with memory safety guarantees but without a garbage collector, which matters for a process that needs to manage large model weights and session state predictably across long-running generation loops. Metal, Apple’s low-level graphics and compute API, gives direct control over the GPU without going through an intermediate abstraction layer like MLX’s array operations. Perplexity’s blog describes the approach as treating “Apple silicon as a distinct inference platform” and mapping Qwen’s operations directly to its compute and memory architecture, rather than routing through a framework designed to generalize across many possible models.
Coverage from ExplainX, which published its own technical breakdown of the release, frames Lily as being “built from scratch for exactly one purpose,” contrasting it directly with MLX-LM as a general-purpose engine that has to leave performance on the table in exchange for flexibility. That tradeoff, generality versus raw speed on one specific target, is the central design bet behind Lily, and it’s a bet that only makes sense when a company has a single model it cares deeply about running as fast as possible on a single class of hardware it can fully control for.
Licensing and Where to Find the Code
Perplexity has described Lily as open source and made the code publicly available on GitHub under the company’s organization, with coverage from Daily.dev and others confirming the repository is live. Neither Perplexity’s announcement nor the early press coverage from MarkTechPost has spelled out the specific open-source license text (whether MIT, Apache 2.0, or another license) in enough detail to state definitively here, so developers evaluating Lily for commercial or derivative use should check the LICENSE file in the repository directly rather than assume a specific license type. What is confirmed is that the code is public, downloadable, and intended for outside developers to build on, not a source-available or view-only release.
How Lily Stacks Up Against the Existing Apple Silicon Inference Toolchain
Before Lily, developers running large language models locally on a Mac had four realistic options: Apple’s own MLX framework, llama.cpp with its GGUF quantization ecosystem, Ollama as a friendlier wrapper around llama.cpp-style backends, and LM Studio as a GUI-first local model host. Each of those tools is general-purpose by design, built to run dozens or hundreds of different model architectures with varying degrees of Apple Silicon optimization. Lily takes the opposite approach, and that tradeoff shapes exactly what kind of comparison is fair to make.
| Tool | Model scope | Backend | Speed vs. Lily on Qwen3.6-35B-A3B | Best fit |
|---|---|---|---|---|
| Lily (Perplexity) | Qwen3.6-35B-A3B only | Rust + hand-written Metal kernels | Baseline (fastest confirmed for this model) | Perplexity Computer’s Hybrid Compute mode |
| MLX-LM (Apple) | Wide model support | MLX array framework | 1.23x-1.35x slower than Lily on this model | General local experimentation on Apple Silicon |
| llama.cpp | Wide model support via GGUF | C/C++, Metal backend | No official Lily comparison published | Cross-platform local inference, widest quantization support |
| Ollama | Wide model support | llama.cpp-derived backend | No official Lily comparison published | Simple local model management via CLI/API |
| LM Studio | Wide model support | llama.cpp-derived backend | No official Lily comparison published | GUI-first local model hosting for non-CLI users |
The honest caveat, one that Perplexity’s own materials do not paper over, is that the company has published a rigorous comparison against MLX-LM but has not released official head-to-head numbers against llama.cpp, Ollama, or LM Studio. Independent community benchmarks show Qwen3.6-35B-A3B running competently under llama.cpp with multi-token prediction, in the neighborhood of 100 tokens per second on suitable hardware, but those figures come from separate projects rather than a controlled comparison against Lily. Any claim that Lily beats llama.cpp by a specific margin would go beyond what’s currently documented.
The Strategic Angle: Why Give Away a Performance Edge
Open-sourcing Lily is a choice, not an obligation, and it’s worth asking why Perplexity made it. One read, offered by Daily.dev’s coverage of the release, frames Hybrid Compute as a system where user-owned Macs perform meaningful inference work that would otherwise run on Perplexity’s own cloud infrastructure, requiring Apple Silicon, macOS 15, and at least 32GB of unified memory to participate. Under that read, faster local inference isn’t just a user-experience win, it’s a cost lever: every task that a fast, well-optimized local engine can handle on a user’s own hardware is a task Perplexity’s cloud doesn’t have to absorb the compute cost for.
A second, complementary read is straightforward developer goodwill and ecosystem positioning. By publishing both the code and a transparent, reproducible benchmark methodology against MLX-LM, Perplexity invites scrutiny rather than avoiding it, and a credible, verifiable performance claim does more for a company’s engineering reputation than an unverifiable marketing number ever could. Given that Perplexity already competes directly with well-resourced AI labs, including ones now buying up Mac hardware at scale for their own AI training pipelines, for both talent and developer mindshare, a well-received open-source release focused on a genuinely hard systems problem, squeezing more tokens per second out of a mixture-of-experts model on unified memory hardware, is a reasonable way to earn credibility in a crowded field.
Historical Context: The Local-Inference Arms Race on Apple Silicon
Apple Silicon’s unified memory architecture, where the CPU and GPU share a single pool of RAM rather than requiring a separate discrete GPU with its own VRAM, has made high-memory MacBook Pro and Mac Studio configurations unusually attractive for local LLM inference since the M1 generation. That advantage became sharper as consumer GPU VRAM has stayed comparatively constrained while Apple has pushed unified memory configurations up to very high capacities on its Max and Ultra chips, letting large mixture-of-experts models fit entirely in memory on a single machine without the multi-GPU setups dense local inference on Nvidia hardware often requires.
That hardware advantage, most recently extended by the Apple M6 and M5 Ultra debut with its larger unified memory ceilings, created the demand that llama.cpp, MLX, Ollama, and LM Studio have spent the past several years serving, each building progressively better Metal-backed support so Mac owners could run open models locally instead of depending entirely on cloud APIs. Qwen’s own team has leaned into that trend directly, building official mlx-lm and mlx-vlm support into its release process so Qwen models work on Apple Silicon out of the box. Lily is best understood as the next step in that same trajectory: rather than a general framework adding support for one more model, it’s a vendor building a dedicated, production-grade runtime around one model it has decided to bet heavily on, and then choosing to share that work instead of keeping it proprietary.
Market Impact for Developers and the Local-AI Ecosystem
For developers building on Apple Silicon, Lily’s release has a few concrete implications worth weighing now rather than waiting to see how the story plays out. First, the OpenAI-compatible chat-completions API means existing tooling built against that interface format, LangChain integrations, chat UIs, agent frameworks, can likely point at a local Lily instance with minimal rewiring, lowering the switching cost for anyone curious enough to try it. Second, because Lily is narrowly scoped to one model, teams already committed to Qwen3.6-35B-A3B specifically stand to gain the most; teams working with other open models gain nothing directly from Lily itself, though the broader technique, hand-written Metal kernels tuned to a known architecture, is a pattern other model-specific runtimes could copy, in a pricing and performance landscape our DeepSeek V4-Flash vs Gemini 3.7 Flash vs Qwen3.8-Flash-Next comparison tracks in more depth for the broader Qwen lineup.
That fits a pattern our own testing found in local AI reliability comparisons against cloud-hosted chatbots, where on-device models increasingly hold their own on cost and consistency. Third, and more speculatively, if other AI labs follow Perplexity’s lead and start shipping model-specific, hand-optimized runtimes instead of contributing improvements back to general frameworks like MLX or llama.cpp, the Apple Silicon local-inference ecosystem could start to fragment into a collection of narrow, high-performance tools rather than consolidating around a small number of general-purpose ones. That would hand developers a real tradeoff: pick a narrow runtime for maximum speed on one specific model, or a general runtime for flexibility across many models at somewhat lower throughput.
What Perplexity Says, In Its Own Words
Perplexity’s public announcement, posted to its official account, framed the release plainly: “Today we’re open-sourcing Lily, the local inference engine we built for hybrid compute in Perplexity Computer,” the company said in its announcement post. The company added that “Lily is specialized for Qwen3.6-35B-A3B on Apple silicon, built so on-device compute doesn’t bottleneck Computer tasks.”
On its own blog, Perplexity described the engine’s scope in similar terms: “Lily, our lightweight local inference engine, is built specifically for Apple silicon and Qwen3.6-35B-A3B, with separate optimizations for prefill and decode,” the company wrote in its post on optimizing on-device inference for Apple Silicon. The company further noted that “Lily implements this specialization end to end in a single process,” and detailed the architecture directly: “A Rust runtime loads the model checkpoint and manages the session state and generation loop, an OpenAI-compatible chat-completions API accepts requests and streams tokens, and custom Metal kernels execute Qwen-specific operations.”
Predictions: Where This Goes From Here
A handful of near-term outcomes look reasonably likely based on how the release has landed so far, though all of them remain forecasts rather than confirmed roadmap items from Perplexity.
- Expect community forks and quantized variants of Lily’s approach to appear within weeks, mirroring how quickly the llama.cpp and MLX communities historically produce derivative optimizations once a technique proves viable in public benchmarks.
- Expect renewed pressure on Apple’s own MLX team to close the gap Perplexity has now publicly documented, since a 1.23x-1.35x throughput deficit against a hand-tuned competitor on a popular open model is the kind of gap framework maintainers tend to respond to.
- Expect other AI companies with consumer-facing Mac apps to evaluate similar model-specific runtimes for their own flagship local models, particularly any company running a Hybrid Compute-style split between local and cloud execution.
- Expect scrutiny of Lily’s exact open-source license terms to intensify as commercial developers evaluate whether they can build products on top of it, given that neither Perplexity’s announcement nor early press coverage has spelled out the license explicitly.
- Expect head-to-head, independently run benchmarks against llama.cpp, Ollama, and LM Studio to surface from the developer community within the coming weeks, filling the gap left by Perplexity’s MLX-only comparison.
The Broader On-Device AI Trend This Fits Into
Lily’s release lands squarely inside a broader 2025-2026 pattern of AI companies pushing more inference work onto user-owned devices, driven by a mix of latency, privacy, and infrastructure-cost incentives. High-memory Apple Silicon Macs have become a favored target for that push specifically because unified memory lets large mixture-of-experts models fit locally without the multi-GPU setups dense inference on Nvidia hardware typically demands. Community projects packaging quantized Qwen3.6-35B-A3B configurations for local coding agents, alongside official mlx-lm and mlx-vlm support from the Qwen team itself, show that developer appetite for running serious open models locally on Mac hardware was already substantial before Lily arrived; Perplexity’s release adds a genuinely faster, if narrower, option to that existing toolkit rather than creating the category from scratch.
Whether Lily becomes a template other companies copy or stays a one-off tool scoped tightly to Perplexity Computer likely depends on how much appetite the wider developer community has for maintaining narrow, model-specific runtimes versus contributing to the general frameworks that already serve dozens of models reasonably well. Both paths have precedent in the broader AI tooling world, and Perplexity’s own bet, judging by the engineering effort visible in Lily’s design, is that specialization wins when the target model and hardware are both known and fixed in advance, a bet worth watching alongside the rest of the AI chip and silicon landscape heading into 2027.
Frequently Asked Questions
What is Lily, exactly?
Lily is a local inference engine, written in Rust with hand-written Metal kernels, that Perplexity AI open-sourced on September 2, 2026. It’s built specifically to run Qwen3.6-35B-A3B on Apple Silicon and powers the local half of Hybrid Compute inside Perplexity Computer.
How much faster is Lily than MLX-LM?
Perplexity’s own benchmarks, run on an M5 Max MacBook Pro across ten prompt lengths and ten decode contexts, show Lily averaging 1.23x higher prefill throughput and 1.35x higher decode throughput than MLX-LM on the same Qwen3.6-35B-A3B model.
Does Lily sacrifice output quality for speed?
According to Perplexity and independent analysis from AlphaSignal, no. Reported figures show roughly 0.04% higher perplexity and 96.35% identical top-token selection compared to MLX-LM, indicating near-identical output quality alongside the throughput gain.
What is Qwen3.6-35B-A3B?
It’s a 35-billion-parameter mixture-of-experts language model from Alibaba’s Qwen team, available on Hugging Face and ModelScope. As an MoE model, it activates only a subset of its total parameters per token rather than running the full parameter count on every forward pass.
Is Lily free and open source?
Yes, Perplexity has made the Lily code publicly available on GitHub. The exact open-source license terms had not been explicitly detailed in early coverage as of this writing, so developers should check the repository’s license file directly before building commercial products on top of it.
Can Lily run models other than Qwen3.6-35B-A3B?
No. Lily is deliberately narrow by design, built for one model on one hardware family. Developers who need to run other open models locally on Apple Silicon should continue using general-purpose tools like MLX-LM, llama.cpp, Ollama, or LM Studio.
What hardware do I need to run Lily?
Lily targets Apple Silicon Macs. Related Hybrid Compute documentation from Perplexity references a requirement of macOS 15 and at least 32GB of unified memory to run local models like Qwen3.6-35B-A3B at usable speeds, though exact minimum requirements for Lily specifically should be confirmed against Perplexity’s own repository documentation.
Has anyone benchmarked Lily against llama.cpp or Ollama?
Not officially. Perplexity has only published a direct comparison against MLX-LM. Community benchmarks of Qwen3.6-35B-A3B running under llama.cpp exist independently, showing roughly 100 tokens per second on suitable Apple Silicon hardware, but no controlled, apples-to-apples comparison against Lily has been published as of this writing.
