What is Swiftlet?
Swiftlet is an open-source runtime for running Qwen Mixture-of-Experts models locally on Apple devices. Written in Swift and Metal, Swiftlet supports the Qwen3-Next and Qwen3.5/3.6 hybrid model family. Swiftlet serves developers embedding local inference in Apple applications and users running models through command-line tools or a compatible chat interface.
Swiftlet addresses memory limits by keeping a model’s small dense core resident while fetching routed expert weights from storage as needed. The repository reports working, validated generation and includes measurements across Macs and iPhone. Swiftlet is designed as a library first, with additional interfaces for direct local use.
What Swiftlet can do
Swiftlet combines local inference, application integration, and model-container tooling. Swiftlet can maintain chat conversations, stream generated text, and expose a local chat-completions endpoint. The supplied tools also download prepared containers or repack raw MLX checkpoints, allowing developers to prepare models and run inference within the same project.
- Swiftlet streams routed expert weights from SSD into a bounded cache.
- SwiftletCore supports streaming chat deltas, conversation caching, and repetition control.
- Swiftlet provides cooperative cancellation and memory-pressure handling.
- Swiftlet offers command-line chat, generation, and benchmarking.
- Swiftlet serves the OpenAI chat-completions API on loopback.
- Swiftlet repacks MLX checkpoints into .qpack containers.
- Swiftlet compiles Metal shaders at runtime without a build-time Metal toolchain.
How Swiftlet works
Swiftlet keeps attention weights, routers, shared experts, embeddings, and other dense components in memory. Routed experts are packed into fixed-stride blobs, so an expert fetch corresponds to a storage read. A bounded cache retains frequently used experts, while Metal executes the forward pass. The source states that fetching an expert from cache or disk does not change model semantics.
Swiftlet’s Mac quick start begins with cloning the repository and building a release with Swift. Users then download a model container with swiftlet-repack and run chat, generation, or the local server. Developers can instead add SwiftletCore to a macOS or iOS app and use SwiftletSession for conversational inference.
Who Swiftlet is for
Swiftlet is for developers building Apple applications with local model inference and users comfortable running models on their own devices. Swiftlet also supports chat interfaces that accept OpenAI-compatible endpoints. For iPhone users, Priv AI embeds SwiftletCore to download a streamed model and chat on-device without a server.
- Apple app developers can embed local chat through SwiftletCore.
- Command-line users can run local conversations and generation.
- Chat-interface users can connect to Swiftlet’s loopback server.
- Model developers can repack MLX checkpoints and benchmark inference.
- iPhone users can access streamed-model chat through Priv AI.
Swiftlet pricing and source access
Swiftlet is distributed as an open-source project under the Apache-2.0 license. Swiftlet’s repository provides source code, build instructions, tests, and model-container download commands rather than subscription plans. The source does not state commercial pricing for Priv AI, so the runtime’s open-source availability should not be treated as a pricing claim about that app.
Swiftlet strengths and limitations
Swiftlet’s main strength is reducing resident memory requirements by streaming expert weights rather than loading every model weight into RAM. Swiftlet also provides several access paths and validates computation against reference implementations. The repository tests individual layers, incremental decoding, and Metal kernels, and supports byte-level verification of containers against their source checkpoints.
- Swiftlet requires Apple Silicon with macOS 14+ or iOS 17+.
- Swiftlet model containers require substantial free SSD space.
- Swiftlet processes long prompts at decode speed, making large system prompts slow.
- Swiftlet’s 4-bit build can produce repetition artifacts during longer writing tasks.
- Swiftlet’s source cautions that factual recall resembles smaller models because only a subset of parameters is active per token.
- Swiftlet’s Priv AI instructions note that the newest Experimental Models feature was still awaiting App Store review.
Swiftlet FAQ
What is Swiftlet?
Swiftlet is an open-source Swift and Metal runtime for running Qwen Mixture-of-Experts models locally on Apple devices. Swiftlet keeps dense model components in memory and streams routed expert weights from storage, reducing the RAM required to run large model containers.
Is Swiftlet free?
Swiftlet is available as an open-source project under the Apache-2.0 license, with source code and build instructions in its repository. Swiftlet’s page does not list subscription plans or commercial pricing for the separate Priv AI app.
What devices does Swiftlet support?
Swiftlet requires Apple Silicon and macOS 14+ or iOS 17+, along with enough SSD space for a model container. Swiftlet’s repository includes Mac setup instructions and describes running the 35B model on iPhone through Priv AI or a source-built app.
Does Swiftlet have an OpenAI-compatible API?
Swiftlet provides a loopback-only server that implements the OpenAI chat-completions API. Swiftlet can connect to chat interfaces that support OpenAI-compatible endpoints, stream local model output, honor stop sequences, and cancel queued or active generation when a client disconnects.
How can developers use Swiftlet in an app?
Swiftlet provides the SwiftletCore package for macOS and iOS applications. SwiftletSession supports chat with streaming deltas, conversation caching, sampling with repetition control, cooperative cancellation, and memory-pressure handling. SwiftletCore also powers the streamed-model engine embedded in Priv AI.
What are Swiftlet’s main limitations?
Swiftlet requires supported Apple hardware and substantial model storage, and long prompts currently process at decode speed. Swiftlet’s source also notes repetition artifacts in the 4-bit build on longer writing tasks and cautions that factual recall resembles smaller models despite the larger total parameter count.
From the community
No reviews yet
Be the first to review this tool. What you write helps the next person choose.




