AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: What It Takes To Run A 180B-Parameter MoE Model On 32GB Of RAM on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

TL;DR

A Hugging Face developer proposed an experimental Transformers option that streams selected mixture-of-experts model components from storage into a bounded device cache. The reported OLMoE test reached 34.6 tokens per second, near the stated 38.1-token baseline; no completed benchmark is provided for a 180B model on a 32 GB Mac.

A Hugging Face developer has proposed an experimental way to run mixture-of-experts models whose weights exceed available fast memory, streaming selected experts from storage into a bounded cache, as described in the original analysis. In a reported test on OLMoE-1B-7B, the modified Transformers path generated 34.6 tokens per second, compared with 38.1 for the standard path; the headline goal of running a 180B-parameter model on a 32 GB Mac has not been demonstrated in the source.

The approach targets how MoE models store and access expert weights. Each token is routed through only a subset of a model’s experts. The developer says experts make up more than 90% of the weights, creating the possibility of keeping only the experts needed for current computation in faster memory while retrieving others from SSD. The post describes Strata as supporting placement across GPU memory, system RAM and disk, depending on capacity.

The developer’s first Transformers implementation processed experts one at a time in a Python loop. That version reached 10.1 tokens per second even when experts were cached, compared with about 32 for the vanilla path. The revised design placed expert data in a shared, preallocated device tensor and used a table to map expert IDs to cache slots. This allowed the existing grouped matrix multiplication kernel to process selected experts together. In the reported OLMoE comparison, that version reached 34.6 tokens per second, against 38.1 for standard Transformers.

The work was submitted as an option to Transformers’ from_pretrained flow in pull request #50717. The source also identifies a Hugging Face Space, tarekziade/moe-expert-streaming, containing project code. It says implementation work involved Transformers, a model-specific package and Metal kernels, with AI used in building the code; the author describes their contribution as selecting experiments, examining profiles and deciding what to keep.

At a glance
reportWhen: Experimental implementation reported; p…
The developmentA developer has submitted an experimental SSD-backed expert-streaming option for Transformers, reporting near-baseline speed on OLMoE but not yet on the 180B model and 32 GB Mac that motivate the work.
AI Inference · Briefing

Streaming MoE Experts With Limited Memory

An experimental Transformers change streams selected experts from SSD into a bounded cache. The reported OLMoE test ran close to vanilla speed, but the 180B model on a 32 GB Mac remains untested in the source.

34.6 tokens/s
OLMoE with expert streaming
38.1 tokens/s
Vanilla Transformers comparison
90%+
Model weights attributed to experts

How the cache works

The router selects experts for each token; the experiment fetches those experts into a shared cache.

1
Route each token

Choose a subset of experts for the token.

2
Load selected experts

Stream them from storage into available device cache slots.

3
Compute together

Use Transformers’ grouped matrix multiplication kernel.

What the test shows

On OLMoE-1B-7B, the shared cache kept the reported speed near the standard path.

Earlier approach
  • Python loop ran experts one at a time.
  • Reached 10.1 tokens per second, even with experts cached.
Shared cache
  • Preallocated tensor mapped expert IDs to cache slots.
  • Reached 34.6 tokens per second versus 38.1 baseline.

The larger device test is open

The 180B model on a 32 GB Mac is a motivating target, not a measured result in the supplied source.

Evidence

Promising engineering

The cache design brought the reported small-model test close to vanilla Transformers speed.

Open question

SSD performance

Storage traffic and cache misses could change speed when experts must be fetched repeatedly.

Next evidence

Measure the target setup

Report speed and memory use on the larger model, with storage, cache settings and workload specified.

StatusThe change was submitted as Transformers pull request #50717; the source does not say whether it was merged or released.
The takeaway
Expert streaming looks close to baseline on the reported OLMoE test, while performance on the 180B model and 32 GB Mac remains unproven.

The Memory Trade-Off for Local Inference

If the method works on models that cannot fit in memory, it could give developers another way to run larger MoE systems on a laptop or single GPU. Instead of requiring all expert weights to reside in fast memory at once, the system could keep a bounded set close to the processor and fetch other experts as routing requires them.

The trade-off is that lower memory needs can come with slower generation. SSD access, data movement and software overhead may outweigh the savings if selected experts are not available quickly enough. The OLMoE result is evidence that the cache design can approach the baseline in that reported test, but it does not establish the speed or practicality of streaming experts from disk for a much larger model.

Products Worth Considering

Bose QuietComfort Wireless Noise-Cancelling Headphones, Moonlight Grey

Bose QuietComfort Wireless Noise-Cancelling Headphones, Moonlight Grey

Logitech MX Master 4 Ergonomic Wireless Mouse with Haptics – Graphite

Logitech MX Master 4 Ergonomic Wireless Mouse with Haptics – Graphite

As an affiliate, we earn on qualifying purchases.

How the Expert Cache Was Changed

The author also reports comparisons on Qwen3.6-35B-A3B using an A10G GPU with 24 GB of memory and ample system RAM. In that setup, Strata decoded at 82–88 tokens per second with Q8_0, while llama.cpp with Q8_0 and its fit option reached 63–64 tokens per second. These figures apply to the specified model and hardware; they are not a direct forecast for a Mac or for the experimental Transformers path.

The post characterizes other offloading strategies differently: it says llama.cpp can place experts on the CPU, vLLM reads offloaded weights over PCIe without a hot-expert cache, and Transformers with Accelerate moves an entire layer’s expert bank before processing it. The new experiment aims to fetch only experts selected by the router while preserving Transformers’ grouped computation. These descriptions and performance figures come from the source post, not a matched independent benchmark.

“34.6 tokens per second, compared with 38.1 for vanilla Transformers.”

— The developer, reporting the OLMoE test

The 32 GB Mac Test Is Pending

The source gives no measured result for running the intended 180B MoE model on a 32 GB Mac. It also does not specify the storage device, cache size, prompt conditions or repeated-run results for the OLMoE comparison, limiting what readers can infer about performance across workloads. A test where experts must be fetched from SSD may behave differently from one where the relevant weights are already cached.

The post identifies pull request #50717 but does not say whether it has been accepted, merged or released. It also does not report how the cache performs when the set of routed experts changes quickly, or provide a same-device, same-model comparison against Strata. The reported results should therefore be treated as an implementation experiment, not as evidence that the 180B use case is already practical.

Merge Status and Larger-Model Results

The next immediate development to watch is the status of Transformers pull request #50717 and whether the experimental option becomes available in a published release. The linked Hugging Face Space offers a way to inspect the project code, according to the source.

To establish whether the motivating use case works, a follow-up benchmark would need to report generation speed and memory use on the 180B model and 32 GB Mac, alongside the storage device, cache settings and workload. A direct comparison with Strata under matching conditions would help show how the approaches differ. Until those results are available, the strongest supported finding remains the near-baseline OLMoE result.

Key Questions

Has the 180B model been shown running on a 32 GB Mac?

No. The source presents that setup as a motivating target but reports no completed benchmark for it.

What speed did the experimental Transformers path reach?

The developer reported 34.6 tokens per second on OLMoE-1B-7B, compared with 38.1 for vanilla Transformers. The source does not provide enough test details to generalize the result to other models or devices.

How does expert streaming reduce memory use?

Rather than keeping every expert in fast device memory, the approach aims to load the experts selected for computation into a bounded cache. Other experts can be fetched from storage when needed, which may lower memory requirements but introduces data-transfer costs.

Is the feature already part of a Transformers release?

The source says it was submitted in pull request #50717, but does not state whether it was merged or released. Its availability in a published version remains unclear.

Primary source: Hugging Face · via ThorstenMeyerAI.com

You May Also Like

Google Is Supporting Water Resilience In Chile.

Google says its $1.2 million contribution will help line 2.5 km of an irrigation canal in Chile’s Maipo River Basin and reduce water loss.

The Dawn Of The Intelligence Age: AI’s Transformative Impact

OpenAI’s essay announces the start of the ‘Intelligence Age,’ highlighting potential breakthroughs and risks in AI’s expanding role in society.

We Asked Grok AI Whether Ethereum Is A Buy Under $2,400

A 24/7 Wall St. headline says it asked Grok whether Ethereum is a buy below $2,400, but the answer and supporting analysis are unavailable.

The Real Effort Of AI At Work May Come After It Delivers

Research finds AI can speed up some work but slow other tasks as workers check and correct its output. The verification burden varies by task.