AI

Running a 100B-Plus AI Model on a Laptop: What It Takes and What It Costs

Memory capacity, memory bandwidth and model design decide what a laptop can run, and a single vendor's 145B claim remains unverified.

12 min read
A laptop with a glowing multicolour screen and backlit keys sits on a dark reflective surface lit in purple and blue.
Image by JoshuaWoroniecki from Pixabay

A laptop with 64 GB to 128 GB of memory can load open-weight models of about 100 billion parameters or more, if they are compressed and built so only a small part works on each token. Speed, quality, cost and battery are the trade-offs. Edge0's new 145B claim comes from one source and has not been independently tested.

A laptop can run an open-weight AI model of roughly 100 billion parameters, but only a well-equipped one, and only certain kinds of model. The machine generally needs 64 GB to 128 GB of memory that the graphics hardware can use. The model has to be compressed, and it has to be built so that only a small slice of it works on each word it produces. A typical thin laptop cannot do this.

The question matters because running a model locally changes the trade-offs of AI use: hardware cost and speed on one side, and control over where the model runs on the other. It surged on November 12, 2026, when a vendor called Edge0 announced a 145-billion-parameter open model that it says runs entirely on a personal computer. That claim rests on a single source so far, and this article treats it as such.

What decides whether a model fits

Parameters are the learned numbers inside a model, and more of them usually means a larger memory footprint. How large depends on how many bits each one is stored in. Secondary guides give a simple rule of thumb: file size in gigabytes is roughly the bits per weight times the parameter count, divided by 8. Storing weights in fewer bits is called quantization.

Model design matters as much as compression. In a mixture-of-experts (MoE) model, only some internal 'experts' run for each token, so total and active parameters differ. Total parameters set the memory needed; active parameters set the work per token. OpenAI's gpt-oss-120b, for example, has about 117 billion parameters in total but 5.1 billion active, while gpt-oss-20b has 21 billion in total and 3.6 billion active, according to the vendor's materials.

OpenAI says the MoE weights of both models are natively 4-bit (a format called MXFP4), which lets the 120b model run within 80 GB and the 20b model within 16 GB. Actual files are smaller than that ceiling. Third-party guides report about 61 GB on Hugging Face, 65 GB through Ollama and a 63.4 GB GGUF file for the 120b model, at roughly 4.25 bits per parameter.

Context length adds a second memory bill

The weights are only part of the footprint. During a conversation the software also keeps a running record called the KV cache, which llama.cpp stores at 16-bit precision by default. Flags exist to store its keys and values at lower precision (q8_0 or q4_0). Longer documents therefore cost more memory, which matters for tool-heavy and retrieval-heavy tasks, as AMD notes.

A third-party guide relaying llama.cpp maintainers' figures puts total usage for gpt-oss-120b at 64.9 GB with a 32K-token context and 68.5 GB at 131K. For the 20b model the totals are 15.5 GB and 17.9 GB. The primary source for these numbers was not retrieved, so they are indicative rather than authoritative.

Reported memory for open models, weights and in-use totals (figures as relayed by vendor and third-party guides, 2025–2026)
Model and setting Reported memory Note
gpt-oss-20b, MXFP4 About 13–14 GB (Ollama tag: 14 GB) Weights only
gpt-oss-120b, MXFP4 About 61 GB (Hugging Face), 65 GB (Ollama), 63.4 GB (GGUF) Weights only
gpt-oss-20b in use, 32K / 131K context 15.5 GB / 17.9 GB Second-hand total
gpt-oss-120b in use, 32K / 131K context 64.9 GB / 68.5 GB Second-hand total
Qwen3.5 122B (AMD test) 76 GB Format not stated
Edge0-145B, 'q3k' checkpoint 29 GB 'peak memory' claimed Vendor claim; full weight size not found

Which laptops have the memory

Two laptop designs dominate this conversation because they let the processor and graphics hardware draw on one large memory pool. That avoids the small, separate video memory that limits many conventional laptops. The evidence on speed for both is thinner than the evidence on capacity, and the brief did not retrieve controlled, independent benchmarks of 100B-class models on laptops.

Apple Silicon

According to Wikipedia's Apple M4 page, which cites Apple, the M4 Max MacBook Pro offers up to 128 GB of unified memory at 546 GB/s of bandwidth, or 410 GB/s for the lower-binned chip. The M4 Pro tops out at 273 GB/s. One buyer's guide says about 96 GB of the 128 GB is available to a model under macOS, though that rests on a single secondary source.

Memory on these machines is soldered and cannot be upgraded later, so capacity is a decision made at purchase. A calculator-style website lists gpt-oss-120b at about 67 tokens per second on an M4 Max with 128 GB. That figure is not from a controlled benchmark and should be treated as unverified.

AMD Ryzen AI Max+ 395

AMD says a driver update lets a 128 GB Ryzen AI Max+ 395 machine use 96 GB as graphics memory and run models of up to 128 billion parameters in llama.cpp on Windows, including the 109-billion-parameter Llama 4 Scout. The analyst firm Futurum reports speeds of up to 15 tokens per second. The numbers appear to have been supplied by AMD, so they are vendor-sourced.

AMD's ROCm engineering blog describes loading a 76 GB Qwen3.5 122B model on a 128 GB machine, with 64 GB allocated to the GPU and the remainder handled by mixed CPU and GPU loading. AMD also says gpt-oss-120b runs about 10 times faster on its chip than Llama 3 70B. That compares a sparse MoE model with a dense one, so it says more about model design than about the hardware. A third-party guide adds that most inference software does not use the chip's neural processing unit.

NVIDIA laptops and the desktop yardstick

The brief did not retrieve specifications for NVIDIA laptop GPUs, so their memory limits are not covered here. The only NVIDIA hardware figures available are for a desktop RTX 4090, which a buyer's guide lists at 24 GB and 1,008 GB/s of bandwidth. Edge0's own requirements, discussed below, call for an NVIDIA GPU with at least 16 GB of memory on Windows or Linux.

Why bandwidth and active parameters set the speed

Generating each token requires reading the active weights from memory. One guide says the M4 Max's 546 GB/s is roughly half of the RTX 4090's. Another says the speed ceiling scales as bandwidth divided by model file size. The principle is widely stated and consistent with the numbers, though no primary benchmark was retrieved to confirm it.

Applying that rule gives a rough picture. A dense 70B model at about 4.85 bits per weight is roughly 42 GB, so 546 GB/s divided by 42 GB gives a best case near 13 tokens per second. An MoE model reading about 5 billion active parameters at about 4.25 bits touches roughly 2.7 GB per token, so its theoretical ceiling is in the low hundreds. Real speeds are lower, but the gap explains why MoE models feel quick on laptops while dense models of similar total size do not. These are Jaroha's back-of-envelope calculations, not measured results.

The software that runs these models

llama.cpp is the common engine underneath many local tools. It converts models to the GGUF file format and can quantize them. A mirror of its documentation recommends the Q4_K_M format as a default, and says that using an 'importance matrix' reduces the quality cost compared with naive quantization. The project's own pages were not retrieved, so these details come from a third-party mirror.

Ollama and LM Studio are the easier front ends. Ollama's gpt-oss tags are 14 GB and 65 GB, according to a third-party guide, and AMD recommends running either tool on its Vulkan backend for Ryzen AI Max+ machines. AMD also advises installing models only from trusted sources, a sensible caution given that weights are large files from many publishers.

None of this requires programming in the simplest case. The harder parts are choosing a quantization level, setting context length so the KV cache does not exhaust memory, and, on some AMD systems, configuring how much memory the graphics side may claim. Documentation for Ollama, LM Studio and NVIDIA's tools was not retrieved for this article.

What gets traded away

Quality lost to compression

The strongest evidence is a peer-reviewed study by Kurtic and colleagues, presented at ACL 2025, which ran more than 500,000 evaluations on the Llama-3.1 family. It found 8-bit FP8 effectively lossless and well-tuned INT8 losing 1–3%. INT4 weight-only quantization was competitive with 8-bit. Average benchmark recovery against the 16-bit baseline was 99.75% for 8-bit and 99.36% for the W4A16-INT scheme.

The same paper reports caveats. The smallest models showed somewhat larger drops at INT4, and smaller models varied more on hard tests such as GPQA and MuSR. Averages can hide task-specific losses.

The study also has limits for laptop users. It covers Llama-3.1 and formats common in server software, not the GGUF K-quants most laptop users run, and not MoE models. Evidence below 4 bits is thinner. One secondary source puts the perplexity increase of the Q2_K format at roughly 16 times that of Q4_K_M, and a preprint that is not peer reviewed says sub-4-bit quantization generally degrades quality substantially, though it can work for narrow tasks.

Secondary figures for Q4_K_M itself disagree: a 0.91% perplexity rise on a 7B LLaMA model in one source, about 1–2% in another. Perplexity is also not the same as task accuracy. Blogs claim code and structured reasoning tolerate compression less well than casual chat, but the brief found no primary support for that.

Speed, cost, battery and the cloud

  • Cost and flexibility: 64–128 GB machines are premium hardware, and on Apple Silicon the memory cannot be expanded after purchase. No reliable laptop price data was gathered for this article.
  • Speed: bandwidth caps dense models, and Macs have roughly half the bandwidth of a top desktop GPU, according to one guide.
  • Battery and heat: no source retrieved measures battery life or sustained speed under load. Edge0's roughly 120 W typical draw is a vendor figure.
  • Capability: Edge0 itself says cloud models remain better for memory across devices and for live information.

The Edge0-145B claim, examined

What Edge0 says

Edge0 published Edge0-145B and Edge0-120B, along with an inference framework and deployment tools, on November 12, 2026, according to its own blog. The company calls the 145B model the world's first open-source on-device model of that size. It reports 29 GB of peak memory, more than 1,100 tokens per second for prompt processing, over 80 tokens per second for generation, about 120 W typical power draw and a 128K–256K context window.

The company describes roughly 10 billion active parameters per token. Groups of 64 experts share two large matrices, so each expert keeps only a small set of unique parameters. Edge0 says the model was trained with its on-device form in mind from the start, and that data is pipelined across GPU memory, system memory and SSD, with CPU and GPU work-splitting and speculative decoding. On that basis it says a 48 GB laptop can run it.

Why the headline needs care

Every Edge0 figure traces to the company's launch post, which labels its performance table 'internal testing'. The aggregators, blogs and trackers that cover it re-report that post. The brief found no independent benchmark and no third-party reproduction, and the model was two days old at the time of writing. The retrieved text also did not show which laptop produced the 80 tokens per second or at what context length.

The 29 GB figure also cannot mean a 145B model fits in 29 GB. Simple division gives about 1.6 bits per parameter, while a roughly 3-bit format would imply about 54 GB of weights and 4.25 bits about 77 GB. The most plausible reading is that 29 GB is the resident accelerator or unified memory, with the rest streamed from system memory or SSD, which matches Edge0's own description. That is Jaroha's interpretation, and whether the figure includes the KV cache at long context is unknown.

The requirements are narrower than the word 'laptop' suggests. A DEV Community post summarizing Edge0's repository says Linux and Windows need an NVIDIA GPU with CUDA 13 and at least 16 GB of GPU memory, and suggests a 48 GB MacBook Pro as the Apple option. One summary of Edge0's post says low-compute, low-memory or low-battery devices cannot run the 145B model.

Benchmarks, lineage and licence

Edge0 reports 64.6 on DeepSWE 1.1, 86.52 on Terminal-Bench 2.1 and 90.4 on GPQA-Diamond. In the same summary, Claude-Opus-4.6 scores 91.3 on GPQA-Diamond, so even Edge0's numbers put it near that model, not above it. A company blog from Activepieces says the model outperforms Claude on several benchmarks, which is a looser paraphrase. The benchmark versions and the comparison scores could not be checked against official sources.

The coverage is inconsistent in smaller ways. The LLM Releases tracker lists a Qwen3.8-Flash-Next base model and an 'Edge0 Community Model License 1.0' for the weights, while DEV Community says the code is Apache 2.0 and Byteiota calls the whole release fully Apache 2.0. These may describe different things, code versus weights, but the licence files were not read. Byteiota also says it runs on a single NVIDIA GPU, and a DEV Community post says it was trained for 4-bit execution while the checkpoint name 'q3k' suggests roughly 3 bits.

Edge0 is also not the first claim of this scale. AMD announced in 2025 that Ryzen AI Max+ machines could run models of up to 128 billion parameters locally. What Edge0 adds, if the claims hold, is a larger model with a much smaller resident footprint.

What it means for local AI

The practical shift is real and fairly well documented: sparse MoE designs and 4-bit weights have brought 100B-class models within reach of 64 GB to 128 GB machines. OpenAI's gpt-oss-120b, with 5.1 billion active parameters, is the clearest documented example. Local models still cost more up front, answer more slowly than the best cloud systems on dense designs, and lose some accuracy to compression, though the peer-reviewed evidence says 8-bit and 4-bit losses are small on average.

What to watch is independent measurement. If outside testers reproduce Edge0's memory and speed figures on stated laptops and confirm quality at roughly 3 bits, the claim will matter. Until then, the more reliable guidance is to check a model's total size, its active parameters and the memory left over for context before buying hardware for it.

Frequently asked questions

How much memory does a laptop need to run a 120B-parameter model?

For gpt-oss-120b, reported file sizes are 61–65 GB, and one second-hand figure puts total use at 64.9 GB with a 32K context and 68.5 GB at 131K. Guides disagree on the minimum, from 64 GB-plus to 96 GB-plus, mostly because of how much headroom they leave for context and the operating system. A 128 GB machine is the usual recommendation.

Is a model's total size or its active size more important?

Both matter, for different reasons. Total parameters set how much memory is needed to load the model, while active parameters set how much data is read for each token and so largely how fast it generates. That is why gpt-oss-120b, with 5.1 billion active parameters, can feel quicker than a dense model of similar total size.

How much quality does 4-bit quantization cost?

A large peer-reviewed study on Llama-3.1 models found 4-bit weight-only quantization competitive with 8-bit, with 99.36% average recovery for one scheme. Smaller models and harder tasks showed larger or more variable losses. The study did not cover the GGUF formats most laptop users run or MoE models, and evidence below 4 bits is thinner.

Can any laptop run Edge0-145B?

No. Per reports summarizing Edge0's repository, Windows and Linux need an NVIDIA GPU with CUDA 13 and at least 16 GB of GPU memory, and Edge0 suggests a MacBook Pro with 48 GB. Its performance figures come only from the company's own testing and have not been independently verified.

Is a local model better than a cloud model?

Not across the board. Local runs keep the model on the machine, but they demand costly memory and trade away some speed and accuracy. Edge0's own post acknowledges that cloud models remain better for memory across devices and for live information.

Sources (18)

  1. Edge0-145B: Open-Source On-Device AI with 145B Parameters , Edge0 (2026-11-12)
  2. Introducing gpt-oss , OpenAI
  3. openai/gpt-oss-120b , Hugging Face
  4. 'Give Me BF16 or Give Me Death'? Accuracy-Performance Trade-Offs in LLM Quantization , ACL Anthology (ACL 2025) (2025-01-01)
  5. AMD Ryzen AI Max+ Upgraded: Run up to 128 Billion parameter LLMs on Windows with LM Studio , AMD
  6. AMD Ryzen AI Max+ AI PCs Deliver Exceptional Intelligence Right on Your Desk , AMD
  7. AI Inference on AMD Ryzen AI Max Processor , AMD ROCm Blogs
  8. AMD Upgrades Ryzen AI Max+ With 128B Parameter Model , Futurum Group
  9. Apple M4 , Wikipedia
  10. gpt-oss Local Deployment: Requirements and Setup , vdf.ai
  11. GPT-OSS 120B Hardware Requirements: GPU, Memory, and How to Run It (2026) , Yotta Labs
  12. Quantizing Models - llama.cpp , Mintlify (mirror of llama.cpp documentation)
  13. Optimizing LLMs Using Quantization for Mobile Execution , arXiv (preprint, not peer reviewed)
  14. Edge0-145B: a 145B Model That Runs on Your Laptop , DEV Community
  15. Edge0-145B: Frontier AI Capabilities on Your Own Computer , daily.dev
  16. Edge0-145B: release date , LLM Releases
  17. Edge0-145B: Run a 145B Open-Source Model on Your Laptop , Byteiota
  18. Running Edge0-145B for Frontier AI on Local Hardware , Activepieces

Share this article

Back to list