Picture an owner who runs a heating and plumbing company out of Lévis. She wants her technicians to dictate a job note on the phone and get a clean work order back, customer name and address included, and she would rather that note never left the phone. Her question is the one this note answers: how much AI fits in a phone, a laptop or a small box in the office, what runs out first, and when the answer is "not enough, use a server". The short version: a small device runs out of memory and memory speed long before it runs out of model quality, and the labs that ship on-device models spend their effort on exactly that.
What limits a model on a small device.
A language model is a large file of numbers, its weights, plus a working area that grows with the length of the conversation, the KV cache. To produce one token of output, a dense model reads every weight once. That gives two hard limits, both about memory. Capacity: the weights, the working area and whatever else the device is doing must fit in RAM at once. Bandwidth: each token means reading the whole model again, so the speed at which memory can be read caps tokens per second. Battery and heat follow from the same bytes, but none of the sources here measures either, so this note puts no numbers on them.
That framing is ours. What the sources show is that the three labs shipping models to phones and laptops each attacked the same two limits, three different ways. Apple shrank the bits: its on-device model is "a compact, approximately 3-billion-parameter model optimized to run efficiently on Apple silicon", compressed "to 2 bits-per-weight using Quantization-Aware-Training (QAT)".[1] Google moved part of the model out of the accelerator's memory: Gemma 3n's raw parameter counts are 5 billion and 8 billion, yet the models run with "as little as 2GB (E2B) and 3GB (E4B) of memory".[2] Microsoft kept Phi-4-Mini at 3.8 billion parameters and spent the effort on training data instead.[3]
Tokens per second follow the memory.
The cleanest public evidence we could read on the bandwidth limit is a GitHub thread rather than a paper: "Performance of llama.cpp on Apple Silicon M-series", opened by the project's maintainer Georgi Gerganov in November 2023 and still collecting results in 2026.[4] Volunteers run one command on their own Macs against the same 7-billion-parameter Llama 2 model in three formats: F16 (16 bits per weight), Q8_0 (8 bits) and Q4_0 (4 bits). The summary table pins one build of the software so chips compare evenly, and reports prompt processing (PP, a 512-token prompt) and text generation (TG, one token at a time, 128 tokens).
Read the TG column against the memory bandwidth column. The same model in the same format generates 14.15 tokens a second on a base M1 (68 GB/s), 21.91 on a base M2 (100 GB/s), 50.74 on an M4 Pro (273 GB/s), 65.95 on an M2 Max (400 GB/s) and 94.27 on an M2 Ultra (800 GB/s).[4] Our arithmetic: from the M1 to the M2 Ultra, bandwidth rises 11.8 times and generation speed 6.7 times, so the relationship is not linear, but the ordering never breaks: more memory speed, more tokens, in every row.
The table also shows why the labs shrink bits. Hold the chip and change the format. On the 20-core M4 Pro, the 7B model generates 17.18 tokens a second at F16, 30.69 at Q8_0 and 50.74 at Q4_0. On the 38-core M2 Max: 24.65, 41.83 and 65.95.[4] Fewer bits per weight means fewer bytes to read per token: a little under three times faster from 16 bits to 4 (our division), not the four times the bit counts suggest, because quantized formats carry scale values too. Prompt processing barely moves across formats (439.78 tokens a second at Q4_0 on that M4 Pro against 464.48 at F16), which fits the maintainer's remark on the thread that at a batch of 512 "the computation is compute bound".[4] Reading a long prompt is limited by arithmetic. Generating the reply is limited by memory.
One caution before anyone orders hardware off these charts. The table is pinned to a 2023 build and a 2023 model so that chips compare evenly; the thread itself reports the 76-core M2 Ultra at Q4_0 rising from 94.27 to 125.21 tokens a second on an August 2026 build with Flash Attention on.[4] The absolute numbers are stale; the shape is the part to keep.
What the labs ship on-device.
Apple. The 2025 tech report describes the on-device model as approximately 3 billion parameters and two design choices aimed at memory. The model is split in two: "Block 1 contained 62.5% of the total transformer layers, while Block 2 contained the remaining 37.5%", and Block 2 reuses Block 1's keys and values instead of computing its own, "reducing KV cache memory usage by 37.5%" and, since prefill can skip that work, cutting time-to-first-token by about the same share.[1] Then the weights are compressed to 2 bits each with quantization-aware training, the embedding table "to 4 bits per weight" and the KV cache "to 8 bits per weight", with low-rank adapters trained afterwards "to recover quality lost during compression".[1] Our arithmetic on the report's figures: 3 billion weights at 2 bits is about 0.75 GB, against about 6 GB at 16 bits. That is the difference between a model that shares a phone with everything else on it and one that does not fit.
What did the 2 bits cost? Table 3: at 16 bits the on-device model scores 67.8 on MMLU and 85.1 on IFEval; the shipped 2-bit version scores 64.4 and 82.3. The server model, compressed to 3.6 bits, goes from 80.0 to 79.2 on MMLU and from 89.1 to 90.2 on IFEval.[1] Our subtraction: 3.4 and 2.8 points lost on device; 0.8 lost and 1.1 gained on the server. Eight times fewer bits, a few points of benchmark.
Google. Gemma 3n takes a different route to the same place. Its two sizes are named E2B and E4B for their effective parameters; the raw counts, the developer guide says, are "5B and 8B respectively". The trick is per-layer embeddings: a large share of the parameters can be "loaded and computed efficiently on the CPU", so only the core transformer weights, about 2 billion for E2B and 4 billion for E4B, need to sit in accelerator memory. Hence the claim that the models run with "as little as 2GB (E2B) and 3GB (E4B) of memory".[2] Google also reports that its own KV-cache sharing gives "a notable 2x improvement on prefill performance compared to Gemma 3 4B".[2] Apple and Google both reached for the same trick, for the same reason.
Microsoft. Phi-4-Mini is "a 3.8-billion-parameter language model" with a 128K context and a 5-trillion-token pre-training corpus weighted toward math and code.[3] The report describes no memory trick; the bet is that a well-trained model under 4 billion parameters is good enough.
| model | size as shipped | how it fits | what the lab reports |
|---|---|---|---|
| Apple AFM On-Device[1] | ≈3B parameters · 2-bit weights, 4-bit embeddings, 8-bit KV cache | quantization-aware training, KV-cache sharing across two blocks (−37.5% KV memory), adapters to recover quality | MMLU 67.8 → 64.4 after compression; beats Qwen-2.5-3B, Gemma-3-4B and Gemma-3n-E4B on MMLU in Apple's runs, trails Qwen-3-4B |
| Google Gemma 3n E2B / E4B[2] | 5B / 8B raw · ≈2B / 4B in accelerator memory | per-layer embeddings computed on the CPU, nested MatFormer sizes (E4B contains E2B), KV-cache sharing | runs in "as little as 2GB (E2B) and 3GB (E4B) of memory"; E2B "up to 2x faster inference" than E4B; E4B "LMArena score over 1300" |
| Microsoft Phi-4-Mini[3] | 3.8B parameters · 128K context | no memory trick described; 5T training tokens weighted to math and code | MMLU 67.3, GSM-8K 88.6, MATH 64.0, HumanEval 74.4 vs Llama-3.1 8B 68.1, 82.4, 47.6, 66.5 and Qwen2.5 7B 72.6, 88.7, 60.4, 75.0; Table 7's average over all its benchmarks: 64.9, 63.9, 67.9 |
Read the numbers carefully.
Every benchmark here was run by the model's own vendor. Apple measured Gemma-3n-E4B at 57.84 on MMLU, ten points below its own model's 67.85.[1] Google's headline for the same model is an LMArena score "over 1300".[2] Different yardsticks, different parties, different settings. Treat every cross-model number as one lab's claim about a competitor.
"Matches models twice its size" has a table behind it, so read the table. Microsoft's abstract says Phi-4-Mini matches models of twice its size on math and coding tasks that need complex reasoning. On Table 7 that holds for MATH and HumanEval against Llama-3.1 8B and roughly holds against Qwen2.5 7B on GSM-8K, while Table 7's average across all its benchmarks lands three points below Qwen (64.9 against 67.9).[3] Both statements are true. Only one is in the abstract.
The compression deltas are on two benchmarks, and the hardware table is a thread. A 3.4-point drop on MMLU says nothing direct about how a 2-bit model handles a dictated French job note with an address in it.[1] And the llama.cpp table is one row per configuration, submitted by whoever owned the machine, no repeats, no variance.[4]
The case that small is enough.
The next question is how much intelligence the job needs. A June 2025 position paper from NVIDIA Research, "Small Language Models are the Future of Agentic AI", revised in September 2026, argues that for most of what an agent does the answer is: less than people assume.[5] Its small model is one that "can fit onto a common consumer electronic device"; the authors "would be comfortable with considering most models below 10bn parameters in size to be SLMs". Everything in the previous section qualifies.
The paper's most quotable line is: "Serving a 7bn SLM is 10–30× cheaper (in latency, energy consumption, and FLOPs) than a 70–175bn LLM".[5] Read it with the footnote in mind: the paper cites four other works for that range rather than measuring it. Its own contribution is a set of estimates on three open-source agents. The authors examine the calls each makes to a language model and judge that "about 60%" of MetaGPT's, "about 40%" of Open Operator's and "about 70%" of Cradle's "could be reliably handled by appropriately specialized SLMs".[5] Those are judgments, not measured substitutions, and the authors say so. The paper calls itself "a value statement" and invites rebuttals.
For a small business the argument lands in a practical place. Most AI calls inside an automation are narrow and repetitive: classify this email, pull five fields from this invoice, turn this dictated note into a work order. The model that fits in the phone may be enough for those, if someone checks that it is.
Sizing the decision: on-device, local box, hosted.
Acceleratech does not build phone models. We help businesses decide where an automation's AI should run, and the sources above give that decision three measurable inputs plus one legal one. The tiers are our framing; the numbers inside them are the sources'.
| input | on-device (phone, laptop) | local box (a small server in the office) | hosted |
|---|---|---|---|
| what fits | roughly 3–4B parameters: Apple's ≈3B compressed to 2 bits, Gemma 3n in 2–3 GB of accelerator memory, Phi-4-Mini at 3.8B uncompressed [1][2][3]. Check the device's RAM in its settings: the weights and the KV cache share it with everything else running. | what the box's memory holds; the thread runs 7B at 16 bits on laptop-class chips [4] | any size; the limit is cost per token |
| tokens per second | 14–24 t/s for a 7B Q4_0 on base chips in the thread [4] | 50–94 t/s on 273–800 GB/s chips, newer builds faster [4] | the provider's; shared |
| context length | the KV cache competes with the weights for memory; Apple quantized it to 8 bits and shares it to fit [1] | longer contexts fit; long prompts are compute bound and cost time on small chips [4] | the provider's problem and your bill |
| where the data goes | stays on the device | stays in the building | leaves, usually outside Québec |
Tokens per second the task needs. Start from who or what consumes the output. A person reading a reply is slow: this site's own read-time convention assumes 240 words a minute, four a second, and a token is a fraction of a word, so even a base laptop chip's 20-odd tokens a second in the thread outruns reading.[4] An agent is not a person. If a step produces 2,000 tokens of structured output and runs 500 times a night, that is a million tokens: about 14 hours at 20 tokens a second, about three at 90 (our arithmetic). The input side is the other half: a 60-page document is prompt processing, compute bound and barely helped by fewer bits.[4] Write down tokens in, tokens out, runs per day and the deadline, and the tier usually picks itself. Every prompt token also adds to the KV cache, which on a small device competes with the weights: shorten the prompt before you upgrade the hardware.
Privacy posture, and why local is the easy case. The Commission d'accès à l'information du Québec's guidance says a privacy impact assessment (an EFVP) is mandatory in three situations, two of them about where AI runs. One is before communicating personal information "à une entité située à l'extérieur du Québec", or entrusting someone outside Québec with collecting, using, communicating or keeping it on the company's behalf. Another is "tout projet lié à un système d'information ou de prestation électronique de services" involving personal information, including "un projet d'acquisition, de développement ou de refonte".[6] A hosted AI tool run from servers outside the province trips both; a model on the technician's phone trips only the second. The CAI adds that a communication outside Québec needs a written agreement and that the enterprise "demeure toujours responsable" for the information it holds.[6]
We read that as a reason to prefer on-device or a local box whenever the job touches names, addresses or anything else personal. It is not legal advice, and the EFVP does not disappear because the box is in your closet: the system is still a project involving personal information. Our notes on where a tool's data goes and on buying hardware for compliance go further; the second argues that owning the box loses on cost and is bought for compliance or not at all. What these sources add is that the compliance case now comes with a model that fits.
What to do on Monday.
Here is how we would size the plumbing company's job note, or any job like it. The steps are ours, not the sources' advice.
- Write the task's shape before you pick a tier.
Four numbers: tokens in (the dictated note, maybe 300), tokens out (the work order, maybe 400), runs a day (one per job, perhaps 30 across the crew) and the deadline (the technician is standing in a basement; ten seconds is fine, two minutes is not). Then the data: names, addresses, phone numbers? For the job note, yes.
→ a job a base laptop chip's 20 tokens a second finishes in about 20 seconds (our arithmetic from the thread) - Measure on the hardware you already own.
The thread's command (
llama-bench, a 512-token prompt, 128 generated tokens) runs on any laptop in the office. The thread covers Apple chips only; llama-bench builds on Windows and Linux too, so the same command gives a number for an office PC, and the bandwidth to compare it against is that machine's memory spec (we found no PC table we could cite). Run it with the model you would actually deploy, not Llama 2 7B. The thread's absolute numbers are stale; yours will not be.→ tokens per second for your machine, your model, this year's software - Test a 3–4B model on your task with a known-good set.
Take twenty real notes, strip the personal details or run the test where the data is allowed to go, and write the correct work order for each by hand. Run the small model and count field-level errors. Apple lost 3.4 MMLU points to 2-bit compression; what you need to know is how many wrong addresses that kind of model produces on your notes. If it fails, check what you fed it before you blame its size: send the note and the fields you need, not the whole customer file.
→ the quality gate, measured on the job, not the benchmark - Pick the tier, name the EFVP, plan for the model to change.
If the small model passes on-device, you have the simplest privacy posture and no per-token bill; the EFVP for the new system still gets done, with the responsable involved from the start. If it fails and the data is personal, the next step is a local box, not a hosted service. Either way, make the model a setting: the 2-bit on-device model of 2025 will not be the one you run in 2027.
→ a decision you can defend, and swap
What this does not claim.
Not that on-device beats hosted on quality. Apple's own table has its ≈3B on-device model at 64.4 MMLU after compression and its server model at 79.2.[1] Small models may be enough for narrow, repetitive calls; nobody here claims they match frontier models on open-ended work.
Not that the bandwidth relationship is a law. One volunteer thread, one model, one 2023 build, Apple chips only. Generation speed rose with bandwidth in every row we used, and the thread does not state the cause. The explanation that each token reads every weight is the standard one and ours; the thread supplies the pattern, not the proof. Battery and heat are not measured by any source here, so they are not measured here.
Not that the vendor benchmarks transfer to your task. MMLU, IFEval, GSM-8K and LMArena measure what they measure. A dictated job note in Québec French with a civic address in it is measured by your twenty-note set and nothing else.
Not legal advice. The CAI's EFVP situations are quoted as the regulator states them.[6] Whether a given deployment falls under them, and what the assessment must contain, is a question for the person responsible for personal information protection and, where needed, counsel.
If one of your workflows puts customer details through an AI step and you want to know whether it could run on devices you already own, describe the step (tokens in and out, how often, what data; no credentials or customer records) through the contact form. We will write back, free, with the tier we would start on and the twenty-case test we would run first. Placing the AI in an automation is part of our workflow automation services.