NPU Explained: What a Neural Processing Unit Is, How It Differs From a CPU and GPU

What an NPU is

An NPU (Neural Processing Unit) is a dedicated processor that runs the low-precision matrix and tensor arithmetic behind neural network inference, at a fraction of the energy a CPU or GPU would use for the same work.

That is the whole definition. Everything else is detail about how it does that and what it refuses to do.

What it is for. Running an already-trained AI model on the device that is in front of you: speech recognition, camera processing, background blur, face unlock, on-device language models, semantic search over your own files. These workloads are sustained, highly parallel, and tolerant of reduced numerical precision (INT4, INT8, FP8, FP16). They are exactly what an NPU is laid out for.

What it is not. It is not a small GPU. It has no graphics pipeline, no shader programming model, and no general-purpose compute language comparable to CUDA. It is also not a training chip. Model training still happens on GPUs and datacenter accelerators. An NPU is an inference engine.

How it differs from a CPU. A CPU is built for low-latency execution of branchy, control-heavy code: one or a few instruction streams, deep caches, out-of-order execution, aggressive branch prediction. It can do tensor maths, and modern cores have vector extensions for it, but per watt it is the worst of the three at sustained inference.

How it differs from a GPU. A GPU is built for parallel throughput across thousands of programmable cores, which is why training runs there. It keeps the flexibility of a programmable pipeline, and it pays for that in power. NVIDIA rates the GeForce RTX 5090 at 575 W total graphics power. An NPU gives up programmability to get the energy cost of inference down far enough that a feature can stay on all day on battery.

Looking for the broader picture rather than the NPU specifically? CPUs, GPUs, NPUs, and TPUs compared is the four-way explainer, including where Google's datacenter TPU fits. This page is the NPU reference: architecture, the TOPS metric, the vendor landscape, and how to actually target one.

NPU vs CPU vs GPU, in one pass

The three are complementary. A single video call on a modern laptop uses all of them at once: the NPU runs background blur and noise suppression, the GPU runs the video codec, the CPU runs the application and the network stack.

CPU. Few powerful cores. Best at serial work, branching, and switching between many different jobs. Worst efficiency for sustained tensor maths. Intel specifies the Core Ultra X9 388H at a 25 W base power and 80 W maximum turbo power for the whole package.

GPU. Thousands of lightweight cores optimised for throughput. Dense matrix maths is its home ground, which is why every frontier model is trained on GPUs. Modern GPUs also carry dedicated matrix hardware: NVIDIA rates a single Rubin GPU at 50 petaflops of NVFP4 compute. The cost is power and, in a laptop, thermal headroom.

NPU. Fixed-function and semi-fixed pipelines for tensor operations, fed by direct memory access into the compute fabric. No occupancy to manage, no graphics work to schedule around. The result is inference that a device can run continuously without wrecking battery life. Qualcomm claims its Snapdragon 8 Elite Gen 5 Hexagon NPU is 37% faster at 16% lower power than the previous generation, which is the shape of the improvement curve here: performance and efficiency together, generation over generation.

The practical division of labour:

One caveat worth stating plainly: vendors publish CPU and GPU power figures but generally do not publish a standalone wattage for the NPU block. Claims that an NPU is a specific multiple more efficient than a GPU are usually derived from benchmarks rather than from published specifications, so treat them as directional.

What TOPS actually measures

TOPS means trillions of operations per second. It counts how many multiply-accumulate operations the accelerator can issue per second at a stated numerical precision. That precision is almost always INT8. Intel, for example, lists the NPU on its Core Ultra X9 388H as 50 peak TOPS at Int8.

Three things make cross-vendor TOPS comparisons unreliable.

1. Precision changes the number. A rating at INT8 is not the same rating at FP16, and INT4 raises it again. A quoted TOPS figure without a precision is not a specification.
2. Peak is not sustained. Peak TOPS assumes the matrix engine is fully utilised. Real models do not keep it fed continuously.
3. TOPS ignores memory bandwidth. Language model inference is frequently memory-bound rather than compute-bound. Bandwidth is the number that often decides real speed, and it is why Apple advertises unified memory bandwidth alongside its Neural Engine: 153GB/s on M5 and up to 170GB/s on M6.

Vendors also increasingly quote a "platform" TOPS number that sums NPU, GPU, and CPU AI throughput. That is a different quantity from the NPU rating, and the two get mixed together in coverage constantly. Check which one a spec sheet means before comparing two machines.

The 40 TOPS Copilot+ line

Microsoft requires an NPU capable of more than 40 trillion operations per second for a Windows machine to carry the Copilot+ PC label, alongside 16 GB of RAM and 256 GB of storage.

It is a go-to-market gate, not a capability cliff. Below it, certain on-device Windows AI features do not run. The number was set to match the first wave of qualifying silicon, not to mark the point where on-device inference becomes possible. Plenty of useful AI runs on NPUs rated well under 40 TOPS, and on phones that have no TOPS rating published at all.

The NPU silicon shipping in September 2026

Consumer NPU ratings have roughly doubled since the Copilot+ threshold was set, and the leading Windows parts now sit at twice the floor.

Four things in that table are worth reading carefully.

Qualcomm doubled the Windows bar. Snapdragon X2 Elite and X2 Elite Extreme carry an 80 TOPS Hexagon NPU, announced in September 2025 and shipping in PCs through 2026. Qualcomm extended the X2 line downward with Snapdragon X2 Plus in January 2026.

Intel moved the NPU to a new process. Core Ultra series 3, codenamed Panther Lake, launched on 5 January 2026 as the first client platform on Intel 18A, with up to 50 NPU TOPS.

AMD brought Copilot+ to the desktop socket. The Ryzen AI 400 series, announced at CES 2026, reaches up to 60 NPU TOPS on mobile parts and is the first socketed desktop processor line to meet the Copilot+ NPU requirement.

Apple stopped publishing TOPS, and moved AI compute into the GPU. The last Apple Neural Engine figure the company published was 38 trillion operations per second on M4 in May 2024. Since M5, Apple has put a Neural Accelerator in each GPU core and now quotes GPU AI compute instead: M5 Ultra puts a Neural Accelerator in up to 80 GPU cores with 1.2TB/s of unified memory bandwidth. The Neural Engine is still there and still growing, but Apple's story is now that AI runs across the whole chip.

That last point is the most consequential trend on this page. The clean separation between "the NPU block" and "the GPU" is eroding. Apple puts matrix hardware in GPU cores and neural accelerators in CPU cores. Intel quotes a platform AI number that spans three engines. The question is shifting from "how big is the NPU" to "how much of this chip can do tensor maths, and can the runtime reach it."

What the NPU actually accelerates today

Shipping on NPU across platforms.

• Camera pipelines: HDR fusion, denoise, scene and subject segmentation, video effects.
• On-device speech recognition, dictation, and live captions.
• Real-time meeting effects: background blur, noise suppression, framing, eye contact correction.
• Biometric authentication.
• Photo indexing and semantic search over local content.

Moving onto the NPU now.

• Small on-device language models in the roughly 1B to 8B parameter range, for summarisation, rewriting, and assistant features. Google says Tensor G6 runs its latest Gemini Nano model up to 3.5 times faster while using up to 3.5 times less energy than the previous generation.
• Live speech-to-speech translation, which Google ships on Pixel 11 as automatic dubbing of media into the user's language.
• Document understanding, local retrieval embeddings, and automatic alt text.

Still on the GPU.

• Large models at usable speed.
• Image and video generation at native resolution.
• Most creative-application neural features.
• Anything that benefits from large batch sizes.

The boundary moves toward the NPU each generation, and memory bandwidth is what moves it. Compute has run ahead of bandwidth on device for three generations now.

Targeting an NPU as a developer

There is still no NPU equivalent of CUDA: no single language, no portable kernel model, no shared toolchain across vendors. Access runs through runtimes.

On Windows, Microsoft has moved its recommendation from DirectML to Windows ML, which sits on ONNX Runtime and selects an execution provider automatically. It detects the hardware, downloads the right provider (QNN for Qualcomm NPUs, OpenVINO for Intel), runs inference there, and falls back to GPU or CPU when the preferred provider is unavailable. Apple platforms use Core ML and MLX. Android and cross-platform mobile work goes through vendor SDKs, LiteRT, or ONNX Runtime.

The workflow that holds across all of them:

1. Train on GPU in the cloud or on a workstation, and export to ONNX or Core ML.
2. Quantise to INT8 or INT4 with the platform's tooling. Microsoft points at Olive for ONNX Runtime targets.
3. Validate accuracy retention on a dataset that represents your actual traffic.
4. Deploy, and let the runtime place operations.

Quantisation is the step that decides whether this works. NPUs are much faster at INT8 than at FP16, and faster again at INT4, but each precision drop costs accuracy. The engineering judgement is picking the tier that keeps the accuracy your task actually needs. Microsoft is explicit that many NPUs only support integer maths in low-bit formats, so conversion is not optional.

For measurement, Windows now surfaces NPU utilisation in Task Manager, and the Windows Performance Recorder ships a Neural Processing profile that records NPU activity and the call stacks submitting work.

Where this goes next

Memory bandwidth is the constraint to watch. Apple's own figures show the trend: 153GB/s on M5, up to 170GB/s on M6, 1.2TB/s on M5 Ultra. Apple also says A20 Pro carries 50 percent more memory bandwidth than A19 Pro. Every step there widens the model size that runs usefully on device.

The accelerator is spreading across the die. Neural accelerators in GPU cores and in CPU cores mean the interesting question is total on-chip tensor throughput and whether the runtime can schedule across it, not the NPU rating alone.

The runtime layer is consolidating. ONNX Runtime underneath Windows ML, Core ML on Apple platforms, and vendor execution providers plugged into both is now the shape of the stack. That is where the remaining differentiation sits, because the silicon ratings are converging.

Last verified

Last verified: September 2026. Checked against vendor product pages and press releases (Qualcomm, Intel, AMD, Apple, Google), Intel ARK specifications, Microsoft Learn Copilot+ developer documentation, and Windows 11 system requirements. Every TOPS figure on this page is quoted from the vendor that makes the part. Where a vendor does not publish a TOPS number, this page says so rather than repeating a third-party estimate.

Related reading

This page is the NPU reference. If you want the ground-level comparison of all four processing units, including Google's datacenter TPU and where each one physically sits in a machine, read CPUs, GPUs, NPUs, and TPUs compared. It answers "what does each of these four do, and which one runs my work." For why memory hierarchy dominates so much of this discussion, see what CPU cache is and how L1, L2, and L3 differ.

Frequently asked questions

What does NPU mean?

NPU stands for Neural Processing Unit. It is a processor built to run the low-precision matrix and tensor maths behind neural network inference, using far less energy than a CPU or GPU would for the same work. It is the block in a phone or laptop chip that handles AI features on the device instead of in the cloud.

What is an NPU processor?

It is the AI accelerator built into a modern phone, laptop, or edge chip. It runs already-trained models locally: speech recognition, camera processing, background blur, face unlock, and small language models. It does not train models, and it does not replace the CPU or GPU. It sits alongside them on the same piece of silicon and takes the inference work.

What is the difference between an NPU, a CPU, and a GPU?

A CPU is a generalist optimised for low-latency execution of branchy code. A GPU is optimised for massively parallel throughput and is what model training runs on. An NPU is a fixed-function accelerator for tensor maths, built for efficiency rather than flexibility. A single application often uses all three at once, with the operating system deciding placement.

Is an NPU faster than a GPU?

Usually not in raw throughput. A discrete GPU has far more compute and far more memory bandwidth. The NPU wins on energy per inference, which is what matters for features that run continuously on battery. Peak speed belongs to the GPU. Sustained, always-on AI belongs to the NPU.

What does TOPS actually measure?

Trillions of operations per second, almost always counted at INT8 precision. Three caveats apply: the precision changes the number, peak figures assume a utilisation that real workloads do not reach, and TOPS ignores memory bandwidth entirely. For language models, inference is often memory-bound, so a higher TOPS rating can still be slower in practice.

What is the 40 TOPS Copilot+ requirement?

Microsoft's threshold for the Copilot+ PC label: an NPU capable of more than 40 trillion operations per second, plus 16 GB of RAM and 256 GB of storage. It gates a specific Windows feature set. It is not the point at which on-device AI becomes possible, and leading 2026 parts from Qualcomm, AMD, and Intel all clear it, at 80, 60, and 50 TOPS respectively.

Can developers program an NPU directly?

Rarely. There is no cross-vendor equivalent of CUDA. On Windows, Microsoft now recommends Windows ML, which runs on ONNX Runtime and picks an execution provider automatically for the hardware present. On Apple platforms it is Core ML or MLX. In practice you export a quantised model and let the runtime decide placement.

Do I need an NPU in my next laptop?

Only if you want on-device AI features that run without draining the battery, or you plan to develop against one. An NPU does not make ordinary work faster. If you buy a Copilot+ Windows machine or any recent Mac, you get one regardless.