What is CUDA, and why does it matter for AI?

In plain English

Graphics chips (GPUs) were built to draw video-game graphics, which means doing thousands of small calculations at once. CUDA is the software toolkit NVIDIA released in 2007 that lets programmers use its GPUs for any kind of calculation, not just graphics. It sits between the chip and the AI frameworks people actually use, such as PyTorch, and turns their instructions into work the GPU can do very fast.

Why it matters

When deep learning took off in the 2010s, researchers built their tools on CUDA, because NVIDIA chips and CUDA were what worked. Nearly two decades of libraries, tutorials and expertise followed: NVIDIA says its CUDA community has around 6 million developers. That software head start is a big reason NVIDIA dominates AI chips: switching to another chip can mean rewriting or retuning software, which is slow and risky.

Rivals are working to loosen that grip. AMD has its own platform, ROCm; Google’s TPUs use different software; and newer tools such as OpenAI’s Triton let developers write fast AI code that can run on more than one kind of chip.

An example

A start-up wants to train a model on cheaper chips from a different maker. Its code runs on NVIDIA GPUs through PyTorch and CUDA. Most of PyTorch now supports other chips too, but a few speed-critical parts were written directly in CUDA, so the team has to rewrite them before the move pays off.

What to watch out for

“It only runs on CUDA” in a meeting means the software is tied to NVIDIA hardware, which affects cost and supply. As AI frameworks add better support for other chips, that lock-in is slowly easing, so check whether it still applies to your case before assuming it does.

See where CUDA sits in The AI tech stack, and the hardware itself in What powers AI.