Understanding GPU Computing: CUDA Cores, Tensor Cores, TOPS and TFLOPS

This is the first part of a series of articles that I intend to write to document progress of a project I started about radio propagation and ways to predict it. Part of this work will make it on paper in an article that will be printed in DUBUS magazine, 4/2026. The rest is just documentation I gathered and tests I made on hardware that is available to me without any additional purchases. All in a better organized format. I wanted to see if I can use to its best whatever equipment I own and refrain for spending any more money. The main reason I published all this on my blog is because I hope somebody else will find it useful, information, code and reference. Enjoy

Part 1 of the series: Exploring GPU Computing — From CUDA Fundamentals to SDR and AI

1. Introduction

Graphics Processing Units (GPUs) have evolved considerably beyond their original purpose of rendering computer graphics. Today, they play an important role in artificial intelligence, computational physics, genomics, radio signal processing, molecular modeling, fluid dynamics, and many other scientific and engineering disciplines.

Modern GPU specifications frequently advertise performance in trillions of operations per second (TOPS) or trillions of floating-point operations per second (TFLOPS). NVIDIA Jetson modules, for example, are marketed with AI performance figures reaching tens or hundreds of TOPS.

But what do these numbers actually mean? Is a GPU capable of 157 TOPS necessarily faster than one capable of 67 TOPS? Can a high-TOPS accelerator solve computational fluid dynamics problems faster? And why are GPUs particularly suitable for neural networks?

The answers depend on several architectural and mathematical considerations, including numerical precision, parallelism, memory bandwidth, and the characteristics of the algorithm being executed.

This article introduces these concepts and provides the theoretical foundation for a series of practical CUDA experiments. In subsequent articles, we will implement CUDA programs, compare CPU and GPU performance, and eventually develop GPU-accelerated digital signal processing applications.

2. CPU and GPU: Two Different Computing Architectures

Central Processing Units (CPUs) and Graphics Processing Units (GPUs) are both programmable processors, but their architectures are optimized for different kinds of computational workloads.

2.1 The CPU: Optimized for General-Purpose Execution

A modern CPU typically contains a relatively small number of sophisticated processor cores. These cores are optimized for low-latency execution, complex instruction sequences, branching, operating-system tasks, and workloads with irregular memory access patterns.

CPU cores incorporate architectural features such as:

• Multiple cache levels.
• Branch prediction and speculative execution.
• Out-of-order instruction execution.
• Vector instruction sets such as AVX2 and AVX-512, depending on the processor.
• Hardware support for multiple execution threads.

For example, the Intel Core i9-9880H processor used in our experiments has eight physical cores and sixteen logical threads. It also supports AVX2 instructions, allowing multiple floating-point values to be processed within a single vector instruction.

CPUs are therefore not inherently sequential processors. They can exploit substantial instruction-level, data-level, and thread-level parallelism. Nevertheless, GPU architectures are designed to expose a much larger number of concurrent execution threads for suitable workloads.

2.2 The GPU: Optimized for Massive Parallelism

A GPU contains many arithmetic execution units organized into processing groups. In NVIDIA terminology, these groups are called Streaming Multiprocessors (SMs).

GPU execution is particularly effective when the same computational operation must be applied to a large collection of independent data elements.

Consider the addition of two vectors:

$$ C[i] = A[i] + B[i] $$

Each output element can be calculated independently of the others.

On a CPU, a conventional implementation might use a loop:

void vectorAddCPU(
    const float* A,
    const float* B,
    float* C,
    int N)
{
    for (int i = 0; i < N; ++i) {
        C[i] = A[i] + B[i];
    }
}

A CUDA implementation expresses the same operation as a GPU kernel:

__global__ void vectorAddGPU(
    const float* A,
    const float* B,
    float* C,
    int N)
{
    int i = blockIdx.x * blockDim.x
          + threadIdx.x;

    if (i < N) {
        C[i] = A[i] + B[i];
    }
}

The expression involving blockIdx, blockDim, and threadIdx determines which vector element is processed by each CUDA thread.

Instead of manually iterating over all elements, the programmer describes the work performed by a single logical GPU thread. CUDA then schedules many such threads for execution.

We will analyze this code and the CUDA execution hierarchy in detail in Part 3 of the series.

3. Understanding CUDA’s SIMT Execution Model

NVIDIA GPUs use a programming and execution model called Single Instruction, Multiple Threads (SIMT).

Under SIMT, many threads execute similar instruction sequences while maintaining their own registers and logical execution state.

CUDA organizes threads into a hierarchy:

• Thread: An individual logical execution instance.
• Block: A group of CUDA threads that can cooperate through shared memory and synchronization.
• Grid: A collection of thread blocks executing the same kernel.
• Warp: The hardware scheduling group, consisting of 32 CUDA threads on NVIDIA architectures discussed here.

For example, a CUDA kernel launched with 4,096 blocks containing 256 threads per block represents:

4,096 × 256 = 1,048,576 logical CUDA threads.

This does not mean the GPU physically executes one million threads simultaneously. The hardware schedules them in groups across its Streaming Multiprocessors.

SIMT execution is efficient when threads within a warp follow similar execution paths. Divergent branches can reduce efficiency because different execution paths may need to be processed separately.

This is one reason why GPUs perform particularly well on regular numerical algorithms, such as matrix multiplication, filtering, image processing, and many neural network operations.

For a detailed description, see the NVIDIA CUDA Programming Guide.

4. CUDA Cores Versus Tensor Cores

NVIDIA GPU specifications distinguish between CUDA cores and Tensor Cores. These are not equivalent computing resources.

4.1 CUDA Cores

CUDA cores provide general-purpose arithmetic capabilities used by a broad range of GPU algorithms.

Examples include:

• Vector and array operations.
• Digital filtering and numerical transforms.
• Complex arithmetic.
• Image and signal processing.
• Numerical simulation.
• Custom CUDA kernels.

The NVIDIA Quadro T2000 GPU used in our experiments is based on the Turing architecture and has 1,024 CUDA cores.

Although it lacks dedicated Tensor Cores, it is fully capable of running CUDA programs and can accelerate many numerical workloads.

This makes it a useful experimental platform for learning GPU programming without requiring an expensive modern accelerator.

4.2 Tensor Cores

Tensor Cores are specialized processing units designed to accelerate supported matrix operations, particularly matrix multiplication and accumulation.

A simplified mathematical operation is:

$$ D = A × B + C $$

where A, B, C, and D represent matrices or matrix fragments.

Matrix multiplication is central to many machine learning algorithms. Neural networks frequently compute transformations of the form:

$$ Y = W × X + b $$

where W represents learned weights, X represents input data, and b represents a bias term.

Modern transformer models, convolutional neural networks, and other deep learning architectures perform enormous numbers of such operations.

Tensor Cores accelerate supported operations using numerical formats such as FP16, BF16, TF32, INT8, and others, depending on the GPU generation.

Not every GPU offers the same Tensor Core capabilities, and not every algorithm can efficiently use them.

For example, an FFT implemented using NVIDIA cuFFT generally relies on CUDA-compatible numerical execution resources and memory operations rather than benefiting automatically from advertised INT8 Tensor Core throughput.

5. What Do TOPS and TFLOPS Mean?

Two commonly advertised performance metrics are TOPS and TFLOPS.

5.1 TOPS: Trillions of Operations Per Second

TOPS stands for Trillion Operations Per Second.

One TOPS represents:

10¹² operations per second.

In AI accelerator specifications, TOPS commonly refers to low-precision integer operations, particularly INT8 arithmetic.

For example, NVIDIA advertises the following maximum AI performance figures for selected Jetson Orin systems:

System	Advertised peak AI performance
Jetson Orin Nano Super	67 TOPS
Jetson Orin NX 16 GB, Super configuration	157 TOPS
Jetson AGX Orin 64 GB	275 TOPS

These are advertised peak sparse INT8 performance figures under the specified operating configurations. They should not be interpreted as measured end-to-end application performance.

Sparsity matters. Some AI hardware can exploit specially structured patterns of zero-valued weights. Performance figures that assume supported sparsity cannot be directly compared with dense matrix-operation performance.

For example, NVIDIA specifies up to 157 sparse INT8 TOPS for the Jetson Orin NX 16 GB in its Super operating configuration, versus approximately 78 dense INT8 TOPS at the overall accelerator level.

Consequently, a device advertised at 157 TOPS does not necessarily execute an arbitrary neural network twice as fast as a device advertised at 78 TOPS.

Actual performance depends on the network architecture, numerical format, supported operators, batch size, memory access patterns, and software implementation.

Reference: NVIDIA Jetson Orin specifications.

5.2 TFLOPS: Trillions of Floating-Point Operations Per Second

TFLOPS stands for Trillion Floating-Point Operations Per Second.

Unlike integer operations, floating-point operations use numerical formats representing values across a wide dynamic range.

Common formats include:

Format	Typical usage	Approximate storage
INT8	Quantized neural network inference	1 byte
FP16	Deep learning inference and training	2 bytes
BF16	Deep learning with wider exponent range	2 bytes
FP32	DSP, machine learning, numerical simulation	4 bytes
FP64	High-precision scientific computation	8 bytes

The numerical format matters enormously.

For example, a computational fluid dynamics solver may require FP64 arithmetic for numerical stability or accuracy, depending on the solver and physical problem.

An AI accelerator offering very high INT8 TOPS may have relatively modest FP64 throughput.

Therefore, TOPS and TFLOPS cannot be compared meaningfully without specifying the operation type, numerical precision, and hardware execution units being measured.

Another important convention is that some performance figures count a fused multiply-add operation as two arithmetic operations: one multiplication and one addition. Benchmark comparisons must use compatible counting conventions.

6. Memory Bandwidth: An Often Overlooked Limitation

Arithmetic performance is only one factor determining how fast a program executes.

Many scientific and engineering programs spend substantial time transferring data between memory and processing units.

Consider vector addition:

$$ C[i] = A[i] + B[i] $$

Using FP32 values, each vector element requires:

• Reading A[i]: 4 bytes.
• Reading B[i]: 4 bytes.
• Writing C[i]: 4 bytes.

Therefore, each element involves approximately 12 bytes of memory traffic for a single floating-point addition, assuming those accesses reach the memory level being analyzed.

The arithmetic intensity is:

I = 1 FLOP / 12 bytes ≈ 0.083 FLOP/byte

Arithmetic intensity describes the number of computational operations performed per byte of data transferred.

Vector addition has very low arithmetic intensity. Even a GPU capable of enormous arithmetic throughput may spend most of its time waiting for data.

This leads to an important distinction:

• Compute-bound algorithms are primarily limited by available arithmetic throughput.
• Memory-bound algorithms are primarily limited by memory bandwidth or data movement.

The distinction is central to the Roofline performance model, which relates achievable computational performance to arithmetic intensity, peak arithmetic throughput, and memory bandwidth.

A simplified expression is:

Achievable performance ≤ min(Peak compute performance, Arithmetic intensity × Memory bandwidth).

Thus, a GPU with much greater theoretical computational capability may provide little benefit for a memory-bound algorithm.

7. The Cost of Moving Data Between CPU and GPU

A conventional desktop or laptop NVIDIA GPU generally has its own dedicated memory.

For example, our Dell Precision 5540 contains:

• 64 GiB of system RAM accessible by the CPU.
• An NVIDIA Quadro T2000 with 4 GiB of dedicated GPU memory.

CUDA applications commonly move data between these memory spaces using the CUDA Runtime API.

A simplified processing sequence is:

1. Allocate memory on the CPU.
2. Allocate memory on the GPU.
3. Transfer the input data to GPU memory.
4. Execute one or more CUDA kernels.
5. Transfer the results back to CPU memory.

For an algorithm using explicit transfers, the overall execution time can be approximated as:

$$ t_{total} = t_{host-to-device} + t_{compute} + t_{device-to-host} + t_{other-overhead} $$

The GPU computation may be much faster than the corresponding CPU operation, yet the complete application can still be slower because of transfer overhead.

This is especially relevant for small workloads or algorithms with low arithmetic intensity.

One strategy is to transfer data once, perform several operations while the data remains in GPU memory, and return only the required results.

This strategy is common in machine learning, image processing, and scientific computing.

Integrated architectures with shared physical memory have different data-movement characteristics, but memory placement, access patterns, synchronization, and bandwidth remain important.

8. Why GPUs Are So Effective for Artificial Intelligence

Modern deep learning algorithms rely extensively on large-scale tensor operations.

Neural networks may contain millions or billions of learned parameters, and inference often involves repeated matrix multiplications, convolutions, normalization operations, and nonlinear transformations.

Many of these operations offer considerable data parallelism.

GPU acceleration therefore benefits AI applications in several ways:

• High arithmetic throughput for supported tensor operations.
• Parallel processing of independent data elements.
• Specialized mixed-precision arithmetic.
• Efficient execution of large matrix operations.
• Hardware and software support for batching.

However, AI applications have very different resource requirements.

8.1 Large Language Models

Large language models are built around transformer architectures and other neural network structures that perform extensive tensor computations.

LLM inference requires both arithmetic throughput and sufficient memory to store model weights and intermediate state.

For example, a model containing seven billion parameters requires approximately 14 GB merely to store its weights in FP16 format:

7 × 10⁹ parameters × 2 bytes = 14 GB.

Quantization can reduce this requirement. Four-bit quantization reduces the theoretical raw weight storage to approximately 3.5 GB, excluding quantization metadata, attention caches, temporary buffers, and runtime overhead.

This explains why GPU memory capacity can be more important than peak TOPS when choosing hardware for local LLM inference.

Memory bandwidth also strongly influences token-generation speed in many inference configurations.

8.2 Computer Vision and Robotics

Computer vision systems process images or video to recognize objects, classify scenes, estimate motion, or interpret their surroundings.

Robotics applications may combine several neural networks with sensor processing and real-time control.

In these applications, embedded GPU systems such as NVIDIA Jetson are particularly attractive because they combine computational acceleration with relatively low power consumption and interfaces for cameras and other sensors.

Latency, power consumption, memory capacity, and sustained processing throughput may matter more than peak TOPS alone.

8.3 Genomics and Molecular Biology

GPU computing is used in genomics, molecular modeling, and computational biology.

Different applications require different kinds of acceleration.

DNA sequence alignment and variant calling can benefit from specialized parallel algorithms, even when neural networks are not involved.

Protein structure prediction and biomolecular machine learning frequently use neural networks, geometric models, and other computationally intensive techniques.

GPU requirements therefore vary from relatively modest inference workloads to training large models requiring extensive accelerator memory and distributed computation.

8.4 Computational Fluid Dynamics

Computational fluid dynamics (CFD) illustrates the difference between conventional scientific computing and AI acceleration.

Traditional CFD solvers discretize physical equations, such as the Navier–Stokes equations, and solve large numerical systems.

Their performance may depend heavily on FP32 or FP64 throughput, sparse linear algebra, memory bandwidth, and communication between processors.

AI-assisted CFD takes a different approach. Neural networks may be trained to approximate certain solution fields or predict physical behavior for specified boundary conditions.

Such models are often called surrogate models.

Once trained and adequately validated, a surrogate model may evaluate particular predictions much faster than repeatedly solving the complete physical system.

However, surrogate models have limitations involving generalization, physical consistency, and prediction error. They do not automatically replace high-fidelity numerical simulation.

8.5 Electromagnetics and Radio Signal Processing

GPU computing also has numerous applications in electrical engineering, telecommunications, and electromagnetics.

Examples include:

• Fast Fourier transforms.
• Digital filtering and convolution.
• Matched filtering and signal correlation.
• Large-scale spectral analysis.
• Radio modulation recognition using neural networks.
• Electromagnetic field simulation.
• AI-assisted antenna optimization.

In software-defined radio (SDR), an incoming stream of complex I/Q samples can be divided into blocks, transformed into the frequency domain, and analyzed for signal detection or classification.

Large batches of FFTs may benefit from GPU acceleration, especially when additional signal-processing stages execute without transferring intermediate arrays back to CPU memory.

This is one of the practical applications we will investigate later in the series.

9. Choosing Between an Embedded AI System and a Desktop GPU

It is useful to distinguish between embedded AI computers and general-purpose GPU workstations.

Characteristic	Embedded AI computer	Desktop/workstation GPU
Typical deployment	Robots, instruments, edge devices	Development workstations, simulation systems
Power consumption	Generally optimized for low power	Usually higher
AI inference	Often highly optimized	Often very powerful
Model training	Possible, but resource-constrained	Generally more suitable
Memory architecture	Often shared CPU/GPU memory	Usually dedicated GPU VRAM
Scientific FP64 computing	Frequently limited	Highly dependent on GPU class
Sensor integration	Often a major design objective	Usually requires additional interfaces

For example, a Jetson Orin NX may be an attractive choice for continuous, low-power neural network inference inside an SDR instrument or autonomous robot.

A desktop NVIDIA GPU may be more appropriate for model training, large-scale numerical experimentation, or scientific workloads requiring substantial compute and memory capacity.

For very demanding FP64 workloads, specialized HPC accelerators may be preferable to consumer-oriented GPUs.

The appropriate hardware therefore depends on the computational problem rather than the largest advertised TOPS figure.

10. Our Experimental Platform

To investigate these concepts experimentally, this series uses a Dell Precision 5540 mobile workstation.

Component	Specification
Computer	Dell Precision 5540
Processor	Intel Core i9-9880H
CPU cores / threads	8 / 16
System memory	64 GiB
GPU	NVIDIA Quadro T2000 Mobile
GPU architecture	Turing
GPU memory	4 GiB
CUDA compute capability	7.5
Operating system	Windows 11
CUDA Toolkit	12.9
Host C++ compiler	Microsoft Visual C++ 2022
Development environment	Visual Studio Code

This is not a high-end modern AI workstation. Nevertheless, it is sufficiently capable for studying CUDA execution, GPU memory management, vector operations, FFT acceleration, and smaller machine learning workloads.

Using modest hardware is also useful from an educational perspective. Performance limitations make architectural tradeoffs more visible.

Rather than assuming a GPU will always be faster, we will implement comparable CPU and GPU programs, measure their execution times, and identify the circumstances under which acceleration is beneficial.

11. How We Will Measure Performance

Throughout the series, we will distinguish between theoretical performance and experimentally observed performance.

We will pay particular attention to:

• Execution time: The measured duration of a computation.
• Throughput: The amount of data processed per unit time.
• Latency: The time required to obtain a result.
• Speedup: The ratio between reference and accelerated execution times.
• Memory bandwidth: The rate at which data moves through the memory system.
• Transfer overhead: The time spent moving data between CPU and GPU memory.
• Numerical accuracy: The difference between CPU and GPU computational results.

Speedup will generally be defined as:

Speedup = CPU execution time / GPU execution time.

A speedup greater than one indicates that the GPU implementation is faster under the specified conditions.

Importantly, we will measure both GPU computation time and complete application time, including relevant data transfers.

We will also distinguish between single-threaded CPU baselines and optimized multicore CPU implementations.

These distinctions are essential for producing technically meaningful benchmark conclusions.

12. Conclusions

GPUs are powerful computational devices, but their performance cannot be adequately described by a single specification such as TOPS or TFLOPS.

Several conclusions are worth retaining:

1. CPUs and GPUs are optimized for different execution patterns.
2. CUDA cores provide general-purpose GPU computing capabilities, while Tensor Cores accelerate supported matrix operations.
3. TOPS and TFLOPS measure different forms of arithmetic throughput and must be interpreted with respect to numerical precision.
4. Memory bandwidth and data-transfer overhead can limit performance even when arithmetic throughput is very high.
5. GPU acceleration is most useful when algorithms expose substantial parallelism and enough computation to justify data movement and scheduling overhead.
6. AI, genomics, fluid dynamics, and digital signal processing can all use GPUs, but their numerical and architectural requirements differ.
7. Real application performance must be measured rather than inferred solely from hardware specifications.

In the next article, we will move from theory to implementation and configure a complete CUDA C++ development environment on Windows 11 using Visual Studio 2022, Visual Studio Code, and the NVIDIA CUDA Toolkit.

Our goal will be to compile and execute a CUDA program on an NVIDIA Quadro T2000, while documenting the installation process and the configuration issues encountered along the way.

13. References and Further Reading

1. NVIDIA — CUDA Programming Guide. CUDA execution model, memory hierarchy, thread organization, and programming principles.
2. NVIDIA — CUDA C++ Programming Guide. CUDA programming model and GPU architecture reference.
3. NVIDIA — Jetson Orin Technical Specifications. Embedded AI hardware, throughput, memory, and performance configurations.
4. NVIDIA Developer Blog — JetPack 6.2 and Super Mode. Discussion of sparse and dense AI throughput and supported power configurations.
5. FFTW — Fastest Fourier Transform in the West. CPU-based FFT library used in subsequent experimental comparisons.
6. NVIDIA — cuFFT Documentation. GPU-accelerated Fourier transform library.