What is tensor parallelism?
Tensor parallelism splits an individual matrix operation inside a model across multiple GPUs, rather than splitting the model layer by layer. A single weight matrix, the kind used in every attention and feed-forward block, gets sliced into chunks, with each GPU holding one chunk and computing its share of the multiplication. The devices then exchange partial results to reassemble the full output before moving to the next operation. This happens many times within a single forward pass, once per split matrix multiply, which is why every GPU in a tensor-parallel group is active on every layer at once.
How it differs from pipeline parallelism
Pipeline parallelism is the other common way to spread a model across devices, and it works on a different axis. Instead of splitting a matrix, it assigns whole layers to different GPUs: layers 1 through 10 on GPU one, layers 11 through 20 on GPU two, and so on. A request flows through the pipeline stage by stage. Tensor parallelism keeps every GPU working on every layer simultaneously, exchanging slices of intermediate results at each step, while pipeline parallelism keeps each GPU working on its own subset of layers and only hands off a completed activation to the next stage. The two are often combined at large scale: tensor parallelism within a node where interconnect is fast, pipeline parallelism across nodes where it is not.
Why it needs fast interconnect
Splitting a matrix multiplication across GPUs means those GPUs have to synchronize on every single split operation, not just once per layer. That communication has to be fast, or the GPUs spend more time waiting on each other than computing. This is the practical reason tensor parallelism is normally confined to GPUs connected by a high-bandwidth, low-latency link on the same board or chassis, rather than machines connected over an ordinary network. Split a matrix across GPUs on opposite ends of a slow link and the synchronization overhead can erase whatever benefit came from splitting the work in the first place.
Where the DGX Spark fits
A single DGX Spark is one GB10 Grace Blackwell superchip, one GPU, not a set of discrete GPUs sharing a board. There's nothing to tensor-parallelize on one Spark, because there's only one device to split work across. The concept becomes relevant once multiple Sparks are connected, such as in our two-node cluster guide, though the interconnect between separate Sparks is a network link rather than the tight on-board interconnect that tensor parallelism usually assumes, so most multi-Spark setups lean more on pipeline-style splitting or running independent model replicas than fine-grained tensor parallelism. It matters more directly for understanding how larger data-center GPU pods, the kind with several discrete GPUs wired together, serve models too large for one device's memory. Our single Spark vs. cluster post covers the practical tradeoffs of scaling a Spark deployment beyond one node.
What it buys you, and what it costs
Tensor parallelism lets a model larger than one GPU's memory run at all, and it can reduce per-token latency because the work for each layer is split across more compute. It costs communication overhead on every layer and it requires every participating GPU to be similarly fast, since the slowest one paces the whole group. For a workload that already fits comfortably in one device's memory, like most models on a single Spark's 128 GB unified pool as described in our unified memory post, there's nothing to gain from tensor parallelism since there's no capacity problem it needs to solve.