Gradient aggregation is the step in distributed machine learning where gradients computed independently on different workers or data shards are combined into a single update for the shared model. Typically realised by summing or averaging local gradients, it lets parallel workers train a consistent global model despite operating on disjoint data. The aggregation strategy and its communication pattern strongly influence training throughput, convergence and, in federated settings, privacy.
Overview
- In data-parallel training each worker holds a replica of the model and computes gradients on its own minibatch via backpropagation.
- These local gradients are then aggregated, typically by averaging, so that every replica applies an identical, globally informed update.
- Aggregation can be centralised through a parameter server or decentralised through all-reduce collectives across peers.
- The choice of aggregation pattern, frequency and any compression governs the balance between communication overhead and convergence quality.
Key aspects
- Summation or averaging of per-worker gradients into a global gradient.
- Synchronous versus asynchronous aggregation schedules.
- Centralised parameter-server versus decentralised all-reduce topologies.
- Communication cost as a primary scaling bottleneck.
- Privacy-preserving variants used in federated learning.
Mechanisms
- All-reduce collectives that sum and redistribute gradients across workers.
- Parameter servers that collect, aggregate and serve updated parameters.
- Ring and tree communication patterns to balance bandwidth and latency.
- Gradient compression and quantisation to cut communication volume.
- Secure aggregation protocols that combine updates without exposing them.
Applications
- Large-scale deep-learning training across GPU and accelerator clusters.
- Federated learning across mobile and edge devices.
- Multi-node training of language and vision models.
- High-performance-computing scientific model fitting.
- Privacy-sensitive collaborative model training.