Long range dependency modelling is the capability of a sequence model to capture relationships between elements that are far apart in a sequence, such as tokens separated by thousands of positions. Recurrent architectures struggle with this because gradients vanish over long horizons, whereas attention mechanisms and structured state-space models provide direct or efficient paths between distant elements. Effective long range modelling is essential for tasks where context far from the current position determines the output.
Overview
- Long range dependency modelling addresses the difficulty of relating distant parts of a sequence. Recurrent networks pass information through a chain of states that attenuates over distance, while self-attention compares every pair of positions directly at the cost of quadratic complexity. Structured state-space models and efficient attention variants aim to preserve long-range reach while reducing that cost.
Mechanisms
- Self-attention provides direct pairwise paths between distant positions
- Positional encodings inject order information attention otherwise lacks
- Recurrent models suffer vanishing gradients over long horizons
- State-space models offer near-linear scaling for very long sequences
- Context window length bounds how much history the model can attend to
Applications
- Document-level language understanding and long-form generation
- Modelling long genomic, audio, and time-series sequences
- Retrieval-augmented reasoning over extended contexts
- Architectures requiring efficient attention over long inputs