Category:
Category:
Long-context Models
Category:
LLM Architecture
Definition
Models capable of processing extremely long sequences of tokens.
Explanation
Long-context models support context windows from 128K to millions of tokens. This enables entire books, multi-day conversations, large documents, datasets, logs, or project histories to be processed at once. They use attention optimizations such as sparse attention, linear attention, or memory-compressed architectures.
Technical Architecture
Tokens → Long-context Attention → LLM Reasoning → Output
Core Component
Extended context window, optimized attention, memory layers
Use Cases
Document QA, research agents, analytics, log processing
Pitfalls
Lost-in-the-middle effect; slow inference; context irrelevant to reasoning
LLM Keywords
Long context LLMs, Extended Context, Large Token Window
Related Concepts
Related Frameworks
Context Window, RAG, Memory Routing
• Extended Attention Architecture