Category:

Token / Tokenization

Category:

AI Foundations

Definition

The process of converting text into tokens that LLMs can process.

Explanation

Tokenization splits raw text into smaller units such as words, subwords, or characters. LLMs operate on tokens rather than raw text. Tokenization directly affects cost, latency, context length, and output quality in enterprise AI systems.

Technical Architecture

Raw Text → Tokenizer → Token IDs → LLM Processing

Core Component

Tokenizer, vocabulary, encoding rules

Use Cases

Prompt design, cost estimation, long-context reasoning

Pitfalls

Unexpected token splits, hidden cost increases

LLM Keywords

Tokenization, LLM Tokens, Context Window

Related Concepts

Related Frameworks

• LLM
• Context Window
• Sampling

• BPE
• SentencePiece
• WordPiece

Back to Glossary Index