Token Concepts
The basic unit of text that AI models process. Prompts are split into tokens (roughly 0.75 words each), and models have a maximum token limit.
Tokens are how AI models “see” your text. Understanding tokens helps you write more effective prompts within the limits.
What Is a Token?
Tokens are pieces of text, roughly:
- Common words = 1 token
- Long/rare words = 2+ tokens
- Punctuation = usually 1 token
- ~0.75 words per token on average
Examples
| Text | Approximate Tokens |
|---|---|
| “cat” | 1 |
| “photograph” | 1-2 |
| “bioluminescent” | 2-3 |
| “a photo of a cat” | 5 |
| “highly detailed digital art” | 5 |
Token Limits
| Model | Limit | Notes |
|---|---|---|
| SD 1.5 | 77 tokens | Standard CLIP |
| SDXL | 77 tokens | Two text encoders |
| DALL-E 3 | ~400 tokens | Much longer prompts |
| Midjourney | ~60 tokens | Varies by version |
Beyond the Limit
When you exceed the token limit:
- Extra tokens are truncated (ignored)
- Some UIs support prompt chunking
- BREAK keyword splits into chunks (SD)
Token Weighting
Some systems allow emphasizing tokens:
(word)or(word:1.2)= more emphasis[word]or(word:0.8)= less emphasis- Stacking:
((word))= even more
Tips
- Put important words early in prompt
- Remove filler words when near limit
- Check token count in your UI
- Longer ≠ better; be concise