Definition
The process by which an AI model breaks down human language into smaller units called ‘tokens’. These tokens can be words, parts of words, or punctuation, which the AI then processes numerically.
Why it matters (in Poovi’s context)
It is a fundamental step in how AI understands and generates text. Differences in tokenisation, especially when combined with unusual phrasing or encoding, can be exploited to bypass safety filters that are trained on different token representations.
Key properties or components
- Breaks text into manageable units
- Enables numerical processing by AI
- Different models may tokenise differently
- Can be exploited via unusual token sequences
Contradictions or debates
None.