Tokens
Words and partial words used within Models
Tokens are the fundamental units of information used by LLMs to process and generate text. They represent the basic building blocks of language and can be words, phrases, or symbols. LLMs use tokens to understand the context, meaning, and structure of the input text and to generate appropriate responses. In natural language processing (NLP), tokens are typically represented as strings of characters. Each character in the string is assigned a unique identifier, and the tokens are organized into sentences and paragraphs based on grammatical rules.
Input tokens are the ones you send to the model.
Output tokens are the ones model is generating and returning for you.
Getting started
Select a page below to learn more.