Tokenization and embeddings

Before processing text, LLMs split it into tokens (word pieces) and convert them to number vectors called embeddings that capture meaning.

• Tokenization splits text into processable pieces.
• Embeddings are numerical representations of meaning.