Skip to content

Tokenizer

The component that splits raw text into tokens the model can process, and converts them back to text.

Most modern tokenizers use subword schemes like BPE, so a rare word may split into several tokens while common words stay whole. This is why token counts differ across languages for the same meaning.

Related resources

Hugging Face Transformers Tutorial
The official tutorial for Hugging Face Transformers, the industry-standard library for NLP and computer vision. It covers text classification, tokenisation, translation, image segmentation, and more, with code examples for both PyTorch and TensorFlow backends. Aimed at practitioners who want to leverage pre-trained models or fine-tune them on custom datasets for production use.
The Complete OpenAI API Guide
A comprehensive, step-by-step guide to the OpenAI API ecosystem, covering GPT, DALL·E, and Whisper models in depth. It walks you through account setup, authentication, making your first API call, prompt design patterns, and cost optimisation strategies—ideal for developers new to the OpenAI platform who want a structured learning path.