Large Language Models (LLMs) are machine learning models trained to understand and generate text. Most modern generative LLMs are trained to predict the next token based on the tokens that came before it.
1. How LLMs Generate Text
Given the input:
Today the weather is
The model predicts possible next tokens and assigns probabilities to them. A decoding strategy selects the next token, which is then added to the input. This process repeats until the model finishes generating its response.
LLMs contain millions or billions of learned parameters that capture patterns in their training data.
2. Decoding Strategies
Decoding determines how a model selects the next token.
- Greedy Decoding: Selects the token with the highest probability at each step.
- Beam Search: Explores multiple candidate sequences and keeps the most promising ones.
- Sampling: Randomly selects tokens according to their probability distribution.
Several parameters can influence sampling:
- Temperature: Controls the sharpness of the probability distribution. Lower values generally produce more predictable outputs, while higher values encourage diversity.
- Top-k: Restricts sampling to the k most probable tokens.
- Top-p: Restricts sampling to the smallest set of tokens whose cumulative probability reaches a specified threshold.
These techniques offer different trade-offs between predictability, diversity, and generation quality.
3. Tokens and Tokenizers
LLMs process numerical representations rather than raw text. A tokenizer converts text into tokens and maps them to numerical token IDs.
A token can represent a complete word, part of a word, punctuation, or another text unit.
Common tokenization approaches include:
- Character-based: Splits text into individual characters.
- Word-based: Splits text into words.
- Subword-based: Splits text into frequently occurring subword units.
Subword tokenization, including Byte Pair Encoding (BPE), is widely used because it balances vocabulary size and flexibility.
The context window defines how many tokens a model can process in a single context. It includes input content and, depending on the model and API, may also account for generated output.
4. Foundation Models vs. Instruction Models
There are two important categories of language models:
- Foundation or base models: Learn language patterns from large datasets, often through next-token prediction.
- Instruction-tuned models: Receive additional training to follow instructions and respond more appropriately to user requests.
For example, a base model may continue a text sequence, while an instruction-tuned model is better suited to answering questions or following a task description.
Chat models also commonly use structured message formats with roles such as system, user, and assistant. Correct formatting helps the model interpret the conversation as intended.
5. Text Generation API Parameters
When using a text generation API, several parameters control how the model behaves:
| Parameter | Purpose |
|---|---|
model | Specifies the model to use. |
messages | Provides conversation messages and their roles. |
max_tokens or an equivalent output limit | Limits generated output length, depending on the API. |
temperature | Controls sampling randomness. |
top_p | Controls nucleus sampling. |
n | Requests multiple output choices when supported. |
stream | Returns output incrementally instead of waiting for the complete response. |
Available parameters and their exact meanings depend on the provider and model.
6. What Is a Local LLM?
A Local LLM runs on your own computer or infrastructure instead of relying entirely on a remotely hosted model.
Advantages
- Greater control over where data is processed.
- Potentially lower recurring API costs.
- Offline usage when the model and required resources are available locally.
- More control over model selection and deployment.
Disadvantages
- Requires sufficient RAM, memory bandwidth, and potentially GPU resources.
- Model setup and optimization can require additional work.
- Smaller local models may perform less effectively on complex tasks.
- Hardware, electricity, and maintenance still have costs.
Tools such as LM Studio make it easier to download and run compatible models locally. It can also expose a local API that supports integration with applications using compatible OpenAI-style API formats.
7. Using vs. Fine-Tuning vs. Training an LLM
These are three different activities:
- Inference: Run an already-trained model to generate responses.
- Fine-tuning: Continue training a pretrained model on a more specific dataset to adapt its behavior.
- Training from scratch: Train a model from an initial state using a large dataset and substantial computing resources.
For most developers, starting with inference is the most practical approach. Fine-tuning can come later when a specific use case requires it.
Conclusion
Understanding LLMs starts with tokenization, next-token prediction, decoding strategies, and context windows. From there, developers can explore model inference, API integration, local deployment, fine-tuning, and retrieval-augmented generation (RAG).
You do not need to train a large model from scratch to build useful AI applications. Learning how to select, run, and integrate existing models is an excellent starting point.
