A: Small Language Models are typically built using three main approaches:
1. Training from scratch: The model learns language patterns from large, carefully curated datasets, with its parameters initialized and trained rather than inherited from an existing pre-trained model.
2. Fine-tuning a pre-trained model: An existing model is further trained on selected data to improve its performance on specific tasks, domains, or instructions.
3. Knowledge distillation: A smaller student model learns from the outputs, predictions, or other training signals of a more capable *teacher model*, transferring useful capabilities into a more compact system.
SLMs are often described as models with fewer than 10 billion parameters, although there is no universally agreed-upon size threshold.
These approaches can also be combined to build models that balance capability, computational efficiency, and deployment costs.
References:
