ref: https://arxiv.org/pdf/2504.21191

Summary

The paper shows that finetuning still matters more than model size for well-defined tasks. The authors compared focused models (SLM + Fine Tune) and large models (LLM) across three scenario: basic binary classification, multi-class labeling, and multi-class with very minimal training data. Focused models trained with domain data consistently performed well, and their advantage became most clear in the low-data scenario. Focused model learned the domain specific vocabulary, jargon, syntax, subtle semantic relationship, not just general knowledge.

Maybe the real win isn’t choosing between SLMs and LLMs, but using finetuned smaller models to ground decisions before handing them to larger models for context.