Toward construction-specialized, small language models: The interplay of domain adaptation, model scale and data volume

초록

While language models (LMs) are central to construction digitalization and automation, existing general-purpose LMs struggle with complex engineering contexts and domain-aligned responses. This study presents construction-specialized LMs at large, medium and small scales using four representative domain adaptation strategies: prompt engineering, retrieval-augmented generation, task-specific fine-tuning and pretraining-and-fine-tuning. Evaluated on a construction-specific question answering (QA) dataset, we show that a small-scale LM adapted via pretraining-and-fine-tuning achieves the best performance, improving F1-score by 14.6 %, S-BERT by 10.2 % and inference speed fourfold over larger-scale counterparts. Further evaluation across data regimes-from zero-shot to many-shot-reveals that training-free adaptations (prompt engineering and retrieval-augmented generation) on large-scale models excels in data-scarce settings, whereas training-required strategies (task-specific fine-tuning and pretraining-and-fine-tuning) unlock the potential of smaller models under sufficient supervision. These findings illuminate the interplay among domain adaptation strategies, model scale and data volume, providing a roadmap for developing more scalable, construction-specialized LMs in diverse field conditions.

키워드