RAG 적용을 위한 기술교범 정비절차 검색방식의 성능 비교 연구: BM25·벡터·하이브리드 검색을 중심으로
Ⓒ 2026 Korea Society for Naval Science & Technology
초록
본 연구는 기술교범 기반 검색증강생성(retrieval-augmented generation, RAG) 적용을 위한 검색단계에서 BM25, 벡터 및 하이브리드 검색방식의 정비절차 검색성능을 비교·분석을 수행하였다. 5종의 기술교범으로부터 구축한 788개의 문서 청크를 대상으로 직접질의와 표현변형질의 각 30건, 총 60건의 평가질의를 구성하였다. 벡터 검색에는 Qwen3-Embedding-8B 모델을 적용하였으며, 하이브리드 검색은 BM25와 벡터 검색의 순위를 Reciprocal Rank Fusion(RRF)으로 결합하였다. 검색성능은 Hit@1, Hit@3, Hit@5 및 Mean Reciprocal Rank(MRR)를 이용하여 평가하였다. 실험 결과 전체 질의의 MRR은 BM25 0.2535, 벡터 검색 0.6355, 하이브리드 검색 0.6415로 나타났다. 벡터 및 하이브리드 검색은 BM25보다 통계적으로 유의하게 높은 성능을 보였으며, 전체 질의에서는 벡터와 하이브리드 검색 간 유의한 차이가 확인되지 않았다. 반면 질의 유형을 구분하여 분석한 결과 직접질의에서는 하이브리드 검색이 상대적으로 높은 성능을 보였으며, 표현변형질의에서는 벡터 검색이 높은 성능을 나타냈다.
Abstract
This study compared BM25, vector and hybrid retrieval methods for maintenance procedure search in the retrieval stage of RAG applied to technical manuals. A corpus comprising 788 document chunks was constructed from five manuals, and 60 evaluation queries, including 30 direct and 30 paraphrased queries, were prepared. Qwen3-Embedding-8B was used for vector retrieval, whereas a hybrid retrieval approach combined BM25 and vector rankings using reciprocal rank fusion (RRF). Performance was evaluated using Hit@1, Hit@3, Hit@5, and mean reciprocal rank (MRR). The results showed MRR values of 0.2535 for BM25, 0.6355 for vector retrieval, and 0.6415 for hybrid retrieval. Both vector and hybrid methods significantly outperformed BM25, with no significant difference between them overall. In terms of query type, hybrid retrieval performed better on direct queries, whereas vector retrieval performed better on paraphrased queries.
Keywords:
Technical Manual, Maintenance Procedure, Maintenance Support, Technical Data, RAG(Retrieval Augmented Generation)키워드:
기술교범, 정비절차, 정비지원, 기술자료References
- Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel et al., ‘Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,’ Advances in Neural Information Processing Systems, VOL. 33, 2020.
-
Stephen Robertson & Hugo Zaragoza, ‘The Probabilistic Relevance Framework: BM25 and Beyond,’ Foundations and Trends in Information Retrieval, VOL. 4, NO. 1-2, 2009.
[https://doi.org/10.1561/1500000019]
-
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih, ‘Dense Passage Retrieval for Open-Domain Question Answering,’ in Proceedings of EMNLP, November 2020, pp. 6769-6781.
[https://doi.org/10.18653/v1/2020.emnlp-main.550]
-
Gordon V. Cormack, Charles L. A. Clarke, & Stefan Büttcher, ‘Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods,’ in Proceedings of ACM SIGIR, 2009, pp. 758-759.
[https://doi.org/10.1145/1571941.1572114]
- Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin et al., ‘Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models,’ arXiv preprint arXiv:2506.05176, , 2025.
- Ye-rin Nam, Han-seon Choi, Jong-Geun Choi, Hyuk-Jin Kwon, & Young-hoon Lee, ‘Small Language Model and RAG-Based Military Maintenance Manual Curation System,’ Journal of Internet Computing and Services, VOL. 26, NO. 1, 2025, pp. 181-192.
-
Sebastian Bruch, Siyu Gai, & Amir Ingber, ‘An Analysis of Fusion Functions for Hybrid Retrieval,’ ACM Transactions on Information Systems, VOL. 42, NO. 1, 2023, Article 20.
[https://doi.org/10.1145/3596512]
- Antoine Louis, Gijs van Dijck, & Gerasimos Spanakis, ‘Know When to Fuse: Investigating Non-English Hybrid Retrieval in the Legal Domain,’ in Proceedings of COLING, Abu Dhabi, UAE, January 2025, pp. 4293-4312.