Welcome to the InfoSci Platform
Efficient Open Domain Question Answering With Delayed Attention in Transformer-Based Models

Efficient Open Domain Question Answering With Delayed Attention in Transformer-Based Models

Wissam Siblini, Mohamed Challal, Charlotte Pasqual
Copyright: © 2022 |Volume: 18 |Issue: 2 |Pages: 16
ISSN: 1548-3924|EISSN: 1548-3932|EISBN13: 9781799893691|DOI: 10.4018/IJDWM.298005
Cite Article Cite Article

MLA

Siblini, Wissam, et al. "Efficient Open Domain Question Answering With Delayed Attention in Transformer-Based Models." IJDWM vol.18, no.2 2022: pp.1-16. http://doi.org/10.4018/IJDWM.298005

APA

Siblini, W., Challal, M., & Pasqual, C. (2022). Efficient Open Domain Question Answering With Delayed Attention in Transformer-Based Models. International Journal of Data Warehousing and Mining (IJDWM), 18(2), 1-16. http://doi.org/10.4018/IJDWM.298005

Chicago

Siblini, Wissam, Mohamed Challal, and Charlotte Pasqual. "Efficient Open Domain Question Answering With Delayed Attention in Transformer-Based Models," International Journal of Data Warehousing and Mining (IJDWM) 18, no.2: 1-16. http://doi.org/10.4018/IJDWM.298005

Export Reference

Mendeley
Favorite Full-Issue Download

Abstract

Open Domain Question Answering (ODQA) on a large-scale corpus of documents (e.g. Wikipedia) is a key challenge in computer science. Although Transformer-based language models such as Bert have shown an ability to outperform humans to extract answers from small pre-selected passages of text, they suffer from their high complexity if the search space is much larger. The most common way to deal with this problem is to add a preliminary information retrieval step to strongly filter the corpus and keep only the relevant passages. In this article, the authors consider a more direct and complementary solution which consists in restricting the attention mechanism in Transformer-based models to allow a more efficient management of computations. The resulting variants are competitive with the original models on the extractive task and allow, in the ODQA setting, a significant acceleration of predictions and sometimes even an improvement in the quality of response.