<  Back to the Polytechnique Montréal portal

MLMLM: link prediction with mean likelihood masked language model

Louis Clouatre, Philippe Trempe, Amal Zouaq and Sarath Chandar Anbil Parthipan

Paper (2021)

Open Acess document in PolyPublie and at official publisher
[img]
Preview
Open Access to the full text of this document
Published Version
Terms of Use: Creative Commons Attribution
Download (474kB)
Show abstract
Hide abstract

Abstract

Knowledge Bases (KBs) are easy to query, verifiable, and interpretable. They however scale with man-hours and high-quality data. Masked Language Models (MLMs), such as BERT, scale with computing power as well as unstructured raw text data. The knowledge contained within these models is however not directly interpretable. We propose to perform link prediction with MLMs to address both the KBs scalability issues and the MLMs interpretability issues. By committing the knowledge embedded in MLMs to a KB, it becomes interpretable. To do that we introduce MLMLM, Mean Likelihood Masked Language Model, an approach comparing the mean likelihood of generating the different entities to perform link prediction in a tractable manner. We obtain State of the Art (SotA) results on the WN18RR dataset and SotA results on the Precision@1 metric on the WikidataM5 inductive and transductive setting. We also obtain convincing results on link prediction on previously unseen entities, making MLMLM a suitable approach to introducing new entities to a KB.

Subjects: 2700 Information technology > 2705 Software and development
2700 Information technology > 2706 Software engineering
Department: Department of Computer Engineering and Software Engineering
Funders: Apogée Canada, Canada First Research Excellence Fund program, École Polytechnique Startup Fund PIED, Canada CIFAR AI Chair, GRSNG / NSERC - Discovery Grant
PolyPublie URL: https://publications.polymtl.ca/10613/
Conference Title: The Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021)
Conference Location: Bangkok, Thailand
Conference Date(s): 2021-08-01 - 2021-08-06
Publisher: Association for Computational Linguistics
DOI: 10.18653/v1/2021.findings-acl.378
Official URL: https://doi.org/10.18653/v1/2021.findings-acl.378
Date Deposited: 19 Dec 2022 13:47
Last Modified: 09 Apr 2024 06:45
Cite in APA 7: Clouatre, L., Trempe, P., Zouaq, A., & Anbil Parthipan, S. C. (2021, August). MLMLM: link prediction with mean likelihood masked language model [Paper]. The Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021), Bangkok, Thailand. https://doi.org/10.18653/v1/2021.findings-acl.378

Statistics

Total downloads

Downloads per month in the last year

Origin of downloads

Dimensions

Repository Staff Only

View Item View Item