We propose a methodology for leveraging aspects of ontology design principles to guide the use of a masked language model (MLM) as a query engine over raw text documents. By using targeted fill-in-the blank-style prompts to define relations, we show how a domain expert could use BERT, a well-known MLM, to extract triples from unseen documents without any fine-tuning. We evaluate our proposed methodology using a modified document-level relation extraction task, highlighting early successes but also numerous areas that need improvement. Despite these shortcomings, we then discuss why we are still hopeful that this paves the way toward flexible text-based query engines which use collections of unstructured documents.

Knowledge graphs (KGs) and their related ontologies constitute a key component in modern knowledge-based systems. However, hand-crafting these is not scalable, particularly due to the rate at which knowledge changes in many real-world applications. Partially automating the process of extracting and even modelling knowledge has therefore been a subject of research for many years. Nevertheless, accurate and reliable KG construction from natural language documents still remains a difficult task with many challenges, even in light of the impressive recent advances in language modelling. This paper focuses on one of those challenges, namely the extraction of accurate entity representations from text documents in order to facilitate relation extraction (RE). We present a novel method for generating document-contextualized input representations for entities using a masked language model (MLM) without the need for any sort of fine-tuning. These representations are then used as inputs to the same MLM that generated them, alleviating the need to include entire documents when prompting. Our results show that these representations 1) improve the ability of the MLMs BERT and RoBERTa to identify statements that represent correct relations between two entities; and 2) allow BERT to perform on par with the fine-tuned MLMs BioBERT and PubMedBERT.

A large part of knowledge is commonly encoded into text documents. While extracting this information into a Knowledge Graph (KG) is a common approach, it suffers from challenges when texts are added, removed, or changed, or when the schema of the intended KG changes. Instead we advocate an approach where text and models evolve together in an interactive manner. We present LINTEXT, a system accompanying a published method which allows users to jointly explore and model the information held within text documents. The modeling is accomplished by specifying fill-in-the-blank prompts along with some metadata which are then recorded as specifications for simple relations that can be used to generate an ontology. The exploration aspect is accomplished by having the system complete each prompt with entities identified from the text and presenting the completions as a ranked list to the user, allowing users to verify the quality of the extracted triples. By elevating the development of the ontology to a visual and interactive level, it has an immediate text connection and users can be more certain that the documents they wish to model contain the information they wish to extract or query. Additionally, our system is designed to support the development of relation extraction (RE) pipelines underlying the document analysis, with a particular focus on supporting methods for improving vector representations of the extracted entities. To this end, users can choose to analyze documents from pre-annotated RE data sets to understand how changes in different elements of the pipeline affect the results.

As language models (LMs) increase in size and complexity, so too does their capacity to capture and recall knowledge from their pre-training data. This property is in some cases desirable, such as when applied to general-domain knowledge graph completion and relation extraction (RE) tasks. However, when extracting information about well-known entities from documents, it can be unclear whether LM-based systems perform well by actually solving the tasks, or instead by leveraging patterns about those entities learned from pre-training data. In realistic scenarios, documents may be domain-specific and unlike anything in the pre-training data, or may use well-known named entities in new ways. In the latter case, LMs need to disregard parts of their background knowledge to succeed, a capability we informally call suspension of disbelief. This paper uses a zero-shot RE setting to investigate whether masked LMs (MLMs) are capable of this. We devise a method called DocShRED to construct adversarial versions of well-known RE data sets like DocRED, by intentionally using named entities in ways that are inconsistent with common pre-training data. We find that BERT and RoBERTa exhibit near-random performance on DocShRED, even when using a zero-shot technique to incorporate the text as supporting information. We also find that both MLMs perform significantly better on DocShRED when entities' surface forms are withheld using a novel method called entity isolation, highlighting the impact of background knowledge on the task. Finally, when we apply entity isolation to the biomedical RE data set BioRED, BERT outperforms BioBERT and PubMedBERT without fine-tuning, suggesting an overall improvement in generalizability.