In recent years, the proliferation of distinct forms of false information online has raised the challenge of verifying the truthfulness of such content. To this aim, the scientific community has developed computational approaches specific to various types of information items as, e.g., misinformation in health-related blog posts, fake news on social media and microblogs, and opinion spam in reviews on dedicated platforms. Usually, this issue is addressed as a classification task by employing Machine/Deep Learning algorithms; alternative approaches rely on model-based algorithms exploiting prior domain knowledge and also focusing on explainability.
The possibility of supplementing automatic approaches with manual ones, relying on human-in-the-loop solutions either to verify the truthfulness of information items or to evaluate the effectiveness of such models, remains almost entirely uninvestigated, except for very recent proposals that have not yet understood in detail how to hybridize the automatic approaches with human computation. Yet, human computation techniques such as crowdsourcing seem a promising compromise between truthfulness assessment performed by a few experts (that does not scale) and by automatic systems (that suffers from imperfect accuracy).
In this context, another crucial aspect that has not been sufficiently considered yet is how to evaluate the effectiveness of systems aimed to assess the truthfulness of information items. How to define rigorous evaluation methodologies? How to build reliable gold standard datasets? While the truthfulness of information items such as news can be verified ex-post in a somehow factual way, the truthfulness of online reviews is difficult to assess objectively: how to avoid the bias that can be introduced by humans assessing content that is ambiguous or on debated topics? What is the granularity of an information item, e.g., is it entirely truthful, or does it contain both true and false statements? Last but not least, which measures can be used to evaluate the effectiveness of such systems? Up to now, adaptations of measures used for other tasks have been employed (typically classification), but the peculiar characteristics of the truthfulness assessment task, and even the datasets already available, call for new measures still to be defined.
The aim of the MoT project is twofold:
- To define a novel evaluation framework to assess the effectiveness of approaches aimed at detecting information truthfulness, including the definition of suitable paradigms, datasets, and measures, defined ad hoc for this task.
- To define novel hybrid solutions that combine state-of-the-art automatic approaches (i.e., based on Machine and Deep Learning, Knowledge-based) with manual ones (i.e., crowdsourcing and experts). Such hybrid solutions could exploit the different characteristics and advantages of the single approaches, keeping the best from all of them, and combining them in an effective way.
Project page