List (non-exhaustive) of publications produced by the members of the laboratory.
2022
Amigó, Enrique; Mizzaro, Stefano; Spina, Damiano
Ranking Interruptus: When Truncated Rankings Are Better and How to Measure That Proceedings Article
In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 588–598, Association for Computing Machinery, New York, NY, USA, 2022, ISBN: 9781450387323.
@inproceedings{10.1145/3477495.3532051,
title = {Ranking Interruptus: When Truncated Rankings Are Better and How to Measure That},
author = {Enrique Amigó and Stefano Mizzaro and Damiano Spina},
url = {https://doi.org/10.1145/3477495.3532051},
doi = {10.1145/3477495.3532051},
isbn = {9781450387323},
year = {2022},
date = {2022-01-01},
booktitle = {Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval},
pages = {588–598},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
series = {SIGIR '22},
abstract = {Most of information retrieval effectiveness evaluation metrics assume that systems appending irrelevant documents at the bottom of the ranking are as effective as (or not worse than) systems that have a stopping criteria to 'truncate' the ranking at the right position to avoid retrieving those irrelevant documents at the end. It can be argued, however, that such truncated rankings are more useful to the end user. It is thus important to understand how to measure retrieval effectiveness in this scenario. In this paper we provide both theoretical and experimental contributions. We first define formal properties to analyze how effectiveness metrics behave when evaluating truncated rankings. Our theoretical analysis shows that de-facto standard metrics do not satisfy desirable properties to evaluate truncated rankings: only Observational Information Effectiveness (OIE) – a metric based on Shannon's information theory – satisfies them all. We then perform experiments to compare several metrics on nine TREC datasets. According to our experimental results, the most appropriate metrics for truncated rankings are OIE and a novel extension of Rank-Biased Precision that adds a user effort factor penalizing the retrieval of irrelevant documents.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Most of information retrieval effectiveness evaluation metrics assume that systems appending irrelevant documents at the bottom of the ranking are as effective as (or not worse than) systems that have a stopping criteria to 'truncate' the ranking at the right position to avoid retrieving those irrelevant documents at the end. It can be argued, however, that such truncated rankings are more useful to the end user. It is thus important to understand how to measure retrieval effectiveness in this scenario. In this paper we provide both theoretical and experimental contributions. We first define formal properties to analyze how effectiveness metrics behave when evaluating truncated rankings. Our theoretical analysis shows that de-facto standard metrics do not satisfy desirable properties to evaluate truncated rankings: only Observational Information Effectiveness (OIE) – a metric based on Shannon's information theory – satisfies them all. We then perform experiments to compare several metrics on nine TREC datasets. According to our experimental results, the most appropriate metrics for truncated rankings are OIE and a novel extension of Rank-Biased Precision that adds a user effort factor penalizing the retrieval of irrelevant documents.
2021
Brand, Erik; Roitero, Kevin; Soprano, Michael; Demartini, Gianluca
E-BART: Jointly Predicting and Explaining Truthfulness Proceedings Article
In: Augenstein, Isabelle; Papotti, Paolo; Wright, Dustin (Ed.): Proceedings of the 2021 Truth and Trust Online Conference (TTO 2021), Virtual, October 7-8, 2021, pp. 18–27, Hacks Hackers, 2021.
@inproceedings{conference-paper-tto-2021,
title = {E-BART: Jointly Predicting and Explaining Truthfulness},
author = {Erik Brand and Kevin Roitero and Michael Soprano and Gianluca Demartini},
editor = {Isabelle Augenstein and Paolo Papotti and Dustin Wright},
url = {https://truthandtrustonline.com/wp-content/uploads/2021/10/TTO2021_paper_16-1.pdf},
year = {2021},
date = {2021-01-01},
booktitle = {Proceedings of the 2021 Truth and Trust Online Conference (TTO 2021), Virtual, October 7-8, 2021},
pages = {18--27},
publisher = {Hacks Hackers},
abstract = {Automated fact-checking (AFC) systems exist to combat disinformation, however their complexity makes them opaque to the end user, making it difficult to foster trust. In this paper, we introduce the E-BART model with the hope of making progress on this front. E-BART is able to provide a veracity prediction for a claim, and jointly generate a human-readable explanation for this decision. We show that E-BART is competitive with the state-of-theart on the e-FEVER and e-SNLI tasks. In addition, we validate the joint-prediction architecture by showing 1) that generating explanations does not significantly impede the model from performing well in its main task of veracity prediction, and 2) that predicted veracity and explanations are more internally coherent when generated jointly than separately. Finally, we also conduct human evaluations on the impact of generated explanations and observe that explanations increase human ability to spot misinformation and make people more skeptical about claims.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Automated fact-checking (AFC) systems exist to combat disinformation, however their complexity makes them opaque to the end user, making it difficult to foster trust. In this paper, we introduce the E-BART model with the hope of making progress on this front. E-BART is able to provide a veracity prediction for a claim, and jointly generate a human-readable explanation for this decision. We show that E-BART is competitive with the state-of-theart on the e-FEVER and e-SNLI tasks. In addition, we validate the joint-prediction architecture by showing 1) that generating explanations does not significantly impede the model from performing well in its main task of veracity prediction, and 2) that predicted veracity and explanations are more internally coherent when generated jointly than separately. Finally, we also conduct human evaluations on the impact of generated explanations and observe that explanations increase human ability to spot misinformation and make people more skeptical about claims.
Roitero, Kevin; Soprano, Michael; Portelli, Beatrice; Luise, Massimiliano De; Spina, Damiano; Mea, Vincenzo Della; Serra, Giuseppe; Mizzaro, Stefano; Demartini, Gianluca
Can The Crowd Judge Truthfulness? A Longitudinal Study on Recent Misinformation About COVID-19 Journal Article
In: Personal and Ubiquitous Computing, 2021, ISSN: 1617-4917.
@article{journal-paper-puc-2021,
title = {Can The Crowd Judge Truthfulness? A Longitudinal Study on Recent Misinformation About COVID-19},
author = {Kevin Roitero and Michael Soprano and Beatrice Portelli and Massimiliano De Luise and Damiano Spina and Vincenzo Della Mea and Giuseppe Serra and Stefano Mizzaro and Gianluca Demartini},
url = {https://doi.org/10.1007/s00779-021-01604-6},
doi = {10.1007/s00779-021-01604-6},
issn = {1617-4917},
year = {2021},
date = {2021-01-01},
journal = {Personal and Ubiquitous Computing},
abstract = {Recently, the misinformation problem has been addressed with a crowdsourcing-based approach: to assess the truthfulness of a statement, instead of relying on a few experts, a crowd of non-expert is exploited. We study whether crowdsourcing is an effective and reliable method to assess truthfulness during a pandemic, targeting statements related to COVID-19, thus addressing (mis)information that is both related to a sensitive and personal issue and very recent as compared to when the judgment is done. In our experiments, crowd workers are asked to assess the truthfulness of statements, and to provide evidence for the assessments. Besides showing that the crowd is able to accurately judge the truthfulness of the statements, we report results on workers' behavior, agreement among workers, effect of aggregation functions, of scales transformations, and of workers background and bias. We perform a longitudinal study by re-launching the task multiple times with both novice and experienced workers, deriving important insights on how the behavior and quality change over time. Our results show that workers are able to detect and objectively categorize online (mis)information related to COVID-19; both crowdsourced and expert judgments can be transformed and aggregated to improve quality; worker background and other signals (e.g., source of information, behavior) impact the quality of the data. The longitudinal study demonstrates that the time-span has a major effect on the quality of the judgments, for both novice and experienced workers. Finally, we provide an extensive failure analysis of the statements misjudged by the crowd-workers.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Recently, the misinformation problem has been addressed with a crowdsourcing-based approach: to assess the truthfulness of a statement, instead of relying on a few experts, a crowd of non-expert is exploited. We study whether crowdsourcing is an effective and reliable method to assess truthfulness during a pandemic, targeting statements related to COVID-19, thus addressing (mis)information that is both related to a sensitive and personal issue and very recent as compared to when the judgment is done. In our experiments, crowd workers are asked to assess the truthfulness of statements, and to provide evidence for the assessments. Besides showing that the crowd is able to accurately judge the truthfulness of the statements, we report results on workers' behavior, agreement among workers, effect of aggregation functions, of scales transformations, and of workers background and bias. We perform a longitudinal study by re-launching the task multiple times with both novice and experienced workers, deriving important insights on how the behavior and quality change over time. Our results show that workers are able to detect and objectively categorize online (mis)information related to COVID-19; both crowdsourced and expert judgments can be transformed and aggregated to improve quality; worker background and other signals (e.g., source of information, behavior) impact the quality of the data. The longitudinal study demonstrates that the time-span has a major effect on the quality of the judgments, for both novice and experienced workers. Finally, we provide an extensive failure analysis of the statements misjudged by the crowd-workers.
Soprano, Michael; Roitero, Kevin; Barbera, David La; Ceolin, Davide; Spina, Damiano; Mizzaro, Stefano; Demartini, Gianluca
The Many Dimensions of Truthfulness: Crowdsourcing Misinformation Assessments on a Multidimensional Scale Journal Article
In: Information Processing & Management, vol. 58, no. 6, pp. 102710, 2021, ISSN: 0306-4573.
@article{journal-paper-ipm-2021,
title = {The Many Dimensions of Truthfulness: Crowdsourcing Misinformation Assessments on a Multidimensional Scale},
author = {Michael Soprano and Kevin Roitero and David La Barbera and Davide Ceolin and Damiano Spina and Stefano Mizzaro and Gianluca Demartini},
url = {https://www.sciencedirect.com/science/article/pii/S0306457321001941},
doi = {https://doi.org/10.1016/j.ipm.2021.102710},
issn = {0306-4573},
year = {2021},
date = {2021-01-01},
journal = {Information Processing & Management},
volume = {58},
number = {6},
pages = {102710},
abstract = {Recent work has demonstrated the viability of using crowdsourcing as a tool for evaluating the truthfulness of public statements. Under certain conditions such as: (1) having a balanced set of workers with different backgrounds and cognitive abilities; (2) using an adequate set of mechanisms to control the quality of the collected data; and (3) using a coarse grained assessment scale, the crowd can provide reliable identification of fake news. However, fake news are a subtle matter: statements can be just biased (“cherrypicked”), imprecise, wrong, etc. and the unidimensional truth scale used in existing work cannot account for such differences. In this paper we propose a multidimensional notion of truthfulness and we ask the crowd workers to assess seven different dimensions of truthfulness selected based on existing literature: Correctness, Neutrality, Comprehensibility, Precision, Completeness, Speaker’s Trustworthiness, and Informativeness. We deploy a set of quality control mechanisms to ensure that the thousands of assessments collected on 180 publicly available fact-checked statements distributed over two datasets are of adequate quality, including a custom search engine used by the crowd workers to find web pages supporting their truthfulness assessments. A comprehensive analysis of crowdsourced judgments shows that: (1) the crowdsourced assessments are reliable when compared to an expert-provided gold standard; (2) the proposed dimensions of truthfulness capture independent pieces of information; (3) the crowdsourcing task can be easily learned by the workers; and (4) the resulting assessments provide a useful basis for a more complete estimation of statement truthfulness.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Recent work has demonstrated the viability of using crowdsourcing as a tool for evaluating the truthfulness of public statements. Under certain conditions such as: (1) having a balanced set of workers with different backgrounds and cognitive abilities; (2) using an adequate set of mechanisms to control the quality of the collected data; and (3) using a coarse grained assessment scale, the crowd can provide reliable identification of fake news. However, fake news are a subtle matter: statements can be just biased (“cherrypicked”), imprecise, wrong, etc. and the unidimensional truth scale used in existing work cannot account for such differences. In this paper we propose a multidimensional notion of truthfulness and we ask the crowd workers to assess seven different dimensions of truthfulness selected based on existing literature: Correctness, Neutrality, Comprehensibility, Precision, Completeness, Speaker’s Trustworthiness, and Informativeness. We deploy a set of quality control mechanisms to ensure that the thousands of assessments collected on 180 publicly available fact-checked statements distributed over two datasets are of adequate quality, including a custom search engine used by the crowd workers to find web pages supporting their truthfulness assessments. A comprehensive analysis of crowdsourced judgments shows that: (1) the crowdsourced assessments are reliable when compared to an expert-provided gold standard; (2) the proposed dimensions of truthfulness capture independent pieces of information; (3) the crowdsourcing task can be easily learned by the workers; and (4) the resulting assessments provide a useful basis for a more complete estimation of statement truthfulness.
Ceolin, Davide; Primiero, Giuseppe; Wielemaker, Jan; Soprano, Michael
Assessing the Quality of Online Reviews Using Formal Argumentation Theory Proceedings Article
In: Brambilla, Marco; Chbeir, Richard; Frasincar, Flavius; Manolescu, Ioana (Ed.): Web Engineering, pp. 71–87, Springer International Publishing, Cham, 2021, ISBN: 978-3-030-74296-6.
@inproceedings{10.1007/978-3-030-74296-6_6,
title = {Assessing the Quality of Online Reviews Using Formal Argumentation Theory},
author = {Davide Ceolin and Giuseppe Primiero and Jan Wielemaker and Michael Soprano},
editor = {Marco Brambilla and Richard Chbeir and Flavius Frasincar and Ioana Manolescu},
doi = {10.1007/978-3-030-74296-6_6},
isbn = {978-3-030-74296-6},
year = {2021},
date = {2021-01-01},
booktitle = {Web Engineering},
pages = {71--87},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {Review scores collect users' opinions in a simple and intuitive manner. However, review scores are also easily manipulable, hence they are often accompanied by explanations. A substantial amount of research has been devoted to ascertaining the quality of reviews, to identify the most useful and authentic scores through explanation analysis. In this paper, we advance the state of the art in review quality analysis. We introduce a rating system to identify review arguments and to define an appropriate weighted semantics through formal argumentation theory. We introduce an algorithm to construct a corresponding graph, based on a selection of weighted arguments, their semantic similarity, and the supported ratings. We provide an algorithm to identify the model of such an argumentation graph, maximizing the overall weight of the admitted nodes and edges. We evaluate these contributions on the Amazon review dataset by McAuley et al. [15], by comparing the results of our argumentation assessment with the upvotes received by the reviews. Also, we deepen the evaluation by crowdsourcing a multidimensional assessment of reviews and comparing it to the argumentation assessment. Lastly, we perform a user study to evaluate the explainability of our method. Our method achieves two goals: (1) it identifies reviews that are considered useful, comprehensible, truthful by online users and does so in an unsupervised manner, and (2) it provides an explanation of quality assessments.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Review scores collect users' opinions in a simple and intuitive manner. However, review scores are also easily manipulable, hence they are often accompanied by explanations. A substantial amount of research has been devoted to ascertaining the quality of reviews, to identify the most useful and authentic scores through explanation analysis. In this paper, we advance the state of the art in review quality analysis. We introduce a rating system to identify review arguments and to define an appropriate weighted semantics through formal argumentation theory. We introduce an algorithm to construct a corresponding graph, based on a selection of weighted arguments, their semantic similarity, and the supported ratings. We provide an algorithm to identify the model of such an argumentation graph, maximizing the overall weight of the admitted nodes and edges. We evaluate these contributions on the Amazon review dataset by McAuley et al. [15], by comparing the results of our argumentation assessment with the upvotes received by the reviews. Also, we deepen the evaluation by crowdsourcing a multidimensional assessment of reviews and comparing it to the argumentation assessment. Lastly, we perform a user study to evaluate the explainability of our method. Our method achieves two goals: (1) it identifies reviews that are considered useful, comprehensible, truthful by online users and does so in an unsupervised manner, and (2) it provides an explanation of quality assessments.
Qu, Yunke; Roitero, Kevin; Mizzaro, Stefano; Spina, Damiano; Demartini, Gianluca
Human-in-the-Loop Systems for Truthfulness: A Study of Human and Machine Confidence Proceedings Article
In: Augenstein, Isabelle; Papotti, Paolo; Wright, Dustin (Ed.): Proceedings of the 2021 Truth and Trust Online Conference (TTO 2021), Virtual, October 7-8, 2021, pp. 40–49, Hacks Hackers, 2021.
@inproceedings{DBLP:conf/tto/QuRMSD21,
title = {Human-in-the-Loop Systems for Truthfulness: A Study of Human and
Machine Confidence},
author = {Yunke Qu and Kevin Roitero and Stefano Mizzaro and Damiano Spina and Gianluca Demartini},
editor = {Isabelle Augenstein and Paolo Papotti and Dustin Wright},
url = {https://truthandtrustonline.com/wp-content/uploads/2021/10/TTO2021_paper_29.pdf},
year = {2021},
date = {2021-01-01},
booktitle = {Proceedings of the 2021 Truth and Trust Online Conference (TTO 2021),
Virtual, October 7-8, 2021},
pages = {40--49},
publisher = {Hacks Hackers},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Roitero, Kevin; Portelli, Beatrice; Popescu, Mihai Horia; Mea, Vincenzo Della
DiLBERT: Cheap Embeddings for Disease Related Medical NLP Journal Article
In: IEEE Access, vol. 9, pp. 159714-159723, 2021.
@article{9628010,
title = {DiLBERT: Cheap Embeddings for Disease Related Medical NLP},
author = {Kevin Roitero and Beatrice Portelli and Mihai Horia Popescu and Vincenzo Della Mea},
doi = {10.1109/ACCESS.2021.3131386},
year = {2021},
date = {2021-01-01},
journal = {IEEE Access},
volume = {9},
pages = {159714-159723},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Demartini, Gianluca; Roitero, Kevin; Mizzaro, Stefano
Managing Bias in Human-Annotated Data: Moving Beyond Bias Removal Journal Article
In: CoRR, vol. abs/2110.13504, 2021.
@article{DBLP:journals/corr/abs-2110-13504,
title = {Managing Bias in Human-Annotated Data: Moving Beyond Bias Removal},
author = {Gianluca Demartini and Kevin Roitero and Stefano Mizzaro},
url = {https://arxiv.org/abs/2110.13504},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2110.13504},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Conde-Sousa, Eduardo; Vale, João; Feng, Ming; Xu, Kele; Wang, Yin; Mea, Vincenzo Della; Barbera, David La; Montahaei, Ehsan; Baghshah, Mahdieh Soleymani; Turzynski, Andreas; Gildenblat, Jacob; Klaiman, Eldad; Hong, Yiyu; Aresta, Guilherme; Araújo, Teresa; Aguiar, Paulo; Eloy, Catarina; Polónia, António
HEROHE Challenge: assessing HER2 status in breast cancer without immunohistochemistry or in situ hybridization Miscellaneous
2021.
@misc{https://doi.org/10.48550/arxiv.2111.04738,
title = {HEROHE Challenge: assessing HER2 status in breast cancer without immunohistochemistry or in situ hybridization},
author = {Eduardo Conde-Sousa and João Vale and Ming Feng and Kele Xu and Yin Wang and Vincenzo Della Mea and David La Barbera and Ehsan Montahaei and Mahdieh Soleymani Baghshah and Andreas Turzynski and Jacob Gildenblat and Eldad Klaiman and Yiyu Hong and Guilherme Aresta and Teresa Araújo and Paulo Aguiar and Catarina Eloy and António Polónia},
url = {https://arxiv.org/abs/2111.04738},
doi = {10.48550/ARXIV.2111.04738},
year = {2021},
date = {2021-01-01},
publisher = {arXiv},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
Barbera, David La; Roitero, Kevin; Mizzaro, Stefano; Mea, Vincenzo Della; Valent, Francesca
A Software Simulator for Optimizing Ambulance Location and Response Time: A Preliminary Report Proceedings Article
In: 2021 IEEE International Conference on Digital Health (ICDH), pp. 209-211, 2021.
@inproceedings{9581242,
title = {A Software Simulator for Optimizing Ambulance Location and Response Time: A Preliminary Report},
author = {David La Barbera and Kevin Roitero and Stefano Mizzaro and Vincenzo Della Mea and Francesca Valent},
doi = {10.1109/ICDH52753.2021.00037},
year = {2021},
date = {2021-01-01},
booktitle = {2021 IEEE International Conference on Digital Health (ICDH)},
pages = {209-211},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2020
Barbera, David La; Polónia, António; Roitero, Kevin; Conde-Sousa, Eduardo; Mea, Vincenzo Della
Detection of HER2 from Haematoxylin-Eosin Slides Through a Cascade of Deep Learning Classifiers via Multi-Instance Learning Journal Article
In: Journal of Imaging, vol. 6, no. 9, 2020, ISSN: 2313-433X.
@article{labarberaher2,
title = {Detection of HER2 from Haematoxylin-Eosin Slides Through a Cascade of Deep Learning Classifiers via Multi-Instance Learning},
author = {David La Barbera and António Polónia and Kevin Roitero and Eduardo Conde-Sousa and Vincenzo Della Mea},
doi = {10.3390/jimaging6090082},
issn = {2313-433X},
year = {2020},
date = {2020-08-23},
urldate = {2020-08-23},
journal = {Journal of Imaging},
volume = {6},
number = {9},
abstract = {Breast cancer is the most frequently diagnosed cancer in woman. The correct identification of the HER2 receptor is a matter of major importance when dealing with breast cancer: an over-expression of HER2 is associated with aggressive clinical behaviour; moreover, HER2 targeted therapy results in a significant improvement in the overall survival rate. In this work, we employ a pipeline based on a cascade of deep neural network classifiers and multi-instance learning to detect the presence of HER2 from Haematoxylin-Eosin slides, which partly mimics the pathologist’s behaviour by first recognizing cancer and then evaluating HER2. Our results show that the proposed system presents a good overall effectiveness. Furthermore, the system design is prone to further improvements that can be easily deployed in order to increase the effectiveness score.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Breast cancer is the most frequently diagnosed cancer in woman. The correct identification of the HER2 receptor is a matter of major importance when dealing with breast cancer: an over-expression of HER2 is associated with aggressive clinical behaviour; moreover, HER2 targeted therapy results in a significant improvement in the overall survival rate. In this work, we employ a pipeline based on a cascade of deep neural network classifiers and multi-instance learning to detect the presence of HER2 from Haematoxylin–Eosin slides, which partly mimics the pathologist’s behaviour by first recognizing cancer and then evaluating HER2. Our results show that the proposed system presents a good overall effectiveness. Furthermore, the system design is prone to further improvements that can be easily deployed in order to increase the effectiveness score.
Roitero, Kevin; Soprano, Michael; Fan, Shaoyang; Spina, Damiano; Mizzaro, Stefano; Demartini, Gianluca
Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Background Proceedings Article
In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 439–448, Association for Computing Machinery, Virtual Event, China, 2020, ISBN: 9781450380164.
@inproceedings{10.1145/3397271.3401112,
title = {Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Background},
author = {Kevin Roitero and Michael Soprano and Shaoyang Fan and Damiano Spina and Stefano Mizzaro and Gianluca Demartini},
url = {https://doi.org/10.1145/3397271.3401112},
doi = {10.1145/3397271.3401112},
isbn = {9781450380164},
year = {2020},
date = {2020-01-01},
booktitle = {Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval},
pages = {439–448},
publisher = {Association for Computing Machinery},
address = {Virtual Event, China},
series = {SIGIR '20},
abstract = {Truthfulness judgments are a fundamental step in the process of fighting misinformation, as they are crucial to train and evaluate classifiers that automatically distinguish true and false statements. Usually such judgments are made by experts, like journalists for political statements or medical doctors for medical statements. In this paper, we follow a different approach and rely on (non-expert) crowd workers. This of course leads to the following research question: Can crowdsourcing be reliably used to assess the truthfulness of information and to create large-scale labeled collections for information credibility systems? To address this issue, we present the results of an extensive study based on crowdsourcing: we collect thousands of truthfulness assessments over two datasets, and we compare expert judgments with crowd judgments, expressed on scales with various granularity levels. We also measure the political bias and the cognitive background of the workers, and quantify their effect on the reliability of the data provided by the crowd.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Truthfulness judgments are a fundamental step in the process of fighting misinformation, as they are crucial to train and evaluate classifiers that automatically distinguish true and false statements. Usually such judgments are made by experts, like journalists for political statements or medical doctors for medical statements. In this paper, we follow a different approach and rely on (non-expert) crowd workers. This of course leads to the following research question: Can crowdsourcing be reliably used to assess the truthfulness of information and to create large-scale labeled collections for information credibility systems? To address this issue, we present the results of an extensive study based on crowdsourcing: we collect thousands of truthfulness assessments over two datasets, and we compare expert judgments with crowd judgments, expressed on scales with various granularity levels. We also measure the political bias and the cognitive background of the workers, and quantify their effect on the reliability of the data provided by the crowd.
Roitero, Kevin; Culpepper, Shane J; Sanderson, Mark; Scholer, Falk; Mizzaro, Stefano
Fewer topics? A million topics? Both?! On topics subsets in test collections Journal Article
In: Inf. Retr. J., vol. 23, no. 1, pp. 49–85, 2020.
@article{DBLP:journals/ir/RoiteroCSSM20,
title = {Fewer topics? A million topics? Both?! On topics subsets in test
collections},
author = {Kevin Roitero and Shane J Culpepper and Mark Sanderson and Falk Scholer and Stefano Mizzaro},
url = {https://doi.org/10.1007/s10791-019-09357-w},
doi = {10.1007/s10791-019-09357-w},
year = {2020},
date = {2020-01-01},
journal = {Inf. Retr. J.},
volume = {23},
number = {1},
pages = {49--85},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Han, Lei; Maddalena, Eddy; Checco, Alessandro; Sarasua, Cristina; Gadiraju, Ujwal; Roitero, Kevin; Demartini, Gianluca
Crowd Worker Strategies in Relevance Judgment Tasks Proceedings Article
In: Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 241–249, Association for Computing Machinery, Houston, TX, USA, 2020, ISBN: 9781450368223.
@inproceedings{10.1145/3336191.3371857,
title = {Crowd Worker Strategies in Relevance Judgment Tasks},
author = {Lei Han and Eddy Maddalena and Alessandro Checco and Cristina Sarasua and Ujwal Gadiraju and Kevin Roitero and Gianluca Demartini},
url = {https://doi.org/10.1145/3336191.3371857},
doi = {10.1145/3336191.3371857},
isbn = {9781450368223},
year = {2020},
date = {2020-01-01},
booktitle = {Proceedings of the 13th International Conference on Web Search and Data Mining},
pages = {241–249},
publisher = {Association for Computing Machinery},
address = {Houston, TX, USA},
series = {WSDM '20},
abstract = {Crowdsourcing is a popular technique to collect large amounts of human-generated labels, such as relevance judgments used to create information retrieval (IR) evaluation collections. Previous research has shown how collecting high quality labels from a crowdsourcing platform can be challenging. Existing quality assurance techniques focus on answer aggregation or on the use of gold questions where ground-truth data allows to check for the quality of the responses.In this paper, we present qualitative and quantitative results, revealing how different crowd workers adopt different work strategies to complete relevance judgment tasks efficiently and their consequent impact on quality. We delve into the techniques and tools that highly experienced crowd workers use to be more efficient in completing crowdsourcing micro-tasks. To this end, we use both qualitative results from worker interviews and surveys, as well as the results of a data-driven study of behavioral log data (i.e., clicks, keystrokes and keyboard shortcuts) collected from crowd workers performing relevance judgment tasks. Our results highlight the presence of frequently used shortcut patterns that can speed-up task completion, thus increasing the hourly wage of efficient workers. We observe how crowd work experiences result in different types of working strategies, productivity levels, quality and diversity of the crowdsourced judgments.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Crowdsourcing is a popular technique to collect large amounts of human-generated labels, such as relevance judgments used to create information retrieval (IR) evaluation collections. Previous research has shown how collecting high quality labels from a crowdsourcing platform can be challenging. Existing quality assurance techniques focus on answer aggregation or on the use of gold questions where ground-truth data allows to check for the quality of the responses.In this paper, we present qualitative and quantitative results, revealing how different crowd workers adopt different work strategies to complete relevance judgment tasks efficiently and their consequent impact on quality. We delve into the techniques and tools that highly experienced crowd workers use to be more efficient in completing crowdsourcing micro-tasks. To this end, we use both qualitative results from worker interviews and surveys, as well as the results of a data-driven study of behavioral log data (i.e., clicks, keystrokes and keyboard shortcuts) collected from crowd workers performing relevance judgment tasks. Our results highlight the presence of frequently used shortcut patterns that can speed-up task completion, thus increasing the hourly wage of efficient workers. We observe how crowd work experiences result in different types of working strategies, productivity levels, quality and diversity of the crowdsourced judgments.
Roitero, Kevin; Brunello, Andrea; Serra, Giuseppe; Mizzaro, Stefano
Effectiveness evaluation without human relevance judgments: A systematic analysis of existing methods and of their combinations Journal Article
In: Information Processing & Management, vol. 57, no. 2, pp. 102149, 2020, ISSN: 0306-4573.
@article{ROITERO2020102149,
title = {Effectiveness evaluation without human relevance judgments: A systematic analysis of existing methods and of their combinations},
author = {Kevin Roitero and Andrea Brunello and Giuseppe Serra and Stefano Mizzaro},
url = {http://www.sciencedirect.com/science/article/pii/S030645731930192X},
doi = {https://doi.org/10.1016/j.ipm.2019.102149},
issn = {0306-4573},
year = {2020},
date = {2020-01-01},
journal = {Information Processing & Management},
volume = {57},
number = {2},
pages = {102149},
abstract = {In test collection based evaluation of retrieval effectiveness, it has been suggested to completely avoid using human relevance judgments. Although several methods have been proposed, their accuracy is still limited. In this paper we present two overall contributions. First, we provide a systematic comparison of all the most widely adopted previous approaches on a large set of 14 TREC collections. We aim at analyzing the methods in a homogeneous and complete way, in terms of the accuracy measures used as well as in terms of the datasets selected, showing that considerably different results may be achieved considering different methods, datasets, and measures. Second, we study the combination of such methods, which, to the best of our knowledge, has not been investigated so far. Our experimental results show that simple combination strategies based on data fusion techniques are usually not effective and even harmful. However, some more sophisticated solutions, based on machine learning, are indeed effective and often outperform all individual methods. Moreover, they are more stable, as they show a smaller variation across datasets. Our results have the practical implication that, when trying to automatically evaluate retrieval effectiveness, researchers should not use a single method, but a (machine-learning based) combination of them.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
In test collection based evaluation of retrieval effectiveness, it has been suggested to completely avoid using human relevance judgments. Although several methods have been proposed, their accuracy is still limited. In this paper we present two overall contributions. First, we provide a systematic comparison of all the most widely adopted previous approaches on a large set of 14 TREC collections. We aim at analyzing the methods in a homogeneous and complete way, in terms of the accuracy measures used as well as in terms of the datasets selected, showing that considerably different results may be achieved considering different methods, datasets, and measures. Second, we study the combination of such methods, which, to the best of our knowledge, has not been investigated so far. Our experimental results show that simple combination strategies based on data fusion techniques are usually not effective and even harmful. However, some more sophisticated solutions, based on machine learning, are indeed effective and often outperform all individual methods. Moreover, they are more stable, as they show a smaller variation across datasets. Our results have the practical implication that, when trying to automatically evaluate retrieval effectiveness, researchers should not use a single method, but a (machine-learning based) combination of them.
Roitero, Kevin; Soprano, Michael; Portelli, Beatrice; Spina, Damiano; Mea, Vincenzo Della; Serra, Giuseppe; Mizzaro, Stefano; Demartini, Gianluca
The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively? Proceedings Article
In: Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM2020). Galway, Ireland (Online). October 19-23, 2020. Conference Rank: GGS A+, Core A, pp. 1305–1314, Association for Computing Machinery, Virtual Event, Ireland, 2020, ISBN: 9781450368599.
@inproceedings{conference-paper-cikm2020,
title = {The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively?},
author = {Kevin Roitero and Michael Soprano and Beatrice Portelli and Damiano Spina and Vincenzo Della Mea and Giuseppe Serra and Stefano Mizzaro and Gianluca Demartini},
url = {https://doi.org/10.1145/3340531.3412048},
doi = {10.1145/3340531.3412048},
isbn = {9781450368599},
year = {2020},
date = {2020-01-01},
booktitle = {Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM2020). Galway, Ireland (Online). October 19-23, 2020. Conference Rank: GGS A+, Core A},
pages = {1305–1314},
publisher = {Association for Computing Machinery},
address = {Virtual Event, Ireland},
series = {CIKM '20},
abstract = {Misinformation is an ever increasing problem that is difficult to solve for the research community and has a negative impact on the society at large. Very recently, the problem has been addressed with a crowdsourcing-based approach to scale up labeling efforts: to assess the truthfulness of a statement, instead of relying on a few experts, a crowd of (non-expert) judges is exploited. We follow the same approach to study whether crowdsourcing is an effective and reliable method to assess statements truthfulness during a pandemic. We specifically target statements related to the COVID-19 health emergency, that is still ongoing at the time of the study and has arguably caused an increase of the amount of misinformation that is spreading online (a phenomenon for which the term "infodemic" has been used). By doing so, we are able to address (mis)information that is both related to a sensitive and personal issue like health and very recent as compared to when the judgment is done: two issues that have not been analyzed in related work.In our experiment, crowd workers are asked to assess the truthfulness of statements, as well as to provide evidence for the assessments as a URL and a text justification. Besides showing that the crowd is able to accurately judge the truthfulness of the statements, we also report results on many different aspects, including: agreement among workers, the effect of different aggregation functions, of scales transformations, and of workers background / bias. We also analyze workers behavior, in terms of queries submitted, URLs found / selected, text justifications, and other behavioral data like clicks and mouse actions collected by means of an ad hoc logger.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Misinformation is an ever increasing problem that is difficult to solve for the research community and has a negative impact on the society at large. Very recently, the problem has been addressed with a crowdsourcing-based approach to scale up labeling efforts: to assess the truthfulness of a statement, instead of relying on a few experts, a crowd of (non-expert) judges is exploited. We follow the same approach to study whether crowdsourcing is an effective and reliable method to assess statements truthfulness during a pandemic. We specifically target statements related to the COVID-19 health emergency, that is still ongoing at the time of the study and has arguably caused an increase of the amount of misinformation that is spreading online (a phenomenon for which the term "infodemic" has been used). By doing so, we are able to address (mis)information that is both related to a sensitive and personal issue like health and very recent as compared to when the judgment is done: two issues that have not been analyzed in related work.In our experiment, crowd workers are asked to assess the truthfulness of statements, as well as to provide evidence for the assessments as a URL and a text justification. Besides showing that the crowd is able to accurately judge the truthfulness of the statements, we also report results on many different aspects, including: agreement among workers, the effect of different aggregation functions, of scales transformations, and of workers background / bias. We also analyze workers behavior, in terms of queries submitted, URLs found / selected, text justifications, and other behavioral data like clicks and mouse actions collected by means of an ad hoc logger.
Roitero, Kevin; Soprano, Michael; Fan, Shaoyang; Spina, Damiano; Mizzaro, Stefano; Demartini, Gianluca
Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Background Proceedings Article
In: Proceedings of the 43st International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2020). Xi’an, China (Online). July 25-30, 2020. Conference Rank: GGS A++, Core A*, pp. 439–448, Association for Computing Machinery, Virtual Event, China, 2020, ISBN: 9781450380164.
@inproceedings{conference-paper-sigir2020,
title = {Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Background},
author = {Kevin Roitero and Michael Soprano and Shaoyang Fan and Damiano Spina and Stefano Mizzaro and Gianluca Demartini},
url = {https://doi.org/10.1145/3397271.3401112},
doi = {10.1145/3397271.3401112},
isbn = {9781450380164},
year = {2020},
date = {2020-01-01},
booktitle = {Proceedings of the 43st International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2020). Xi’an, China (Online). July 25-30, 2020. Conference Rank: GGS A++, Core A*},
pages = {439–448},
publisher = {Association for Computing Machinery},
address = {Virtual Event, China},
series = {SIGIR '20},
abstract = {Truthfulness judgments are a fundamental step in the process of fighting misinformation, as they are crucial to train and evaluate classifiers that automatically distinguish true and false statements. Usually such judgments are made by experts, like journalists for political statements or medical doctors for medical statements. In this paper, we follow a different approach and rely on (non-expert) crowd workers. This of course leads to the following research question: Can crowdsourcing be reliably used to assess the truthfulness of information and to create large-scale labeled collections for information credibility systems? To address this issue, we present the results of an extensive study based on crowdsourcing: we collect thousands of truthfulness assessments over two datasets, and we compare expert judgments with crowd judgments, expressed on scales with various granularity levels. We also measure the political bias and the cognitive background of the workers, and quantify their effect on the reliability of the data provided by the crowd.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Truthfulness judgments are a fundamental step in the process of fighting misinformation, as they are crucial to train and evaluate classifiers that automatically distinguish true and false statements. Usually such judgments are made by experts, like journalists for political statements or medical doctors for medical statements. In this paper, we follow a different approach and rely on (non-expert) crowd workers. This of course leads to the following research question: Can crowdsourcing be reliably used to assess the truthfulness of information and to create large-scale labeled collections for information credibility systems? To address this issue, we present the results of an extensive study based on crowdsourcing: we collect thousands of truthfulness assessments over two datasets, and we compare expert judgments with crowd judgments, expressed on scales with various granularity levels. We also measure the political bias and the cognitive background of the workers, and quantify their effect on the reliability of the data provided by the crowd.
Barbera, David La; Roitero, Kevin; Demartini, Gianluca; Mizzaro, Stefano; Spina, Damiano
Crowdsourcing Truthfulness: The Impact of Judgment Scale and Assessor Bias Proceedings Article
In: Jose, Joemon M.; Yilmaz, Emine; Magalhães, João; Castells, Pablo; Ferro, Nicola; Silva, Mário J.; Martins, Flávio (Ed.): Advances in Information Retrieval - 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14-17, 2020, Proceedings, Part II, pp. 207–214, Springer, 2020.
@inproceedings{DBLP:conf/ecir/BarberaRDMS20,
title = {Crowdsourcing Truthfulness: The Impact of Judgment Scale and Assessor
Bias},
author = {David La Barbera and Kevin Roitero and Gianluca Demartini and Stefano Mizzaro and Damiano Spina},
editor = {Joemon M. Jose and Emine Yilmaz and João Magalhães and Pablo Castells and Nicola Ferro and Mário J. Silva and Flávio Martins},
url = {https://doi.org/10.1007/978-3-030-45442-5_26},
doi = {10.1007/978-3-030-45442-5_26},
year = {2020},
date = {2020-01-01},
booktitle = {Advances in Information Retrieval - 42nd European Conference on IR
Research, ECIR 2020, Lisbon, Portugal, April 14-17, 2020, Proceedings,
Part II},
volume = {12036},
pages = {207--214},
publisher = {Springer},
series = {Lecture Notes in Computer Science},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Demartini, Gianluca; Mizzaro, Stefano; Spina, Damiano
Human-in-the-loop Artificial Intelligence for Fighting Online Misinformation: Challenges and Opportunities Journal Article
In: IEEE Data Eng. Bull., vol. 43, no. 3, pp. 65–74, 2020.
@article{DBLP:journals/debu/DemartiniMS20,
title = {Human-in-the-loop Artificial Intelligence for Fighting Online Misinformation:
Challenges and Opportunities},
author = {Gianluca Demartini and Stefano Mizzaro and Damiano Spina},
url = {http://sites.computer.org/debull/A20sept/p65.pdf},
year = {2020},
date = {2020-01-01},
journal = {IEEE Data Eng. Bull.},
volume = {43},
number = {3},
pages = {65--74},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Barbera, David La; Polónia, António; Roitero, Kevin; Conde-Sousa, Eduardo; Mea, Vincenzo Della
Detection of HER2 from Haematoxylin-Eosin Slides Through a Cascade of Deep Learning Classifiers via Multi-Instance Learning Journal Article
In: Journal of Imaging, vol. 6, no. 9, 2020, ISSN: 2313-433X.
@article{jimaging6090082,
title = {Detection of HER2 from Haematoxylin-Eosin Slides Through a Cascade of Deep Learning Classifiers via Multi-Instance Learning},
author = {David La Barbera and António Polónia and Kevin Roitero and Eduardo Conde-Sousa and Vincenzo Della Mea},
url = {https://www.mdpi.com/2313-433X/6/9/82},
doi = {10.3390/jimaging6090082},
issn = {2313-433X},
year = {2020},
date = {2020-01-01},
journal = {Journal of Imaging},
volume = {6},
number = {9},
abstract = {Breast cancer is the most frequently diagnosed cancer in woman. The correct identification of the HER2 receptor is a matter of major importance when dealing with breast cancer: an over-expression of HER2 is associated with aggressive clinical behaviour; moreover, HER2 targeted therapy results in a significant improvement in the overall survival rate. In this work, we employ a pipeline based on a cascade of deep neural network classifiers and multi-instance learning to detect the presence of HER2 from Haematoxylin-Eosin slides, which partly mimics the pathologist’s behaviour by first recognizing cancer and then evaluating HER2. Our results show that the proposed system presents a good overall effectiveness. Furthermore, the system design is prone to further improvements that can be easily deployed in order to increase the effectiveness score.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Breast cancer is the most frequently diagnosed cancer in woman. The correct identification of the HER2 receptor is a matter of major importance when dealing with breast cancer: an over-expression of HER2 is associated with aggressive clinical behaviour; moreover, HER2 targeted therapy results in a significant improvement in the overall survival rate. In this work, we employ a pipeline based on a cascade of deep neural network classifiers and multi-instance learning to detect the presence of HER2 from Haematoxylin–Eosin slides, which partly mimics the pathologist’s behaviour by first recognizing cancer and then evaluating HER2. Our results show that the proposed system presents a good overall effectiveness. Furthermore, the system design is prone to further improvements that can be easily deployed in order to increase the effectiveness score.
Roitero, Kevin; Carterette, Ben; Mehrotra, Rishabh; Lalmas, Mounia
Leveraging Behavioral Heterogeneity Across Markets for Cross-Market Training of Recommender Systems Book Chapter
In: Companion Proceedings of the Web Conference 2020, pp. 694–702, Association for Computing Machinery, New York, NY, USA, 2020, ISBN: 9781450370240.
@inbook{10.1145/3366424.3384362,
title = {Leveraging Behavioral Heterogeneity Across Markets for Cross-Market Training of Recommender Systems},
author = {Kevin Roitero and Ben Carterette and Rishabh Mehrotra and Mounia Lalmas},
url = {https://doi.org/10.1145/3366424.3384362},
isbn = {9781450370240},
year = {2020},
date = {2020-01-01},
booktitle = {Companion Proceedings of the Web Conference 2020},
pages = {694–702},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
abstract = {Modern recommender systems are optimised to deliver personalised recommendations to millions of users spread across different geographic regions exhibiting various forms of heterogeneity, including behavioural-, content- and trend specific heterogeneity. System designers often face the challenge of deploying either a single global model across all markets, or developing custom models for different markets. In this work, we focus on the specific case of music recommendation across 21 different markets, and consider the trade-off between developing global model versus market specific models. We begin by investigating behavioural differences across users of different markets, and motivate the need for considering market as an important factor when training models. We propose five different training styles, covering the entire spectrum of models: from a single global model to individual market specific models, and in the process, propose ways to identify and leverage users abroad, and data from similar markets. Based on a large scale experimentation with data for 100M users across 21 different markets, we present insights which highlight that markets play a key role, and describe models that leverage market specific data in serving personalised recommendations.},
keywords = {},
pubstate = {published},
tppubtype = {inbook}
}
Modern recommender systems are optimised to deliver personalised recommendations to millions of users spread across different geographic regions exhibiting various forms of heterogeneity, including behavioural-, content- and trend specific heterogeneity. System designers often face the challenge of deploying either a single global model across all markets, or developing custom models for different markets. In this work, we focus on the specific case of music recommendation across 21 different markets, and consider the trade-off between developing global model versus market specific models. We begin by investigating behavioural differences across users of different markets, and motivate the need for considering market as an important factor when training models. We propose five different training styles, covering the entire spectrum of models: from a single global model to individual market specific models, and in the process, propose ways to identify and leverage users abroad, and data from similar markets. Based on a large scale experimentation with data for 100M users across 21 different markets, we present insights which highlight that markets play a key role, and describe models that leverage market specific data in serving personalised recommendations.
Roitero, Kevin; Bozzato, Cristian; Mea, Vincenzo Della; Mizzaro, Stefano; Serra, Giuseppe
Twitter goes to the Doctor: Detecting Medical Tweets using Machine Learning and BERT. Proceedings Article
In: SIIRH@ ECIR, 2020.
@inproceedings{roitero2020twitter,
title = {Twitter goes to the Doctor: Detecting Medical Tweets using Machine Learning and BERT.},
author = {Kevin Roitero and Cristian Bozzato and Vincenzo Della Mea and Stefano Mizzaro and Giuseppe Serra},
year = {2020},
date = {2020-01-01},
booktitle = {SIIRH@ ECIR},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Amigó, Enrique; Gonzalo, Julio; Mizzaro, Stefano; Carrillo-de-Albornoz, Jorge
An Effectiveness Metric for Ordinal Classification: Formal Properties and Experimental Results Proceedings Article
In: Jurafsky, Dan; Chai, Joyce; Schluter, Natalie; Tetreault, Joel (Ed.): Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3938–3949, Association for Computational Linguistics, Online, 2020.
@inproceedings{amigo-etal-2020-effectiveness,
title = {An Effectiveness Metric for Ordinal Classification: Formal Properties and Experimental Results},
author = {Enrique Amigó and Julio Gonzalo and Stefano Mizzaro and Jorge Carrillo-de-Albornoz},
editor = {Dan Jurafsky and Joyce Chai and Natalie Schluter and Joel Tetreault},
url = {https://aclanthology.org/2020.acl-main.363},
doi = {10.18653/v1/2020.acl-main.363},
year = {2020},
date = {2020-01-01},
booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
pages = {3938–3949},
publisher = {Association for Computational Linguistics},
address = {Online},
abstract = {In Ordinal Classification tasks, items have to be assigned to classes that have a relative ordering, such as ``positive'', ``neutral'', ``negative'' in sentiment analysis. Remarkably, the most popular evaluation metrics for ordinal classification tasks either ignore relevant information (for instance, precision/recall on each of the classes ignores their relative ordering) or assume additional information (for instance, Mean Average Error assumes absolute distances between classes). In this paper we propose a new metric for Ordinal Classification, Closeness Evaluation Measure, that is rooted on Measurement Theory and Information Theory. Our theoretical analysis and experimental results over both synthetic data and data from NLP shared tasks indicate that the proposed metric captures quality aspects from different traditional tasks simultaneously. In addition, it generalizes some popular classification (nominal scale) and error minimization (interval scale) metrics, depending on the measurement scale in which it is instantiated.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
In Ordinal Classification tasks, items have to be assigned to classes that have a relative ordering, such as ``positive'', ``neutral'', ``negative'' in sentiment analysis. Remarkably, the most popular evaluation metrics for ordinal classification tasks either ignore relevant information (for instance, precision/recall on each of the classes ignores their relative ordering) or assume additional information (for instance, Mean Average Error assumes absolute distances between classes). In this paper we propose a new metric for Ordinal Classification, Closeness Evaluation Measure, that is rooted on Measurement Theory and Information Theory. Our theoretical analysis and experimental results over both synthetic data and data from NLP shared tasks indicate that the proposed metric captures quality aspects from different traditional tasks simultaneously. In addition, it generalizes some popular classification (nominal scale) and error minimization (interval scale) metrics, depending on the measurement scale in which it is instantiated.
Amigó, Enrique; Mizzaro, Stefano
On the nature of information access evaluation metrics: a unifying framework Journal Article
In: Information Retrieval Journal, vol. 23, no. 3, pp. 318–386, 2020, ISSN: 1573-7659.
@article{Amigo2020,
title = {On the nature of information access evaluation metrics: a unifying framework},
author = {Enrique Amigó and Stefano Mizzaro},
url = {https://doi.org/10.1007/s10791-020-09374-0},
doi = {10.1007/s10791-020-09374-0},
issn = {1573-7659},
year = {2020},
date = {2020-01-01},
journal = {Information Retrieval Journal},
volume = {23},
number = {3},
pages = {318–386},
abstract = {We provide a uniform, general, and complete formal account of evaluation metrics for ranking, classification, clustering, and other information access problems. We leverage concepts from measurement theory, such as scale types and permissible transformation functions, and we capture the nature of evaluation metrics in many tasks by two formal definitions, which lead to a distinction of two metric/tasks families, and provide a comprehensive classification of the tasks that have been proposed so far. We derive some theorems to analyze the suitability (or otherwise) of some common metrics. Within our model we can derive and explain the theoretical properties and drawbacks of the state of the art metrics for multiple tasks. The main contributions of this paper are that, differently from previous studies, the formalization is well grounded on a solid discipline, it is general as it can take into account most effectiveness metrics as well as most existing tasks, and it allows to derive important consequences on metrics and their limitations.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
We provide a uniform, general, and complete formal account of evaluation metrics for ranking, classification, clustering, and other information access problems. We leverage concepts from measurement theory, such as scale types and permissible transformation functions, and we capture the nature of evaluation metrics in many tasks by two formal definitions, which lead to a distinction of two metric/tasks families, and provide a comprehensive classification of the tasks that have been proposed so far. We derive some theorems to analyze the suitability (or otherwise) of some common metrics. Within our model we can derive and explain the theoretical properties and drawbacks of the state of the art metrics for multiple tasks. The main contributions of this paper are that, differently from previous studies, the formalization is well grounded on a solid discipline, it is general as it can take into account most effectiveness metrics as well as most existing tasks, and it allows to derive important consequences on metrics and their limitations.
Amigó, Enrique; Fang, Hui; Mizzaro, Stefano; Zhai, Chengxiang
Axiomatic thinking for information retrieval: introduction to special issue Journal Article
In: Information Retrieval Journal, vol. 23, no. 3, pp. 187–190, 2020, ISSN: 1573-7659.
@article{Amigo2020b,
title = {Axiomatic thinking for information retrieval: introduction to special issue},
author = {Enrique Amigó and Hui Fang and Stefano Mizzaro and Chengxiang Zhai},
url = {https://doi.org/10.1007/s10791-020-09376-y},
doi = {10.1007/s10791-020-09376-y},
issn = {1573-7659},
year = {2020},
date = {2020-01-01},
journal = {Information Retrieval Journal},
volume = {23},
number = {3},
pages = {187–190},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2019
Roitero, Kevin; Soprano, Michael; Mizzaro, Stefano
Bias and Fairness in Effectiveness Evaluation by Means of Network Analysis and Mixture Models Proceedings Article
In: CEUR Workshop Proceedings, pp. 2, CEUR-WS, 2019.
@inproceedings{bias-fairness-19,
title = {Bias and Fairness in Effectiveness Evaluation by Means of Network Analysis and Mixture Models},
author = {Kevin Roitero and Michael Soprano and Stefano Mizzaro},
year = {2019},
date = {2019-10-14},
booktitle = {CEUR Workshop Proceedings},
volume = {2441},
pages = {2},
publisher = {CEUR-WS},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Han, L; Roitero, K; Gadiraju, U; Sarasua, C; Checco, A; Maddalena, E; Demartini, G
The Impact of Task Abandonment in Crowdsourcing Journal Article
In: IEEE Transactions on Knowledge & Data Engineering, no. 01, pp. 1-1, 2019, ISSN: 1558-2191.
@article{8873609,
title = {The Impact of Task Abandonment in Crowdsourcing},
author = {L Han and K Roitero and U Gadiraju and C Sarasua and A Checco and E Maddalena and G Demartini},
doi = {10.1109/TKDE.2019.2948168},
issn = {1558-2191},
year = {2019},
date = {2019-10-01},
journal = {IEEE Transactions on Knowledge & Data Engineering},
number = {01},
pages = {1-1},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Kevin,; J, Culpepper Shane; Mark, Sanderson; Falk, Scholer; Roitero, Mizzaro Stefano
Fewer topics? A million topics? Both?! On topics subsets in test collections Journal Article
In: Information Retrieval Journal, 2019, ISSN: 1573-7659.
@article{Roitero2019ffew,
title = {Fewer topics? A million topics? Both?! On topics subsets in test collections},
author = {Kevin and Culpepper Shane J and Sanderson Mark and Scholer Falk and Mizzaro Stefano Roitero},
url = {https://doi.org/10.1007/s10791-019-09357-w},
doi = {10.1007/s10791-019-09357-w},
issn = {1573-7659},
year = {2019},
date = {2019-05-08},
journal = {Information Retrieval Journal},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Roitero, Kevin; Mizzaro, Stefano; Soprano, Michael
Bias and Fairness in Effectiveness Evaluation by Means of Network Analysis and Mixture Models Proceedings Article
In: Proceedings of the 10th Italian Information Retrieval Workshop, Padova, Italy, September 16-18, 2019., pp. 6–7, 2019.
@inproceedings{DBLP:conf/iir/RoiteroMS19,
title = {Bias and Fairness in Effectiveness Evaluation by Means of Network
Analysis and Mixture Models},
author = {Kevin Roitero and Stefano Mizzaro and Michael Soprano},
url = {http://ceur-ws.org/Vol-2441/paper4.pdf},
year = {2019},
date = {2019-01-01},
booktitle = {Proceedings of the 10th Italian Information Retrieval Workshop, Padova,
Italy, September 16-18, 2019.},
pages = {6--7},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Soprano, Michael; Roitero, Kevin; Mizzaro, Stefano
HITS Hits Readersourcing: Validating Peer Review Alternatives Using Network Analysis Proceedings Article
In: Proceedings of the 4th Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL 2019) co-located with the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019), Paris, France, July 25, 2019., pp. 70–82, 2019.
@inproceedings{DBLP:conf/sigir/SopranoRM19,
title = {HITS Hits Readersourcing: Validating Peer Review Alternatives Using
Network Analysis},
author = {Michael Soprano and Kevin Roitero and Stefano Mizzaro},
url = {http://ceur-ws.org/Vol-2414/paper7.pdf},
year = {2019},
date = {2019-01-01},
booktitle = {Proceedings of the 4th Joint Workshop on Bibliometric-enhanced Information
Retrieval and Natural Language Processing for Digital Libraries (BIRNDL 2019) co-located with the 42nd International ACM SIGIR Conference
on Research and Development in Information Retrieval (SIGIR 2019),
Paris, France, July 25, 2019.},
pages = {70--82},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Zampieri, Fabio; Roitero, Kevin; Culpepper, Shane J; Kurland, Oren; Mizzaro, Stefano
On Topic Difficulty in IR Evaluation: The Effect of Systems, Corpora, and System Components Proceedings Article
In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019., pp. 909–912, 2019.
@inproceedings{DBLP:conf/sigir/ZampieriRCKM19,
title = {On Topic Difficulty in IR Evaluation: The Effect of Systems, Corpora,
and System Components},
author = {Fabio Zampieri and Kevin Roitero and Shane J Culpepper and Oren Kurland and Stefano Mizzaro},
url = {https://doi.org/10.1145/3331184.3331279},
doi = {10.1145/3331184.3331279},
year = {2019},
date = {2019-01-01},
booktitle = {Proceedings of the 42nd International ACM SIGIR Conference on
Research and Development in Information Retrieval, SIGIR 2019, Paris,
France, July 21-25, 2019.},
pages = {909--912},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Han, Lei; Roitero, Kevin; Gadiraju, Ujwal; Sarasua, Cristina; Checco, Alessandro; Maddalena, Eddy; Demartini, Gianluca
All Those Wasted Hours: On Task Abandonment in Crowdsourcing Proceedings Article
In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, WSDM 2019, Melbourne, VIC, Australia, February 11-15, 2019, pp. 321–329, 2019.
@inproceedings{DBLP:conf/wsdm/HanRGSCMD19,
title = {All Those Wasted Hours: On Task Abandonment in Crowdsourcing},
author = {Lei Han and Kevin Roitero and Ujwal Gadiraju and Cristina Sarasua and Alessandro Checco and Eddy Maddalena and Gianluca Demartini},
url = {https://doi.org/10.1145/3289600.3291035},
doi = {10.1145/3289600.3291035},
year = {2019},
date = {2019-01-01},
booktitle = {Proceedings of the Twelfth ACM International Conference on Web Search
and Data Mining, WSDM 2019, Melbourne, VIC, Australia, February
11-15, 2019},
pages = {321--329},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Soprano, Michael; Mizzaro, Stefano
Crowdsourcing Peer Review: As We May Do Proceedings Article
In: Manghi, Paolo; Candela, Leonardo; Silvello, Gianmaria (Ed.): Digital Libraries: Supporting Open Science, pp. 259–273, Springer International Publishing, Cham, 2019, ISBN: 978-3-030-11226-4.
@inproceedings{10.1007/978-3-030-11226-4_21,
title = {Crowdsourcing Peer Review: As We May Do},
author = {Michael Soprano and Stefano Mizzaro},
editor = {Paolo Manghi and Leonardo Candela and Gianmaria Silvello},
isbn = {978-3-030-11226-4},
year = {2019},
date = {2019-01-01},
booktitle = {Digital Libraries: Supporting Open Science},
pages = {259--273},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {This paper describes Readersourcing 2.0, an ecosystem providing an implementation of the Readersourcing approach proposed by Mizzaro [10]. Readersourcing is proposed as an alternative to the standard peer review activity that aims to exploit the otherwise lost opinions of readers. Readersourcing 2.0 implements two different models based on the so-called codetermination algorithms. We describe the requirements, present the overall architecture, and show how the end-user can interact with the system. Readersourcing 2.0 will be used in the future to study also other topics, like the idea of shepherding the users to achieve a better quality of the reviews and the differences between a review activity carried out with a single-blind or a double-blind approach.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
This paper describes Readersourcing 2.0, an ecosystem providing an implementation of the Readersourcing approach proposed by Mizzaro [10]. Readersourcing is proposed as an alternative to the standard peer review activity that aims to exploit the otherwise lost opinions of readers. Readersourcing 2.0 implements two different models based on the so-called codetermination algorithms. We describe the requirements, present the overall architecture, and show how the end-user can interact with the system. Readersourcing 2.0 will be used in the future to study also other topics, like the idea of shepherding the users to achieve a better quality of the reviews and the differences between a review activity carried out with a single-blind or a double-blind approach.
Roitero, Kevin; Brunello, Andrea; Urbano, Julián; Mizzaro, Stefano
Towards Stochastic Simulations of Relevance Profiles Proceedings Article
In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 2217–2220, Association for Computing Machinery, Beijing, China, 2019, ISBN: 9781450369763.
@inproceedings{10.1145/3357384.3358123,
title = {Towards Stochastic Simulations of Relevance Profiles},
author = {Kevin Roitero and Andrea Brunello and Julián Urbano and Stefano Mizzaro},
url = {https://doi.org/10.1145/3357384.3358123},
doi = {10.1145/3357384.3358123},
isbn = {9781450369763},
year = {2019},
date = {2019-01-01},
booktitle = {Proceedings of the 28th ACM International Conference on Information and Knowledge Management},
pages = {2217–2220},
publisher = {Association for Computing Machinery},
address = {Beijing, China},
series = {CIKM '19},
abstract = {Recently proposed methods allow the generation of simulated scores representing the values of an effectiveness metric, but they do not investigate the generation of the actual lists of retrieved documents. In this paper we address this limitation: we present an approach that exploits an evolutionary algorithm and, given a metric score, creates a simulated relevance profile (i.e., a ranked list of relevance values) that produces that score. We show how the simulated relevance profiles are realistic under various analyses.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Recently proposed methods allow the generation of simulated scores representing the values of an effectiveness metric, but they do not investigate the generation of the actual lists of retrieved documents. In this paper we address this limitation: we present an approach that exploits an evolutionary algorithm and, given a metric score, creates a simulated relevance profile (i.e., a ranked list of relevance values) that produces that score. We show how the simulated relevance profiles are realistic under various analyses.
Han, Lei; Roitero, Kevin; Maddalena, Eddy; Mizzaro, Stefano; Demartini, Gianluca
On Transforming Relevance Scales Proceedings Article
In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 39–48, Association for Computing Machinery, Beijing, China, 2019, ISBN: 9781450369763.
@inproceedings{10.1145/3357384.3357988,
title = {On Transforming Relevance Scales},
author = {Lei Han and Kevin Roitero and Eddy Maddalena and Stefano Mizzaro and Gianluca Demartini},
url = {https://doi.org/10.1145/3357384.3357988},
doi = {10.1145/3357384.3357988},
isbn = {9781450369763},
year = {2019},
date = {2019-01-01},
booktitle = {Proceedings of the 28th ACM International Conference on Information and Knowledge Management},
pages = {39–48},
publisher = {Association for Computing Machinery},
address = {Beijing, China},
series = {CIKM '19},
abstract = {Information Retrieval (IR) researchers have often used existing IR evaluation collections and transformed the relevance scale in which judgments have been collected, e.g., to use metrics that assume binary judgments like Mean Average Precision. Such scale transformations are often arbitrary (e.g., 0,1 mapped to 0 and 2,3 mapped to 1) and it is assumed that they have no impact on the results of IR evaluation. Moreover, the use of crowdsourcing to collect relevance judgments has become a standard methodology. When designing the crowdsourcing relevance judgment task, one of the decision to be made is the how granular the relevance scale used to collect judgments should be. Such decision has then repercussions on the metrics used to measure IR system effectiveness. In this paper we look at the effect of scale transformations in a systematic way. We perform extensive experiments to study the transformation of judgments from fine-grained to coarse-grained. We use different relevance judgments expressed on different relevance scales and either expressed by expert annotators or collected by means of crowdsourcing. The objective is to understand the impact of relevance scale transformations on IR evaluation outcomes and to draw conclusions on how to best transform judgments into a different scale, when necessary.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Information Retrieval (IR) researchers have often used existing IR evaluation collections and transformed the relevance scale in which judgments have been collected, e.g., to use metrics that assume binary judgments like Mean Average Precision. Such scale transformations are often arbitrary (e.g., 0,1 mapped to 0 and 2,3 mapped to 1) and it is assumed that they have no impact on the results of IR evaluation. Moreover, the use of crowdsourcing to collect relevance judgments has become a standard methodology. When designing the crowdsourcing relevance judgment task, one of the decision to be made is the how granular the relevance scale used to collect judgments should be. Such decision has then repercussions on the metrics used to measure IR system effectiveness. In this paper we look at the effect of scale transformations in a systematic way. We perform extensive experiments to study the transformation of judgments from fine-grained to coarse-grained. We use different relevance judgments expressed on different relevance scales and either expressed by expert annotators or collected by means of crowdsourcing. The objective is to understand the impact of relevance scale transformations on IR evaluation outcomes and to draw conclusions on how to best transform judgments into a different scale, when necessary.
Soprano, Kevin Roitero
Bias and Fairness in Effectiveness Evaluation by Means of Network Analysis and Mixture Models Proceedings Article
In: Agosti, Maristella; Buccio, Emanuele Di; Melucci, Massimo; Mizzaro, Stefano; Pasi, Gabriella; Silvestri, Fabrizio (Ed.): Proceedings of the 10th Italian Information Retrieval Workshop, Padova, Italy, September 16-18, 2019, pp. 6–7, CEUR-WS.org, 2019.
@inproceedings{workshop-paper-iir2019,
title = {Bias and Fairness in Effectiveness Evaluation by Means of Network Analysis and Mixture Models},
author = {Kevin Roitero Soprano},
editor = {Maristella Agosti and Emanuele Di Buccio and Massimo Melucci and Stefano Mizzaro and Gabriella Pasi and Fabrizio Silvestri},
url = {http://ceur-ws.org/Vol-2441/paper4.pdf},
year = {2019},
date = {2019-01-01},
booktitle = {Proceedings of the 10th Italian Information Retrieval Workshop, Padova, Italy, September 16-18, 2019},
volume = {2441},
pages = {6--7},
publisher = {CEUR-WS.org},
series = {CEUR Workshop Proceedings},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Soprano, Michael; Roitero, Kevin; Mizzaro, Stefano
HITS Hits Readersourcing: Validating Peer Review Alternatives Using Network Analysis Proceedings Article
In: Chandrasekaran, Muthu Kumar; Mayr, Philipp (Ed.): Proceedings of the 4th Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL 2019) co-located with the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019), Paris, France, July 25, 2019, pp. 70–82, CEUR-WS.org, 2019.
@inproceedings{workshop-paper-birnld2019,
title = {HITS Hits Readersourcing: Validating Peer Review Alternatives Using Network Analysis},
author = {Michael Soprano and Kevin Roitero and Stefano Mizzaro},
editor = {Muthu Kumar Chandrasekaran and Philipp Mayr},
url = {http://ceur-ws.org/Vol-2414/paper7.pdf},
year = {2019},
date = {2019-01-01},
booktitle = {Proceedings of the 4th Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL 2019) co-located with the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019), Paris, France, July 25, 2019},
volume = {2414},
pages = {70--82},
publisher = {CEUR-WS.org},
series = {CEUR Workshop Proceedings},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Soprano, Michael; Mizzaro, Stefano
Crowdsourcing Peer Review: As We May Do Proceedings Article
In: Manghi, Paolo; Candela, Leonardo; Silvello, Gianmaria (Ed.): Digital Libraries: Supporting Open Science - 15th Italian Research Conference on Digital Libraries, IRCDL 2019, Pisa, Italy, January 31 - February 1, 2019, Proceedings, pp. 259–273, Springer, 2019.
@inproceedings{conference-paper-ircdl2019,
title = {Crowdsourcing Peer Review: As We May Do},
author = {Michael Soprano and Stefano Mizzaro},
editor = {Paolo Manghi and Leonardo Candela and Gianmaria Silvello},
url = {https://doi.org/10.1007/978-3-030-11226-4_21},
doi = {10.1007/978-3-030-11226-4_21},
year = {2019},
date = {2019-01-01},
booktitle = {Digital Libraries: Supporting Open Science - 15th Italian Research Conference on Digital Libraries, IRCDL 2019, Pisa, Italy, January 31 - February 1, 2019, Proceedings},
volume = {988},
pages = {259--273},
publisher = {Springer},
series = {Communications in Computer and Information Science},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Soprano, Michael; Mizzaro, Stefano
Crowdsourcing Peer Review in the Digital Humanities? Proceedings Article
In: Book of Abstracts, 8th AIUCD Conference 2019 – Pedagogy, Teaching, and Research in the Age of Digital Humanities, page 251. Udine, Italy, 2019, pp. 251, Udine, Italy, 2019, ISBN: 9788894253535.
@inproceedings{conference-paper-aiudc2019,
title = {Crowdsourcing Peer Review in the Digital Humanities?},
author = {Michael Soprano and Stefano Mizzaro},
url = {http://aiucd2019.uniud.it/wp-content/uploads/2020/03/AIUCD2019-BoA_DEF.pdf},
isbn = {9788894253535},
year = {2019},
date = {2019-01-01},
booktitle = {Book of Abstracts, 8th AIUCD Conference 2019 – Pedagogy, Teaching, and Research in the Age of Digital Humanities, page 251. Udine, Italy, 2019},
pages = {251},
address = {Udine, Italy},
series = {AIUCD '19},
abstract = {We propose an alternative approach to the standard peer review activity that aims to exploit the otherwise lost opinions of readers of publications which is called Readersourcing, originally proposed by Mizzaro [1]. Such an approach can be formalized by means of different models which share the same general principles. These models should be able to define a way, to measure the overall quality of a publication as well the reputation of a reader as an assessor; moreover, from these measures it should be possible to derive the reputation of a scholar as an author. We describe an ecosystem called Readersourcing 2.0 which provides an implementation for two Readersourcing models [2, 3] by outlining its goals and requirements. Readersourcing 2.0 will be used in the future to gather fresh data to analyze and validate.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
We propose an alternative approach to the standard peer review activity that aims to exploit the otherwise lost opinions of readers of publications which is called Readersourcing, originally proposed by Mizzaro [1]. Such an approach can be formalized by means of different models which share the same general principles. These models should be able to define a way, to measure the overall quality of a publication as well the reputation of a reader as an assessor; moreover, from these measures it should be possible to derive the reputation of a scholar as an author. We describe an ecosystem called Readersourcing 2.0 which provides an implementation for two Readersourcing models [2, 3] by outlining its goals and requirements. Readersourcing 2.0 will be used in the future to gather fresh data to analyze and validate.
2018
Roitero, Kevin; Spina, Damiano; Demartini, Gianluca; Mizzaro, Stefano
How Many Truth Levels? Six? One Hundred? Even more? Validating Truthfulness of Statements via Crowdsourcing Proceedings Article
In: To appear in RDSM 2018 : Workshop on Rumours and Deception in Social Media., 2018.
@inproceedings{toapp:howmany,
title = {How Many Truth Levels? Six? One Hundred? Even more? Validating Truthfulness of Statements via Crowdsourcing},
author = {Kevin Roitero and Damiano Spina and Gianluca Demartini and Stefano Mizzaro},
year = {2018},
date = {2018-01-01},
booktitle = {To appear in RDSM 2018 : Workshop on Rumours and Deception in Social Media.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Roitero, Kevin; Soprano, Michael; Mizzaro, Stefano
Effectiveness Evaluation with a Subset of Topics: A Practical Approach Proceedings Article
In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1145–1148, Association for Computing Machinery, Ann Arbor, MI, USA, 2018, ISBN: 9781450356572.
@inproceedings{10.1145/3209978.3210108,
title = {Effectiveness Evaluation with a Subset of Topics: A Practical Approach},
author = {Kevin Roitero and Michael Soprano and Stefano Mizzaro},
url = {https://doi.org/10.1145/3209978.3210108},
doi = {10.1145/3209978.3210108},
isbn = {9781450356572},
year = {2018},
date = {2018-01-01},
booktitle = {The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval},
pages = {1145–1148},
publisher = {Association for Computing Machinery},
address = {Ann Arbor, MI, USA},
series = {SIGIR '18},
abstract = {Several researchers have proposed to reduce the number of topics used in TREC-like initiatives. One research direction that has been pursued is what is the optimal topic subset of a given cardinality that evaluates the systems/runs in the most accurate way. Such a research direction has been so far mainly theoretical, with almost no indication on how to select the few good topics in practice. We propose such a practical criterion for topic selection: we rely on the methods for automatic system evaluation without relevance judgments, and by running some experiments on several TREC collections we show that the topics selected on the basis of those evaluations are indeed more informative than random topics.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Several researchers have proposed to reduce the number of topics used in TREC-like initiatives. One research direction that has been pursued is what is the optimal topic subset of a given cardinality that evaluates the systems/runs in the most accurate way. Such a research direction has been so far mainly theoretical, with almost no indication on how to select the few good topics in practice. We propose such a practical criterion for topic selection: we rely on the methods for automatic system evaluation without relevance judgments, and by running some experiments on several TREC collections we show that the topics selected on the basis of those evaluations are indeed more informative than random topics.
Roitero, Kevin; Maddalena, Eddy; Ponte, Yannick; Mizzaro, Stefano
IRevalOO: An Object Oriented Framework for Retrieval Evaluation Proceedings Article
In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 913–916, Association for Computing Machinery, Ann Arbor, MI, USA, 2018, ISBN: 9781450356572.
@inproceedings{10.1145/3209978.3210084,
title = {IRevalOO: An Object Oriented Framework for Retrieval Evaluation},
author = {Kevin Roitero and Eddy Maddalena and Yannick Ponte and Stefano Mizzaro},
url = {https://doi.org/10.1145/3209978.3210084},
doi = {10.1145/3209978.3210084},
isbn = {9781450356572},
year = {2018},
date = {2018-01-01},
booktitle = {The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval},
pages = {913–916},
publisher = {Association for Computing Machinery},
address = {Ann Arbor, MI, USA},
series = {SIGIR '18},
abstract = {We propose IRevalOO, a flexible Object Oriented framework that (i) can be used as-is as a replacement of the widely adopted trec_eval software, and (ii) can be easily extended (or "instantiated'', in framework terminology) to implement different scenarios of test collection based retrieval evaluation. Instances of IRevalOO can provide a usable and convenient alternative to the state-of-the-art software commonly used by different initiatives (TREC, NTCIR, CLEF, FIRE, etc.). Also, those instances can be easily adapted to satisfy future customization needs of researchers, as: implementing and experimenting with new metrics, even based on new notions of relevance; using different formats for system output and "qrels''; and in general visualizing, comparing, and managing retrieval evaluation results.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
We propose IRevalOO, a flexible Object Oriented framework that (i) can be used as-is as a replacement of the widely adopted trec_eval software, and (ii) can be easily extended (or "instantiated'', in framework terminology) to implement different scenarios of test collection based retrieval evaluation. Instances of IRevalOO can provide a usable and convenient alternative to the state-of-the-art software commonly used by different initiatives (TREC, NTCIR, CLEF, FIRE, etc.). Also, those instances can be easily adapted to satisfy future customization needs of researchers, as: implementing and experimenting with new metrics, even based on new notions of relevance; using different formats for system output and "qrels''; and in general visualizing, comparing, and managing retrieval evaluation results.
Roitero, Kevin
CHEERS: CHeap & Engineered Evaluation of Retrieval Systems Proceedings Article
In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1467, Association for Computing Machinery, Ann Arbor, MI, USA, 2018, ISBN: 9781450356572.
@inproceedings{10.1145/3209978.3210229,
title = {CHEERS: CHeap & Engineered Evaluation of Retrieval Systems},
author = {Kevin Roitero},
url = {https://doi.org/10.1145/3209978.3210229},
doi = {10.1145/3209978.3210229},
isbn = {9781450356572},
year = {2018},
date = {2018-01-01},
booktitle = {The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval},
pages = {1467},
publisher = {Association for Computing Machinery},
address = {Ann Arbor, MI, USA},
series = {SIGIR '18},
abstract = {In test collection based evaluation of retrieval effectiveness, many research investigated different directions for an economical and a semi-automatic evaluation of retrieval systems. Although several methods have been proposed and experimentally evaluated, their accuracy seems still limited. In this paper we present our proposal for a more engineered approach to information retrieval evaluation.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
In test collection based evaluation of retrieval effectiveness, many research investigated different directions for an economical and a semi-automatic evaluation of retrieval systems. Although several methods have been proposed and experimentally evaluated, their accuracy seems still limited. In this paper we present our proposal for a more engineered approach to information retrieval evaluation.
Roitero, Kevin; Maddalena, Eddy; Demartini, Gianluca; Mizzaro, Stefano
On Fine-Grained Relevance Scales Proceedings Article
In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 675–684, Association for Computing Machinery, Ann Arbor, MI, USA, 2018, ISBN: 9781450356572.
@inproceedings{10.1145/3209978.3210052,
title = {On Fine-Grained Relevance Scales},
author = {Kevin Roitero and Eddy Maddalena and Gianluca Demartini and Stefano Mizzaro},
url = {https://doi.org/10.1145/3209978.3210052},
doi = {10.1145/3209978.3210052},
isbn = {9781450356572},
year = {2018},
date = {2018-01-01},
urldate = {2018-01-01},
booktitle = {The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval},
pages = {675–684},
publisher = {Association for Computing Machinery},
address = {Ann Arbor, MI, USA},
series = {SIGIR '18},
abstract = {In Information Retrieval evaluation, the classical approach of adopting binary relevance judgments has been replaced by multi-level relevance judgments and by gain-based metrics leveraging such multi-level judgment scales. Recent work has also proposed and evaluated unbounded relevance scales by means of Magnitude Estimation (ME) and compared them with multi-level scales. While ME brings advantages like the ability for assessors to always judge the next document as having higher or lower relevance than any of the documents they have judged so far, it also comes with some drawbacks. For example, it is not a natural approach for human assessors to judge items as they are used to do on the Web (e.g., 5-star rating). In this work, we propose and experimentally evaluate a bounded and fine-grained relevance scale having many of the advantages and dealing with some of the issues of ME. We collect relevance judgments over a 100-level relevance scale (S100) by means of a large-scale crowdsourcing experiment and compare the results with other relevance scales (binary, 4-level, and ME) showing the benefit of fine-grained scales over both coarse-grained and unbounded scales as well as highlighting some new results on ME. Our results show that S100 maintains the flexibility of unbounded scales like ME in providing assessors with ample choice when judging document relevance (i.e., assessors can fit relevance judgments in between of previously given judgments). It also allows assessors to judge on a more familiar scale (e.g., on 10 levels) and to perform efficiently since the very first judging task.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
In Information Retrieval evaluation, the classical approach of adopting binary relevance judgments has been replaced by multi-level relevance judgments and by gain-based metrics leveraging such multi-level judgment scales. Recent work has also proposed and evaluated unbounded relevance scales by means of Magnitude Estimation (ME) and compared them with multi-level scales. While ME brings advantages like the ability for assessors to always judge the next document as having higher or lower relevance than any of the documents they have judged so far, it also comes with some drawbacks. For example, it is not a natural approach for human assessors to judge items as they are used to do on the Web (e.g., 5-star rating). In this work, we propose and experimentally evaluate a bounded and fine-grained relevance scale having many of the advantages and dealing with some of the issues of ME. We collect relevance judgments over a 100-level relevance scale (S100) by means of a large-scale crowdsourcing experiment and compare the results with other relevance scales (binary, 4-level, and ME) showing the benefit of fine-grained scales over both coarse-grained and unbounded scales as well as highlighting some new results on ME. Our results show that S100 maintains the flexibility of unbounded scales like ME in providing assessors with ample choice when judging document relevance (i.e., assessors can fit relevance judgments in between of previously given judgments). It also allows assessors to judge on a more familiar scale (e.g., on 10 levels) and to perform efficiently since the very first judging task.
Mizzaro, Stefano; Mothe, Josiane; Roitero, Kevin; Ullah, Md Zia
Query Performance Prediction and Effectiveness Evaluation Without Relevance Judgments: Two Sides of the Same Coin Proceedings Article
In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1233–1236, Association for Computing Machinery, Ann Arbor, MI, USA, 2018, ISBN: 9781450356572.
@inproceedings{10.1145/3209978.3210146,
title = {Query Performance Prediction and Effectiveness Evaluation Without Relevance Judgments: Two Sides of the Same Coin},
author = {Stefano Mizzaro and Josiane Mothe and Kevin Roitero and Md Zia Ullah},
url = {https://doi.org/10.1145/3209978.3210146},
doi = {10.1145/3209978.3210146},
isbn = {9781450356572},
year = {2018},
date = {2018-01-01},
booktitle = {The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval},
pages = {1233–1236},
publisher = {Association for Computing Machinery},
address = {Ann Arbor, MI, USA},
series = {SIGIR '18},
abstract = {Some methods have been developed for automatic effectiveness evaluation without relevance judgments. We propose to use those methods, and their combination based on a machine learning approach, for query performance prediction. Moreover, since predicting average precision as it is usually done in query performance prediction literature is sensitive to the reference system that is chosen, we focus on predicting the average of average precision values over several systems. Results of an extensive experimental evaluation on ten TREC collections show that our proposed methods outperform state-of-the-art query performance predictors.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Some methods have been developed for automatic effectiveness evaluation without relevance judgments. We propose to use those methods, and their combination based on a machine learning approach, for query performance prediction. Moreover, since predicting average precision as it is usually done in query performance prediction literature is sensitive to the reference system that is chosen, we focus on predicting the average of average precision values over several systems. Results of an extensive experimental evaluation on ten TREC collections show that our proposed methods outperform state-of-the-art query performance predictors.
Amigó, Enrique; Giner, Fernando; Mizzaro, Stefano; Spina, Damiano
A Formal Account of Effectiveness Evaluation and Ranking Fusion Proceedings Article
In: Proceedings of the 2018 ACM SIGIR International Conference on Theory of Information Retrieval, pp. 123–130, Association for Computing Machinery, Tianjin, China, 2018, ISBN: 9781450356565.
@inproceedings{10.1145/3234944.3234958,
title = {A Formal Account of Effectiveness Evaluation and Ranking Fusion},
author = {Enrique Amigó and Fernando Giner and Stefano Mizzaro and Damiano Spina},
url = {https://doi.org/10.1145/3234944.3234958},
doi = {10.1145/3234944.3234958},
isbn = {9781450356565},
year = {2018},
date = {2018-01-01},
booktitle = {Proceedings of the 2018 ACM SIGIR International Conference on Theory of Information Retrieval},
pages = {123–130},
publisher = {Association for Computing Machinery},
address = {Tianjin, China},
series = {ICTIR '18},
abstract = {This paper proposes a theoretical framework which models the information provided by retrieval systems in terms of Information Theory. The proposed framework allows to formalize: (i) system effectiveness as an information theoretic similarity between system outputs and human assessments, and (ii) ranking fusion as an information quantity measure. As a result, the proposed effectiveness metric improves popular metrics in terms of formal constraints. In addition, our empirical experiments suggest that it captures quality aspects from traditional metrics, while the reverse is not true. Our work also advances the understanding of theoretical foundations of the empirically known phenomenon of effectiveness increase when combining retrieval system outputs in an unsupervised manner.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
This paper proposes a theoretical framework which models the information provided by retrieval systems in terms of Information Theory. The proposed framework allows to formalize: (i) system effectiveness as an information theoretic similarity between system outputs and human assessments, and (ii) ranking fusion as an information quantity measure. As a result, the proposed effectiveness metric improves popular metrics in terms of formal constraints. In addition, our empirical experiments suggest that it captures quality aspects from traditional metrics, while the reverse is not true. Our work also advances the understanding of theoretical foundations of the empirically known phenomenon of effectiveness increase when combining retrieval system outputs in an unsupervised manner.
Amigó, Enrique; Fang, Hui; Mizzaro, Stefano; Zhai, ChengXiang
Are We on the Right Track? An Examination of Information Retrieval Methodologies Proceedings Article
In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 997–1000, Association for Computing Machinery, Ann Arbor, MI, USA, 2018, ISBN: 9781450356572.
@inproceedings{10.1145/3209978.3210131b,
title = {Are We on the Right Track? An Examination of Information Retrieval Methodologies},
author = {Enrique Amigó and Hui Fang and Stefano Mizzaro and ChengXiang Zhai},
url = {https://doi.org/10.1145/3209978.3210131},
doi = {10.1145/3209978.3210131},
isbn = {9781450356572},
year = {2018},
date = {2018-01-01},
booktitle = {The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval},
pages = {997–1000},
publisher = {Association for Computing Machinery},
address = {Ann Arbor, MI, USA},
series = {SIGIR '18},
abstract = {The unpredictability of user behavior and the need for effectiveness make it difficult to define a suitable research methodology for Information Retrieval (IR). In order to tackle this challenge, we categorize existing IR methodologies along two dimensions: (1) empirical vs. theoretical, and (2) top-down vs. bottom-up. The strengths and drawbacks of the resulting categories are characterized according to 6 desirable aspects. The analysis suggests that different methodologies are complementary and therefore, equally necessary. The categorization of the 167 full papers published in the last SIGIR (2016 and 2017) and ICTIR (2017) conferences suggest that most of existing work is empirical bottom-up, suggesting lack of some desirable aspects. With the hope of improving IR research practice, we propose a general methodology for IR that integrates the strengths of existing research methods.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
The unpredictability of user behavior and the need for effectiveness make it difficult to define a suitable research methodology for Information Retrieval (IR). In order to tackle this challenge, we categorize existing IR methodologies along two dimensions: (1) empirical vs. theoretical, and (2) top-down vs. bottom-up. The strengths and drawbacks of the resulting categories are characterized according to 6 desirable aspects. The analysis suggests that different methodologies are complementary and therefore, equally necessary. The categorization of the 167 full papers published in the last SIGIR (2016 and 2017) and ICTIR (2017) conferences suggest that most of existing work is empirical bottom-up, suggesting lack of some desirable aspects. With the hope of improving IR research practice, we propose a general methodology for IR that integrates the strengths of existing research methods.
Amigó, Enrique; Fang, Hui; Mizzaro, Stefano; Zhai, ChengXiang
Report on the SIGIR 2017 Workshop on Axiomatic Thinking for Information Retrieval and Related Tasks (ATIR) Journal Article
In: SIGIR Forum, vol. 51, no. 3, pp. 99–106, 2018, ISSN: 0163-5840.
@article{10.1145/3190580.3190596,
title = {Report on the SIGIR 2017 Workshop on Axiomatic Thinking for Information Retrieval and Related Tasks (ATIR)},
author = {Enrique Amigó and Hui Fang and Stefano Mizzaro and ChengXiang Zhai},
url = {https://doi.org/10.1145/3190580.3190596},
doi = {10.1145/3190580.3190596},
issn = {0163-5840},
year = {2018},
date = {2018-01-01},
journal = {SIGIR Forum},
volume = {51},
number = {3},
pages = {99–106},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
abstract = {The SIGIR 2017 workshop on Axiomatic Thinking for Information Retrieval and Related Tasks took place on August 11, 2017 in Tokyo, Japan. The workshop aimed to help foster collaboration of researchers working on different perspectives of axiomatic thinking and encourage discussion and research on general methodological issues related to applying axiomatic thinking to information retrieval and related tasks. The program consisted of one keynote talk, four research presentations and a final panel discussion. This report outlines the events of the workshop and summarizes the major outcomes. More information about the workshop is available at https://www.eecis.udel.edu/~hfang/ATIR.html.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
The SIGIR 2017 workshop on Axiomatic Thinking for Information Retrieval and Related Tasks took place on August 11, 2017 in Tokyo, Japan. The workshop aimed to help foster collaboration of researchers working on different perspectives of axiomatic thinking and encourage discussion and research on general methodological issues related to applying axiomatic thinking to information retrieval and related tasks. The program consisted of one keynote talk, four research presentations and a final panel discussion. This report outlines the events of the workshop and summarizes the major outcomes. More information about the workshop is available at https://www.eecis.udel.edu/~hfang/ATIR.html.
Roitero, Kevin; Passon, Marco; Serra, Giuseppe; Mizzaro, Stefano
Reproduce. Generalize. Extend. On Information Retrieval Evaluation without Relevance Judgments Journal Article
In: J. Data and Information Quality, vol. 10, no. 3, 2018, ISSN: 1936-1955.
@article{10.1145/3241064,
title = {Reproduce. Generalize. Extend. On Information Retrieval Evaluation without Relevance Judgments},
author = {Kevin Roitero and Marco Passon and Giuseppe Serra and Stefano Mizzaro},
url = {https://doi.org/10.1145/3241064},
doi = {10.1145/3241064},
issn = {1936-1955},
year = {2018},
date = {2018-01-01},
journal = {J. Data and Information Quality},
volume = {10},
number = {3},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
abstract = {The evaluation of retrieval effectiveness by means of test collections is a commonly used methodology in the information retrieval field. Some researchers have addressed the quite fascinating research question of whether it is possible to evaluate effectiveness completely automatically, without human relevance assessments. Since human relevance assessment is one of the main costs of building a test collection, both in human time and money resources, this rather ambitious goal would have a practical impact. In this article, we reproduce the main results on evaluating information retrieval systems without relevance judgments; furthermore, we generalize such previous work to analyze the effect of test collections, evaluation metrics, and pool depth. We also expand the idea to semi-automatic evaluation and estimation of topic difficulty. Our results show that (i) previous work is overall reproducible, although some specific results are not; (ii) collection, metric, and pool depth impact the automatic evaluation of systems, which is anyway accurate in several cases; (iii) semi-automatic evaluation is an effective methodology; and (iv) automatic evaluation can (to some extent) be used to predict topic difficulty.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
The evaluation of retrieval effectiveness by means of test collections is a commonly used methodology in the information retrieval field. Some researchers have addressed the quite fascinating research question of whether it is possible to evaluate effectiveness completely automatically, without human relevance assessments. Since human relevance assessment is one of the main costs of building a test collection, both in human time and money resources, this rather ambitious goal would have a practical impact. In this article, we reproduce the main results on evaluating information retrieval systems without relevance judgments; furthermore, we generalize such previous work to analyze the effect of test collections, evaluation metrics, and pool depth. We also expand the idea to semi-automatic evaluation and estimation of topic difficulty. Our results show that (i) previous work is overall reproducible, although some specific results are not; (ii) collection, metric, and pool depth impact the automatic evaluation of systems, which is anyway accurate in several cases; (iii) semi-automatic evaluation is an effective methodology; and (iv) automatic evaluation can (to some extent) be used to predict topic difficulty.
Roitero, Kevin; Soprano, Michael; Mizzaro, Stefano
Effectiveness Evaluation with a Subset of Topics: A Practical Approach Proceedings Article
In: Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2018). Ann Arbor Michigan, U.S.A, July 8-12, 2018. Conference Rank: GGS A++, Core A*, pp. 1145–1148, Association for Computing Machinery, Ann Arbor, MI, USA, 2018, ISBN: 9781450356572.
@inproceedings{conference-paper-sigir2018,
title = {Effectiveness Evaluation with a Subset of Topics: A Practical Approach},
author = {Kevin Roitero and Michael Soprano and Stefano Mizzaro},
url = {https://doi.org/10.1145/3209978.3210108},
doi = {10.1145/3209978.3210108},
isbn = {9781450356572},
year = {2018},
date = {2018-01-01},
booktitle = {Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2018). Ann Arbor Michigan, U.S.A, July 8-12, 2018. Conference Rank: GGS A++, Core A*},
pages = {1145–1148},
publisher = {Association for Computing Machinery},
address = {Ann Arbor, MI, USA},
series = {SIGIR '18},
abstract = {Several researchers have proposed to reduce the number of topics used in TREC-like initiatives. One research direction that has been pursued is what is the optimal topic subset of a given cardinality that evaluates the systems/runs in the most accurate way. Such a research direction has been so far mainly theoretical, with almost no indication on how to select the few good topics in practice. We propose such a practical criterion for topic selection: we rely on the methods for automatic system evaluation without relevance judgments, and by running some experiments on several TREC collections we show that the topics selected on the basis of those evaluations are indeed more informative than random topics.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Several researchers have proposed to reduce the number of topics used in TREC-like initiatives. One research direction that has been pursued is what is the optimal topic subset of a given cardinality that evaluates the systems/runs in the most accurate way. Such a research direction has been so far mainly theoretical, with almost no indication on how to select the few good topics in practice. We propose such a practical criterion for topic selection: we rely on the methods for automatic system evaluation without relevance judgments, and by running some experiments on several TREC collections we show that the topics selected on the basis of those evaluations are indeed more informative than random topics.