TY - GEN
T1 - Experiment-driven improvements in Human-in-the-loop Machine Learning Annotation via significance-based A/B testing
AU - Alfaro-Flores, Rafael
AU - Salas-Bonilla, José
AU - Juillard, Loic
AU - Esquivel-Rodríguez, Juan
N1 - Publisher Copyright:
©2021 IEEE
PY - 2021
Y1 - 2021
N2 - We present an end-to-end experimentation framework to improve the human annotation of data sets used in the training process of Machine Learning models. It covers the instrumentation of the annotation tool, the aggregation of metrics that highlight usage patterns and hypothesis-testing tools that enable the comparison of experimental groups, to decide whether improvements in the annotation process significantly impact the overall results. We show the potential of the protocol using two real-life annotation use cases.
AB - We present an end-to-end experimentation framework to improve the human annotation of data sets used in the training process of Machine Learning models. It covers the instrumentation of the annotation tool, the aggregation of metrics that highlight usage patterns and hypothesis-testing tools that enable the comparison of experimental groups, to decide whether improvements in the annotation process significantly impact the overall results. We show the potential of the protocol using two real-life annotation use cases.
KW - Data warehouse and repository
KW - Experimental Design
KW - Human performance
KW - Machine Learning
KW - Statistical Methods
UR - http://www.scopus.com/inward/record.url?scp=85123835546&partnerID=8YFLogxK
U2 - 10.1109/CLEI53233.2021.9639977
DO - 10.1109/CLEI53233.2021.9639977
M3 - Contribución a la conferencia
AN - SCOPUS:85123835546
T3 - Proceedings - 2021 47th Latin American Computing Conference, CLEI 2021
BT - Proceedings - 2021 47th Latin American Computing Conference, CLEI 2021
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 47th Latin American Computing Conference, CLEI 2021
Y2 - 25 October 2021 through 29 October 2021
ER -