Dropping the Baton?: Understanding Errors and Bottlenecks in a Crowdsourced Sensemaking Pipeline

TitleDropping the Baton?: Understanding Errors and Bottlenecks in a Crowdsourced Sensemaking Pipeline
Publication TypeJournal Article
Year of Publication2019
AuthorsLi, T, Manns, CJ, North, C, Luther, K
JournalProc. ACM Hum.-Comput. Interact.
Volume3
Pagination136:1–136:26
Date Published11/2019
ISSN2573-0142
Keywordscrowdsourcing, Intelligence Analysis, investigations, mysteries, sensemaking, text analytics
Abstract

Crowdsourced sensemaking has shown great potential for enabling scalable analysis of complex data sets, from planning trips, to designing products, to solving crimes. Yet, most crowd sensemaking approaches still require expert intervention because of worker errors and bottlenecks that would otherwise harm the output quality. Mitigating these errors and bottlenecks would significantly reduce the burden on experts, yet little is known about the types of mistakes crowds make with sensemaking micro-tasks and how they propagate in the sensemaking loop. In this paper, we conduct a series of studies with 325 crowd workers using a crowd sensemaking pipeline to solve a fictional terrorist plot, focusing on understanding why errors and bottlenecks happen and how they propagate. We classify types of crowd errors and show how the amount and quality of input data influence worker performance. We conclude by suggesting design recommendations for integrated crowdsourcing systems and speculating how a complementary top-down path of the pipeline could refine crowd analyses.

URLhttp://doi.acm.org/10.1145/3359238
DOI10.1145/3359238
File Upload: