TY - JOUR T1 - Dropping the Baton?: Understanding Errors and Bottlenecks in a Crowdsourced Sensemaking Pipeline JF - Proc. ACM Hum.-Comput. Interact. Y1 - 2019 A1 - Li, Tianyi A1 - Manns, Chandler J. A1 - North, Chris A1 - Luther, Kurt KW - crowdsourcing KW - Intelligence Analysis KW - investigations KW - mysteries KW - sensemaking KW - text analytics AB - Crowdsourced sensemaking has shown great potential for enabling scalable analysis of complex data sets, from planning trips, to designing products, to solving crimes. Yet, most crowd sensemaking approaches still require expert intervention because of worker errors and bottlenecks that would otherwise harm the output quality. Mitigating these errors and bottlenecks would significantly reduce the burden on experts, yet little is known about the types of mistakes crowds make with sensemaking micro-tasks and how they propagate in the sensemaking loop. In this paper, we conduct a series of studies with 325 crowd workers using a crowd sensemaking pipeline to solve a fictional terrorist plot, focusing on understanding why errors and bottlenecks happen and how they propagate. We classify types of crowd errors and show how the amount and quality of input data influence worker performance. We conclude by suggesting design recommendations for integrated crowdsourcing systems and speculating how a complementary top-down path of the pipeline could refine crowd analyses. PB - ACM CY - New York, NY, USA VL - 3 UR - http://doi.acm.org/10.1145/3359238 ER -