Herde, M.; Huseljic, D. & Sick, B. (2023), 'Multi-annotator Deep Learning: A Probabilistic Framework for Classification', Transactions on Machine Learning Research .
[BibTeX] [Endnote]

Solving complex classification tasks using deep neural networks typically requires large amounts of annotated data. However, corresponding class labels are noisy when provided by error-prone annotators, e.g., crowdworkers. Training standard deep neural networks leads to subpar performances in such multi-annotator supervised learning settings. We address this issue by presenting a probabilistic training framework named multi-annotator deep learning (MaDL). A downstream ground truth and an annotator performance model are jointly trained in an end-to-end learning approach. The ground truth model learns to predict instances' true class labels, while the annotator performance model infers probabilistic estimates of annotators' performances. A modular network architecture enables us to make varying assumptions regarding annotators' performances, e.g., an optional class or instance dependency. Further, we learn annotator embeddings to estimate annotators' densities within a latent space as proxies of their potentially correlated annotations. Together with a weighted loss function, we improve the learning from correlated annotation patterns. In a comprehensive evaluation, we examine three research questions about multi-annotator supervised learning. Our findings show MaDL's state-of-the-art performance and robustness against many correlated, spamming annotators.

Kaur, J.; JafariAsbagh, M.; Radicchi, F. & Menczer, F. (2014), Scholarometer: A System for Crowdsourcing Scholarly Impact Metrics, in 'Proceedings of the 2014 ACM Conference on Web Science' , ACM, New York, NY, USA , pp. 285--286 .
[BibTeX] [Endnote]

Scholarometer (scholarometer.indiana.edu) is a social tool developed to facilitate citation analysis and help evaluate the impact of authors. The Scholarometer service allows scholars to compute various citation-based impact measures. In exchange, users provide disciplinary annotations of authors, which allow for the computation of discipline-specific statistics and discipline-neutral impact metrics. We present here two improvements of our system. First, we integrated a new universal impact metric hs that uses crowdsourced data to calculate the global rank of a scholar across disciplinary boundaries. Second, improvements made in ambiguous name classification have increased the accuracy from 80% to 87%.

2013

Jäschke, R. & Rudolph, S. (2013), Attribute Exploration on the Web, in Peggy Cellier; Felix Distel & Bernhard Ganter, ed., 'Contributions to the 11th International Conference on Formal Concept Analysis' , pp. 19--34 .
[BibTeX] [Endnote]

Zogaj, S. & Bretschneider, U. (2013), Crowdtesting with testCloud – Managing the Challenges of an Intermediary, in 'European Conference on Information Systems (ECIS 2013)' .
[BibTeX] [Endnote]

2012

Leimeister, J. M. (2012), 'Crowdsourcing: Crowdfunding, Crowdvoting, Crowdcreation ', Zeitschrift für Controlling und Management (ZFCM) (56) , 388-392 .
[BibTeX] [Endnote]

Liu, X.; Lu, M.; Ooi, B. C.; Shen, Y.; Wu, S. & Zhang, M. (2012), 'CDAS: a crowdsourcing data analytics system', Proceedings of the VLDB Endowment 5 (10) , 1040--1051 .
[BibTeX] [Endnote]

Some complex problems, such as image tagging and natural language processing, are very challenging for computers, where even state-of-the-art technology is yet able to provide satisfactory accuracy. Therefore, rather than relying solely on developing new and better algorithms to handle such tasks, we look to the crowdsourcing solution -- employing human participation -- to make good the shortfall in current technology. Crowdsourcing is a good supplement to many computer tasks. A complex job may be divided into computer-oriented tasks and human-oriented tasks, which are then assigned to machines and humans respectively.</p> <p>To leverage the power of crowdsourcing, we design and implement a Crowdsourcing Data Analytics System, CDAS. CDAS is a framework designed to support the deployment of various crowdsourcing applications. The core part of CDAS is a quality-sensitive answering model, which guides the crowdsourcing engine to process and monitor the human tasks. In this paper, we introduce the principles of our quality-sensitive model. To satisfy user required accuracy, the model guides the crowdsourcing query engine for the design and processing of the corresponding crowdsourcing jobs. It provides an estimated accuracy for each generated result based on the human workers' historical performances. When verifying the quality of the result, the model employs an online strategy to reduce waiting time. To show the effectiveness of the model, we implement and deploy two analytics jobs on CDAS, a twitter sentiment analytics job and an image tagging job. We use real Twitter and Flickr data as our queries respectively. We compare our approaches with state-of-the-art classification and image annotation techniques. The results show that the human-assisted methods can indeed achieve a much higher accuracy. By embedding the quality-sensitive model into crowdsourcing query engine, we effectively reduce the processing cost while maintaining the required query answer quality.

Lofi, C.; Selke, J. & Balke, W.-T. (2012), 'Information Extraction Meets Crowdsourcing: A Promising Couple', Datenbank-Spektrum 12 (2) , 109--120 .
[BibTeX] [Endnote]

Recent years brought tremendous advancements in the area of automated information extraction. But still, problem scenarios remain where even state-of-the-art algorithms do not provide a satisfying solution. In these cases, another aspiring recent trend can be exploited to achieve the required extraction quality: explicit crowdsourcing of human intelligence tasks. In this paper, we discuss the synergies between information extraction and crowdsourcing. In particular, we methodically identify and classify the challenges and fallacies that arise when combining both approaches. Furthermore, we argue that for harnessing the full potential of either approach, true hybrid techniques must be considered. To demonstrate this point, we showcase such a hybrid technique, which tightly interweaves information extraction with crowdsourcing and machine learning to vastly surpass the abilities of either technique.

Selke, J.; Lofi, C. & Balke, W.-T. (2012), 'Pushing the boundaries of crowd-enabled databases with query-driven schema expansion', Proceedings of the VLDB Endowment 5 (6) , 538--549 .
[BibTeX] [Endnote]

By incorporating human workers into the query execution process crowd-enabled databases facilitate intelligent, social capabilities like completing missing data at query time or performing cognitive operators. But despite all their flexibility, crowd-enabled databases still maintain rigid schemas. In this paper, we extend crowd-enabled databases by flexible query-driven schema expansion, allowing the addition of new attributes to the database at query time. However, the number of crowd-sourced mini-tasks to fill in missing values may often be prohibitively large and the resulting data quality is doubtful. Instead of simple crowd-sourcing to obtain all values individually, we leverage the usergenerated data found in the Social Web: By exploiting user ratings we build <i>perceptual spaces</i>, i.e., highly-compressed representations of opinions, impressions, and perceptions of large numbers of users. Using few training samples obtained by expert crowd sourcing, we then can extract all missing data automatically from the perceptual space with high quality and at low costs. Extensive experiments show that our approach can boost both performance and quality of crowd-enabled databases, while also providing the flexibility to expand schemas in a query-driven fashion.

2011

Bretschneider, U. & Leimeister, J. M. (2011), Schöne neue Crowdsourcing Welt: Billige Arbeitskräfte, Weisheit der Massen?, in K. Meißner & M. Engelien, ed., 'Proceedings zum Workshop Gemeinschaft in Neuen Medien (GeNeMe 11)' , pp. 1-13 .
[BibTeX] [Endnote]

Doan, A.; Ramakrishnan, R. & Halevy, A. Y. (2011), 'Crowdsourcing systems on the World-Wide Web', Communications of the ACM 54 (4) , 86--96 .
[BibTeX] [Endnote]

Franklin, M. J.; Kossmann, D.; Kraska, T.; Ramesh, S. & Xin, R. (2011), CrowdDB: answering queries with crowdsourcing, in 'Proceedings of the 2011 international conference on Management of data' , ACM, New York, NY, USA , pp. 61--72 .
[BibTeX] [Endnote]

Some queries cannot be answered by machines only. Processing such queries requires human input for providing information that is missing from the database, for performing computationally difficult functions, and for matching, ranking, or aggregating results based on fuzzy criteria. CrowdDB uses human input via crowdsourcing to process queries that neither database systems nor search engines can adequately answer. It uses SQL both as a language for posing complex queries and as a way to model data. While CrowdDB leverages many aspects of traditional database systems, there are also important differences. Conceptually, a major change is that the traditional closed-world assumption for query processing does not hold for human input. From an implementation perspective, human-oriented query operators are needed to solicit, integrate and cleanse crowdsourced data. Furthermore, performance and cost depend on a number of new factors including worker affinity, training, fatigue, motivation and location. We describe the design of CrowdDB, report on an initial set of experiments using Amazon Mechanical Turk, and outline important avenues for future work in the development of crowdsourced query processing systems.

Kittur, A.; Smus, B.; Khamkar, S. & Kraut, R. E. (2011), CrowdForge: crowdsourcing complex work, in 'Proceedings of the 24th annual ACM symposium on User interface software and technology' , ACM, New York, NY, USA , pp. 43--52 .
[BibTeX] [Endnote]

Micro-task markets such as Amazon's Mechanical Turk represent a new paradigm for accomplishing work, in which employers can tap into a large population of workers around the globe to accomplish tasks in a fraction of the time and money of more traditional methods. However, such markets have been primarily used for simple, independent tasks, such as labeling an image or judging the relevance of a search result. Here we present a general purpose framework for accomplishing complex and interdependent tasks using micro-task markets. We describe our framework, a web-based prototype, and case studies on article writing, decision making, and science journalism that demonstrate the benefits and limitations of the approach.

Marcus, A.; Wu, E.; Madden, S. & Miller, R. C. (2011), Crowdsourced Databases: Query Processing with People, in 'Proceedings of the 5th Biennial Conference on Innovative Data Systems Research' , CIDR, , pp. 211--214 .
[BibTeX] [Endnote]

Amazon's Mechanical Turk ( service allows users to post short tasks ( that other users can receive a small amount of money for completing. Common tasks on the system include labelling a collection of images, com- bining two sets of images to identify people which appear in both, or extracting sentiment from a corpus of text snippets. Designing a work ow of various kinds of HITs for ltering, aggregating, sorting, and joining data sources together is common, and comes with a set of challenges in optimizing the cost per HIT, the overall time to task completion, and the accuracy of MTurk results. We propose Qurk, a novel query system for managing these work ows, allowing crowd- powered processing of relational databases. We describe a number of query execution and optimization challenges, and discuss some potential solutions.

Minder, P. & Bernstein, A. (2011), CrowdLang - First Steps Towards Programmable Human Computers for General Computation, in 'Proceedings of the 3rd Human Computation Workshop' , AAAI Press, , pp. 103--108 .
[BibTeX] [Endnote]

Crowdsourcing markets such as Amazon’s Mechanical Turk provide an enormous potential for accomplishing work by combining human and machine computation. Today crowdsourcing is mostly used for massive parallel information processing for a variety of tasks such as image labeling. However, as we move to more sophisticated problem-solving there is little knowledge about managing dependencies between steps and a lack of tools for doing so. As the contribution of this paper, we present a concept of an executable, model-based programming language and a general purpose framework for accomplishing more sophisticated problems. Our approach is inspired by coordination theory and an analysis of emergent collective intelligence. We illustrate the applicability of our proposed language by combining machine and human computation based on existing interaction patterns for several general computation problems.

Parameswaran, A.; Park, H.; Garcia-Molina, H.; Polyzotis, N. & Widom, J. (2011), 'Deco: Declarative Crowdsourcing' (1015) , Technical report, Stanford University , Stanford InfoLab .
[BibTeX] [Endnote]

Crowdsourcing enables programmers to incorporate ``human computation'' as a building block in algorithms that cannot be fully automated, such as text analysis and image recognition. Similarly, humans can be used as a building block in data-intensive applications --- providing, comparing, and verifying data used by applications. Building upon the decades-long success of declarative approaches to conventional data management, we use a similar approach for data-intensive applications that incorporate humans. Specifically, declarative queries are posed over stored relational data as well as data computed on-demand from the crowd, and the underlying system orchestrates the computation of query answers. We present Deco, a database system for declarative crowdsourcing. We describe Deco's data model, query language, and our initial prototype. Deco's data model was designed to be general (it can be instantiated to other proposed models), flexible (it allows methods for uncertainty resolution and external access to be plugged in), and principled (it has a precisely-defined semantics). Syntactically, Deco's query language is a simple extension to SQL. Based on Deco's data model, we define a precise semantics for arbitrary queries involving both stored data and data obtained from the crowd. We then describe the Deco query processor, which respects our semantics while coping with the unique combination of latency, monetary cost, and uncertainty introduced in the crowdsourcing environment. Finally, we describe our current system implementation, and we discuss the novel query optimization challenges that form the core of our ongoing work.

Quinn, A. J. & Bederson, B. B. (2011), Human computation: a survey and taxonomy of a growing field, in 'Proceedings of the SIGCHI Conference on Human Factors in Computing Systems' , ACM, New York, NY, USA , pp. 1403--1412 .
[BibTeX] [Endnote]

The rapid growth of human computation within research and industry has produced many novel ideas aimed at organizing web users to do great things. However, the growth is not adequately supported by a framework with which to understand each new system in the context of the old. We classify human computation systems to help identify parallels between different systems and reveal "holes" in the existing work as opportunities for new research. Since human computation is often confused with "crowdsourcing" and other terms, we explore the position of human computation with respect to these related topics.

2010

Brew, A.; Greene, D. & Cunningham, P. (2010), Using Crowdsourcing and Active Learning to Track Sentiment in Online Media, in Helder Coelho; Rudi Studer & Michael Wooldridge, ed., 'Proceedings of the 19th European Conference on Artificial Intelligence' , IOS Press, Amsterdam, The Netherlands, The Netherlands , pp. 145--150 .
[BibTeX] [Endnote]

Tracking sentiment in the popular media has long been of interest to media analysts and pundits. With the availability of news content via online syndicated feeds, it is now possible to automate some aspects of this process. There is also great potential to crowdsource Crowdsourcing is a term, sometimes associated with Web 2.0 technologies, that describes outsourcing of tasks to a large often anonymous community. much of the annotation work that is required to train a machine learning system to perform sentiment scoring. We describe such a system for tracking economic sentiment in online media that has been deployed since August 2009. It uses annotations provided by a cohort of non-expert annotators to train a learning system to classify a large body of news items. We report on the design challenges addressed in managing the effort of the annotators and in making annotation an interesting experience.

Finin, T.; Murnane, W.; Karandikar, A.; Keller, N.; Martineau, J. & Dredze, M. (2010), Annotating named entities in Twitter data with crowdsourcing, in 'Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk' , Association for Computational Linguistics, Stroudsburg, PA, USA , pp. 80--88 .
[BibTeX] [Endnote]

We describe our experience using both Amazon Mechanical Turk (MTurk) and Crowd-Flower to collect simple named entity annotations for Twitter status updates. Unlike most genres that have traditionally been the focus of named entity experiments, Twitter is far more informal and abbreviated. The collected annotations and annotation techniques will provide a first step towards the full study of named entity recognition in domains like Facebook and Twitter. We also briefly describe how to use MTurk to collect judgements on the quality of "word clouds."

Raykar, V. C.; Yu, S.; Zhao, L. H.; Valadez, G. H.; Florin, C.; Bogoni, L. & Moy, L. (2010), 'Learning From Crowds', Journal of Machine Learning Research 11 , 1297--1322 .
[BibTeX] [Endnote]

For many supervised learning tasks it may be infeasible (or very expensive) to obtain objective and reliable labels. Instead, we can collect subjective (possibly noisy) labels from multiple experts or annotators. In practice, there is a substantial amount of disagreement among the annotators, and hence it is of great practical interest to address conventional supervised learning problems in this scenario. In this paper we describe a probabilistic approach for supervised learning when we have multiple annotators providing (possibly noisy) labels but no absolute gold standard. The proposed algorithm evaluates the different experts and also gives an estimate of the actual hidden labels. Experimental results indicate that the proposed method is superior to the commonly used majority voting baseline.

2008

Alonso, O.; Rose, D. E. & Stewart, B. (2008), 'Crowdsourcing for relevance evaluation', SIGIR Forum 42 (2) , 9--15 .
[BibTeX] [Endnote]

Relevance evaluation is an essential part of the development and maintenance of information retrieval systems. Yet traditional evaluation approaches have several limitations; in particular, conducting new editorial evaluations of a search system can be very expensive. We describe a new approach to evaluation called TERC, based on the crowdsourcing paradigm, in which many online users, drawn from a large community, each performs a small evaluation task.