TY - JOUR AU - Herde, Marek AU - Huseljic, Denis AU - Sick, Bernhard T1 - Multi-annotator Deep Learning: A Probabilistic Framework for Classification JO - Transactions on Machine Learning Research PY - 2023/ VL - IS - SP - EP - UR - https://openreview.net/forum?id=MgdoxzImlK M3 - KW - imported KW - itegpub KW - isac-www KW - NoisyLabels KW - DeepLearning KW - Crowdsourcing L1 - SN - N1 - N1 - AB - Solving complex classification tasks using deep neural networks typically requires large amounts of annotated data. However, corresponding class labels are noisy when provided by error-prone annotators, e.g., crowdworkers. Training standard deep neural networks leads to subpar performances in such multi-annotator supervised learning settings. We address this issue by presenting a probabilistic training framework named multi-annotator deep learning (MaDL). A downstream ground truth and an annotator performance model are jointly trained in an end-to-end learning approach. The ground truth model learns to predict instances' true class labels, while the annotator performance model infers probabilistic estimates of annotators' performances. A modular network architecture enables us to make varying assumptions regarding annotators' performances, e.g., an optional class or instance dependency. Further, we learn annotator embeddings to estimate annotators' densities within a latent space as proxies of their potentially correlated annotations. Together with a weighted loss function, we improve the learning from correlated annotation patterns. In a comprehensive evaluation, we examine three research questions about multi-annotator supervised learning. Our findings show MaDL's state-of-the-art performance and robustness against many correlated, spamming annotators. ER - TY - CONF AU - Kaur, Jasleen AU - JafariAsbagh, Mohsen AU - Radicchi, Filippo AU - Menczer, Filippo A2 - T1 - Scholarometer: A System for Crowdsourcing Scholarly Impact Metrics T2 - Proceedings of the 2014 ACM Conference on Web Science PB - ACM CY - New York, NY, USA PY - 2014/ M2 - VL - IS - SP - 285 EP - 286 UR - http://doi.acm.org/10.1145/2615569.2615669 M3 - 10.1145/2615569.2615669 KW - citation KW - crowdsourcing KW - impact KW - scholarometer KW - tagging L1 - SN - 978-1-4503-2622-3 N1 - Scholarometer N1 - AB - Scholarometer (scholarometer.indiana.edu) is a social tool developed to facilitate citation analysis and help evaluate the impact of authors. The Scholarometer service allows scholars to compute various citation-based impact measures. In exchange, users provide disciplinary annotations of authors, which allow for the computation of discipline-specific statistics and discipline-neutral impact metrics. We present here two improvements of our system. First, we integrated a new universal impact metric hs that uses crowdsourced data to calculate the global rank of a scholar across disciplinary boundaries. Second, improvements made in ambiguous name classification have increased the accuracy from 80% to 87%. ER - TY - CONF AU - Jäschke, Robert AU - Rudolph, Sebastian A2 - Cellier, Peggy A2 - Distel, Felix A2 - Ganter, Bernhard T1 - Attribute Exploration on the Web T2 - Contributions to the 11th International Conference on Formal Concept Analysis PB - CY - PY - 2013/05 M2 - VL - IS - SP - 19 EP - 34 UR - http://nbn-resolving.de/urn:nbn:de:bsz:14-qucosa-113133 M3 - KW - 2013 KW - acquisition KW - analysis KW - attribute KW - computing KW - concept KW - crowdsourcing KW - data KW - exploration KW - fca KW - formal KW - human KW - information KW - ir KW - iteg KW - knowledge KW - l3s KW - linked KW - lod KW - open KW - retrieval KW - search KW - sparql KW - web L1 - SN - N1 - N1 - AB - We propose an approach for supporting attribute exploration by web information retrieval, in particular by posing appropriate queries to search engines, crowd sourcing systems, and the linked open data cloud. We discuss underlying general assumptions for this to work and the degree to which these can be taken for granted. ER - TY - CONF AU - Zogaj, S. AU - Bretschneider, U. A2 - T1 - Crowdtesting with testCloud – Managing the Challenges of an Intermediary T2 - European Conference on Information Systems (ECIS 2013) PB - CY - Utrecht, Netherlands (accepted for publication) PY - 2013/ M2 - VL - IS - SP - EP - UR - M3 - KW - Crowdsourcing KW - Crowdtesting KW - Intermediary KW - itegpub KW - pub_szo KW - pub_ubr L1 - SN - N1 - N1 - AB - ER - TY - JOUR AU - Leimeister, J. M. T1 - Crowdsourcing: Crowdfunding, Crowdvoting, Crowdcreation JO - Zeitschrift für Controlling und Management (ZFCM) PY - 2012/ VL - IS - 56 SP - 388 EP - 392 UR - http://pubs.wi-kassel.de/wp-content/uploads/2013/03/JML_391.pdf M3 - KW - crowdsourcing KW - itegpub KW - pub_jml L1 - SN - N1 - N1 - AB - ER - TY - JOUR AU - Liu, Xuan AU - Lu, Meiyu AU - Ooi, Beng Chin AU - Shen, Yanyan AU - Wu, Sai AU - Zhang, Meihui T1 - CDAS: a crowdsourcing data analytics system JO - Proceedings of the VLDB Endowment PY - 2012/06 VL - 5 IS - 10 SP - 1040 EP - 1051 UR - http://dl.acm.org/citation.cfm?id=2336664.2336676 M3 - KW - analytics KW - cdas KW - collective KW - crowdsourcing KW - data KW - intelligence KW - mining KW - web L1 - SN - N1 - N1 - AB - Some complex problems, such as image tagging and natural language processing, are very challenging for computers, where even state-of-the-art technology is yet able to provide satisfactory accuracy. Therefore, rather than relying solely on developing new and better algorithms to handle such tasks, we look to the crowdsourcing solution -- employing human participation -- to make good the shortfall in current technology. Crowdsourcing is a good supplement to many computer tasks. A complex job may be divided into computer-oriented tasks and human-oriented tasks, which are then assigned to machines and humans respectively.

To leverage the power of crowdsourcing, we design and implement a Crowdsourcing Data Analytics System, CDAS. CDAS is a framework designed to support the deployment of various crowdsourcing applications. The core part of CDAS is a quality-sensitive answering model, which guides the crowdsourcing engine to process and monitor the human tasks. In this paper, we introduce the principles of our quality-sensitive model. To satisfy user required accuracy, the model guides the crowdsourcing query engine for the design and processing of the corresponding crowdsourcing jobs. It provides an estimated accuracy for each generated result based on the human workers' historical performances. When verifying the quality of the result, the model employs an online strategy to reduce waiting time. To show the effectiveness of the model, we implement and deploy two analytics jobs on CDAS, a twitter sentiment analytics job and an image tagging job. We use real Twitter and Flickr data as our queries respectively. We compare our approaches with state-of-the-art classification and image annotation techniques. The results show that the human-assisted methods can indeed achieve a much higher accuracy. By embedding the quality-sensitive model into crowdsourcing query engine, we effectively reduce the processing cost while maintaining the required query answer quality. ER - TY - JOUR AU - Lofi, Christoph AU - Selke, Joachim AU - Balke, Wolf-Tilo T1 - Information Extraction Meets Crowdsourcing: A Promising Couple JO - Datenbank-Spektrum PY - 2012/ VL - 12 IS - 2 SP - 109 EP - 120 UR - http://dx.doi.org/10.1007/s13222-012-0092-8 M3 - 10.1007/s13222-012-0092-8 KW - cirg KW - collective KW - computing KW - crowdsourcing KW - extraction KW - human KW - ie KW - information KW - intelligence KW - social L1 - SN - N1 - N1 - AB - Recent years brought tremendous advancements in the area of automated information extraction. But still, problem scenarios remain where even state-of-the-art algorithms do not provide a satisfying solution. In these cases, another aspiring recent trend can be exploited to achieve the required extraction quality: explicit crowdsourcing of human intelligence tasks. In this paper, we discuss the synergies between information extraction and crowdsourcing. In particular, we methodically identify and classify the challenges and fallacies that arise when combining both approaches. Furthermore, we argue that for harnessing the full potential of either approach, true hybrid techniques must be considered. To demonstrate this point, we showcase such a hybrid technique, which tightly interweaves information extraction with crowdsourcing and machine learning to vastly surpass the abilities of either technique. ER - TY - JOUR AU - Selke, Joachim AU - Lofi, Christoph AU - Balke, Wolf-Tilo T1 - Pushing the boundaries of crowd-enabled databases with query-driven schema expansion JO - Proceedings of the VLDB Endowment PY - 2012/02 VL - 5 IS - 6 SP - 538 EP - 549 UR - http://dl.acm.org/citation.cfm?id=2168651.2168655 M3 - KW - cirg KW - collective KW - computing KW - crowdsourcing KW - database KW - expansion KW - human KW - intelligence KW - schema KW - social L1 - SN - N1 - N1 - AB - By incorporating human workers into the query execution process crowd-enabled databases facilitate intelligent, social capabilities like completing missing data at query time or performing cognitive operators. But despite all their flexibility, crowd-enabled databases still maintain rigid schemas. In this paper, we extend crowd-enabled databases by flexible query-driven schema expansion, allowing the addition of new attributes to the database at query time. However, the number of crowd-sourced mini-tasks to fill in missing values may often be prohibitively large and the resulting data quality is doubtful. Instead of simple crowd-sourcing to obtain all values individually, we leverage the usergenerated data found in the Social Web: By exploiting user ratings we build perceptual spaces, i.e., highly-compressed representations of opinions, impressions, and perceptions of large numbers of users. Using few training samples obtained by expert crowd sourcing, we then can extract all missing data automatically from the perceptual space with high quality and at low costs. Extensive experiments show that our approach can boost both performance and quality of crowd-enabled databases, while also providing the flexibility to expand schemas in a query-driven fashion. ER - TY - CONF AU - Bretschneider, U. AU - Leimeister, J. M. A2 - Meißner, K. A2 - Engelien, M. T1 - Schöne neue Crowdsourcing Welt: Billige Arbeitskräfte, Weisheit der Massen? T2 - Proceedings zum Workshop Gemeinschaft in Neuen Medien (GeNeMe 11) PB - CY - Dresden, Germany PY - 2011/ M2 - VL - IS - SP - 1 EP - 13 UR - http://pubs.wi-kassel.de/wp-content/uploads/2013/03/JML_297.pdf M3 - KW - Crowdsourcing KW - itegpub KW - pub_jml KW - pub_ubr L1 - SN - N1 - N1 - AB - ER - TY - JOUR AU - Doan, Anhai AU - Ramakrishnan, Raghu AU - Halevy, Alon Y. T1 - Crowdsourcing systems on the World-Wide Web JO - Communications of the ACM PY - 2011/04 VL - 54 IS - 4 SP - 86 EP - 96 UR - http://doi.acm.org/10.1145/1924421.1924442 M3 - 10.1145/1924421.1924442 KW - cirg KW - collective KW - computing KW - crowdsourcing KW - human KW - intelligence KW - social L1 - SN - N1 - N1 - AB - The practice of crowdsourcing is transforming the Web and giving rise to a new field. ER - TY - CONF AU - Franklin, Michael J. AU - Kossmann, Donald AU - Kraska, Tim AU - Ramesh, Sukriti AU - Xin, Reynold A2 - T1 - CrowdDB: answering queries with crowdsourcing T2 - Proceedings of the 2011 international conference on Management of data PB - ACM CY - New York, NY, USA PY - 2011/ M2 - VL - IS - SP - 61 EP - 72 UR - http://doi.acm.org/10.1145/1989323.1989331 M3 - 10.1145/1989323.1989331 KW - cirg KW - collective KW - computation KW - crowddb KW - crowdsourcing KW - database KW - human KW - intelligence KW - social L1 - SN - 978-1-4503-0661-4 N1 - N1 - AB - Some queries cannot be answered by machines only. Processing such queries requires human input for providing information that is missing from the database, for performing computationally difficult functions, and for matching, ranking, or aggregating results based on fuzzy criteria. CrowdDB uses human input via crowdsourcing to process queries that neither database systems nor search engines can adequately answer. It uses SQL both as a language for posing complex queries and as a way to model data. While CrowdDB leverages many aspects of traditional database systems, there are also important differences. Conceptually, a major change is that the traditional closed-world assumption for query processing does not hold for human input. From an implementation perspective, human-oriented query operators are needed to solicit, integrate and cleanse crowdsourced data. Furthermore, performance and cost depend on a number of new factors including worker affinity, training, fatigue, motivation and location. We describe the design of CrowdDB, report on an initial set of experiments using Amazon Mechanical Turk, and outline important avenues for future work in the development of crowdsourced query processing systems. ER - TY - CONF AU - Kittur, Aniket AU - Smus, Boris AU - Khamkar, Susheel AU - Kraut, Robert E. A2 - T1 - CrowdForge: crowdsourcing complex work T2 - Proceedings of the 24th annual ACM symposium on User interface software and technology PB - ACM CY - New York, NY, USA PY - 2011/ M2 - VL - IS - SP - 43 EP - 52 UR - http://doi.acm.org/10.1145/2047196.2047202 M3 - 10.1145/2047196.2047202 KW - cirg KW - collective KW - computing KW - crowdsourcing KW - human KW - intelligence KW - social L1 - SN - 978-1-4503-0716-1 N1 - N1 - AB - Micro-task markets such as Amazon's Mechanical Turk represent a new paradigm for accomplishing work, in which employers can tap into a large population of workers around the globe to accomplish tasks in a fraction of the time and money of more traditional methods. However, such markets have been primarily used for simple, independent tasks, such as labeling an image or judging the relevance of a search result. Here we present a general purpose framework for accomplishing complex and interdependent tasks using micro-task markets. We describe our framework, a web-based prototype, and case studies on article writing, decision making, and science journalism that demonstrate the benefits and limitations of the approach. ER - TY - CONF AU - Marcus, Adam AU - Wu, Eugene AU - Madden, Samuel AU - Miller, Robert C. A2 - T1 - Crowdsourced Databases: Query Processing with People T2 - Proceedings of the 5th Biennial Conference on Innovative Data Systems Research PB - CIDR CY - PY - 2011/01 M2 - VL - IS - SP - 211 EP - 214 UR - http://dspace.mit.edu/handle/1721.1/62827 M3 - 1721.1/62827 KW - cirg KW - collective KW - computing KW - crowdsourcing KW - database KW - intelligence KW - query KW - social L1 - SN - N1 - N1 - AB - Amazon's Mechanical Turk ( service allows users to post short tasks ( that other users can receive a small amount of money for completing. Common tasks on the system include labelling a collection of images, com- bining two sets of images to identify people which appear in both, or extracting sentiment from a corpus of text snippets. Designing a work ow of various kinds of HITs for ltering, aggregating, sorting, and joining data sources together is common, and comes with a set of challenges in optimizing the cost per HIT, the overall time to task completion, and the accuracy of MTurk results. We propose Qurk, a novel query system for managing these work ows, allowing crowd- powered processing of relational databases. We describe a number of query execution and optimization challenges, and discuss some potential solutions. ER - TY - CONF AU - Minder, Patrick AU - Bernstein, Abraham A2 - T1 - CrowdLang - First Steps Towards Programmable Human Computers for General Computation T2 - Proceedings of the 3rd Human Computation Workshop PB - AAAI Press CY - PY - 2011/ M2 - VL - IS - SP - 103 EP - 108 UR - https://www.aaai.org/ocs/index.php/WS/AAAIW11/paper/viewFile/3891/4251 M3 - KW - cirg KW - collective KW - computing KW - crowdsourcing KW - human KW - intelligence KW - social L1 - SN - N1 - N1 - AB - Crowdsourcing markets such as Amazon’s Mechanical Turk provide an enormous potential for accomplishing work by combining human and machine computation. Today crowdsourcing is mostly used for massive parallel information processing for a variety of tasks such as image labeling. However, as we move to more sophisticated problem-solving there is little knowledge about managing dependencies between steps and a lack of tools for doing so. As the contribution of this paper, we present a concept of an executable, model-based programming language and a general purpose framework for accomplishing more sophisticated problems. Our approach is inspired by coordination theory and an analysis of emergent collective intelligence. We illustrate the applicability of our proposed language by combining machine and human computation based on existing interaction patterns for several general computation problems. ER - TY - RPRT AU - Parameswaran, Aditya AU - Park, Hyunjung AU - Garcia-Molina, Hector AU - Polyzotis, Neoklis AU - Widom, Jennifer A2 - T1 - Deco: Declarative Crowdsourcing PB - Stanford University AD - PY - 2011/ VL - IS - 1015 SP - EP - UR - http://ilpubs.stanford.edu:8090/1015/ M3 - KW - cirg KW - collective KW - computing KW - crowdsourcing KW - database KW - deco KW - human KW - intelligence KW - programming KW - social L1 - N1 - N1 - N1 - AB - Crowdsourcing enables programmers to incorporate ``human computation'' as a building block in algorithms that cannot be fully automated, such as text analysis and image recognition. Similarly, humans can be used as a building block in data-intensive applications --- providing, comparing, and verifying data used by applications. Building upon the decades-long success of declarative approaches to conventional data management, we use a similar approach for data-intensive applications that incorporate humans. Specifically, declarative queries are posed over stored relational data as well as data computed on-demand from the crowd, and the underlying system orchestrates the computation of query answers. We present Deco, a database system for declarative crowdsourcing. We describe Deco's data model, query language, and our initial prototype. Deco's data model was designed to be general (it can be instantiated to other proposed models), flexible (it allows methods for uncertainty resolution and external access to be plugged in), and principled (it has a precisely-defined semantics). Syntactically, Deco's query language is a simple extension to SQL. Based on Deco's data model, we define a precise semantics for arbitrary queries involving both stored data and data obtained from the crowd. We then describe the Deco query processor, which respects our semantics while coping with the unique combination of latency, monetary cost, and uncertainty introduced in the crowdsourcing environment. Finally, we describe our current system implementation, and we discuss the novel query optimization challenges that form the core of our ongoing work. ER - TY - CONF AU - Quinn, Alexander J. AU - Bederson, Benjamin B. A2 - T1 - Human computation: a survey and taxonomy of a growing field T2 - Proceedings of the SIGCHI Conference on Human Factors in Computing Systems PB - ACM CY - New York, NY, USA PY - 2011/ M2 - VL - IS - SP - 1403 EP - 1412 UR - http://doi.acm.org/10.1145/1978942.1979148 M3 - 10.1145/1978942.1979148 KW - crowdsourcing KW - human_computation KW - social_computing L1 - SN - 978-1-4503-0228-9 N1 - Human computation N1 - AB - The rapid growth of human computation within research and industry has produced many novel ideas aimed at organizing web users to do great things. However, the growth is not adequately supported by a framework with which to understand each new system in the context of the old. We classify human computation systems to help identify parallels between different systems and reveal "holes" in the existing work as opportunities for new research. Since human computation is often confused with "crowdsourcing" and other terms, we explore the position of human computation with respect to these related topics. ER - TY - CONF AU - Brew, Anthony AU - Greene, Derek AU - Cunningham, Pádraig A2 - Coelho, Helder A2 - Studer, Rudi A2 - Wooldridge, Michael T1 - Using Crowdsourcing and Active Learning to Track Sentiment in Online Media T2 - Proceedings of the 19th European Conference on Artificial Intelligence PB - IOS Press CY - Amsterdam, The Netherlands, The Netherlands PY - 2010/ M2 - VL - 215 IS - SP - 145 EP - 150 UR - http://dl.acm.org/citation.cfm?id=1860967.1860997 M3 - KW - active KW - analysis KW - crowdsourcing KW - datamining KW - learning KW - media KW - online KW - sentiment KW - web L1 - SN - 978-1-60750-605-8 N1 - N1 - AB - Tracking sentiment in the popular media has long been of interest to media analysts and pundits. With the availability of news content via online syndicated feeds, it is now possible to automate some aspects of this process. There is also great potential to crowdsource Crowdsourcing is a term, sometimes associated with Web 2.0 technologies, that describes outsourcing of tasks to a large often anonymous community. much of the annotation work that is required to train a machine learning system to perform sentiment scoring. We describe such a system for tracking economic sentiment in online media that has been deployed since August 2009. It uses annotations provided by a cohort of non-expert annotators to train a learning system to classify a large body of news items. We report on the design challenges addressed in managing the effort of the annotators and in making annotation an interesting experience. ER - TY - CONF AU - Finin, Tim AU - Murnane, Will AU - Karandikar, Anand AU - Keller, Nicholas AU - Martineau, Justin AU - Dredze, Mark A2 - T1 - Annotating named entities in Twitter data with crowdsourcing T2 - Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk PB - Association for Computational Linguistics CY - Stroudsburg, PA, USA PY - 2010/ M2 - VL - IS - SP - 80 EP - 88 UR - http://dl.acm.org/citation.cfm?id=1866696.1866709 M3 - KW - crowdsourcing KW - entity KW - ner KW - recognition KW - twitter L1 - SN - N1 - N1 - AB - We describe our experience using both Amazon Mechanical Turk (MTurk) and Crowd-Flower to collect simple named entity annotations for Twitter status updates. Unlike most genres that have traditionally been the focus of named entity experiments, Twitter is far more informal and abbreviated. The collected annotations and annotation techniques will provide a first step towards the full study of named entity recognition in domains like Facebook and Twitter. We also briefly describe how to use MTurk to collect judgements on the quality of "word clouds." ER - TY - JOUR AU - Raykar, Vikas C. AU - Yu, Shipeng AU - Zhao, Linda H. AU - Valadez, Gerardo Hermosillo AU - Florin, Charles AU - Bogoni, Luca AU - Moy, Linda T1 - Learning From Crowds JO - Journal of Machine Learning Research PY - 2010/08 VL - 11 IS - SP - 1297 EP - 1322 UR - http://dl.acm.org/citation.cfm?id=1756006.1859894 M3 - KW - cirg KW - collective KW - computing KW - crowdsourcing KW - extraction KW - human KW - ie KW - information KW - intelligence KW - learning KW - machine KW - ml KW - social L1 - SN - N1 - N1 - AB - For many supervised learning tasks it may be infeasible (or very expensive) to obtain objective and reliable labels. Instead, we can collect subjective (possibly noisy) labels from multiple experts or annotators. In practice, there is a substantial amount of disagreement among the annotators, and hence it is of great practical interest to address conventional supervised learning problems in this scenario. In this paper we describe a probabilistic approach for supervised learning when we have multiple annotators providing (possibly noisy) labels but no absolute gold standard. The proposed algorithm evaluates the different experts and also gives an estimate of the actual hidden labels. Experimental results indicate that the proposed method is superior to the commonly used majority voting baseline. ER - TY - JOUR AU - Alonso, Omar AU - Rose, Daniel E. AU - Stewart, Benjamin T1 - Crowdsourcing for relevance evaluation JO - SIGIR Forum PY - 2008/november VL - 42 IS - 2 SP - 9 EP - 15 UR - http://doi.acm.org/10.1145/1480506.1480508 M3 - 10.1145/1480506.1480508 KW - crowdsourcing KW - evaluation KW - ir KW - relevance L1 - SN - N1 - N1 - AB - Relevance evaluation is an essential part of the development and maintenance of information retrieval systems. Yet traditional evaluation approaches have several limitations; in particular, conducting new editorial evaluations of a search system can be very expensive. We describe a new approach to evaluation called TERC, based on the crowdsourcing paradigm, in which many online users, drawn from a large community, each performs a small evaluation task. ER -