Solving complex classification tasks using deep neural networks typically requires large amounts of annotated data. However, corresponding class labels are noisy when provided by error-prone annotators, e.g., crowdworkers. Training standard deep neural networks leads to subpar performances in such multi-annotator supervised learning settings. We address this issue by presenting a probabilistic training framework named multi-annotator deep learning (MaDL). A downstream ground truth and an annotator performance model are jointly trained in an end-to-end learning approach. The ground truth model learns to predict instances' true class labels, while the annotator performance model infers probabilistic estimates of annotators' performances. A modular network architecture enables us to make varying assumptions regarding annotators' performances, e.g., an optional class or instance dependency. Further, we learn annotator embeddings to estimate annotators' densities within a latent space as proxies of their potentially correlated annotations. Together with a weighted loss function, we improve the learning from correlated annotation patterns. In a comprehensive evaluation, we examine three research questions about multi-annotator supervised learning. Our findings show MaDL's state-of-the-art performance and robustness against many correlated, spamming annotators.

@article{herde2023multi, author = {Herde, Marek and Huseljic, Denis and Sick, Bernhard}, title = {Multi-annotator Deep Learning: A Probabilistic Framework for Classification}, journal = {Transactions on Machine Learning Research}, year = {2023}, url = {https://openreview.net/forum?id=MgdoxzImlK}, keywords = {imported, itegpub, isac-www, NoisyLabels, DeepLearning, Crowdsourcing}, abstract = {Solving complex classification tasks using deep neural networks typically requires large amounts of annotated data. However, corresponding class labels are noisy when provided by error-prone annotators, e.g., crowdworkers. Training standard deep neural networks leads to subpar performances in such multi-annotator supervised learning settings. We address this issue by presenting a probabilistic training framework named multi-annotator deep learning (MaDL). A downstream ground truth and an annotator performance model are jointly trained in an end-to-end learning approach. The ground truth model learns to predict instances' true class labels, while the annotator performance model infers probabilistic estimates of annotators' performances. A modular network architecture enables us to make varying assumptions regarding annotators' performances, e.g., an optional class or instance dependency. Further, we learn annotator embeddings to estimate annotators' densities within a latent space as proxies of their potentially correlated annotations. Together with a weighted loss function, we improve the learning from correlated annotation patterns. In a comprehensive evaluation, we examine three research questions about multi-annotator supervised learning. Our findings show MaDL's state-of-the-art performance and robustness against many correlated, spamming annotators.} }

%0 = article %A = Herde, Marek and Huseljic, Denis and Sick, Bernhard %D = 2023 %T = Multi-annotator Deep Learning: A Probabilistic Framework for Classification %U = https://openreview.net/forum?id=MgdoxzImlK

Scholarometer (scholarometer.indiana.edu) is a social tool developed to facilitate citation analysis and help evaluate the impact of authors. The Scholarometer service allows scholars to compute various citation-based impact measures. In exchange, users provide disciplinary annotations of authors, which allow for the computation of discipline-specific statistics and discipline-neutral impact metrics. We present here two improvements of our system. First, we integrated a new universal impact metric hs that uses crowdsourced data to calculate the global rank of a scholar across disciplinary boundaries. Second, improvements made in ambiguous name classification have increased the accuracy from 80% to 87%.

@inproceedings{kaur2014scholarometer, author = {Kaur, Jasleen and JafariAsbagh, Mohsen and Radicchi, Filippo and Menczer, Filippo}, title = {Scholarometer: A System for Crowdsourcing Scholarly Impact Metrics}, booktitle = {Proceedings of the 2014 ACM Conference on Web Science}, series = {WebSci '14}, publisher = {ACM}, address = {New York, NY, USA}, year = {2014}, pages = {285--286}, url = {http://doi.acm.org/10.1145/2615569.2615669}, doi = {10.1145/2615569.2615669}, isbn = {978-1-4503-2622-3}, keywords = {citation, crowdsourcing, impact, scholarometer, tagging}, abstract = {Scholarometer (scholarometer.indiana.edu) is a social tool developed to facilitate citation analysis and help evaluate the impact of authors. The Scholarometer service allows scholars to compute various citation-based impact measures. In exchange, users provide disciplinary annotations of authors, which allow for the computation of discipline-specific statistics and discipline-neutral impact metrics. We present here two improvements of our system. First, we integrated a new universal impact metric hs that uses crowdsourced data to calculate the global rank of a scholar across disciplinary boundaries. Second, improvements made in ambiguous name classification have increased the accuracy from 80% to 87%.} }

%0 = inproceedings %A = Kaur, Jasleen and JafariAsbagh, Mohsen and Radicchi, Filippo and Menczer, Filippo %B = Proceedings of the 2014 ACM Conference on Web Science %C = New York, NY, USA %D = 2014 %I = ACM %T = Scholarometer: A System for Crowdsourcing Scholarly Impact Metrics %U = http://doi.acm.org/10.1145/2615569.2615669

We propose an approach for supporting attribute exploration by web information retrieval, in particular by posing appropriate queries to search engines, crowd sourcing systems, and the linked open data cloud. We discuss underlying general assumptions for this to work and the degree to which these can be taken for granted.

@inproceedings{jaschke2013attribute, author = {Jäschke, Robert and Rudolph, Sebastian}, title = {Attribute Exploration on the Web}, editor = {Cellier, Peggy and Distel, Felix and Ganter, Bernhard}, booktitle = {Contributions to the 11th International Conference on Formal Concept Analysis}, year = {2013}, pages = {19--34}, url = {http://nbn-resolving.de/urn:nbn:de:bsz:14-qucosa-113133}, keywords = {2013, acquisition, analysis, attribute, computing, concept, crowdsourcing, data, exploration, fca, formal, human, information, ir, iteg, knowledge, l3s, linked, lod, open, retrieval, search, sparql, web}, abstract = {We propose an approach for supporting attribute exploration by web information retrieval, in particular by posing appropriate queries to search engines, crowd sourcing systems, and the linked open data cloud. We discuss underlying general assumptions for this to work and the degree to which these can be taken for granted.} }

%0 = inproceedings %A = Jäschke, Robert and Rudolph, Sebastian %B = Contributions to the 11th International Conference on Formal Concept Analysis %D = 2013 %T = Attribute Exploration on the Web %U = http://nbn-resolving.de/urn:nbn:de:bsz:14-qucosa-113133

@inproceedings{zogaj2013crowdtesting, author = {Zogaj, S. and Bretschneider, U.}, title = {Crowdtesting with testCloud – Managing the Challenges of an Intermediary}, booktitle = {European Conference on Information Systems (ECIS 2013)}, address = {Utrecht, Netherlands (accepted for publication)}, year = {2013}, keywords = {Crowdsourcing, Crowdtesting, Intermediary, itegpub, pub_szo, pub_ubr} }

%0 = inproceedings %A = Zogaj, S. and Bretschneider, U. %B = European Conference on Information Systems (ECIS 2013) %C = Utrecht, Netherlands (accepted for publication) %D = 2013 %T = Crowdtesting with testCloud – Managing the Challenges of an Intermediary

@article{ls_leimeister, author = {Leimeister, J. M.}, title = {Crowdsourcing: Crowdfunding, Crowdvoting, Crowdcreation }, journal = {Zeitschrift für Controlling und Management (ZFCM)}, year = {2012}, number = {56}, pages = {388-392}, note = {JML_391}, url = {http://pubs.wi-kassel.de/wp-content/uploads/2013/03/JML_391.pdf}, keywords = {crowdsourcing, itegpub, pub_jml} }

%0 = article %A = Leimeister, J. M. %D = 2012 %T = Crowdsourcing: Crowdfunding, Crowdvoting, Crowdcreation %U = http://pubs.wi-kassel.de/wp-content/uploads/2013/03/JML_391.pdf

Some complex problems, such as image tagging and natural language processing, are very challenging for computers, where even state-of-the-art technology is yet able to provide satisfactory accuracy. Therefore, rather than relying solely on developing new and better algorithms to handle such tasks, we look to the crowdsourcing solution - employing human participation - to make good the shortfall in current technology. Crowdsourcing is a good supplement to many computer tasks. A complex job may be divided into computer-oriented tasks and human-oriented tasks, which are then assigned to machines and humans respectively.</p> <p>To leverage the power of crowdsourcing, we design and implement a Crowdsourcing Data Analytics System, CDAS. CDAS is a framework designed to support the deployment of various crowdsourcing applications. The core part of CDAS is a quality-sensitive answering model, which guides the crowdsourcing engine to process and monitor the human tasks. In this paper, we introduce the principles of our quality-sensitive model. To satisfy user required accuracy, the model guides the crowdsourcing query engine for the design and processing of the corresponding crowdsourcing jobs. It provides an estimated accuracy for each generated result based on the human workers' historical performances. When verifying the quality of the result, the model employs an online strategy to reduce waiting time. To show the effectiveness of the model, we implement and deploy two analytics jobs on CDAS, a twitter sentiment analytics job and an image tagging job. We use real Twitter and Flickr data as our queries respectively. We compare our approaches with state-of-the-art classification and image annotation techniques. The results show that the human-assisted methods can indeed achieve a much higher accuracy. By embedding the quality-sensitive model into crowdsourcing query engine, we effectively reduce the processing cost while maintaining the required query answer quality.

@article{liu2012crowdsourcing, author = {Liu, Xuan and Lu, Meiyu and Ooi, Beng Chin and Shen, Yanyan and Wu, Sai and Zhang, Meihui}, title = {CDAS: a crowdsourcing data analytics system}, journal = {Proceedings of the VLDB Endowment}, publisher = {VLDB Endowment}, year = {2012}, volume = {5}, number = {10}, pages = {1040--1051}, url = {http://dl.acm.org/citation.cfm?id=2336664.2336676}, issn = {2150-8097}, keywords = {analytics, cdas, collective, crowdsourcing, data, intelligence, mining, web}, abstract = {Some complex problems, such as image tagging and natural language processing, are very challenging for computers, where even state-of-the-art technology is yet able to provide satisfactory accuracy. Therefore, rather than relying solely on developing new and better algorithms to handle such tasks, we look to the crowdsourcing solution -- employing human participation -- to make good the shortfall in current technology. Crowdsourcing is a good supplement to many computer tasks. A complex job may be divided into computer-oriented tasks and human-oriented tasks, which are then assigned to machines and humans respectively.</p> <p>To leverage the power of crowdsourcing, we design and implement a Crowdsourcing Data Analytics System, CDAS. CDAS is a framework designed to support the deployment of various crowdsourcing applications. The core part of CDAS is a quality-sensitive answering model, which guides the crowdsourcing engine to process and monitor the human tasks. In this paper, we introduce the principles of our quality-sensitive model. To satisfy user required accuracy, the model guides the crowdsourcing query engine for the design and processing of the corresponding crowdsourcing jobs. It provides an estimated accuracy for each generated result based on the human workers' historical performances. When verifying the quality of the result, the model employs an online strategy to reduce waiting time. To show the effectiveness of the model, we implement and deploy two analytics jobs on CDAS, a twitter sentiment analytics job and an image tagging job. We use real Twitter and Flickr data as our queries respectively. We compare our approaches with state-of-the-art classification and image annotation techniques. The results show that the human-assisted methods can indeed achieve a much higher accuracy. By embedding the quality-sensitive model into crowdsourcing query engine, we effectively reduce the processing cost while maintaining the required query answer quality.} }

%0 = article %A = Liu, Xuan and Lu, Meiyu and Ooi, Beng Chin and Shen, Yanyan and Wu, Sai and Zhang, Meihui %D = 2012 %I = VLDB Endowment %T = CDAS: a crowdsourcing data analytics system %U = http://dl.acm.org/citation.cfm?id=2336664.2336676

Recent years brought tremendous advancements in the area of automated information extraction. But still, problem scenarios remain where even state-of-the-art algorithms do not provide a satisfying solution. In these cases, another aspiring recent trend can be exploited to achieve the required extraction quality: explicit crowdsourcing of human intelligence tasks. In this paper, we discuss the synergies between information extraction and crowdsourcing. In particular, we methodically identify and classify the challenges and fallacies that arise when combining both approaches. Furthermore, we argue that for harnessing the full potential of either approach, true hybrid techniques must be considered. To demonstrate this point, we showcase such a hybrid technique, which tightly interweaves information extraction with crowdsourcing and machine learning to vastly surpass the abilities of either technique.

@article{lofi2012information, author = {Lofi, Christoph and Selke, Joachim and Balke, Wolf-Tilo}, title = {Information Extraction Meets Crowdsourcing: A Promising Couple}, journal = {Datenbank-Spektrum}, publisher = {Springer}, address = {Berlin/Heidelberg}, year = {2012}, volume = {12}, number = {2}, pages = {109--120}, url = {http://dx.doi.org/10.1007/s13222-012-0092-8}, doi = {10.1007/s13222-012-0092-8}, issn = {1618-2162}, keywords = {cirg, collective, computing, crowdsourcing, extraction, human, ie, information, intelligence, social}, abstract = {Recent years brought tremendous advancements in the area of automated information extraction. But still, problem scenarios remain where even state-of-the-art algorithms do not provide a satisfying solution. In these cases, another aspiring recent trend can be exploited to achieve the required extraction quality: explicit crowdsourcing of human intelligence tasks. In this paper, we discuss the synergies between information extraction and crowdsourcing. In particular, we methodically identify and classify the challenges and fallacies that arise when combining both approaches. Furthermore, we argue that for harnessing the full potential of either approach, true hybrid techniques must be considered. To demonstrate this point, we showcase such a hybrid technique, which tightly interweaves information extraction with crowdsourcing and machine learning to vastly surpass the abilities of either technique.} }

%0 = article %A = Lofi, Christoph and Selke, Joachim and Balke, Wolf-Tilo %C = Berlin/Heidelberg %D = 2012 %I = Springer %T = Information Extraction Meets Crowdsourcing: A Promising Couple %U = http://dx.doi.org/10.1007/s13222-012-0092-8

By incorporating human workers into the query execution process crowd-enabled databases facilitate intelligent, social capabilities like completing missing data at query time or performing cognitive operators. But despite all their flexibility, crowd-enabled databases still maintain rigid schemas. In this paper, we extend crowd-enabled databases by flexible query-driven schema expansion, allowing the addition of new attributes to the database at query time. However, the number of crowd-sourced mini-tasks to fill in missing values may often be prohibitively large and the resulting data quality is doubtful. Instead of simple crowd-sourcing to obtain all values individually, we leverage the usergenerated data found in the Social Web: By exploiting user ratings we build <i>perceptual spaces</i>, i.e., highly-compressed representations of opinions, impressions, and perceptions of large numbers of users. Using few training samples obtained by expert crowd sourcing, we then can extract all missing data automatically from the perceptual space with high quality and at low costs. Extensive experiments show that our approach can boost both performance and quality of crowd-enabled databases, while also providing the flexibility to expand schemas in a query-driven fashion.

@article{selke2012pushing, author = {Selke, Joachim and Lofi, Christoph and Balke, Wolf-Tilo}, title = {Pushing the boundaries of crowd-enabled databases with query-driven schema expansion}, journal = {Proceedings of the VLDB Endowment}, publisher = {VLDB Endowment}, year = {2012}, volume = {5}, number = {6}, pages = {538--549}, url = {http://dl.acm.org/citation.cfm?id=2168651.2168655}, issn = {2150-8097}, keywords = {cirg, collective, computing, crowdsourcing, database, expansion, human, intelligence, schema, social}, abstract = {By incorporating human workers into the query execution process crowd-enabled databases facilitate intelligent, social capabilities like completing missing data at query time or performing cognitive operators. But despite all their flexibility, crowd-enabled databases still maintain rigid schemas. In this paper, we extend crowd-enabled databases by flexible query-driven schema expansion, allowing the addition of new attributes to the database at query time. However, the number of crowd-sourced mini-tasks to fill in missing values may often be prohibitively large and the resulting data quality is doubtful. Instead of simple crowd-sourcing to obtain all values individually, we leverage the usergenerated data found in the Social Web: By exploiting user ratings we build <i>perceptual spaces</i>, i.e., highly-compressed representations of opinions, impressions, and perceptions of large numbers of users. Using few training samples obtained by expert crowd sourcing, we then can extract all missing data automatically from the perceptual space with high quality and at low costs. Extensive experiments show that our approach can boost both performance and quality of crowd-enabled databases, while also providing the flexibility to expand schemas in a query-driven fashion.} }

%0 = article %A = Selke, Joachim and Lofi, Christoph and Balke, Wolf-Tilo %D = 2012 %I = VLDB Endowment %T = Pushing the boundaries of crowd-enabled databases with query-driven schema expansion %U = http://dl.acm.org/citation.cfm?id=2168651.2168655

@inproceedings{bretschneider2011schne, author = {Bretschneider, U. and Leimeister, J. M.}, title = {Schöne neue Crowdsourcing Welt: Billige Arbeitskräfte, Weisheit der Massen?}, editor = {Meißner, K. and Engelien, M.}, booktitle = {Proceedings zum Workshop Gemeinschaft in Neuen Medien (GeNeMe 11)}, address = {Dresden, Germany }, year = {2011}, pages = {1-13}, note = {356}, url = {http://pubs.wi-kassel.de/wp-content/uploads/2013/03/JML_297.pdf}, keywords = {Crowdsourcing, itegpub, pub_jml, pub_ubr} }

%0 = inproceedings %A = Bretschneider, U. and Leimeister, J. M. %B = Proceedings zum Workshop Gemeinschaft in Neuen Medien (GeNeMe 11) %C = Dresden, Germany %D = 2011 %T = Schöne neue Crowdsourcing Welt: Billige Arbeitskräfte, Weisheit der Massen? %U = http://pubs.wi-kassel.de/wp-content/uploads/2013/03/JML_297.pdf

The practice of crowdsourcing is transforming the Web and giving rise to a new field.

@article{doan2011crowdsourcing, author = {Doan, Anhai and Ramakrishnan, Raghu and Halevy, Alon Y.}, title = {Crowdsourcing systems on the World-Wide Web}, journal = {Communications of the ACM}, publisher = {ACM}, address = {New York, NY, USA}, year = {2011}, volume = {54}, number = {4}, pages = {86--96}, url = {http://doi.acm.org/10.1145/1924421.1924442}, doi = {10.1145/1924421.1924442}, issn = {0001-0782}, keywords = {cirg, collective, computing, crowdsourcing, human, intelligence, social}, abstract = {The practice of crowdsourcing is transforming the Web and giving rise to a new field.} }

%0 = article %A = Doan, Anhai and Ramakrishnan, Raghu and Halevy, Alon Y. %C = New York, NY, USA %D = 2011 %I = ACM %T = Crowdsourcing systems on the World-Wide Web %U = http://doi.acm.org/10.1145/1924421.1924442

Some queries cannot be answered by machines only. Processing such queries requires human input for providing information that is missing from the database, for performing computationally difficult functions, and for matching, ranking, or aggregating results based on fuzzy criteria. CrowdDB uses human input via crowdsourcing to process queries that neither database systems nor search engines can adequately answer. It uses SQL both as a language for posing complex queries and as a way to model data. While CrowdDB leverages many aspects of traditional database systems, there are also important differences. Conceptually, a major change is that the traditional closed-world assumption for query processing does not hold for human input. From an implementation perspective, human-oriented query operators are needed to solicit, integrate and cleanse crowdsourced data. Furthermore, performance and cost depend on a number of new factors including worker affinity, training, fatigue, motivation and location. We describe the design of CrowdDB, report on an initial set of experiments using Amazon Mechanical Turk, and outline important avenues for future work in the development of crowdsourced query processing systems.

@inproceedings{franklin2011crowddb, author = {Franklin, Michael J. and Kossmann, Donald and Kraska, Tim and Ramesh, Sukriti and Xin, Reynold}, title = {CrowdDB: answering queries with crowdsourcing}, booktitle = {Proceedings of the 2011 international conference on Management of data}, publisher = {ACM}, address = {New York, NY, USA}, year = {2011}, pages = {61--72}, url = {http://doi.acm.org/10.1145/1989323.1989331}, doi = {10.1145/1989323.1989331}, isbn = {978-1-4503-0661-4}, keywords = {cirg, collective, computation, crowddb, crowdsourcing, database, human, intelligence, social}, abstract = {Some queries cannot be answered by machines only. Processing such queries requires human input for providing information that is missing from the database, for performing computationally difficult functions, and for matching, ranking, or aggregating results based on fuzzy criteria. CrowdDB uses human input via crowdsourcing to process queries that neither database systems nor search engines can adequately answer. It uses SQL both as a language for posing complex queries and as a way to model data. While CrowdDB leverages many aspects of traditional database systems, there are also important differences. Conceptually, a major change is that the traditional closed-world assumption for query processing does not hold for human input. From an implementation perspective, human-oriented query operators are needed to solicit, integrate and cleanse crowdsourced data. Furthermore, performance and cost depend on a number of new factors including worker affinity, training, fatigue, motivation and location. We describe the design of CrowdDB, report on an initial set of experiments using Amazon Mechanical Turk, and outline important avenues for future work in the development of crowdsourced query processing systems.} }

%0 = inproceedings %A = Franklin, Michael J. and Kossmann, Donald and Kraska, Tim and Ramesh, Sukriti and Xin, Reynold %B = Proceedings of the 2011 international conference on Management of data %C = New York, NY, USA %D = 2011 %I = ACM %T = CrowdDB: answering queries with crowdsourcing %U = http://doi.acm.org/10.1145/1989323.1989331

Micro-task markets such as Amazon's Mechanical Turk represent a new paradigm for accomplishing work, in which employers can tap into a large population of workers around the globe to accomplish tasks in a fraction of the time and money of more traditional methods. However, such markets have been primarily used for simple, independent tasks, such as labeling an image or judging the relevance of a search result. Here we present a general purpose framework for accomplishing complex and interdependent tasks using micro-task markets. We describe our framework, a web-based prototype, and case studies on article writing, decision making, and science journalism that demonstrate the benefits and limitations of the approach.

@inproceedings{kittur2011crowdforge, author = {Kittur, Aniket and Smus, Boris and Khamkar, Susheel and Kraut, Robert E.}, title = {CrowdForge: crowdsourcing complex work}, booktitle = {Proceedings of the 24th annual ACM symposium on User interface software and technology}, publisher = {ACM}, address = {New York, NY, USA}, year = {2011}, pages = {43--52}, url = {http://doi.acm.org/10.1145/2047196.2047202}, doi = {10.1145/2047196.2047202}, isbn = {978-1-4503-0716-1}, keywords = {cirg, collective, computing, crowdsourcing, human, intelligence, social}, abstract = {Micro-task markets such as Amazon's Mechanical Turk represent a new paradigm for accomplishing work, in which employers can tap into a large population of workers around the globe to accomplish tasks in a fraction of the time and money of more traditional methods. However, such markets have been primarily used for simple, independent tasks, such as labeling an image or judging the relevance of a search result. Here we present a general purpose framework for accomplishing complex and interdependent tasks using micro-task markets. We describe our framework, a web-based prototype, and case studies on article writing, decision making, and science journalism that demonstrate the benefits and limitations of the approach.} }

%0 = inproceedings %A = Kittur, Aniket and Smus, Boris and Khamkar, Susheel and Kraut, Robert E. %B = Proceedings of the 24th annual ACM symposium on User interface software and technology %C = New York, NY, USA %D = 2011 %I = ACM %T = CrowdForge: crowdsourcing complex work %U = http://doi.acm.org/10.1145/2047196.2047202

Amazon's Mechanical Turk ( service allows users to post short tasks ( that other users can receive a small amount of money for completing. Common tasks on the system include labelling a collection of images, com- bining two sets of images to identify people which appear in both, or extracting sentiment from a corpus of text snippets. Designing a work ow of various kinds of HITs for ltering, aggregating, sorting, and joining data sources together is common, and comes with a set of challenges in optimizing the cost per HIT, the overall time to task completion, and the accuracy of MTurk results. We propose Qurk, a novel query system for managing these work ows, allowing crowd- powered processing of relational databases. We describe a number of query execution and optimization challenges, and discuss some potential solutions.

@inproceedings{marcus2011crowdsourced, author = {Marcus, Adam and Wu, Eugene and Madden, Samuel and Miller, Robert C.}, title = {Crowdsourced Databases: Query Processing with People}, booktitle = {Proceedings of the 5th Biennial Conference on Innovative Data Systems Research}, publisher = {CIDR}, year = {2011}, pages = {211--214}, url = {http://dspace.mit.edu/handle/1721.1/62827}, doi = {1721.1/62827}, keywords = {cirg, collective, computing, crowdsourcing, database, intelligence, query, social}, abstract = {Amazon's Mechanical Turk ( service allows users to post short tasks ( that other users can receive a small amount of money for completing. Common tasks on the system include labelling a collection of images, com- bining two sets of images to identify people which appear in both, or extracting sentiment from a corpus of text snippets. Designing a work ow of various kinds of HITs for ltering, aggregating, sorting, and joining data sources together is common, and comes with a set of challenges in optimizing the cost per HIT, the overall time to task completion, and the accuracy of MTurk results. We propose Qurk, a novel query system for managing these work ows, allowing crowd- powered processing of relational databases. We describe a number of query execution and optimization challenges, and discuss some potential solutions.} }

%0 = inproceedings %A = Marcus, Adam and Wu, Eugene and Madden, Samuel and Miller, Robert C. %B = Proceedings of the 5th Biennial Conference on Innovative Data Systems Research %D = 2011 %I = CIDR %T = Crowdsourced Databases: Query Processing with People %U = http://dspace.mit.edu/handle/1721.1/62827

Crowdsourcing markets such as Amazon’s Mechanical Turk provide an enormous potential for accomplishing work by combining human and machine computation. Today crowdsourcing is mostly used for massive parallel information processing for a variety of tasks such as image labeling. However, as we move to more sophisticated problem-solving there is little knowledge about managing dependencies between steps and a lack of tools for doing so. As the contribution of this paper, we present a concept of an executable, model-based programming language and a general purpose framework for accomplishing more sophisticated problems. Our approach is inspired by coordination theory and an analysis of emergent collective intelligence. We illustrate the applicability of our proposed language by combining machine and human computation based on existing interaction patterns for several general computation problems.

@inproceedings{minder2011crowdlang, author = {Minder, Patrick and Bernstein, Abraham}, title = {CrowdLang - First Steps Towards Programmable Human Computers for General Computation}, booktitle = {Proceedings of the 3rd Human Computation Workshop}, series = {AAAI Workshops}, publisher = {AAAI Press}, year = {2011}, pages = {103--108}, url = {https://www.aaai.org/ocs/index.php/WS/AAAIW11/paper/viewFile/3891/4251}, keywords = {cirg, collective, computing, crowdsourcing, human, intelligence, social}, abstract = {Crowdsourcing markets such as Amazon’s Mechanical Turk provide an enormous potential for accomplishing work by combining human and machine computation. Today crowdsourcing is mostly used for massive parallel information processing for a variety of tasks such as image labeling. However, as we move to more sophisticated problem-solving there is little knowledge about managing dependencies between steps and a lack of tools for doing so. As the contribution of this paper, we present a concept of an executable, model-based programming language and a general purpose framework for accomplishing more sophisticated problems. Our approach is inspired by coordination theory and an analysis of emergent collective intelligence. We illustrate the applicability of our proposed language by combining machine and human computation based on existing interaction patterns for several general computation problems.} }

%0 = inproceedings %A = Minder, Patrick and Bernstein, Abraham %B = Proceedings of the 3rd Human Computation Workshop %D = 2011 %I = AAAI Press %T = CrowdLang - First Steps Towards Programmable Human Computers for General Computation %U = https://www.aaai.org/ocs/index.php/WS/AAAIW11/paper/viewFile/3891/4251

Crowdsourcing enables programmers to incorporate ``human computation'' as a building block in algorithms that cannot be fully automated, such as text analysis and image recognition. Similarly, humans can be used as a building block in data-intensive applications -- providing, comparing, and verifying data used by applications. Building upon the decades-long success of declarative approaches to conventional data management, we use a similar approach for data-intensive applications that incorporate humans. Specifically, declarative queries are posed over stored relational data as well as data computed on-demand from the crowd, and the underlying system orchestrates the computation of query answers. We present Deco, a database system for declarative crowdsourcing. We describe Deco's data model, query language, and our initial prototype. Deco's data model was designed to be general (it can be instantiated to other proposed models), flexible (it allows methods for uncertainty resolution and external access to be plugged in), and principled (it has a precisely-defined semantics). Syntactically, Deco's query language is a simple extension to SQL. Based on Deco's data model, we define a precise semantics for arbitrary queries involving both stored data and data obtained from the crowd. We then describe the Deco query processor, which respects our semantics while coping with the unique combination of latency, monetary cost, and uncertainty introduced in the crowdsourcing environment. Finally, we describe our current system implementation, and we discuss the novel query optimization challenges that form the core of our ongoing work.

@techreport{parameswaran2011declarative, author = {Parameswaran, Aditya and Park, Hyunjung and Garcia-Molina, Hector and Polyzotis, Neoklis and Widom, Jennifer}, title = {Deco: Declarative Crowdsourcing}, publisher = {Stanford InfoLab}, year = {2011}, number = {1015}, url = {http://ilpubs.stanford.edu:8090/1015/}, keywords = {cirg, collective, computing, crowdsourcing, database, deco, human, intelligence, programming, social}, abstract = {Crowdsourcing enables programmers to incorporate ``human computation'' as a building block in algorithms that cannot be fully automated, such as text analysis and image recognition. Similarly, humans can be used as a building block in data-intensive applications --- providing, comparing, and verifying data used by applications. Building upon the decades-long success of declarative approaches to conventional data management, we use a similar approach for data-intensive applications that incorporate humans. Specifically, declarative queries are posed over stored relational data as well as data computed on-demand from the crowd, and the underlying system orchestrates the computation of query answers. We present Deco, a database system for declarative crowdsourcing. We describe Deco's data model, query language, and our initial prototype. Deco's data model was designed to be general (it can be instantiated to other proposed models), flexible (it allows methods for uncertainty resolution and external access to be plugged in), and principled (it has a precisely-defined semantics). Syntactically, Deco's query language is a simple extension to SQL. Based on Deco's data model, we define a precise semantics for arbitrary queries involving both stored data and data obtained from the crowd. We then describe the Deco query processor, which respects our semantics while coping with the unique combination of latency, monetary cost, and uncertainty introduced in the crowdsourcing environment. Finally, we describe our current system implementation, and we discuss the novel query optimization challenges that form the core of our ongoing work.} }

%0 = techreport %A = Parameswaran, Aditya and Park, Hyunjung and Garcia-Molina, Hector and Polyzotis, Neoklis and Widom, Jennifer %D = 2011 %I = Stanford InfoLab %T = Deco: Declarative Crowdsourcing %U = http://ilpubs.stanford.edu:8090/1015/

The rapid growth of human computation within research and industry has produced many novel ideas aimed at organizing web users to do great things. However, the growth is not adequately supported by a framework with which to understand each new system in the context of the old. We classify human computation systems to help identify parallels between different systems and reveal "holes" in the existing work as opportunities for new research. Since human computation is often confused with "crowdsourcing" and other terms, we explore the position of human computation with respect to these related topics.

@inproceedings{quinn2011human, author = {Quinn, Alexander J. and Bederson, Benjamin B.}, title = {Human computation: a survey and taxonomy of a growing field}, booktitle = {Proceedings of the SIGCHI Conference on Human Factors in Computing Systems}, series = {CHI '11}, publisher = {ACM}, address = {New York, NY, USA}, year = {2011}, pages = {1403--1412}, url = {http://doi.acm.org/10.1145/1978942.1979148}, doi = {10.1145/1978942.1979148}, isbn = {978-1-4503-0228-9}, keywords = {crowdsourcing, human_computation, social_computing}, abstract = {The rapid growth of human computation within research and industry has produced many novel ideas aimed at organizing web users to do great things. However, the growth is not adequately supported by a framework with which to understand each new system in the context of the old. We classify human computation systems to help identify parallels between different systems and reveal "holes" in the existing work as opportunities for new research. Since human computation is often confused with "crowdsourcing" and other terms, we explore the position of human computation with respect to these related topics.} }

%0 = inproceedings %A = Quinn, Alexander J. and Bederson, Benjamin B. %B = Proceedings of the SIGCHI Conference on Human Factors in Computing Systems %C = New York, NY, USA %D = 2011 %I = ACM %T = Human computation: a survey and taxonomy of a growing field %U = http://doi.acm.org/10.1145/1978942.1979148

Tracking sentiment in the popular media has long been of interest to media analysts and pundits. With the availability of news content via online syndicated feeds, it is now possible to automate some aspects of this process. There is also great potential to crowdsource Crowdsourcing is a term, sometimes associated with Web 2.0 technologies, that describes outsourcing of tasks to a large often anonymous community. much of the annotation work that is required to train a machine learning system to perform sentiment scoring. We describe such a system for tracking economic sentiment in online media that has been deployed since August 2009. It uses annotations provided by a cohort of non-expert annotators to train a learning system to classify a large body of news items. We report on the design challenges addressed in managing the effort of the annotators and in making annotation an interesting experience.

@inproceedings{brew2010using, author = {Brew, Anthony and Greene, Derek and Cunningham, Pádraig}, title = {Using Crowdsourcing and Active Learning to Track Sentiment in Online Media}, editor = {Coelho, Helder and Studer, Rudi and Wooldridge, Michael}, booktitle = {Proceedings of the 19th European Conference on Artificial Intelligence}, series = {Frontiers in Artificial Intelligence and Applications}, publisher = {IOS Press}, address = {Amsterdam, The Netherlands, The Netherlands}, year = {2010}, volume = {215}, pages = {145--150}, url = {http://dl.acm.org/citation.cfm?id=1860967.1860997}, isbn = {978-1-60750-605-8}, keywords = {active, analysis, crowdsourcing, datamining, learning, media, online, sentiment, web}, abstract = {Tracking sentiment in the popular media has long been of interest to media analysts and pundits. With the availability of news content via online syndicated feeds, it is now possible to automate some aspects of this process. There is also great potential to crowdsource Crowdsourcing is a term, sometimes associated with Web 2.0 technologies, that describes outsourcing of tasks to a large often anonymous community. much of the annotation work that is required to train a machine learning system to perform sentiment scoring. We describe such a system for tracking economic sentiment in online media that has been deployed since August 2009. It uses annotations provided by a cohort of non-expert annotators to train a learning system to classify a large body of news items. We report on the design challenges addressed in managing the effort of the annotators and in making annotation an interesting experience.} }

%0 = inproceedings %A = Brew, Anthony and Greene, Derek and Cunningham, Pádraig %B = Proceedings of the 19th European Conference on Artificial Intelligence %C = Amsterdam, The Netherlands, The Netherlands %D = 2010 %I = IOS Press %T = Using Crowdsourcing and Active Learning to Track Sentiment in Online Media %U = http://dl.acm.org/citation.cfm?id=1860967.1860997

We describe our experience using both Amazon Mechanical Turk (MTurk) and Crowd-Flower to collect simple named entity annotations for Twitter status updates. Unlike most genres that have traditionally been the focus of named entity experiments, Twitter is far more informal and abbreviated. The collected annotations and annotation techniques will provide a first step towards the full study of named entity recognition in domains like Facebook and Twitter. We also briefly describe how to use MTurk to collect judgements on the quality of "word clouds."

@inproceedings{finin2010annotating, author = {Finin, Tim and Murnane, Will and Karandikar, Anand and Keller, Nicholas and Martineau, Justin and Dredze, Mark}, title = {Annotating named entities in Twitter data with crowdsourcing}, booktitle = {Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk}, publisher = {Association for Computational Linguistics}, address = {Stroudsburg, PA, USA}, year = {2010}, pages = {80--88}, url = {http://dl.acm.org/citation.cfm?id=1866696.1866709}, keywords = {crowdsourcing, entity, ner, recognition, twitter}, abstract = {We describe our experience using both Amazon Mechanical Turk (MTurk) and Crowd-Flower to collect simple named entity annotations for Twitter status updates. Unlike most genres that have traditionally been the focus of named entity experiments, Twitter is far more informal and abbreviated. The collected annotations and annotation techniques will provide a first step towards the full study of named entity recognition in domains like Facebook and Twitter. We also briefly describe how to use MTurk to collect judgements on the quality of "word clouds."} }

%0 = inproceedings %A = Finin, Tim and Murnane, Will and Karandikar, Anand and Keller, Nicholas and Martineau, Justin and Dredze, Mark %B = Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk %C = Stroudsburg, PA, USA %D = 2010 %I = Association for Computational Linguistics %T = Annotating named entities in Twitter data with crowdsourcing %U = http://dl.acm.org/citation.cfm?id=1866696.1866709

For many supervised learning tasks it may be infeasible (or very expensive) to obtain objective and reliable labels. Instead, we can collect subjective (possibly noisy) labels from multiple experts or annotators. In practice, there is a substantial amount of disagreement among the annotators, and hence it is of great practical interest to address conventional supervised learning problems in this scenario. In this paper we describe a probabilistic approach for supervised learning when we have multiple annotators providing (possibly noisy) labels but no absolute gold standard. The proposed algorithm evaluates the different experts and also gives an estimate of the actual hidden labels. Experimental results indicate that the proposed method is superior to the commonly used majority voting baseline.

@article{raykar2010learning, author = {Raykar, Vikas C. and Yu, Shipeng and Zhao, Linda H. and Valadez, Gerardo Hermosillo and Florin, Charles and Bogoni, Luca and Moy, Linda}, title = {Learning From Crowds}, journal = {Journal of Machine Learning Research}, publisher = {JMLR.org}, year = {2010}, volume = {11}, pages = {1297--1322}, url = {http://dl.acm.org/citation.cfm?id=1756006.1859894}, issn = {1532-4435}, keywords = {cirg, collective, computing, crowdsourcing, extraction, human, ie, information, intelligence, learning, machine, ml, social}, abstract = {For many supervised learning tasks it may be infeasible (or very expensive) to obtain objective and reliable labels. Instead, we can collect subjective (possibly noisy) labels from multiple experts or annotators. In practice, there is a substantial amount of disagreement among the annotators, and hence it is of great practical interest to address conventional supervised learning problems in this scenario. In this paper we describe a probabilistic approach for supervised learning when we have multiple annotators providing (possibly noisy) labels but no absolute gold standard. The proposed algorithm evaluates the different experts and also gives an estimate of the actual hidden labels. Experimental results indicate that the proposed method is superior to the commonly used majority voting baseline.} }

%0 = article %A = Raykar, Vikas C. and Yu, Shipeng and Zhao, Linda H. and Valadez, Gerardo Hermosillo and Florin, Charles and Bogoni, Luca and Moy, Linda %D = 2010 %I = JMLR.org %T = Learning From Crowds %U = http://dl.acm.org/citation.cfm?id=1756006.1859894

Relevance evaluation is an essential part of the development and maintenance of information retrieval systems. Yet traditional evaluation approaches have several limitations; in particular, conducting new editorial evaluations of a search system can be very expensive. We describe a new approach to evaluation called TERC, based on the crowdsourcing paradigm, in which many online users, drawn from a large community, each performs a small evaluation task.

@article{alonso2008crowdsourcing, author = {Alonso, Omar and Rose, Daniel E. and Stewart, Benjamin}, title = {Crowdsourcing for relevance evaluation}, journal = {SIGIR Forum}, publisher = {ACM}, address = {New York, NY, USA}, year = {2008}, volume = {42}, number = {2}, pages = {9--15}, url = {http://doi.acm.org/10.1145/1480506.1480508}, doi = {10.1145/1480506.1480508}, issn = {0163-5840}, keywords = {crowdsourcing, evaluation, ir, relevance}, abstract = {Relevance evaluation is an essential part of the development and maintenance of information retrieval systems. Yet traditional evaluation approaches have several limitations; in particular, conducting new editorial evaluations of a search system can be very expensive. We describe a new approach to evaluation called TERC, based on the crowdsourcing paradigm, in which many online users, drawn from a large community, each performs a small evaluation task.} }

%0 = article %A = Alonso, Omar and Rose, Daniel E. and Stewart, Benjamin %C = New York, NY, USA %D = 2008 %I = ACM %T = Crowdsourcing for relevance evaluation %U = http://doi.acm.org/10.1145/1480506.1480508