| Data | 2.3 Ground Truth |
| Data | The ground truth for training and evaluating our regression models is formed by voting intention polls from YouGov (UK) and a collection of Austrian pollsters2 — as none performed high frequency polling — for the Austrian case study. |
| Experiments | (a) Ground Truth (polls) |
| Experiments | Sub-figure 3a is a plot of ground truth as presented in voting inten- |
| Experiments | 5 _ + oVP + FPQ + GRU 0 5 10 15 20 25 30 35 40 45 Time (a) Ground Truth (polls) |
| Conclusions | 14In some domains, such as medical diagnosis, it makes perfect sense to assume that there is a ground truth . |
| Related Work | Although in our case study we have tested our aggregators by comparing their outcomes to a gold standard, our approach to collective annotation itself does not assume that there is in fact a ground truth . |
| Related Work | In application domains where it is reasonable to assume the existence of a ground truth and where we are able to model the manner in which individual judgments are being distorted relative to this ground truth, social choice theory provides tools (using again maximum-likelihood estimators) for the design of aggregators that maximise chances of recovering the ground truth for a given model of distortion (Young, 1995; Conitzer and Sandholm, 2005). |
| Related Work | Specifically, they have designed an experiment in which the ground truth is defined unambiguously and known to the experiment designer, so as to be able to extract realistic models of distortion from the data collected in a crowdsourcing exercise. |