| Experimental Results 11 | About 300 unique Turkers participated the evaluation tasks. |
| Experimental Results 11 | Otherwise we treat them as ambiguous cases.17 Figure 3 shows a part of the AMT task, where Turkers are presented with questions that help judges to determine the subtle connotative polarity of each word, then asked to rate the degree of connotation on a scale from -5 (most negative) and 5 (most positive). |
| Experimental Results 11 | 17We allow Turkers to mark words that can be used with both positive and negative connotation, which results in about 7% of words that are excluded from the gold standard set. |
| Human Perception of Biased Language | Turkers were shown Wikipedia’s definition of a “biased statement” and two example sentences that illustrated the two types of bias, framing and epistemological. |
| Human Perception of Biased Language | Before the 10 sentences, turkers were asked to list the languages they spoke as well as their primary language in primary school. |
| Human Perception of Biased Language | On average, it took turkers about four minutes to complete each HIT. |
| Evaluation | To evaluate the quality of types assigned to emerging entities, we presented turkers with sentences from the news tagged with out-of-KB entities and the types inferred by the methods under test. |
| Evaluation | The turkers task was to assess the correctness of types assigned to an entity mention. |
| Evaluation | To make it easy to understand the task for the turkers , we combined the extracted entity and type into a sentence. |