Crowdsourcing services such as AMT have made large-scale data annotation and collection efforts financially affordable by granting anyone with basic programming skills access to a marketplace of anonymous online workers (known as Turkers ) willing to complete small tasks.

To ensure that opinions are written by unique authors, we allow only a single submission per Turker .

We also restrict our task to Turkers who are located in the United States, and who maintain an approval rating of at least 90%.