Emory NLP is hiring Emory students for the evaluation of Conversational AI Research. Accepted applicants are expected to go through a training session before the actual annotation. Please fill this form if you want to apply for this job.
Although human-computer interaction has recently made enormous progress, chatbots still struggle with certain conversational aspects natural to humans. This project involves talking with chatbots in text and evaluating various characteristics of dialogues such as:
This annotation job is split into two stages:
STAGE 1: Have conversations with chatbots
In this stage, annotators will chat with several different chatbots using a web browser interface (see screenshots below):
STAGE 2: Annotate conversations to identify various chatbot behaviors
In this stage, annotators read dialogues collected in Stage 1 and assign labels to certain chatbot responses using a web browser interface (see screenshots below). Annotators can choose how many dialogues they annotate (provided they meet a required minimum number), and are compensated based on the number of dialogues annotated:
* Note that 25 hours of work may not be available to each annotator if many annotators participate. We cannot give people more work than what we have!
Compensation will be distributed to annotators at the conclusion of Stage 2 of our study. Each annotator will be compensated $10 if they completed the Stage 1 task, plus an amount proportional to the number of Stage 2 annotations completed. Payment will be provided in the form of an Amazon gift card.
Here is an example of a dialogue between a human and a chatbot:
Here is an example of the evaluation: