Annotation Job: Conversational AI
This application is closed.
Emory NLP is hiring Emory students for the evaluation of Conversational AI Research. Accepted applicants are expected to go through a training session before the actual annotation. Please fill this form if you want to apply for this job.
- Prerequisite: fluent in English.
- The project begins in the week of March 1st, 2022.
- You must be a current Emory student to apply for this job.
- Your name will be acknowledged in research papers describing this annotation work.
Contact
- Jinho D. Choi (firstname.lastname@emory.edu)
Description
Although human-computer interaction has recently made enormous progress, chatbots still struggle with certain conversational aspects natural to humans. This project involves talking with chatbots in text and evaluating various characteristics of dialogues such as:
- The flow of topics introduced during the conversation.
- The internal consistency of contents shared by the chatbot.
- Whether or not the claims made by the chatbot are factually true.
This annotation job is split into two stages:
STAGE 1: Have conversations with chatbots
In this stage, annotators will chat with several different chatbots using a web browser interface (see screenshots below):
- Number of conversations per annotator: 12
- Estimated time per conversation: 5 mins
- Estimated total time to complete per annotator: 60 mins
- Compensation: $10
STAGE 2: Annotate conversations to identify various chatbot behaviors
In this stage, annotators read dialogues collected in Stage 1 and assign labels to certain chatbot responses using a web browser interface (see screenshots below). Annotators can choose how many dialogues they annotate (provided they meet a required minimum number), and are compensated based on the number of dialogues annotated:
- Estimated time to complete the minimum required number of annotations: 90 mins
- Compensation for completing the minimum required number of annotations: $15
- Estimated time to complete the maximum allowed number of annotations: 25 hours*
- Compensation for completing the maximum allowed number of annotations: $250*
* Note that 25 hours of work may not be available to each annotator if many annotators participate. We cannot give people more work than what we have!
Compensation will be distributed to annotators at the conclusion of Stage 2 of our study. Each annotator will be compensated $10 if they completed the Stage 1 task, plus an amount proportional to the number of Stage 2 annotations completed. Payment will be provided in the form of an Amazon gift card.
Projected Timeline
- 3/3: Kickoff meeting (video recording)
- 3/8: Start of Stage 1: Conversation Collection
- 3/16: End of Stage 1
- 3/18: Start of Stage 2: Conversation Labeling
- 3/31: End of Stage 2
- 4/7: Compensation sent to annotators
Annotation
Here is an example of a dialogue between a human and a chatbot:
Here is an example of the evaluation:
