04 Jan

Emory NLP is hiring Emory students for annotation related to conversational AI research. Accepted applicants are expected to go through a training session before the actual annotation. Please fill this form if you want to apply for this job.

  • Prerequisite: fluent in English.
  • The project begins in the week of Jaunary 24th, 2022.
  • You must be a current Emory student to apply for this job.
  • Your name will be acknowledged in research papers describing this annotation work.



Although human-computer interaction has recently made enormous progress, chatbots still struggle with certain conversational aspects natural to humans. This project involves talking with chatbots in text and evaluating various characteristics of dialogues such as:

  • The flow of topics introduced during the conversation.
  • The internal consistency of contents shared by the chatbot.
  • Whether or not the claims made by the chatbot are factually true.

Each annotator is expected to complete the following three tasks:

  1. To have conversations with 2 chatbots
    • Estimated time per conversation: 5 mins
  2. To evaluate 4 human-chatbot conversations
    • Estimated time per conversation: 20 mins
  3. To have conversations with 4 chatbots while evaluating
    • Estimated time per conversation: 25 mins

Thus, the total estimated time is 190 minutes for all three tasks. Once the annotation is done and passes our quality assurance, we will pay $32 with an Amazon gift card.


Here is an example of a dialogue between a human and a chatbot:

Here is an example of the evaluation:

Apply Now!

* The email will not be published on the website.