MTurk/Qualtrics Confirmation Codes

qualtrics_confirm_header
  • Tyler Burleigh
  • Hacks
  • 13 / 05 / 2013
  • 27 Comments
Previously I blogged about how to setup an MTurk confirmation code with SurveyGizmo. I’ve been working with Qualtrics recently, which is an online survey system like SurveyGizmo. I’ve adapted the confirmation code system to work with Qualtrics, and in this post I show how it’s done.

 

The logic:

MTurk Verification Code Diagram

 

First, send them to the external survey with a link on your MTurk HIT. Next, provide a confirmation code somewhere in the external survey (e.g., at the end). Finally, send participants back to MTurk where they enter and submit the confirmation code.

 

Setting up MTurk:

Setting up the MTurk is easy. When creating a New Project you are presented with a list of templates that you can use, and one of these is “Survey Link“. This template has all of the elements that you will need built-in: a HIT title, description, survey link, and a text field with a submit button (for the verification code).

 
2013-01-22_11-44-14

 

Setting up Qualtrics:

We will require three elements: a random number generator, a hidden variable to hold the random number (one that is also recorded to the response file), and some way to display the hidden variable to the participant at the end of the survey.

 

In Qualtrics, we will use the Random Number Generatorweb service” to generate the number, Embedded Data to store the number, and Piped Text to display the number on a custom Survey Termination message. Let’s take this step-by-step.

 

Step One: Setting up the RNG web service.

 

First navigate to Edit Survey -> Survey Flow.

 

2013-05-13_21-45-10

 

On the Survey Flow page, click “Add New Element Here“, then click Web Service.

 

2013-05-13_22-22-22

 

For URL enter: http://reporting.qualtrics.com/projects/randomNumGen.php

 

Click “Add a parameter to send to web service…” two times, and then enter the following values:

This will generate a random 8-digit number. Feel free to increase or decrease the min/max. Personally, I think 8-digits is a good compromise between security (i.e., it’s virtually impossible to guess) and convenience.

 

Untitled-1

 

Click Test URL, then check the box next to “random” and click Add Embedded Data.

 

Untitled-2

 

Name the Embedded Data “confirmation_code“, and then Move the Web Service element to the top of the Survey Flow.

 

Untitled-3

 

Step Two: Creating a custom Survey Termination Message.

 

First navigate to Library -> Message Library and then click Create a New Message.

 

2013-05-13_22-49-42

 

Set the Category to End of Survey Messages, and the Description to Confirmation Code Message. Now, to display the confirmation code you would enter ${e://Field/confirmation_code}. Here is an example message that I’ve used:

 

Untitled-2

 

Step Three: Enabling the custom Survey Termination Message.

 

Finally, we need to activate the message. Navigate to Edit Survey -> Survey Options. Then find the Survey Termination section, and click the End of survey message from a library… radio button. The message we created will be located at My Library -> Confirmation Code Message. Click Save Changes, and that’s it!

 

2013-05-13_23-20-00 copy

 

Your termination page will look something like the following (in this case, styled with a little bit of custom CSS):

 

2013-05-13_23-31-00

Tags: , , , ,

27 Responses

  1. Nicki Rucinski says:

    Hi! This was extremely helpful! I am working as a research assistant for my psychology professor and have been assisting in producing our survey. I was able to follow your instructions for setting up the confirmation code in Qualtrics. I just have one problem. We have an informed consent notice at the very beginning of the survey to which the participant is forced to chose “yes I agree to participate in the study”. I have a Skip Logic in place so that if the person selects “no I do not agree to participate in the study” it automatically takes them to the end of the survey. So it takes the person to the ending message, giving them the confirmation code, even though in this situation they didn’t actually complete the survey. I’m wondering if you have any tips or ideas on how I should remedy this issue! Anything would be greatly appreciated!

    Thanks,
    Nicki Rucinski
    Undergrad at University of WI-La Crosse

    • Tyler Burleigh says:

      Hi Nicki,

      One solution is the following:

      1) Create TWO “end of survey” pages: one with the code, and one without.
      2) Add “Display Logic” to each questions on the “without” end of survey page which say:
      Display this question if [answer to consent question] is [no]
      3) Go into the “Survey Flow” for your survey (this is next to “Survey Options”)
      4) For the Consent Page click “Add Below”. Click “Branch”, then “Add Condition”. The condition should be the same as the Display Logic in step 2: if [answer to consent question] is [no]. Click OK.
      5) Under this branch, click “Add New Element Here”. Click “Block”, then choose the “without” end of survey page as the block to show.
      6) Again, click “Add New Element Here” under the same branch. Click “End of Survey”.

      That should do the trick. Here are a couple of screenshots from an example I whipped up:
      https://dl.dropbox.com/u/485892/ShareX/2013-07/2013-07-11_09-08-48.png
      https://dl.dropbox.com/u/485892/ShareX/2013-07/2013-07-11_09-09-22.png

      Best of luck with your research!

  2. Ben says:

    Thanks! Great tutorial.

  3. Nori Comello says:

    Awesome tutorial on this topic – the clearest set of instructions out there. Just wanted to check if it would work in the following situation: I would like to assign one of five surveys to MTurk workers as their task. Based on other tips I’ve gleaned on the web, I was thinking of creating a separate “randomizer” survey that would redirect all participants to one of the five surveys. Your method for generating the random 8-digit number should still work as long as l do all the steps you outlined in each of the five surveys, right? If there is an easier way to get Mturk workers to take one of five surveys, please let me know. Thanks!!!

  4. Erik says:

    Thanks for the tutorial. I am new to qualtrics and mTurk. How do you verify that the correct code was input by participants? This there a place to check on this in qualtrics?

    Thanks
    e

  5. Erik says:

    cool thanks so much!

  6. François Dessart says:

    Hi Tyler,
    Just wanted to thank you for all these very clear explanations. They really saved me a lot of time.
    I wish you a lot of success for your PhD.
    François (Namur, Belgium)

  7. Amy says:

    Very helpful! Thanks!!

  8. Ryan L says:

    Hi Tyler,

    Thank you for the nice tutorial. Do you happen to know of a way to modify the random number generator so that it only provides each random number once (no duplicates)?

    Thank you,
    Ryan

    • Tyler Burleigh says:

      Hi Ryan,

      Can you tell me more about what you mean by each number appearing only once? Are you referring to unique number sequences (e.g., the sequence “12345″ will only appear once, across participants) or unique numbers within the sequence (e.g., “11234″ would contain a duplicate of “1″)?

      Tyler

      • Ryan L says:

        Hi Tyler, thank you for the prompt reply. One of the issues my colleagues and I have been having has been the issue of MTurk survey participants being able to provide the survey code at the end of the survey to receive compensation without actually completing the survey. Given our use of forced responses in the survey, we assume that there is a way in which people who complete an MTurk survey send the completion code out to others, who are then able to enter the completion code into MTurk without taking the survey. As you know, the way around this is to use the random number generator as your unique survey code. I followed your advice on my last survey and it worked very well (thank you!). However, I noticed that because it’s truly random, the same number can and was generated twice. So the random generator may give two survey participants the same number (i.e. 273). I would like the random number generator to only create the number once. That way, if a second MTurk participant provides the same randomly generated number, I will know it did not come from the survey itself. This person then, would not receive compensation. Does that make sense?

        • Tyler Burleigh says:

          Hi Ryan,

          Given your circumstances, my suggestion would be to generate number sequences that are sufficiently long such that any two people receiving the same number would be an extremely rare event, statistically speaking.

          If you were using a number sequence with 3 digits, then duplicates would NOT be rare. Since there are only 1000 (10^3) permutations, the odds of any number duplicating another would be less than 1 in 1000.

          In this tutorial, I use an 8-digit sequence. This results in a total of 100 million permutations, so the odds of any number duplicating another would be less than 1 in 100 million. If you plan to test thousands of people, then you might consider a longer sequence, like a 10-digit code (for 1 in 10 billion odds), but for anything less an 8-digit code would be more than sufficient.

          Tyler

  9. Rene Kizilcec says:

    Hi Tyler, have you come across QualTurk? It’s designed to integrate MTurk and Qualtrics and ensure some extent of data quality. Check it out here: http://www.QualTurk.com

  10. Meritg says:

    Hi there, I hope you can help me!! I am creating an anonymous survey with Qualtrics but I would like to be able to provide participants a unique URL (a pre survey element after they consent to participate. I was able to create unique links (e.g., email them to myself) but I am not sure how to display them to each participant so that when part#1 consents they get that link, and part#2 gets a new link, and so forth.

    Any suggestions???

    Thanks!
    Meri

    • Tyler Burleigh says:

      Hi Meri,

      Can you tell me more about the unique links? What are they linking to? What do they look like? A specific, detailed example would go a long way.

      Tyler

  11. Dan says:

    Dear Tyler,

    I think my question is similar to Meri’s. I followed the layout you posted and it works great but there is one part I’m unclear on. It seems the only way to get the link from Qualtrics is if the survey is accessible to anyone, if not the link is greyed out. That’s a problem because within minutes of activating it I got several complete surveys and none were from MTurk as I hadn’t yet posted the link there. Do you know of a solution where the survey in Qualtrics is not “public” but I can still use a link to paste in MTurk?

    Thank you,
    Dan

    • Tyler Burleigh says:

      Hi Dan,

      I haven’t experienced what you describe, where “minutes after activating it” you have received complete surveys. How are people finding your survey to complete, unless it’s being advertised somewhere? Are you paying for Qualtrics panel respondents? One solution I can think of, would be to give people visiting the survey from MTurk a code to enter when they get to Qualtrics (in this case, it wouldn’t have to be unique, just something to identify them as MTurk respondents).

  12. Dan says:

    Thanks for replying. I will try your suggestion with the identifying word.

    What happened was bizarre. Mine is a new account, but I didn’t advertise the link anywhere and I am not paying for respondents. Almost instantly I received 6 or 7 replies. At first I thought it was from me previewing the survey but that doesn’t make sense. I will call Qualtrics’ customer service and see if they can explain.

  13. Dan says:

    Turns out it was counting my previews as responses. Thanks for the suggestion though.

  14. Suzanne says:

    This was very helpful! Thanks very much!!

  15. Phil says:

    Dude…thanks a million for this tutorial! You just made my life a lot easier.

    Thank you

    Phil

Leave a Reply