How to design and administer surveys

 

 

Module #10: How to design and administer surveys

READING

 

1. In this assignment, you will be constructing and administering online surveys. To learn

why it’s important that you learn how to design and administer online surveys:

1. Review from Brooklyn College’s summary of Coplin’s (2012) book, 10 Things Employers Want You to Learn in College. Note that on p. 12, under the heading “Gathering Information,” the skill of “Construct Surveys” is
2. Read Tague’s (2004a) brief introduction, “When to Use a Survey,” which addresses the use of surveys outside of psychological science (i.e., beyond the purposes of basic scientific research).

1. To learn whether your administering surveys as a class assignment requires IRB (Institutional Review Board) ethical approval for protection of human research participants:
   1. Read the University of Michigan’s (2004) “Research Ethics and Compliance Policy.” Pay attention to the “Student Class Assignment ”
      • Note: Policies vary across
   2. Search the web for CUNY York’s policy about whether or not in-class research projects that involve collecting data from human participants require ethical approval from the IRB

 

1. You will design and administer two surveys in this class. The first survey will be on a topic chosen from the Online Surveys Topics handout (read closely through this handout). The second survey will explore a research topic of your own choice. As per Module #5, where you searched for two different research topics, it will be important that your topic is neither too broad nor too

 

 

 

1. Note: You will be able to post a link to your surveys on the discussion board where fellow students can go to take your surveys. However, you will not be able to compel anyone’s participation. So, best not to count on this as your primary means of acquiring data. As such, be sure to assess your access to research participants (e.g., friends, family and otherwise) when deciding on your

 

2. You will be required to conduct both descriptive and inferential analysis of your data and present the results of both. Descriptive analysis will involve summarizing your data and presenting it in graphical form. Inferential analysis will involve generalizing from your sample to the population at large. For your inferential analysis, you will statistically test the relationship between two variables. The kind of test you decide to conduct will depend upon the kinds of variables you’re

 

 

 

• If you have two continuous variables (e.g., age vs income), you could do a correlational analysis or a regression.
• If you have one categorical variable with two levels (e.g., high school vs college student) and one continuous variable (hours worked per week), you could conduct an independent samples t-test.
• If you’re measuring changes in some variable at two different time points, you could run a paired-samples t-test
• If you have a categorical variable with more than two levels (e.g., Hunter College, York College, Brooklyn College or City College student) and one continuous variable (commute time, one way, in minutes) you could do an
• If you’re going to compare two categorial variables, you can run a chi

square test.

 

The variables you decide to examine will be described in your hypothesis. For example, let’s say I wanted to test the relationship between political affiliation and mask wearing behavior in NYC adults.

 

We’d want to think carefully about how to operationally define (ala Module #8) these variables (e.g., exactly what information do I need to obtain and how will it obtained?)

 

First, I would need to decide how I’m going to ask people their political affiliation. Do I just ask them whether they are democrats or republicans? Do I want to include independents? Should I have them choose one of these labels? Or should I have them indicate their political affiliation on a sliding numerical scale of some kind where one end = R, the other end = D and the middle = I?

These are the kinds of decisions we need to make in order to operationally define this variable.

 

Second, I need to figure out how to measure the % of the time they wear/wore a mask. Do I ask them how much they wear one now (when people are doing this less and many are vaccinated?) or do I want to focus on when COVID was at its worst and ask them about what they were doing in April 2020? Or during

 

different waves? How should the question be worded? Do they provide a numerical value (0 to 100%) or do we give them categories to choose from?

 

Let’s say we decide to make the predictor variable, political affiliation, a categorical variable (dem, rep, or indie) and the outcome variable a number (pick a % from 0 to 100). Because my predictor is categorical and my outcome variable is continuous, I would probably want to run an ANOVA as my statistical test. If I don’t feel confident doing this, I might decide to just limit political affiliation to two levels, Democrat or Republican, and do a t-test. You should have at least run a t-test in your stats course and possibly an ANOVA at some point (or not). T- tests can be conducted in SPSS or even in Excel.

 

So, before you design your survey, it’s important to think hard about your research question and how you’re going to analyze the data. Try to come up with research questions that are interesting to you (and hopefully to others) and think about how you’re going to analyze the date before you create the survey.

 

 

 

WRITING ASSIGNMENT

 

1. Select one survey topic and one topic of your own choosing. Be sure to select topics
   1. that you’re interested in investigating, and
   2. for which you have access to the appropriate survey

 

1. Read and synthesize previously published psychological science on each of the two topics you
   1. For each of the two survey topics you’ve selected, search Google Scholar for

relevant scientific articles (using the procedure you learned in Module #5).

• Remember you can also find related articles by examining the “Cited By” tool and by examining an article’s reference

2. For each of the two survey topics you’ve selected, find three scientific articles (via Google Scholar or other search engine).
   • The three articles can be at any level in the Hierarchy of Scientific Evidence, and
   • all three articles can be at the same level in the Hierarchy of Scientific Evidence.
3. For each of the two survey topics you’ve selected:
   • Read and analyze the three scientific articles you’ve found (using the

procedure you learned in Module #5).

• Write one paragraph synthesizing the three articles you’ve found (using

the procedure you learned in Module #6).

 

• Remember to synthesize the findings and not Mad Lib them; to write about behavior and phenomena, not researchers and their studies; to place each article’s in-text citation at the end of the sentence (in

parentheses); and to synthesize conflicting results (e.g., using “However,

….”).

• Lastly, include the three scientific articles’ full APA-style citations after each of your two synthesis paragraphs

 

1. Go to the discussion board forum Module #10, Part 1: Survey Development and make a new post that includes the following five parts:
   1. In at least 50 words,
      • explain why you think constructing and administering surveys is a skill employers want you to learn in college, and
      • state at least two uses of online surveys for purposes other than basic scientific research. (These uses can come from Tague’s article or from surveys you’ve seen )
   2. Search the Internet for the York College policy on human subject recruitment for in-class research projects. In at least 50 words, describe the policy, whether you agree with it, and why you agree or disagree with it. Include the link to the page where you found the policy. Note: research conducted for an in-class research project is generally viewed differently than research conducted outside of class, from an ethics perspective. Keep this in mind when searching for the
   3. State the two topics you’ve selected for your two
   4. Post the two paragraphs you’ve written that synthesize previous research on

your two survey topics.

5. Include the APA-style full citations at the end of both

READING

 

1. To get a basic introduction to writing survey items, read Science Buddies’ (no date) article, “Designing a Survey.” This article provides a basic introduction; if you already

have some experience writing survey items, you may skim (though not skip) this article.

 

1. To become more informed about writing survey items:
   1. Read Beretta’s (2014) article, “Top Ten Common Problems in Designing Effective Survey Questions.” Make sure you understand all ten problems Beretta describes – and you know how to avoid all ten
   2. Read Pew Research Center’s (no date) article, “Questionnaire Design.” Make

sure you understand

• open- versus closed-ended questions (the latter are what Science Buddies

refers to as ‘structured questions’);

• the importance of asking questions that are clear and well-specified;

 

• what double-barreled questions and double-negatives are (and how to avoid them);
• what acquiescence bias and social desirability bias are (and how to avoid them);
• factors to consider in ordering your questions; and
• the importance of placing demographic items

 

3. To appreciate the power of question wording, look at Britain Elects’ (2017) pair of tweets.

 

4. Read Harvard University’s Program on Survey Research (2007) “Tip Sheet on Question Wording.” Make sure you understand
   • how to avoid technical jargon, vague or imprecise terms;
   • how to avoid double-barreled questions (again!);
   • how to avoid leading, emotional, or evocative language; and
   • how to use ordinal scales, reference frames, and unique answer choices that cover all response options.
5. Read Peters’ (no date) article, “How to Design a Survey.” Make sure you understand the differences between
   • categorical (also known as nominal) and
   • ordinal survey items
6. Make a teaching document that captures all the information you learned in steps a. and
7. above.
8. Your audience for your teaching document needs to be either other college students or people in industry (not psychological scientists).
9. Your medium for your teaching document needs to be either a power point document or an
10. Save your teaching document as a PDF, named pdf.
11. Test yourself on Professor (no date), “Examples of Bad Questions & How to Fix Them.”

 

1. Go to the discussion board forum “Module #10, Part 2: Teaching Document” and make a new post in which you
   1. attach your teaching document;
   2. tell us the intended audience of your teaching document and why you chose that intended audience; and
   3. tell us how well you did identifying the examples of bad questions and fixing

them (on Professor  quiz).

Writing Assignment

 

1. Now it’s time to apply everything you’ve learned about writing survey items and write the items for your two
   1. For each of your two surveys, you must write no fewer than 6 and no more than 12
   2. For each of your two surveys, you must include at least one and no more than two open-ended items (which count toward your total 6 to 12 total items per survey).
   3. For each of your two surveys, you must include
      • at least one categorical/nominal survey item and
      • at least one ordinal survey item.
   4. For each of your two surveys, you should include at least one demographic item, you must place this (these) at the end (unless there’s a good reason to place them earlier), and your demographic items count toward your total 6 to 12 total items per

 

1. After you’ve written a draft of your items for each of your two surveys:
   1. Check your items against the teaching document you made in Part 2 of this assignment.
   2. Be sure to follow the beneficial procedures you told others to
   3. Be sure you don’t commit any errors you told others not to
2. Save your two sets of items in one PDF, making it clear in your PDF which survey topic each set of items was written to assess. Name your

PDF YourLastname_SurveyItems.pdf.

 

1. Go to the discussion board forum “Module #10, Part 3, Survey Items” and attach your PDF that includes your two sets of

Action Assignment

 

1. You will use Google Forms to create your

 

 

1. Using Google Forms, create each of your two surveys. Be sure to create a meaningful (and inviting) title for each of your two

 

1. Read Tague’s (2004b) steps for pilot-testing a survey, highlighted in yellow in the brief

article, “How to Administer a Survey.”

 

1. Collect pilot data from three participants on each of your two surveys.

 

1. Your three pilot participants for each of your two surveys can be the same three people.
2. Or your pilot participants can be two different sets of three people or a mixture of same and different.

 

3. But you must collect pilot data from three participants on each of your two surveys.

1. Go to the discussion board “Module #10, Part 3, Survey Items” and reply to the original post you made there with a new post in which you
   1. provide a link to each of your two surveys (using the title of each of your surveys for the words that are linked); and
   2. write at least 200 words about what you learned from collecting pilot data about each of your two surveys, including what changes you need to make to each survey before you administer the survey to non-pilot

READING

 

1. Before you administer your surveys, use Beretta’s (2014) article, “Top Ten Common Problems in Designing Effective Survey Questions,” to assess them according to each of Beretta’s ten common

 

 

 

ACTION ASSIGNMENT

 

1. Make changes to your surveys based upon the pilot data you collected (this should be based on what you indicated needed to be changed in part 4 of this assignment).
2. Proceed to collect data (you will email links to your participants).
   • You will collect 15 participants per survey, and the 15 participants can be the same participants for both surveys or some combination of same and different;
   • However, none of the 15 participants on either survey can be the same as your pilot-participants for that survey
   • Post the links to your two revised surveys on the discussion board forum “Module #10, Part 4: Official Surveys”. The reason you’re posting the link here is in the hopes that some fellow students will complete your survey. But, remember, we can’t force anyone to do this! So, make sure to actively pursue participation in your study from people you
3. You don’t have to turn the data in yet (but you’ll be using it in the next

assignment). For now, just hold on to it!