The Questionnaire and Observation Research Strategies 9 Internal and 8 External Killers: Truth About Your Surveys and Observations Lying to You? While surveys and observations can be misleading, the degree to which this is the case depends on their design and human behavior.
Truth About Your Surveys and Observations Lying to You? Questionnaire and Observation Research Strategies 9 Internal and 8 External Killers
Factors such as social desirability, memorability, and incentives influence truthful responses. To get closer to the truth, you should implement fraud protection measures in online surveys, use triangulation by combining multiple methods, and carefully phrase sensitive questions to avoid bias and ensure data reliability.
The Two-Sided Validity Challenge in Postal Questionnaires
Validity of postal questionnaires can be seen from two viewpoints according to Belson (1986).
The Honesty Problem: Are Respondents Telling the Truth?
First, whether respondents who complete questionnaires do so accurately, honestly and correctly.
The Missing Voices Problem: What About Non-Respondents?
Second, whether those who fail to return their questionnaires would have given the same distribution of answers as the returnees did.
Checking for Accuracy: The Intensive Interview Method
The question of accuracy can be checked by means of the intensive interview method, a technique consisting of twelve principal tactics that include familiarization, temporal reconstruction, probing and challenging. The interested reader should consult Belson (1986:35–8).
Tackling Non-Response Bias: The Volunteer Problem
The problem of non-response (the issue of ‘volunteer bias’ as Belson calls it) can, in part, be checked on and controlled for, particularly when the postal questionnaire is sent out on a continuous basis. It involves follow-up contact with non-respondents by means of interviewers trained to secure interviews with such people. A comparison is then made between the replies of respondents and non-respondents.
12 Proven Strategies to Boost Your Response Rates
Further, Hudson and Miller (1997) suggest several strategies for maximizing the response rate to postal questionnaires (and, thereby to increase reliability). They involve:
Strategy 1: Make It Easy – Include Stamped Envelopes
Including stamped addressed envelopes;
Strategy 2: Don’t Give Up – Follow Up Multiple Times
Multiple rounds of follow-up to request re turns;
Strategy 3: Show the Value – Stress Importance and Benefits
Stressing the importance and benefits of the questionnaire;
Strategy 4: Appeal to Their Cause – Highlight Client Group Benefits
Stressing the importance of, and benefits to, the client group being targeted (particularly if it is a minority group that is struggling to have a voice);
Strategy 5: Share Results – Keep Non-Respondents Engaged
Providing interim data from returns to non-returners to involve and engage them in the research;
Strategy 6: Update Your Database – Check and Change Addresses
Checking addresses and changing them if necessary;
Strategy 7: Go Personal – Follow Up with Phone Calls
Following up questionnaires with a personal telephone call;
Strategy 8: Make It Personal – Tailor Your Approach
Tailoring follow-up requests to individuals (with indications to them that they are personally known and/or important to the research—including providing respondents with clues by giving some personal information to show that they are known) rather than blanket generalized letters;
Strategy 9: Design Smart – Consider Questionnaire Features
Features of the questionnaire itself (ease of completion, time to be spent, sensitivity of the questions asked, length of the questionnaire);
Strategy 10: Offer More – Invite Follow-Up Interviews
Invitations to a follow-up interview (face to face or by telephone);
Strategy 11: Get Help – Use Friendly Third Parties
Encouragement to participate by a friendly third party;
Strategy 12: Know Your Audience – Understand Your Sample
Understanding the nature of the sample population in depth, so that effective targeting strategies can be used.
Questionnaires vs. Interviews: The Great Debate
The advantages of the questionnaire over inter views, for instance, are:
Why Questionnaires Win: The Four Key Advantages
It tends to be more reliable; because it is anonymous, it encourages greater honesty (though, of course, dishonesty and falsification might not be able to be discovered in a questionnaire); it is more economical than the interview in terms of time and money; and there is the possibility that it may be mailed.
Its disadvantages, on the other hand, are:
Why Questionnaires Fail: The Seven Major Drawbacks
there is often too low a percentage of returns; the interviewer is able to answer questions concerning both the purpose of the interview and any misunderstandings experienced by the inter viewee, for it sometimes happens in the case of the latter that the same questions have different meanings for different people; if only closed items are used, the questionnaire may lack coverage or authenticity.
If only open items are used, respondents may be unwilling to write their answers for one reason or another; questionnaires present problems to people of limited literacy; and an interview can be conducted at an appropriate speed whereas questionnaires are often filled in hurriedly. There is a need, therefore, to pilot questionnaires and refine their contents, wording, length, etc. as appropriate for the sample being targeted.
The Sampling Crisis: When Your Data Tells Lies
One central issue in considering the reliability and validity of questionnaire surveys is that of sampling. An unrepresentative, skewed sample, one that is too small or too large, can easily distort the data, and indeed, in the case of very small samples, prohibit statistical analysis (Morrison, 1993).
The Double Validity Challenge in Observation Studies
There are questions about two types of validity in observation-based research.
External Validity: The Generalization Problem
In effect, comments about the subjective and idiosyncratic nature of the participant observation study are to do with its external validity. How do we know that the results of this one piece of research are applicable to other situations?
Internal Validity: The Observer Bias Problem
Fears that observers’ judgements will be affected by their close involvement in the group relate to the internal validity of the method.
When Observations Go Wrong: The Reality Check
How do we know that the results of this one piece of research represent the real thing, the experimental treatments, in fact, make a difference in the specific experiments under scrutiny? External validity, on the other hand, asks the question, given these demonstrable effects, to what populations or settings can they be generalized?
The 9 Internal Validity Threats That Apply Everywhere
1. History Effects: External Events That Confound Results
History Frequently in educational research, events other than the experimental treatments occur during the time between pretest and post-test observations. Such events produce effects that can mistakenly be attributed to differences in treatment.
2. Maturation: Natural Changes Over Time
Maturation Between any two observations subjects change in a variety of ways. Such changes can produce differences that are in dependent of the experimental treatments. The problem of maturation is more acute in protracted educational studies than in brief laboratory experiments.
3. Statistical Regression: The Pull Toward Average
Statistical regression like maturation effects, regression effects increase systematically with the time interval between pretests and posttests. Statistical regression occurs in educational (and other) research due to the unreliability of measuring instruments and to extraneous factors unique to each experimental group.
Regression means simply that subjects scoring highest on a pretest are likely to score relatively lower on a post-test; conversely, those scoring lowest on a pretest are likely to score relatively higher on a post-test. In short, in pretest-post-test situations, there is regression to the mean. Regression effects can lead the educational researcher mistakenly to attribute post-test gains and losses to low scoring and high scoring respectively.
4. Testing Effects: When Measurement Changes Behavior
Testing Pretests at the beginning of experiments can produce effects other than those due to the experimental treatments. Such effects can include sensitizing subjects to the true purposes of the experiment and practice effects which produce higher scores on posttest measures.
5. Instrumentation Problems: Unreliable Tools
Instrumentation Unreliable tests or instruments can introduce serious errors into experiments. With human observers or judges or changes in instrumentation and calibration, error can result from changes in their skills and levels of concentration over the course of the experiment.
6. Selection Bias: Wrong Participants, Wrong Results
Selection Bias may be introduced as a result of differences in the selection of subjects for the comparison groups or when intact classes are employed as experimental or control groups. Selection bias, moreover, may inter act with other factors (history, maturation, etc.) to cloud even further the effects of the comparative treatments.
7. Experimental Mortality: The Dropout Dilemma
Experimental mortality the loss of subjects through dropout often occurs in long-running experiments and may result in confounding the effects of the experimental variables, for whereas initially the groups may have been randomly selected, the residue that stays the course is likely to be different from the unbiased sample that began it.
8. Instrument Reactivity: When Tools Change What They Measure
Instrument reactivity the effects that the instruments of the study exert on the people in the study (see also Vulliamy, Lewin and Stephens, 1990).
9. Selection-Maturation Interaction: The Confusing Combination
Selection-maturation interaction where there is confusion between the research design effects and the variables’ effects.
The 8 External Validity Threats That Limit Generalization
Threats to external validity are likely to limit the degree to which generalizations can be made from the particular experimental conditions to other populations or settings. Below, we summarize a number of factors (adapted from Campbell and Stanley, 1963; Bracht and Glass, 1968; Hammersley and Atkinson, 1983; Vulliamy, 1990; Lewis-Beck, 1993) that jeopardize external validity.
1. Vague Variable Descriptions: The Replication Problem
Failure to describe independent variables explicitly unless independent variables are adequately described by the researcher, future replications of the experimental conditions is virtually impossible.
2. Unrepresentative Samples: The Population Mismatch
Lack of representativeness of available and target populations Whilst those participating in the experiment may be representative of an available population, they may not be representative of the population to which the experimenter seeks to generalize her findings, i.e. poor sampling and/or randomization.
3. Hawthorne Effects: When Being Studied Changes Everything
Hawthorne effect Medical research has long recognized the psychological effects that arise out of mere participation in drug experiments, and placebos and double-blind designs are commonly employed to counteract the biasing effects of participation. Similarly, so-called Hawthorne effects threaten to contaminate experimental treatments in educational research when subjects realize their role as guinea pigs.
4. Poor Operationalization: When Lab Doesn’t Match Life
Inadequate operationalizing of dependent variables Dependent variables that the experimenter operationalizes must have validity in the non-experimental setting to which she wishes to generalize her findings. A paper and pencil questionnaire on career choice, for example, may have little validity in respect of the actual employment decisions made by undergraduates on leaving university.
5. Pretest Sensitization: The Measurement Effect
Sensitization/reactivity to experimental conditions as with threats to internal validity, pretests may cause changes in the subjects’ sensitivity to the experimental variables and thus cloud the true effects of the experimental treatment.
6. Interaction Effects: When Multiple Factors Collide
Interaction effects of extraneous factors and experimental treatments all of the above threats to external validity represent interactions of various clouding factors with treatments. As well as these, interaction effects may also arise as a result of any or all of those factors identified under the section on ‘Threats to internal validity’.
7. Unreliable Instruments: When Tools Fail You
Invalidity or unreliability of instruments the use of instruments which yield data in which confidence cannot be placed (see below on tests).
8. Ecological Validity: The Context Transfer Problem
Ecological validity, and its partner, the extent to which behavior observed in one con text can be generalized to another.
The Fundamental Relationship: Internal vs External Validity
Hammersley and Atkinson (1983:10) comment on the serious problems that surround attempts to relate inferences from responses gained under experimental conditions, or from interviews, to everyday life. By way of summary, we have seen that an experiment can be said to be internally valid to the extent that within its own confines, its results are credible (Pilliner, 1973); but for those results to be useful, they must be generalizable beyond the confines of the particular experiment; in a word, they must be externally valid also.
Pilliner points to a lopsided relationship between internal and external validity. Without internal validity an experiment cannot possibly be externally valid. But the converse does not necessarily follow; an internally valid experiment may or may not have external validity.
The Welsh Children Example: Why Perfect Internal Validity Isn’t Enough
Thus, the most carefully designed experiment involving a sample of Welsh-speaking children is not necessarily generalizable to a target population which includes non-Welsh-speaking subjects. It follows, then, that the way to good experimentation in schools, or indeed any other organizational setting, lies in maximizing both internal and external validity.
Read More:
https://nurseseducator.com/didactic-and-dialectic-teaching-rationale-for-team-based-learning/
https://nurseseducator.com/high-fidelity-simulation-use-in-nursing-education/
First NCLEX Exam Center In Pakistan From Lahore (Mall of Lahore) to the Global Nursing
Categories of Journals: W, X, Y and Z Category Journal In Nursing Education
AI in Healthcare Content Creation: A Double-Edged Sword and Scary
Social Links:
https://www.facebook.com/nurseseducator/
https://www.instagram.com/nurseseducator/
https://www.pinterest.com/NursesEducator/
https://www.linkedin.com/in/nurseseducator/
https://www.researchgate.net/profile/Afza-Lal-Din
https://scholar.google.com/citations?hl=en&user=F0XY9vQAAAAJ
I like this site so much, saved to bookmarks.