Theses and dissertations

As you go ...

During your fieldwork, you have other tasks besides collecting data:

1. Keep written notes of methodology in fieldwork, that is, what you actually did, not just what you planned to do.
2. Put your raw data into a form that is easy to read and interpret.
3. Stay alert to what the data means and how it relates to your research problem. This will be necessary for your analysis of the data. for some kinds of methods, it is good practice to start analyzing data before you have finished collecting it.
4. Identify any new assumptions or delineations as you go. Besides, as you reflect on them, they might form a pattern of core and peripheral issues.
5. As you use or discover terminology, you might find that it has either unique or multiple meanings. Write and refine definitions.
6. Keep backups, because computer data can be lost very easily.

You might also have time to start working on you full draft, progressing chapter by chapter.

Recruiting participants

If you had to get organizational approvals to do research within those organizations, you probably already have an idea of how you will recruit participants.

However, you might still have to recruit individuals. People tend to respond positively if they already know you personally or are somehow already connected to you. And if they honestly can't help you, they'll probably tell you why. Cold calling and junk mail are not recommended; they are simply more clutter in people's lives and people frequently ignore them.

Prepare the information for prospective participants. This typically includes:

1. What kind of people do you need? (Description of research population)
2. What are you asking them to do and why?
3. Any conditions (risks, confidentiality, voluntary, participation)
4. How to apply
5. How to ask questions for more information
6. Your contact details

Pre-fieldwork

The purpose of pre-fieldwork is to identify issues that could arise and affect the validity of your findings, but could not be anticipated during the literature review.

In many cases, before you start research proper, you can use a simple interview questionnaire with open ended questions, simply to trawl for unexpected relevant issues. You might then decide to include those new issues in your research plan. Another pre-fieldwork starting point is to ask Grand Tour questions. That is, get some people to explain the basics, the main points, or the scenario. This will help get a feel for all that is involved (so you don't miss out important things) and how they perceive it (for example, split it into categories).

Submit a report of your pre-fieldwork to your supervisor. It should show that you have identified any otherwise unforeseen issues and that you are ready to proceed to the next stage.

Draft and test data collection tools

Write any data collection tools that you plan to use and validate them. For example, you might write a questionnaire, test it with a sample of respondents from the target population, and then revise it, going through the process until it works.

In your report, say exactly what you did and how you did it, and any modifications needed. It should show that they are now appropriate for your planned research. The report should be suitable for eventual inclusion in the methodology chapter of your dissertation. You also need to decide how much data will you need to get for your results to be valid.

Other than improving your tools, you should ask several otehr questions:

1. What kinds of other problems might you face collecting data? For example, if you use on-line forms, does the technology work as planned? Do people find it easy to use?
2. What is feasible and what is not? For example, consider your time-frame and budget. How much do you need to travel? If you need meeting rooms, are they available?
3. How effective were your methods of recruiting participants? Is it easy to get enough people?
4. Now that you have some data, how good are your processes for analyzing it?

Why you need to pre-test questionnaires

Even if a question is very well written, it does not mean that respondents will automatically understand it straight away. All questionnaires need testing with people before you use them widely. You might understand your questions perfectly, but you can't know how others might interpret them. Time spent in testing is well-spent. Imagine the frustration of having to start over from scratch because your questionnaire was found to have a serious flaw late in the fieldwork.

Even some very simple questions cause confusion. They are so simple that people don't take them on face value. They look like a trap or seem to ask for some kind of hidden, deeper answer. For example, How much is a one-dollar candy? The answer is so obvious that it can be frustrating, and it seems like you're fishing for another answer.

Expect to be surprised that some people will not understand even the most obvious (to you) of questions. Then, when you ask, they produce the most logical and obvious responses that had never occurred to you. Your questions could:

• use unfamiliar terms
• sound complicated
• sound vague
• sound foreign (if you're using another language)
• encompass more than one issue
• make people feel embarrassed, frustrated or stupid. For example, Have you stopped beating your wife? (Whether you answer yes or no, it still means that you have been beating your wife.)

Other causes of confusion might be:

• unclear layout
• innocent questions that are not intended to indicate confusion
• re-reading things to figure out what the materials say
• unexplained assumptions
• language, literacy or numeracy difficulties
• out-of-sequence explanations
• not enough instructions on what to do
• look like you are trying to trap them into giving a particular answer
• confusing procedures
• too much or too little information
• incorrect information
• boredom: lack of options
• frustration: activities that are too difficult for the target population.

Questionnaires with open questions are much more flexible, especialy if you use them as a basis for interviews in which you can spontaneously add follow-up questions or talk people around in a circle. It might also be possible to let the questionnaire evolve during fieldwork so that you can follow up on emerging trends.

Questionnaires with closed questions, however, are very different. They need thorough, extensive testing because respondents must choose between the answers you provide. Consequently, your list of answers must allow all possibilities. Don't even think of using the questionnaire until you know it works very well.

Steps for testing questionnaires

Your goal is that respondents can read and understand each question straight away, and give an honest answer.

After proof-reading, test your first draft with colleagues to eliminate the most obvious glitches.

Then do your testing with persons of the target population who have not yet seen the materials nor been asked your questions in any other form. People who already know the subject matter or saw previous drafts of materials won't be tripped up as if they were doing the questionnaire from scratch.

Obviously, you will also ask for verbal feedback, but the non-verbal feedback can just as useful. If possible, observe them for non-verbal clues. If something is unclear, some will stop and re-read it. Others will be clearly frustrated. Others will just decide to skip that bit, and might come back to it later. Some might even give up and choose random answers.

If the topic causes embarrassment, you will need culturally-appropriate questions, and some indirect questions might work better. Some people will try to guess the answers you want. In any case, thorough testing will indicate the quality of your data.

Note how much extra help you give people. Even your unwitting use of body language or perceptible attitude may alter people's responses. Don't help people if they must be able to do the questionnaire with no help at all.

You should see how long it takes them. Respondents normally should be able to do a written survey questionnaire in 15-30 minutes. Some are less.

Then make corrections and test your questionnaire again on a new group of people from the target population who have never seen the questions before, and collate suggestions for improvement. You need to use different people each time, because they need to be doing the questionnaire for the first time; they can no longer give their first impression of the meaning of the questions. If necessary, you can repeat this kind of test more widely with other new groups of people from the target population.

Decide when to stop. You can keep improving forever, so stop when your questionnaire is good enough, that is, when the incoming suggestions are trivial and you are satisfied with it.

See also: https://www.youtube.com/watch?v=DRL4PF2u9XA

Gather first round of data

Gather your first round of data and present a report on it. With some kinds of methodologies, this will be a pilot project, that is, a small-scale project to test the metholodogy.

In your report, include:

• a description of the amount and quality of data collected
• the recording technique
• an assessment of your methodology and any adaptations you needed to make
• any threats to validity of your research
• anything odd or unexpected, such as adverse incidents or counter-intuitive findings
• any emerging trends
• any required further research for the project to meet its research goals. (Some methodologies permit changes of direction partway through, while others disallow it.)

If you find deficiencies in your methodology, report any corrections or modifications you had to make or plan to make.

Note: If you are giving interviews, some supervisors recommend that you use two separate audio recorders and keep both running so you have a backup.

Midway review

The midway review is your supervisory committee's check on your progress so far. A positive assessment indicates that you are making good progress and are on track to complete. The committee may also decide to set any conditions or corrective actions that it believes are necessary for you to achieve satisfactory completion.

As you learn more about the topic, you might find that the question you started with is not the best question. You find that the real problem is not what you thought it was, despite being based on your literature review. This is quite normal in some kinds of research, and doesn't necessarily mean that you made a mistake. It is more likely a sign of progress, because the new understanding of the issue is a result of your research. You could use the better question, and either simply go back to the fieldwork stage and do another iteration to get more data, or plan to do new research fieldwork.

A more concerning eventuality is the publication of research that solves your research problem, making your topic obsolete as a contribution to knowledge. Ask your supervisor. (In the best case, it is simply an addition to your literature review.)

You might be required to provide a written report to your supervisory committee, indicating what you have done so far. Typical questions that you might need to answer are:

1. To what extent has your research complied with the original proposal?
2. Are you on schedule?
3. Is your original definition of the research question still adequate?
4. How effective has your methodology plan been?
5. How much raw data have you already gathered?
6. Does your raw data so far indicate that you will be able to draw conclusions?
7. Have any patterns, trends or anomalies already arisen in the data? If so, describe them.
8. What major obstacles do you currently face?

Fieldwork

Continue gathering data in the field, and start on a rough draft. You don't have to send it in, but you should make regular progress reports.

Implement your method(s). Get out there and meet people. If you're naturally a little afraid, this will be a major challenge, and procrastinating won't help.

Keep complete field notes as you go. Note down observations either at the time you make them or immediately after. (Contemporaneous is the key word.) DON'T trust your memory. If you collect enough information it is inevitable that you'll forget something important.

Keep records of:

• Exactly what you do, why, and information that you get
• When, where, who, why, etc. of interviews
• Assumptions arising or that you must make
• Factors that help define your project more precisely
• Changes to planned method (and the reasons for those changes)
• Ethical issues and how you handled them
• Special factors that you had to consider
• Terms with special meanings
• Data that were unexpected or don't seem to fit
• Patterns that appear
• In fact, all your ideas, observations and anything else that might be relevant.

You probably won't use them all in the end, because your thinking will develop over the time of the research. But even the ones you don't use will be valuable in helping your thinking to develop. And of course, some will survive into the final work. The way to have them is to write them down and you'll be able to quote them directly later on.

Safeguard your notes. Many good projects never get finished because the notes are lost. Keeping them organized will make them more useful.

Start on a write-up. You'll probably want to start a rough draft that is a couple of chapters behind what you're doing in the field. You might then try to polish some earlier chapters while you're still working on drafts of the later chapters.

A gentle warning: be careful not to get so busy writing that you become less productive in the field. On the other hand, don't get so busy researching that you procrastinate on writing.

When to stop collecting data

You know when stop collecting data when you meet three criteria:

1. The data adequately represents the population.
2. Further data-gathering would not uncover any new information.
3. You have enough good data to draw valid conclusions.

In quantitative research, your sampling system should ensure that you meet meet these conditions.

In qualitative research, it is not quite so simple. Respondents have almost certainly given you a variety of different answers. Do you really know why, or are you just guessing? If you feel like this, you still need to keep collecting data. So how do you know when do you stop?

As you collect more data, a clear pattern should eventually emerge in the data regarding the phenomenon being researched. Just noticing a pattern is not adequate; you need enough data to confirm the pattern and incontravertably answer the original research question. In this case, more data would not improve, change, strengthen, nor add to your conclusion. In theory, it mightn't be the only possible pattern, but in practice there's usually only one. Later on, demonstrating the pattern in the data is a large part of your analysis.

In ethnography, it is usually possible to keep collecting more and more valuable data, but the three criteria still apply. You can stop collecting data when you have enough to answer your research question and to confirm your conclusions.

The point of enough data depends on:

1. The clarity of the research question. It is easier to gather enough data if the research question is very clear, the definition of the population is very clear, and your data-gathering approach is focussed. (Put another way, it will be easier to know when you have found something if you have a clear idea of what you're looking for.) In practice, you keep collecting data until it produces no new information, and then you can identify the saturation point in hindsight.
2. The capacity of the method to give sufficient detail. For example, a survey is good for collecting smaller items of data from a large number of respondents, while detailed interviews are normally best used with a smaller number of participants.
3. The quality of data (detail or richness) in interviews. Your interviewees might not have given enough detail for you to answer aspects of your research question.
4. The complexity of the phenomena under investigation. A phenomenon with more different aspects requires more data.
5. Whether data is triangulated, that is, you collect data through different methodologies on the same topic.
6. Whether the phenomenon under investigation is common or rare. The rarer the phenomenon, the less likely it is to occur in any one respondent, so you need more data.
7. The exent of homogeneity in the population. It will be easier to reach saturation in an homogenous group, but a very heterogenous group might give a random selection of very different answers. Consequently, you will probably need more respondents in a heterogenous group to get enough data to support a specific conclusion.

Grounded theory and saturation

In grounded theory, the point when of enough data is called saturation. The rule of thumb is that data saturation occurs when three consecutive interviews do not reveal any new data. However, this is only a rule of thumb, and some data topics might still be unexplained, necessitating more interviews.

Identifying the saturation point is normally a matter of judgement; there is no mathematical formula for saturation, although many have tried. In a dissertation, it will to some extent depend on your supervisor or committee chair. When you have believe you have reached saturation, present your case and see whether he/she agrees. If so, move on. If not, why not?

The role of iteration

The Research Cycle answers the question: I think I might have done enough fieldwork, but do I need to go back and do more?

For this reason, qualitative research is called iterative; researchers can often to do another iteration of data collection. A theoretical sample is a sample that evolves according to findings as they emerge. If a particular line of inquiry is fruitful, the researcher can add more repondents that will help to further explore it. If you want to use a theoretical sample, you should specify it in your proposal.

❓ What if the data does not lead to a firm conclusion?
You have these options:
• Did you fail to notice the pattern in the data?
• Should you do another iteration of data collection and analysis?
• Should you qualify your conclusions rather than make definitive statements. For example, you might say that something tends to ...
• Should you suggest further research with a different or larger population, or with a different sampling technique?

❓ What should I do? I don't have enough data because I can't see any pattern.
Your problem might be that you can't see the pattern, not insufficient data. It might be more difficult to see a pattern if it is quite different from what you expected. Speak to your supervisor. (Note: If your research is examining the correlation of two variables, the pattern could be a null hypothesis, that is, the two variables do not correlate.)

❓ Do I also have to explore thoroughly all emergent themes?
You should explore them if they indicate new findings that affect your conclusions or the direction of your research. Otherwise, it might be enough to confirm that they exist and define them accurately.

❓ I've collected all the data that I'd planned. But the data still has gaps and raises questions that I can't answer. Should I collect more data or should I simply accept it and stop collecting data?
Collecting more data would be appropriate if you have a reason to believe that more data fill would fill those gaps. Otherwise, you should probably accept that it represents the current state of knowledge out there. Analyze the data you have and consider suggesting topics for further research. (By the way, if you have already noticed gaps in the data, then you have already identified a pattern. That's progress.)

References PLoS One. 2018; 13(6): e0198606. Published online 2018 Jun 20. doi: 10.1371/journal.pone.0198606.
Fusch, P. I., & Ness, L. R. (2015). Are We There Yet? Data Saturation in Qualitative Research. The Qualitative Report, 20(9), 1408-1416. Retrieved from https://nsuworks.nova.edu/tqr/vol20/iss9/3
Ray Galvin How many interviews are enough? Do qualitative interviews in building energy consumption research produce reliable knowledge? Just Solutions Cambridge Working Paper 2014B. Published in The Journal of Building Engineering, 2015.
https://www.researchgate.net/profile/David_Morgan19/post/Whats_is_ideal_sample_size_in_qualitative_research/attachment/59d6301679197b807798e33b/AS%3A360974889570309%401463074528665/download/Galvin%2B15%2BHow%2Bmany%2BInterviews.pdf. (Viewed 1 October 2018)

About outliers and anomalies

Your data must be ready for the next stage of your thesis or dissertation. This includes accounting for any outliers and anomalies, which are items of data that do not fit the data pattern. Some sources refer to these as discrepant data, that is, data that is inconsistent with either the assumptions of the research, the literature, or any emerging hypotheses.

While it can arise through good research methods, it can also arise from methodological mistakes where the researcher only looks for what he/she wants to find.

Whatever the case, you will need to account for them. How these are handled depends greatly on the kind of research and the particular field:

• They might indicate an error in methodology or data collection, which must be corrected for the dataset to be considered valid.
• They might be normal and not problemmatical. For example:
• In ethnography and sociology, they might indicate individuals who come from an unusual set of circumstances that was not anticipated in the definition of the population.
• In linguistics, residue items are fairly normal. They might indicate idioms, archaeisms, or sociolinguistic effects.
• In some statistical methods, small numbers of outliers are insignificant and inconsequential, so researchers can simply delete them.
• Exceptions to the rule might indicate an unusual phenomenon that invites further research. Some of these surprises turn the research in another direction, or create new topics for further resesarch. This is a good thing and not unusual; it is especially the case when some data items are counter-intuitive.

New, completely unexpected themes in a qualitative study might emerge during the fieldwork or data analysis. These themes might even appear unrelated to the research problem and conceptual framework, and it would be dishonest to re-interpret the data to make it fit a set of expected conclusions. It can be especially annoying if something shows that the research problem/question should be reframed differently. If the loose end prevents you from drawing any conclusions, your research might be stuck with nowhere to go.

The question is then What you do to tie up such a major loose end?

My preferred option would be to explore the topic in an implications chapter, and perhaps revise definitions and create further research questions. That's high value. If it’s not possible, consider these other options:

• Check the data for errors.
• Take the whole research through another iteration.
• Do some further fieldwork to explore those particular themes in enough depth to draw conclusions. (Common in ethnography.)
• Go back to the literature review and add anything written about those themes. (Not a good option; it presumes that the surprise was foreseeable in the literature, and the whole point is that it wasn’t.)

In most cases, this will blow out the time it takes to finish the research.