Theses and dissertations

Methodology

To start this stage, your first objective is to decide on one or more particular methodologies suited to your topic.

The research problem should drive the research, not the methodology. The truism for people with only one research method is that: If the only tool you have is a hammer, then everything looks like a nail. But if you should choose from a variety of tools, you can select the right one for the job.

💡 Tips

1. Give the reasons for your methodology decisions. If you do not do so, your whole dissertation is at risk of failing. Your examiners must be able to see why you made those decisions.
2. If your results might be controversial, use a conservative methodology with minimal adaptation. (An innovative but unproven methodology would attract criticism, even if it were very sound.)
3. Some methods change with time. Check that your proposed methods are up to date. It might also be helpful to know the reasons for the changes.

Do you need more than one method?

If you collect data in the field, it might be preferable to use at least two different methods. For example, the results of one method may be used to confirm the results of another method (called triangulation). Alternatively, you might use one method to get small amounts of data from a large sample of subjects, and another method to get detailed data from a small sample. Consider these ways of usiing multiple methods:

Mixed methods.

In brief, you have two kinds of mixed method options. In the first, use a qualitative method first to find and explore dynamics, then use a quantitative method to confirm their occurance is the wider population. In the second, use a quantitative method to establish a set of phenomena, then use a qualitative method to explore the reasons for that phenonema.

Triangulation.

This is the collection of different kinds of data and then comparing them. It has three main purposes:

1. to confirm or evaluate the accuracy of data and improve the validity of your conclusions and,
2. to depict in greater detail the dynamics of what is occurring, and
3. to explore different ways of understanding the research problem.1

It need not be difficult. Consider, for example, these different methods: document review, observations, interviews, focus groups, and surveys.

The data sets might converge if they give the same or very consistent results, but comparison does not always simply confirm your data. They might complement each other, perhaps by adding explanations for phenomena. For example, a survey might get feedback from a large population, while interviews and focus groups tend to explore causes and perceptions. Alternatively, the data sets might lead to very different conclusions, and you should explore the reasons for the divergence. This can be very illuminating and rewarding, so do not consider it an error if they are inconsistent.

1 Triangulation, Andrea J. Nightingale, in International Encyclopedia of Human Geography Second ed., 2020. See also Triangulation, A. Nightingale, in International Encyclopedia of Human Geography, 2009.

Describe the limitations of your method

Your chosen methodology might be quite suitable for your purposes, but is inappropriate in particular circumstances or if used in particular ways. You can strengthen your statement of appropriate usage by clarifying its limitations.

Do you need to write a critical review of your methodology?

In some circumstances, your supervisor might ask you to write a critical review of your methodology, for example:

1. You have modified a common methodology so much that your supervisor thinks it needs clarification and defense.
2. You have adapted a methodology from another discipline.
3. Your proposed methodology might have an unusual epistemology or ontology, suggesting that it might not produce valid data.
4. Your proposed methodology might have an unusual epistemology or ontology, suggesting that it might not align with your research goals.
5. The methodology is old-fashioned.
6. The methodology is not normally used in original research. For example, it might be used mainly outside original research, or be so rarely used that the supervisor is unsure about it.
7. The methodology is so rarely used that the supervisor is unsure about it.

In these cases, it is much better to prevent any problem at this stage than to face awkward questions later on.

💡 Hint. This kind of critical review might make an excellent journal article.

Tool development

If you will develop your own new tools, you will need a procedure for writing and testing them.

Students normally write their own questionnaires, because questions need to be focused according to the particular research problem. Then, as part of the methodology, students must pilot and refine them.

Some kinds of questionnaires require complex quantitative validations and have rigid rules for collecting and analyzing data. In these cases, writing and validating the questionnaire might be the whole dissertation. If you need this kind of questionnaire, then you should find one that already exists, although some of them are tightly controlled, licenced, and available only for a fee.

Your supervisor might permit you to get questions (or whole questionnaires) from published dissertations as long as you reference the source and don't breach copyright. In some cases, your supervisor might require you to ask permission of the original writer. In this case, you already have evidence that the questionniare is valid and reliable.

Assistants

You may use assistants for routine work as long as you take full responsibility for them. If you use assistants, be sure to describe in detail the training and supervision you give them and accountability arrangements in sufficient detail for another person to replicate them. This description probably goes best either in the methodology chapter or (if it is long and complex) in an appendix. For ethical reasons, recognize the intellectual contributions made by assistants, either as references or in the preface, depending on the kind of contribution they make.

Data analysis

Your statement of methods should also include a plan for your data analysis. This will vary in complexity according to your method. For example, the interpretation of statistical data needs a clear procedure and might take considerable explanation to demonstrate validity. In contrast, qualitative ethnographic studies usually require relatively little explanation of a data analysis method.

The market currently has many different software packages for analyzing qualitative data. Working with large amounts of text or multimedia data, they can classify, sort and arrange it, and then examine relationships within it. You are not required to use software, and hand-coding might not be any more difficult. If you use software, choose a package that is be easy enough to learn and is suited your particular purpose. Compare the packages. Some are free, and might be as good as proprietary software. Some packages are perhaps not as good. See the wikipedia article.

Some universities allow students to subcontract a data analyst or statitician as long as students describe the details in their methodology. (It's like using research assistants.) It has pros and contras. It might be faster and, for some people, necessary to finish. However, it might be very expensive, and it's best if you know how the analysis works and can do it yourself.

Intercoder reliabillity

The coding process is the interpretation of raw data so that it is in a form that is useful for reaching a conclusion. If you are coding data manually, you might need to have someone else check the coding to increase its objectivity and reliability.

💡 Tips

1. Read the methodology chapters of dissertations:
• You can find a range of available methodologies if they are on the same kind of topic.
• When you have chosen a methodology, these chapters normally contain enough detail for you to replicate those methods. You will see how other researchers have implemented them and perhaps adapted them for particular situations. You might need to make adaptations of your own.
• You might be able to use existing tools. Validated commercial tools are often expensive, but you might be able to find free tools by looking in the main databases and reading dissertations on similar topics. Look at their literature reviews. If they use a tool that is useful to you, the dissertation will give procedures and comments for implementation and data interpretation.
2. You may justify your methodology by referring to research handbooks or other researches.
3. Laerd statistics can be very helpful for students doing quantitative work. It not expensive and gives step by step instructions on everything, including assumptions. https://statistics.laerd.com/

Selection and sampling

How will you select people? Will you research everybody in the population, a core group, a random sample, a purposive sample (group carefully selected to represent everybody in the community), or a snowball sample? In fact, some research guides include many other kinds of sampling techniques.

At doctoral level, your total population might be very large, for example, a whole state school system. You might decide to have several populations and compare them. In this case, you would choose a sample of schools, and in each one a sample of students. This would produce a data set that would better represent the whole state, rather than using only one school. (See stratified sampling and cluster sampling below.)

Everybody in the population

If you want to research everybody in the population you will need to use either a set of statistics, such as whole-of-instititution statistics (which may be quite unreliable or need very careful interpretation), or have a small enough population that you can conduct your own kind of census of everybody, tailored specifically for your needs.

You might have a population size as small as one, for example, a biography or a unique case.

Random sample

Mathematitions have developed various formulae for determining a sample size that fully represents the whole population, called a statistically representative random sample. Your supervisor might view it as the gold standard for quantitative methods. You can use an automated system from the internet as long as you reference it correctly.

However, it is not always that easy. It requires that you know the size of the population and who is in it to be able to calculate the sample size. Then, the size of the sample might be too big to be useful in practical research.

A smaller random sample to some extent still represents the whole population, because every member of the population has an equal chance of being selected.

Core group

A core group means the group of people who are either the key decision-makers, the influencers, or the gatekeepers of its knowledge. Ethnographers often deliberately seek out these people.

Purposive sample

To create a purposive sample (also called a controlled sample) is to define a population according to specific criteria relevant to the topic (e.g. demography, occupation, specific experiences) and carefully select a group to represent everybody in the community. It might be more or less random, depending on how the you recruit respondents:

• It is more like a random sample if you select respondents randomly from the population.
• It is less like a random sample if you use advertisments to recruit respondents.
• It is also less like a random sample if you select respondents. You can select people to whom you have access, while still being a good representation of the whole population. Select typical subjects or cases that are representative of the research population.

You must give compelling evidence for the selection criteria. For example, choose group members carefully so that the group has the same balance for gender, class, employment, age group, etc. as the general population. If it is big enough it should start to lessen the role of individual characteristics. Its advantage is that it represents the population but allows a smaller number than would be required for a random sample.

Theoretical sample

A theoretical sample is to use an intial purposeful sample, analyze thematic categories in the data to find emerging theory, and then add more repondents to the sample to test the validity of the theory and explore relationships between catagories. (Ref.)

Accidental sample

An accidental sample is to select subjects or cases based on convenience or on their availability. It really only means “the group of people who were easy to recruit.” It is difficult to claim that it is representative, but might be the only available method in some circumstances. This is sometimes called an opportunistic sample, or a sample of convenience.

It has several particular uses. First, it is a good option for early trials of procedures. In this case, the representativeness of a sample doesn't matter so much as long as the feedback is helpful in improving the procedure. Second, like snowball sampling below, it might be the best way to recruit people in a particular circumstance, where other methods are not practical or might distort findings. In qualitative studies, you must carefully qualify your conclusions so that they are still valid.

Otherwise, a convenience sample is not useful for quantitative studies where a random sample is required. It is also not recommended if a researcher is using it only as a “lazy person's option” and another kind of sample would be better.

Snowball sample

A snowball sample (sometimes called a networked sample) works well when groups of people are very secretive. It is a network of friends, relatives, and acquaintances, where one contact will lead to others. With this approach, you can easily get more people by having those you know introduce you to their friends and relatives. Because they are socially connected, you can only claim that the research accurately represents your sample—they cannot be considered fully representative of everybody else in the population. However, the aspect of trust can make the information more reliable, and the ability to access more informants through networks makes it easier to contact people.

Stratified sampling

Stratified sampling and cluster sampling are different methods of random sampling when a population comprises different groups (i.e. sub-populations). However, different sources use different (and conflicting) definitions so check them first, and choose the one that will work best for your topic.

Ethics

In many countries, any institution that conducts research must have a specialist committee for research ethics; in the US, the committee is often called the Institutional Review Board (IRB). Researchers must write an application for ethical approval before commencing research. The application is a separate document and must contain an ethical compliance strategy. It is assessed by the ethics committee, and you may not commence resarch until the IRB has approved your ethics proposal.

Some requirements vary according to the particular context, and you might need to have ethical reasons to support your proposed methodology.

No ethical guideline can be comprehensive enough to cover all disciplines and all circumstances, so students need to examine the particular circumstances of their proposed research. Different fields of study have different ethical guidelines, and students should also look for information in professional standards, dissertations on similar topics or done in similar situations, and other published research.

Different levels

Some projects are classified as practicum and not research because they do not aim to create new, generalizable knowldege. In this case, your supervisor might not require an ethical proposal at all. Otherwise, ethical compliance has different levels:

In some cases, the IRB can expedite (speed up) the approval process, while in others, it must go through a full review process.

1. Some research projects are ethically simple because they have no human or animal subjects.
2. Some projects might be classified as research if they explore improved professional practice.
3. Some categories of research are not subject to IRB legislation and only the institution's internal ethical standards apply.
4. Some research topics are ethically very simple and the ethical statement need not take much space. Some educational and management projects are done in existing organizations and do not involve any risks other than the organization’s normal ethical compliance. This especially applies to archival data and secondary data analysis because there is almost no potential harm to the sample. If it does not involve human or animal subjects, approval is often quite simple.
5. In other cases, however, ethical compliance is more complex. For example, you might be unable to give participants exact details of research purposes or ask them to sign a document if it would make them respond unnaturally and make the research invalid. In other cases, students cannot propose an ideal solution and must weigh up any factors pro and contra, such as asking whether the potential benefits outweigh any risks to participants. Students must also ensure that any data on individuals cannot be leaked. If a student contracts someone else to do data management (e.g. transcription, statistics, analysis), the institution needs to know that data is still secure. For example, a contract might be adequate security.

Research with human subjects

If you will collect information on individual human subjects, you will need to address a specific set of guidelines. Proposals must include as a minimum:

1. An interpretation of the research against the requirements of Code of Federal Regulations, Title 45 Part 46, Protection of human subjects.
2. The disclosure of information to subjects.
3. The means for obtaining written permission from research subjects and for storing those documents.
4. The methods of data collection.
5. The security of data against leak or theft, and the methods of archiving, destroying, or deleting it.
6. The method of de-identifying data on individuals, if applicable.
7. An assessment of the risk that de-identified data could be re-identified, if applicable.
8. The risk of any foreseeable harm to subjects.

Risk analysis

In any research that involves human or animal subjects, students must do a risk analysis:

1. Analyze the context (i.e. the particular circumstances of the proposed research);
2. Identify the risks, if any;
3. Assess the likelihood and severity of the risks;
4. Identify whom (participants, others) the risks might affect;
5. Put in place strategies to minimize those risks or reduce them to an acceptable level.

Proposal and methodology chapter

Your report should be suitable for inclusion in your proposal, and, ultimately, in the methodology chapter of your dissertation. Your readers need to know that your research was ethical, especially anybody attempting to replicate your research.

Writing the methodology plan

A methodology plan can be quite involved, and you need to be able to answer these questions:

1. How is the method you have chosen appropriate to what you wish to investigate? Explain your reasons clearly.
2. How will the method you have chosen affect your results? In other words, what kind of results will this method produce?
3. Why did you select that method rather than others?
4. Explain what would have happened if you had chosen those other methods.
5. Will your method work in all the situations you anticipate?
6. If you need to adapt it to your particular situation, how will you do so? (For example, it may not work as it is, and local factors may incline you to one method rather than another.)
7. How will you analyze data?

A possible plan outline for a qualitative study

1. Introductory paragraph, including a restatement of the purpose of your study.
2. Population
1. Describe the selection criteria for respondents.
2. Describe the sample size, and how you will select the sample.
3. Decribe how you plan to recruit participants.
3. Rationale
   1. Give your rationale for selecting a qualitative study, in contrast to a quantitative or mixed methods study. (If you need to include a critical review, this is the place to put it.)
   2. Give your rationale for selecting a specific design (e.g. phenomenological design compared to other designs such as narrative, case study, etc.)
   3. If you use a questionnaire, describe the rationale for the questions and the order you put them in.
   4. If you use interviews or focus groups, how structured or unstructured will meetings be?
   5. Explain how you will ensure reliability and validity of your data, including field-testing questionnaires.
4. Your planned procedures for gathering and analyzing data, for example:
1. Describe how you will conduct interviews (e.g. online or face to face). If online, which app will you use and why? (E.g. Zoom has a recording and transcribing feature.)
2. Describe your recording and transcribing procedure.
3. Will you allow participants to review the interview recordings and as a way to triangulate data?
4. Describe your proposed data analysis methods and any software that you will you use to code and analyze data.
5. Ethics
1. Discuss the ethical considerations of the approach and getting IRB approval before conducting the study.
2. How will you ensure privacy and confidentiality of data? (E.g. use pseudonyms instead of real names.)
3. Say how you will get consent from participants before conducting interviews.
4. How will you store data, the time period you will keep it, and how you will eventually destroy or delete it.
6. Closing paragraph.

💡 Hint

The methodology plan is much the same as the final methodology chapter. Later on, you simply edit the methodology plan to reflect what actually happened. The main differences between the plan and the chapter are:

• The plan is in future tense because it describes what you plan to do in future, while the chapter is in past tense because it reports something you did in the past.
• The chapter will contain a report of any adjustments that needed to be made later on.
• The chapter will report what actually happened. For example, you might have planned for between 100 and 200 respondents to a survey, with 50% male and 50% female. What actually happened is that you had 132 respondents, 43% male, 49% female, and 8% did not give a gender.

With thanks to Τεrry Ηαmmιlτοn.