Reporting on Qualitative Components of a Research Project

Reporting on Qualitative Components of a Research Project Lyn D. English1* 1 School of Teacher Education & Leadership , Faculty of Creative Industries, Education & Social Justice, Queensland University of Technology, Australia ARTICLE INFO ABSTRACT Article History Received: Oct 21, 2021 1 Revision: Oct 24, 2021 Available Online: Oct 30, 2021 In this brief commentary, I first overview the strengths and weaknesses of qualitative research in education and also briefly consider mixed methods. I then offer a few suggestions for reporting on particular methodological components and the results of qualitative research. I do not address the reporting on other significant components such as the literature review and theoretical framework, given that these aspects are generic to both qualitative and quantitative research. The points I make are brief and are by no means exhaustive. Nevertheless, it is hoped that they offer some guidance for authors in education who adopt primarily qualitative methods.


INTRODUCTION
Qualitative research methodology has expanded significantly in the last few decades, with several articles (e.g. Walsham, 2006), books and book chapters addressing the topic (e.g. Bikner-Ahasbahs, Knipping, & Presmeg, 2015;Kaiser & Presmeg, 2019). Mixed methods research has also grown in popularity as researchers capitalise on the strengths of both qualitative and quantitative approaches (e.g. Buchholtz, 2019). Choosing an appropriate research methodology depends on the research questions asked and the theoretical framework adopted. Furthermore, as Cai et al. (2019) emphasised, authors need to justify the research methods chosen.

WHY QUALITATIVE RESEARCH?
Qualitative research is undertaken when one wants to discover or learn more about a topic or a particular phenomenon; it is commonly used to understand people's experiences and perspectives (Johnson & Christensen, 2017). Qualitative data are far more diverse and complex than quantitative data. Qualitative data may comprise transcripts of video/audio recordings of classroom interactions, face-to-face interviews, and peer group reporting, together with students' written documents. Given the nature of the extensive data corpus, qualitative research can be very time-consuming and is thus usually confined to a smaller number of cases than would be addressed in quantitative research. On the other hand, the smaller number of cases in qualitative research enables greater insights into complex situations or dynamic processes. However, it is usually not possible to generalise the results or to verify/falsify hypotheses as one can with quantitative research (Buchholtz, 2019). To draw on the strengths of both research forms, a mixed methods approach is frequently adopted. Kelle and Buchholtz's (2015) metaphor for this approach is worth citing: Quantitative methods provide us with a general picture of the surface of the research field, while qualitative research can be used to drill deep holes into the field yielding the information necessary for indepth explanations (p. 354).
In-depth explanations are most frequently derived from qualitative content analysis, which can apply to both qualitative and mixed methods approaches. Irrespective of the methodology chosen, it is essential that the theoretical framework, together with the research questions, inform decisions about the design and conduct of an education study, including those from mathematics and STEM education. Hence, in reporting a qualitative study, it is important to indicate how and why the research methodology was adopted.

REPORTING ON QUALITATIVE RESEARCH METHODOLOGY
Given the diversity and complexity of qualitative research, the quality of its reporting is especially important (Twining, Heller, Nussbaum, & Tsai, 2017). Detailing the nature, number and background of the participants, the population from which they were drawn, and how they were selected should appear in the methodology section of a report. Where appropriate, this section should also describe the instruments administered, including their appropriateness for addressing the research questions, how the instruments were developed or sourced, their reliability for yielding the required data, and samples of items (or the entire instrument). Furthermore, information on who conducted the data collection, who was present and the nature of the relationships between them is often recommended (e.g. Twining et al., 2017). In other words, the context in which the research took place needs to be included. When authors omit aspects or all of this information, it can be difficult for reviewers to determine the validity of the results and any subsequent claims made (English, 2019).
Likewise, reports on studies involving treatments or classroom interventions should describe them in sufficient detail to enable their key features to be identified and applied in interpreting the results (AERA, 2006). The types of approaches adopted, examples of instructional materials or treatments implemented, and the duration and frequency of implementation or administration should also be stated. Again, the nature of the intervention should be guided/supported by the theoretical framework, which enables the reader to see how the study stemmed from its conceptual foundations.
The nature of the data analysis and how it was conducted are important features to include in the report. As previously noted, qualitative content analysis is one of the favoured means of data analysis. Several articles address the nature of qualitative content analysis, describing it as a systematic approach where the focus is on working with categories (codes) and developing a category system (or a coding frame; Kuckartz, 2019). It is important to provide examples of responses to illustrate the categories adopted in the coding process. It is beyond the scope of this commentary to undertake a detailed discussion of qualitative content analysis, but it is worth noting just a few points from . These include: • The categories of the final coding frame should be described as precisely as possible and the coding procedure should be reliable, that is, at least two coders agree in their coding. Categories for which no agreement can be reached must be excluded from the analysis: 'Content analysis stands and falls by its categories' (Kuckartz, 2019, p. 195). • Qualitative content analysis is conducted in several phases, ranging from preparing data, creating categories (which often comprises several cycles) and undertaking analysis, through to writing a report and presenting the results. Qualitative content analysis thus means more than just data coding. While coding is a key step in the analysis, it is ultimately preparation for the subsequent analytical steps. • The actual analysis phase 'consists of summarizing the data, and constantly comparing and contrasting the data. The analysis techniques can be qualitative as well as quantitative.' (Kuckartz,p. 195). • The quantitative analyses may comprise comparing the frequency of certain categories and/or subcategories for certain groups.

REPORTING RESULTS
In reporting the results of data analysis, it is essential to keep in mind the research questions being investigated. It should be made clear how the analysis addresses the research questions and leads to the outcomes (AERA, 2006). One approach to reporting the results is to organise them according to each research question, that is, revisit each question in turn (assuming more than one question). Only those results that actually answer the questions should be included. Other pertinent findings might emerge from the data analysis, in which case they could be incorporated within the discussion and cited as unanticipated outcomes. Such results could serve as one area for further research.
Ensuring that all claims and conclusions made are supported by the data is especially important (AERA, 2006) and is an aspect that can be easily overlooked even by experienced researchers. It is easy to over-interpret a study's findings, that is, to make claims that are not supported by the data produced. In documenting data outcomes, it is recommended that tables should be used only when they clarify or summarise outcomes involving multiple data points (Saracho, 2013). Generally, the fewer the tables the better, as too many can detract from a manuscript and extend its length, which can be problematic for those journals that have a strict page limit. While tables should be readily interpreted, the messages they convey need to be summarised in the related text. The inclusion of appropriate anecdotes to illustrate aspects of the results can enhance the reporting. Such anecdotes could be drawn from various data transcripts, teacher and student interviews, students' written explanations, and so on. The recent article by Budiyanto et al. (2021) exploring computational thinking in early childhood education provides powerful examples of such anecdotes. Conducted in children's homes due to Covid-19 restrictions, the study revealed how young learners' home environment and interactions with their family supported their development of computational thinking. Anecdotes from parent interviews supported the authors' three primary constructs on the role of educational robots in early childhood. Other examples of anecdotal evidence in interpretive research also appear in the case studies explored by Budiyanto, Prananto, and Tan (2019). Their article highlights how case studies can enable in-depth investigation and rich descriptions.
In concluding this commentary, I think it is worth noting the words of a highly experienced reviewer in her assessment of a journal submission: … let me offer my perspective on what constitutes quality empirical research, whether quantitative, qualitative, or mixed. Fundamentally research is undertaken to question, explore, probe, or to disprove; ideally research is undertaken with an open mind and with researchers' inevitable biases held in check to the extent possible. This means that research can be distinguished from persuasive or promotion writing where the objective can be to convince, support, or endorse. In empirical research, assumptions are not to be forwarded but tested and scrutinized.
The reviewer's words are worth keeping in mind as we embark on any research programme and subsequently report on its findings.