

Introduction
Partly because of the steep competition that pervades the research industry, investigations have revealed many instances in which academics engaged in improper research practices, called research misconduct. Research misconduct encompasses a diverse variety of improper activities. Many of these activities revolve around
-
falsification—of the tendency of researchers to manipulate the materials, procedures, or analyses to distort the results and thus improve the publication
-
fabrication—in which researchers concoct, rather than only distort, data or information.
-
plagiarism—in which researchers may reporting the ideas or words of an author, without acknowledging this person appropriately
-
improper authorship—in which individuals who should be granted authorship on a publication are not designated as authors or vice versa
Many researchers assume they would never be susceptible to this misconduct. However, some variants of falsification, fabrication, plagiarism, and improper authorship are common even in researchers who perceive themselves as upright and honorable. Some improper activities, although seemingly insignificant, can mislead readers and are manifestations of falsification. As examples of these improper activities
-
the researchers analyze the data repeatedly, but with minor variations, such as changes to which variables are controlled or which outliers are excluded, to generate significant results, called p hacking
-
similarly, researchers may continue to collect data until they achieve a significant result—a practice that appreciably increases the likelihood of type I errors or false positives
-
the sample is too small to identify outliers reliably; thus, outliers might distort the results, diminishing the likelihood these findings would be replicated
-
a funding body may request amendments to the methods that could potentially bias the results
-
in a literature review, the researcher may disregard the bulk of evidence that contradicts a key argument
Some researchers who perceive themselves as upright might still acknowledge the original author inappropriately, manifesting as plagiarism. For example, they may
-
fail to cite the original source of some argument, but only cite more renowned authors, called the Matthew effect (e.g., Merton, 1968), or recent variations
-
forget the source of one of their arguments—and even believe they established this argument themselves—called inadvertent plagiarism
-
replicate their own work in other publications and thus, in essence, plagiarize themselves; the inclination to publish similar articles in many journals is called salami slicing (Jackson et al, 2014)
Likewise, authorship is also often inappropriate. For example, according to the Vancouver protocol (ICMJE, 2016), individuals should be assigned authorship to scientific publications if and only if they
-
contributed intellectually towards the conception, design, analysis, or interpretation of this research—at least to the degree that no other person would have imparted the same advice or contribution
-
contributed intellectually to the drafting or revising of this publication
-
approved the final version of this publication and, therefore, are willing to assume responsibility if problems are uncovered
In practice, however, one contributor might not be assigned this authorship, because the other authors might want to conceal the participation of this person. For example, if the name of this contributor was identified, reviewers might be concerned about a conflict of interest. Conversely, senior figures, such as a founder of a research program, might be granted authorship but not contribute to the publication.
A landmark study, published by Martinson et. al. (2005), imparted some insight into the prevalence of research misconduct. A survey of over 3200 researchers, all funded by the National Institutes of Health in America, revealed that 33% of researchers conceded they had engaged in at least one of the most common improper behaviors:
-
falsification of research data
-
circumventing or disregarding human ethical requirements
-
questionable relationships with students or research participants
-
the decision not to disclose the participation of a firm with an interest in the research—because perhaps their products were the subject of this research
-
use of an idea from someone else, without permission or acknowledgement
-
unauthorized use of confidential information in their research
-
concealing data that conflicts with their past publications
-
overlooking research misconduct of other researchers
-
changing the research in response to pressure from a funding agency
Typically, the discourse about research misconduct tends to revolve around researchers, such as academics. However, with the proliferation of citizen science, in which the public collect data to contribute to research programs, Rasmussen (2019) argues that research misconduct could extend to the public as well. For example, like academics, citizen scientists could also fabricate their data, perhaps to underscore a particular agenda. However, the strategies and responses that institutions or other agencies would usually consider do not apply to misconduct that is committed by the public.
How to detect research misconduct: Questions to contemplate
Many individuals—such as reviewers, journal editors, and research administrators at universities—want to assess the research integrity of publications. Journal editors want to maintain the reputation of their journals. Research administrators, such as research integrity officers, might want to prevent research misconduct at their institution.
Grey et al. (2020) developed a checklist, called REAPPRAISED, that can help these individuals fulfill this goal. Specifically, according to these authors, individuals who want to assess the research integrity of publications should consider a range of questions, classified into 11 clusters, such as research governance, ethics, and authorships, generating the acronym REAPPRAISED. Some of the questions include
-
did the authors specify the location of data collection, the source of funding, and the dates of data collection—and is this information plausible and consistent with the registration documents—to gauge research governance
-
has an ethics committee approved the research and do any practices seek unethical—to gauge ethics
-
have the authors specified their contributions anywhere and is authorship of similar papers consistent—to gauge authorship
-
would the reported levels of staffing be adequate to conduct the study—to gauge productivity
-
do the publications include evidence of recycling text across papers—to gauge plagiarism
-
does the number of participants who were recruited and who withdrew, as well as other methods, seem plausible given the timelines, resources, and circumstances—and is the delay between the completion of a study and submission of a manuscript plausible—to gauge research misconduct
-
does the analyses show evidence of problems with data analysis or data management, such as unaccounted missing data, the tendency to report only significant results, multiple tests of the same hypothesis without a suitable adjustment, or analysis that was not specified in the registered analysis plan--to gauge analysis and methods
-
do the images reveal evidence of manipulation or duplication?
-
are subgroup means incompatible with means of the entire sample, are summary data—such as means and frequencies—plausible, do the figures, table, and text present discrepancies, are variables surprisingly consistent
-
are the results internally consistent, are the correct units reported, and are the number of participants consistent throughout the publication—to detect possible errors
-
have the data been published elsewhere; are the data, if reported elsewhere, consistent across publications—to assess data
How to detect research misconduct: Tools that detect erroneous p values
If researchers generate significant results—such as p values that are less than .05—journals are more likely to publish these findings. Consequently, to publish more papers, authors may distort their results to generate significant findings.
The tool called statcheck is an R package and a web app that can identify some instances in which authors may have distorted the p value (Epskamp & Nuijten, 2014). To illustrate, in some instances, authors might generate a p value that exceeds .05, such as .075. However, they might report the p value inaccurately, such as .045, to generate a significant result, but might not change the degrees of freedom and test statistics, such as t values. Indeed, as Nuijten et al. (2016) revealed, in about one in eight publications in psychology between 1985 and 2003, this error, if corrected, would have generated a different conclusion.
The statcheck tool is designed to uncover instances in which a p value that authors report deviates from the p value they should have reported, as derived from the degrees of freedom and test statistic. This tool is effective only if the statistics are reported in the text, rather than in tables, and complies with APA style. Nuijten and Polanin (2020) presents instructions on how individuals can utilize this tool at no cost.
The algorithm comprises a sequence of four procedures (Epskamp & Nuijten, 2014; Nuijten & Polanin, 2020). Specifically
-
the tool converts a pdf or HTML article, or set of articles, to plain text
-
the tool then identifies test statistics, such as t and F, as well as p values
-
next, the tool utilizes the test statistic and degrees of freedom to estimate the p value
-
finally, the tool uncovers inconsistencies between the estimated and reported p value—recognizing that some deviations can be ascribed to rounding errors, one-tailed tests, or other variations
When compared to manual attempts to calculate these p values, statcheck has been shown to be accurate in about 96% to 99.9% of instances (Nuijten & Polanin, 2020) except in specific circumstances. That is, the tool is not accurate if
-
the authors applied a Bonferroni correction or a similar adjustment because they tested the same hypothesis multiple times
-
the data violate the assumption of sphericity, and the authors reported corrected degrees of freedom but uncorrected test statistics and p-values.
How to detect research misconduct: Techniques that detect impossible patterns of statistics
If researchers distort the results of their data analyses, they sometimes report means and standard deviations that actually conflict with one another. To demonstrate
-
some data, such as the frequency of some event, are integers, called a granularity of 1
-
other data, such as data recorded with a particular instrument, might include one decimal place, called a granularity of .1, and so forth
-
the granularity is, therefore, the lowest possible number above 0.
The granularity of data affects the granularity of means. For example
-
if the data comprise 10 integers, the mean of these integers will be a granularity of .1 or higher
-
to illustrate, the mean of 1, 5, 4, 6, 3, 6, 1, 8, 6, and 6 is 4.6
-
if researchers reported a granularity of .01, such as 4.61, this number must be incorrect (Brown & Heathers, 2016)
Brown and Heathers (2016) calculated the minimum granularity of means, given the granularity of the data, G, and the sample size, N. Specifically, according to Brown and Heathers (2016), the granularity of a mean is G/N. To illustrate
-
if the data comprise 10 integers, the granularity of this mean is 1/10 or .01
-
if the data comprise 100 integers, the granularity of this mean is 1/100 or .001
This procedure, called the Granularity-Related Inconsistency of Means or GRIM test, is effective only if authors report the means to enough decimal places. If authors, for example, round the mean to one decimal, distortions are not as likely to be identified.
Anaya (2016) extended this rationale to other statistics, such as standard deviations, generating the GRIMMER test or Granularity-Related Inconsistency of Means Mapped to Error Repeats. This GRIMMER test can reveal unexpected patterns of means and variances, given the sample size and granularity of the data. To illustrate, Anaya (2016) showed that
-
only certain means are consistent with certain variances in particular circumstances
-
for example, in particular circumstance, if the variance is .16, the means can be .2, .8, 1.2, 1.8, and so forth, but nothing else
Likewise, Anaya (2016) uncovered some other interesting patterns. To illustrate, Anaya calculated the variance of hundreds of samples that comprise 5 integers—and then arranged the results from lowest to highest, such as
-
0, 0.16, 0.24, 0.40, 0.56, 0.64, 0.80, 0.96
-
1, 1.16, 1.24, 1.40, 1.56, 1.64, 1.80, 1.96
-
2, 2.16, 2.24, 2.40, 2.56, 2.64, 2.80, 2.96
-
3, 3.16, 3.24, 3.40, 3.56, 3.64, 3.80, 3.96
-
…
In this example, the possible decimals are the same regardless of whether the first number is 0, 1, 2, 3 and so forth. However, when Anaya (2016) repeated this procedure, but varied the number of data points in each sample, an interesting pattern emerged.
-
if the number of data points was odd, the decimals that followed odd numbers—such as 1, 3, 5—although identical to each other, sometimes differed from the decimals that followed even numbers
-
if the number of data points was even, the decimals that followed odd numbers and even numbers were the same.
Conceivably, individuals can utilize this pattern, and several other patterns, to identify possible attempts of authors to distort their results. Subsequently, Heather et al. (2018) developed a complementary tool, called SPRITE—or Sample Parameter Reconstruction via Interative Techniques—that overcomes some of the limitations in GRIMMER. For example, GRIMMER is applicable only when the sample size is less than 100, whereas SPRITE is applicable regardless of the sample size.
As Simonsohn (2013) implied, tools like statcheck, GRIMMER, and SPRITE may not be necessary if authors must share their original data. In these instances, other researchers can, at least in principle, analyze the data themselves and check the results (for a similar perspective, see Shamoo, 2013).
How to detect research misconduct: Patterns that epitomize human intervention and manipulation
Tools, such as statcheck, GRIMMER, and SPRITE, identify statistics in a report that are mathematically inconsistent. However, researchers sometimes report statistics that are possible, in theory, but unlikely in practice. That is, as several researchers, such as Simonsohn (2013), observed, when individuals manipulate data, they tend to naturally generate patterns that are possible but unlikely.
For example, if researchers fabricate their results, they might present means that vary considerably across conditions. To illustrate, they might show the average performance of students greatly varies across teaching conditions. But the researchers might present standard deviations that are similar across conditions. Simonsohn (2013) did indeed present a case study that conforms to this pattern and challenged the legitimacy of these findings. A simulation revealed this pattern is unlikely, but not impossible, in practice. Nevertheless, this pattern might be sufficient to warrant further investigation.
Likewise, if researchers fabricate their results, but the study comprises many conditions, another pattern is possible. The means of some conditions will be particularly different, to generate significant p values. But the means of other conditions will be very similar—more similar than predicted. Simonsohn (2013) indeed showed one team of authors generated this same pattern in multiple publications, raising suspicion.
An analysis of coefficient of variations—the standard deviation divided by the mean in each condition—can underscore some of these unusual patterns. For example, as Hudes et al. (2009) revealed, when researchers fabricate their data, their coefficients of variation tend to be similar across diverse outcomes. To illustrate, the mean divided by the standard deviation approach may 0.5 for both objective measures, such as blood pressure, and subjective ratings, such as perceived health. Yet, when the data are legitimate, distinct outcomes, such as objective indices and subjective measures, will tend to generate diverse coefficients of variation.
To confirm the legitimacy of concerns and to prevent false accusations, Simonsohn (2013) recommends that administrators should
-
check whether the same anomalies are observed in multiple publications
-
seek access to the raw data and then re-analyze these data
-
contact the authors privately and transparently, granting these authors time to respond
-
if still concerned, contact the relevant authorities discreetly
How to detect research misconduct: Participant data
Sometimes, information about the participants of studies might also reveal misconduct. Some researchers, for example, believe that raw data should include the precise date and time data were collected from each participant. If this information is retained, specialists might be able to uncover patterns that signify fabrication. For example, the duration between consecutive participants might be too regular or data might be collected at unlikely times, such as weekends, even when a clinic is usually closed (van den Bor et al., 2017). If each individual needs to participate on several occasions over time, missing data are likely; negligible missing data might also signify fabrication (van den Bor et al., 2017).
How to detect research misconduct: Duplication or manipulation of images
Some authors will include duplicated or manipulated images and graphs in publications. To illustrate, authors sometimes
-
breach copyright and utilize an image without permission—sometimes rotating, cropping, or adjusting the colors and other properties to conceal this duplication
-
utilize tools, such as the Photoshop cloning tool, to delete unwanted data from images. For example, to substantiate the benefits of a cancer treatment, researchers could manipulate images to exaggerate the decrease in a melanoma or other cancer cells.
-
utilize similar tools to insert unwanted data from images.
Researchers have developed several tools to detect the duplication or manipulation of images, such as FraudDetTools (Koppers et al., 2017), a package in R, and the algorithm that Acuna et al. (2018) developed to identify duplication. Koppers et al. (2017) delineated some of the principles that researchers could apply to develop algorithms that detect this duplication or manipulation. To illustrate
-
if data are inserted into an image, the inserted region tends to be surrounded by sharper edges—and some algorithms can detect these edges. Admittedly, when the image is compressed, these artificial edges may be hard to discriminate from legitimate edges. In the future, journals might accept only uncompressed image data to check quality.
-
to delete data from an image, individuals will tend to replace this segment with another segment of the image—usually a segment that displays a neutral background. Yet, this background will often differ from the surrounding background in luminance and other features—and some algorithms can detect this shift.
These breaches are not scarce and seem to be increasing in frequency (for a review, see Koppers et al., 2017). One of the most infamous cases revolved around Dr. Hwang Woo Suk, recounted in depth by Hong (2008). In 1999, he cloned a dairy cow in Korea and became a national celebrity. In 2002, he claimed to have cloned a pig that could be used to transplant organs. And in 2002, he announced that he had cloned a cow that is resistant to mad cow disease. He also apparently applied stem cell treatment to cure a severely injured dog—but had not reported these achievements in scientific papers. However, in 2004 and 2005, following suspicions that other scientists had raised, he did publish these achievements, in collaboration with Dr. Gerald Schatten.
Dr. Gerald Schatten soon terminated this collaboration, after he discovered that Dr. Hwang had acquired human eggs from two junior female researchers in his laboratory. Dr. Hwang later resigned because of these matters. A TV investigative journalism shows then investigated his work, interviewing one of his junior researchers, who indicated that he had manipulated data under instruction. Consequently, the journalists organized another DNA test company to replicate some of the results—an attempt that was unsuccessful. Later, despite conflicting opinions about this case, individuals revealed that Hwang or his collaborators had utilized PhotoShop to manipulate photos of 11 stem cells, printed in the 2005 paper to Science. Problems with the DNA fingerprinting data were also revealed.
The prevention of research misconduct: Overview
To prevent research misconduct, the prevailing approach in scientific discourse revolves around detection and retraction. For example as Shamoo (2013) argued, institutions tend to depend on five approaches to stem research misconduct:
-
voluntary compliance to ethical norms
-
strict regulations and standards
-
education and training of researchers
-
whistleblowers
-
data audits—the approach that Shamoo (2013) recommended
To explore the range of approaches that institutions and agencies have introduced to prevent research misconduct and the effects of these approaches, Marusic et al. (2016) conducted a systematic review. Thirty-one studies met their selection criteria: research studies that examine the effects of some intervention on research integrity or research misconduct, provided these studies included a control group. Their findings reveal that
-
to stem research misconduct, institutions and agencies tend to introduce some training, such as workshops in person, online lectures, interactive online modules, discussion groups, and practical exercises
-
these interventions tend to improve the attitudes of participants significantly but the knowledge of these individuals about this topic only minimally or transiently
-
training in which participants completed practical exercises, such as used text-matching software to identify plagiarism, tended to be most useful
The prevention of research misconduct: Social psychology principles
According to some researchers, such as Redman and Caplan (2017), managers, institutions, and peak scientific bodies could apply some of the principles and insights derived from social psychology to deter research misconduct. For example, as these researchers suggest
-
if the level of competition to succeed, such as to publish, is too pronounced, individuals experience a sense of moral disengagement. That is, the usual guilt and shame they would experience if they behaved improperly dissipates. Indeed, several authors have alluded to the notion that competitive pressures—and the obsession with performance over integrity—may promote research misconduct (e.g., Asai et al., 2016).
-
when individuals are exposed to many reminders of morality—such as codes of ethics—this moral disengagement is not likely
Bandura (1986, 1999) identified eight strategies that individuals might deploy to elicit this moral disengagement. Three of these strategies enable individuals to judge misconduct as ethical. First, individuals might argue this behavior could translate into some unexpected benefit and is thus moral. If they fabricated some of their data, they might argue their publication could help communities immediately—assistance that might otherwise be delayed if they actually completed the study. Second, individuals might invoke euphemisms to describe their misconduct. They might indicate they had ”expedited their research” rather than “fabricated their data”. Third, they might contrast their behavior with even more unethical examples; their choices, therefore, might seem reasonable in comparison.
Two of the strategies devolve responsibility. For example, individuals might contend they were coerced to engage in some act, such as felt pressure from their manager to publish immediately. Alternatively, they might argue they were behaving as part of a team. In addition, to disengage morally, individuals might trivialize the consequences of their actions, reminding themselves perhaps that few people will read their publication anyway.
Fortunately, many studies have explored the characteristics of individuals or circumstances that can diminish or prevent moral disengagement. To illustrate, individuals are not as likely to disengage morally if
-
they tend to feel confident in their capacity to organize, coordinate, and inspire other people, like a leader (Hinrichs et al., 2011)—because these individuals feel more responsible for their actions
-
they work in a supportive environment, in which they believe that colleagues will offer assistance whenever they feel stressed (Chugh et al., 2013)
-
they tend to be empathic (Detert & Trevino, 2008)
The prevention of research misconduct: Suggestions from researchers
Buljan et al. (2018) conducted three focus groups, comprising with doctoral candidates and senior researchers, to gauge their suggestions on which practices might stem research misconduct. As these participants suggested
-
institutions should be more explicit about the procedures that researchers should follow to manage, to retain, and to share their data
-
institutions should assign early career and graduate researchers mentors to discuss their uncertainties, concerns, and doubts. Mentors can both clarify boundaries as well as increase the confidence of these researchers and thus diminish the need to behave expediently.
-
institutions should be more accountable; research misconduct is often blamed on individuals, whereas the practices and culture of institutions is seldom recognized as part of the problem.
References
-
Acuna, D. E., Brookes, P. S., & Kording, K. P. (2018). Bioscience-scale automated detection of figure element reuse. BioRxiv.
-
Anaya, J. (2016). The GRIMMER test: A method for testing the validity of reported measures of variability. PeerJ Preprints, 4, e2400v1.
-
Asai, A., Okita, T., & Enzo, A. (2016). Conflicting messages concerning current strategies against research misconduct in Japan: a call for ethical spontaneity. Journal of Medical Ethics, 42(8), 524-527.
-
Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice Hall.
-
Bandura, A. (1999). Moral disengagement in the preparation of inhumanities. Personal and Social Psychology Review, 3, 193-209.
-
Bonito, A. J., Titus, S. L., & Wright, D. E. (2012). Assessing the preparedness of research integrity officers (RIOs) to appropriately handle possible research misconduct cases. Science and Engineering Ethics, 18(4), 605-619.
-
Bordewijk, E. M., Li, W., van Eekelen, R., Wang, R., Showell, M., Mol, B. W., & van Wely, M. (2021). Methods to assess research misconduct in health-related research: A scoping review. Journal of Clinical Epidemiology, 136, 189-202.
-
Brown, N. J. L. and Heathers, J. A. J. (2016). The grim test: A simple technique detects numerous anomalies in the reporting of results in psychology. PeerJ Preprints.
-
Buljan, I., Barać, L., & Marušić, A. (2018). How researchers perceive research misconduct in biomedicine and how they would prevent it: A qualitative study in a small scientific community. Accountability in Research, 25(4), 220-238.
-
Chugh, D., Kern, M. C., Zhu, Z., & Lee, S. (2013). Withstanding moral disengagement: Attachment security as an ethical intervention. Journal of Experimental Social Psychology, 51, 88-93.
-
Dahlberg, J. E., & Davidian, N. M. (2010). Scientific forensics: how the Office of Research Integrity can assist institutional investigations of research misconduct during oversight review. Science and Engineering Ethics, 16(4), 713-735.
-
Detert, J. R.,& Trevino, L. K. (2008). Moral disengagement in ethical decision making: A study of antecedents and outcomes. Journal of Applied Psychology, 93, 374-391.
-
Epskamp, S., & Nuijten, M. B. (2014). statcheck: Extract statistics from articles and recompute p values (R package version 1.0. 0.).
-
Grey, A., Bolland, M. J., Avenell, A., Klein, A. A., & Gunsalus, C. K. (2020). Check for publication integrity before misconduct. Nature Publishing Group
-
Heathers, J. A., Anaya, J., van der Zee, T., & Brown, N. J. (2018). Recovering data from summary statistics: Sample parameter reconstruction via iterative techniques (SPRITE) (No. e26968v1). PeerJ Preprints.
-
Hinrichs, K. T., Wang, L., Hinrichs, A. T., & Romero, E. J. (2012). Moral disengagement through displacement of responsibility: The role of leadership beliefs. Journal of Applied Social Psychology, 42, 62-8
-
Hong, S. (2008). The Hwang Scandal that “shook the world of science”. East Asian Science, Technology and Society: An International Journal, 2(1), 1-7.
-
Hudes, M. L., McCann, J. C., & Ames, B. N. (2009). Unusual clustering of coefficients of variation in published articles from a medical biochemistry department in India. The FASEB Journal, 23(3).
-
ICMJE (2016). International committee of medical journal editors. Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journal
-
Jackson, D., Walter, G., Daly, J., & Cleary, M. (2014). Multiple outputs from single studies: Acceptable division of findings vs.‘salami’slicing. Journal of Clinical Nursing, 1-2.
-
Koppers, L., Wormer, H., & Ickstadt, K. (2017). Towards a systematic screening tool for quality assurance and semiautomatic fraud detection for images in the life sciences. Science and Engineering Ethics, 23(4), 1113-1128.
-
Martinson, B. C., Anderson, M. S., & De Vries, R. (2005). Scientists behaving badly. Nature, 435(7043), 737-738.
-
Marusic, A., Wager, E., Utrobicic, A., Rothstein, H. R., & Sambunjak, D. (2016). Interventions to prevent misconduct and promote integrity in research and publication. Cochrane Database of Systematic Reviews, (4).
-
Merton, R. K. (1968). The Matthew Effect in Science: The reward and communication systems of science are considered. Science, 159(3810), 56-63.
-
Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., ... & Van der Laan, M. (2014). Promoting Transparency in Social Science Research. Science, 343(6166), 30-31.
-
Nuijten, M. B., Hartgerink, C. H., Van Assen, M. A., Epskamp, S., & Wicherts, J. M. (2016). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior research methods, 48(4), 1205-1226.
-
Nuijten, M. B., & Polanin, J. R. (2020). "statcheck": Automatically detect statistical reporting inconsistencies to increase reproducibility of meta‚Äêanalyses. Research synthesis methods, 11(5), 574-579.
-
O'Kelly, M. (2004). Using statistical techniques to detect fraud: a test case. Pharmaceutical Statistics: The Journal of Applied Statistics in the Pharmaceutical Industry, 3(4), 237-246.
-
Rasmussen, L. M. (2019). Confronting research misconduct in citizen science. Citizen Science: Theory and Practice, 4(1).
-
Redman, B. K., & Caplan, A. L. (2017). Improving research misconduct policies: Evidence from social psychology could inform better policies to prevent misconduct in research. EMBO reports, 18(4), 511-514.
-
Shamoo, A. E. (2013). Data audit as a way to prevent/contain misconduct. Accountability in research, 20(5-6), 369-379.
-
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2018). False-positive citations. Perspectives on Psychological Science, 13(2), 255-259.
-
Simonsohn, U. (2013). Just post it: The lesson from two cases of fabricated data detected by statistics alone. Psychological Science, 24(10), 1875-1888.
-
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534.
-
van den Bor, R. M., Vaessen, P. W., Oosterman, B. J., Zuithoff, N. P., Grobbee, D. E., & Roes, K. C. (2017). A computationally simple central monitoring procedure, effectively applied to empirical trial data with known fraud. Journal of Clinical Epidemiology, 87, 59-69.

Introduction: Publication bias
Health researchers often conduct systematic reviews and meta-analyses to ascertain whether some intervention, such as a novel drug, affects some health outcome, such as diabetes. These researchers collate all the studies that have explored this topic, distil the results of these studies, and then calculate various statistics to determine primarily
-
the average extent to which the intervention enhances health
-
other characteristics or circumstances that affect the impact of this intervention
Systematic reviews and meta-analyses are suitable only if the studies that researchers collate are representative of the studies that have been conducted. Unfortunately
-
studies that did not demonstrate significant benefits are sometimes discarded rather than published
-
some researchers publish only a subset of one study—such as disregard the measures that did not reveal the intended benefits
-
some researchers publish overlapping results, from one trial in more than one publication, called salami slicing (for a discussion, see Gray et al., 2020).
Because of these concerns, the studies that researchers collate may not be representative of the studies that have been conducted, called a publication bias. Consequently, systematic reviews and meta-analyses may be misleading.
Introduction: Registration of clinical trials
To minimize publication biases, several journals and agencies have recommended, and often mandated, that researchers first register clinical trials. Although this recommendation to register clinical trials can be traced to the 1980s, only since the International Committee of Medical Journal Editors imposed this activity in their journals has registration become prevalent. That is, since 2005, clinical trials are not published in the journals these individuals edit, unless registered (De Angelis et al., 2004).
This mandate applies to all clinical trials. Clinical trials, as defined by the World Health Organization, comprises studies that assign humans to one or more interventions that are related to health and evaluate the effect of these interventions on some facet of health. Interventions may include drugs, surgical procedures, behavioral treatments, lifestyle changes, and modifications to health procedures. Studies in which participants are not assigned to conditions do not need to be registered but can be registered. If uncertain whether studies should be registered, the International Committee of Medical Journal Editors recommends the authors register the study anyway.
Introduction: Procedure to register clinical trials
To register a clinical trial, authors need to access one of the registers endorsed by the International Committee of Medical Journal Editors. For example, authors might access
-
www.clinicaltrials.gov: a register that stores clinical trials from around the globe
-
www.anzctr.org.au: a register that stores clinical trials in Australia and New Zealand
-
the Pan African Clinical Trials Registry
-
www.ISRCTN.org, and several other similar websites
Each website presents instructions on how to register a clinical trial (for details on how to register in India, for example, see Bhaskar, 2018). In general, authors need to complete forms in which they specify
-
the title of this study or trial
-
the funding bodies and collaborators—such as hospitals
-
a summary of the trial, like an abstract to a research proposal
-
keywords
In addition, authors also choose various options or enter information to characterize the desgn and methods, such as
-
whether the study is a randomized control trial
-
the number of participants
-
whether participants, health practitioners, investigators, and outcome assessors will be blind to the condition in which individuals are assigned—and how this goal is achieved
-
whether allocations are concealed from the person who assigns participants—and how this allocation is concealed
-
the dates during which data will be collected
-
the key procedure in each condition, such as drug doses
-
how the primary outcome will be measured
-
how secondary outcomes will be measured
-
the inclusion and exclusion criteria of participants
Although usually straightforward, provided the design has been chosen carefully, researchers should be aware of some potential complications or ambiguities. For example, as Tse et al. (2018) identified
-
longitudinal studies only need to be registered once—unless researchers plan to recruit participants who were not included in the original study or need to shift the plan and seek additional consent—in which case this update should be registered as a separate study. The proposal should refer to the previous registered trial
-
the researchers can modify the proposals they registered, and each modified version is retained
Other benefits of clinical trial registration: Post hoc tests
Besides decreases in publication biases, the registration of clinical trials can generate other benefits.
For example, the registration of clinical trials may also stem unsuitable statistical practices.
To illustrate, when researchers do not generate significant results, they may be tempted repeat the analyses many times, after introducing slight modifications, until more compelling findings surface. These researchers might, for example, change the techniques they use to identify and to delete outliers. They could extend or reduce the number of control variables. Or they could even collect more data and then repeat the analysis.
As Zarin et al. (2020) argued, registration of clinical trials may curb some of these practices. Or, at the very least, this registration would compel authors to indicate which statistical procedures had been planned and which statistical procedures should be deemed as post hoc and thus tenuous, warranting further validation.
Other benefits of clinical trial registration: Fewer biases in the research
Arguably, mandatory trial registration could improve the quality of research. That is, when researchers register a clinical trial, they need to answer questions about the design, such as whether participants, personnel, and assessors are blind to the condition in which these participants were allocated. These questions may prompt researchers to consider the key methodological principles they should observe. The researchers might be more likely to design studies that are not as susceptible to various biases.
To explore whether trials that are registered prospectively are indeed less susceptible to various biases, Tan et al. (2019) conducted a systematic review of all clinical trials, published in medical journals, within the first half of 2017. The review uncovered 370 clinical trials that fulfill the eligibility criteria. Most of these trials were registered before publication. Specifically
-
60% of the clinical trials had definitely been registered before the research commenced
-
12% of the clinical trials were registered in the same month the research commenced—and, therefore, whether these trials were registered prospectively or retrospectively was uncertain
-
23% of the clinical trials were registered after the research commenced but before all participants had been recruited
-
1% of clinical trials were registered after all participants had been recruited
-
4% of clinical trials were not registered.
The degree to which the trial was designed effectively differed between trials that were registered prospectively and other trials. For instance, allocation concealment was more likely in trials registered prospectively than trials registered after participants had been recruited or not at all. That is, to diminish bias in trials, researchers should observe a principle called allocation concealment. To illustrate
-
if possible, one person or body should generate a random sequence of binary numbers, such as 0, 1, 1, 1, 0, 0, 1; these numbers represented the order in which participants will be assigned to the control condition and treatment condition
-
a separate person or body should recruit participants and assign these participants to conditions, such as present these individuals with instructions on how to proceed
-
importantly, the person who assign participants to conditions should not be aware of which condition the individuals will be assigned—until after this person has agreed to participate;
-
otherwise, this person might attempt to influence which individuals agree to participate
-
to illustrate, if these individuals feel a participant might not respond well to this treatment and they know this participant will be assigned to the treatment condition, they might deter this person from participating
-
therefore, to conceal the allocation, this person should contact the body that generated the random sequence of binary numbers only after individuals have agreed to participate.
Other measures to reduce bias were also more common in trials registered prospectively than trials registered after participants had been recruited or not at all, such as
-
whether a random sequence generator was deployed to determine which participants will be assigned to the treatment condition
-
whether the participants and individuals administering the treatment were unaware of which participants were allocated to the treatment condition, called double blinding
-
whether the individuals measuring the outcome were unaware of which participants were allocated to the treatment condition
Riemer et al. (2021), in a sample of 585 randomized control trials, published between 1965 and 2017, to examined drugs that reduce nausea or prevent vomiting, also showed that registration coincides with lower risk of bias. Specifically, these researchers subjected the trials to the Cochrane Risk of Bias assessment tool—a tool that identifies limitations in the procedure that increase the risk of biased results. For example, this tool identifies instances in which
-
the researchers did not use a random generator to assign participants to conditions
-
the person allocating participants to conditions was aware of the next condition these individuals would be assigned, compromising the principle of allocation concealment
-
the participants, research administrators, or outcome assessors were aware of the condition in which they had been assigned
-
some of the measures were not reported
The results were striking. About 64% of the registered trials were deemed as low in risk. In contrast, only 28% of the unregistered trials were deemed as low in risk.
Other studies have, in general, confirmed the assumption that registered trials tend to be less susceptible to biases. For example, Lindsley et al. (2022) examined 1177 clinical trials, derived from a sample of 100 Cochrane systematic reviews, published between 2014 and 2019. As these researchers showed, compared to trials that had not been registered, trials that had been registered were more likely to
-
generate a random sequence to assign participants to conditions
-
conceal the allocation from the individuals who assign participants to conditions until after these participants have consented
-
confirm that participants, research administrators, or outcome assessors were blind to the condition in which they had been assigned
Furthermore, compared to trials that had been registered retrospectively, trials that had been registered prospectively were also less susceptible to bias. That is, trials that had been registered prospectively were more likely to confirm that participants, research administrators, or outcome assessors were blind to the condition in which they had been assigned as well as introduce allocation concealment.
These results, therefore, suggest that registration of clinical trials tend to be associated with fewer risks of bias. However, the cause of this association is uncertain. Perhaps, the act of registration could help researchers improve the design. Alternatively, researchers who are more attuned to good practice might be more inclined to both register the trial prospectively and construct an effective design.
Other benefits of clinical trial registration: Quality of research in general
Some findings, however, indicate that merely the act of registration could help researchers improve the quality of their research. To illustrate, Kakkar et al. (2019) explored whether mandatory trial registration improved other features of the research besides the risk of bias. Specifically, Kakkar et al. examined clinical trials in India—a nation in which clinical trial registration was mandated in June 2009. The researchers examined 75 research protocols or proposals submitted in the two years before this mandate and 75 research protocols submitted after this mandate.
The results suggested the quality of protocols improved after clinical trial registration was mandated. For example, the percentage of protocols that justified the sample size increased from 38% to 70%. The description of statistical methods as well as the management of premature withdrawals also improved appreciably.
Other benefits of clinical trial registration: Efficiency of systematic reviews
These registers might facilitate attempts to update systematic reviews as well. Specifically, systematic reviews may consume significant time. Consequently, systematic reviews are often published years after the studies have been collated. The results, once published, may no longer be valid if many other studies on this topic have been published since.
Surian et al. (2021) recommended that registers of clinical trials could be utilized to update systematic reviews more expeditiously. They recommended an approach, called document similarity, to readily identify clinical trials that had not been included in the original systematic review but could be considered in forthcoming updates. In essence, to apply this approach, researchers need to
-
extract all the studies that were included in the original systematic review
-
use a tool to determine the relative frequency of each word in this pool of reports—such as “diabetes” might be 2% of the words and “drug” might be 3% of the words
-
extract other trials from the register—but only trials that are too recent to have been included in this systematic review
-
for each trial, determine the relative frequency of each word
-
calculate the difference—strictly the Euclidean distance—between the relative frequency of words in the original studies and the relative frequency of words in each more recent study
-
lower distances represent trials that are most likely to be relevant to an updated systematic review
After researchers apply this method, they could then decide which clinical trials should be included in the updated systematic review. Specifically, they would
-
first determine whether the trial that generated the lowest Euclidean distance, and is thus more similar to other studies in the systematic review, fulfills the eligibility criteria of this review
-
if so, they would next determine whether the trial that generated the second lowest Euclidean distance fulfills the eligibility criteria of this review, and so forth
-
hey would then terminate this search once a specific number, such as 5, consecutive trials did not fulfill the eligibility criteria.
As Surian et al. (2021) revealed, this method was able to identify trials that should be included in updates to systematic reviews more efficiently than other approaches, such as hierarchical agglomerative clustering. That is, the document similarity could uncover trials that would be eligible to an updated systematic review after screening a relatively small number of publications.
Impediments to the registration of clinical trials: Overview
Despite the recommendations of International Committee of Medical Journal Editors, authors do not always register clinical trials before they commence the research. Instead,
-
some authors register clinical trials after they commence the research
-
some journals do not mandate registration, sometimes to accommodate developed nations, in which registration may not be as straightforward
-
some controversy persists over which studies should be deemed as clinical trials
Despite these concerns, as studies indicate, the percentage of clinical trials that have been registered is increasing over time (Trinquart et al., 2018). However, a significant portion of these trials are not registered before the recruitment of participants commences (Trinquart et al., 2018).
Impediments to the registration of clinical trials: Protection of intellectual property
When researchers register a clinical trial, anyone can access the research they plan to conduct. In some instances, other people could exploit this information and compromise the intellectual property. According to Zarin et al. (2020), clinical trials, after they are registered, could be inaccessible to the public for a predetermined duration.
Impediments to the registration of clinical trials: Determinants of registration
To clarify why researchers do not always register their clinical trials prospectively—that is, before the research commences—studies have explored the determinants of registration. For example, in their review of 370 clinical trials, published in the first half of 2017, Tan et al. (2019) revealed that several features of the trials decreased the likelihood they were registered before the research commenced. For example
-
when the sample size was small, clinical trials were not as likely to be registered prospectively
-
when only one center and one nation contributed to the recruitment of participants, these trials were not as likely to be registered prospectively
Impediments to the registration of clinical trials: Attitudes of journal editors and publishers
To encourage registration of clinical trials prospectively, journal editors could reject manuscripts if the trial was not registered before the research commenced. Yet, many journal editors accept manuscripts even if the trial was registered after the research commenced. Indeed, some journal editors accept manuscripts in which trial has not been registered at all—or do not check registration.
To explore the motivations behind these choices, Wager et al. (2013) identified journals in which the policies around trial registration diverge from conventional practice. The editors or publishers of 15 journals agreed to participate in an interview about this policy. The participants offered several reasons to justify their reluctance to mandate the registration of clinical trials. Specifically
-
some journal editors or publishers did not want to mandate the registration of clinical trials merely because they were concerned these manuscripts could be submitted to rivals instead; they were thus waiting until their rivals changed or clarified their policy
-
some journals publish few of the primary results of clinical trials—but might only publish secondary data, observational data, or other findings in which registration is not as significant
-
some journal editors perceived trials that were peripheral—such as comparisons of drugs that have already been substantiated—as not worthy of registration
-
a few participants felt that registration of clinical trials does not prevent or greatly reduce publication bias; that is, despite registration, some researchers will not be able to publish null results
-
some participants also felt this policy disadvantaged researchers from developed nations in which clinical trials are not as likely to be registered
Admittedly, since 2013, increasingly more journals mandate registration of clinical trials and increasingly more clinical trials are registered prospectively, even in developing nations (e.g., Ndwandwe et al., 2022). Nevertheless, this study is still informative, highlighting some of the concerns or attitudes around registration of clinical trials.
Limitations to the benefits of clinical trial registration: Enduring publication biases
The registration of clinical trials was partly intended to diminish publication bias. But this registration does not entirely overcome publication bias. To illustrate
-
researchers may not publish all the trials they registered
-
researchers may apply procedures that diverge from the registered trial, and so forth
For example, Riemer et al. (2021) explored discrepancies between the registered trial and the final report. Specifically, Riemer et al. extracted 585 randomized control trials, published between 1965 and 2017, that examined drugs that reduce nausea or prevent vomiting. The researchers examined which of these trials had been registered, the research that was proposed in this register, and the final publication of this trial. The first registered trial was published in 2004. Of the trials that had been registered, 36% had been registered prospectively
The analyses revealed some discrepancies between the registered protocol and the final study. For example, in almost 60% of registered trials, the authors had changed the primary outcome measure. To illustrate, the primary outcome measure in the registered protocol was often labelled as a secondary outcome measure in the final publication. Or, sometimes, the primary outcome in the final publication was not even mentioned in the registered protocol. Occasionally, the times at which the outcomes were assessed differed between the registered proposal and final study, suggesting that perhaps researchers may not have reported the outcomes at each time.
Huiskens et al. (2020) also explored discrepancies between the registered protocols and the final studies. This study examined 168 randomized control trials, initiated by Dutch investigators, registered in the relevant Netherlands database between December 2010 and January 2012. Their analysis revealed that
-
almost a quarter of the registered trials were still not published five years later
-
in about 45% of trials, the planned sample size had not been reached
-
the hypotheses had changed in about 16% of instances
Admittedly, the unpublished trials were ascribed to a range of challenges, such as problems with recruitment and rejections of the manuscripts. According to the authors, to increase the percentage of registered trials that are published, all investigators should be able to initially submit the results to a register.
Approaches to encourage registration of clinical trials prospectively
Although clinical trials must be registered, some trials continue to be registered after the trial commences or sometimes not at all. To prevent this problem, the Medical University of South Carolina introduced a series of practices. For example
-
the university employed a coordinator, full time, to encourage registration with ClinicalTrials.gov, to identify unregistered trials, and to develop standard operating procedures
-
assigned the institutional review board, sometimes called an ethics committee, the responsibility to identify trials that must be registered as early as possible
-
arranged individual consulting and assistance to help researchers navigate the registration system
-
organized workshops over lunch, called Lunch and Learns
These practices were largely effective, increasing compliance to 98%.
References
-
Bhaskar, S. B. (2018). Clinical trial registration: A practical perspective. Indian journal of Anaesthesia, 62(1).
-
De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., ... & Van Der Weyden, M. B. (2004). Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Annals of internal medicine, 141(6), 477-478.
-
Gray, R., Water, A., & MacKay, B. (2022). How prospective trial registration can prevent selective outcome reporting and salami slicing. Women and Birth, 35(2), 105-107.
-
Huiskens, J., Kool, B. R., Bakker, J. M., Bruns, E. R., de Jonge, S. W., Olthof, P. B., ... & Punt, C. J. (2020). From registration to publication: A study on Dutch academic randomized controlled trials. Research Synthesis Methods, 11(2), 218-226.
-
Kakkar, A. K., Padhy, B. M., Sarangi, S. C., & Gupta, Y. K. (2019). Methodological characteristics of clinical trials: Impact of mandatory trial registration. Journal of Pharmacy & Pharmaceutical Sciences, 22, 131-141.
-
Lindsley, K., Fusco, N., Li, T., Scholten, R., & Hooft, L. (2022). Clinical trial registration was associated with lower risk of bias compared with non-registered trials among trials included in systematic reviews. Journal of Clinical Epidemiology, 145, 164-173.
-
Ndwandwe, D. E., Runeyi, S., Pienaar, E., Mathebula, L., Hohlfeld, A., & Wiysonge, C. S. (2022). Practices and trends in clinical trial registration in the Pan African Clinical Trials Registry (PACTR): A descriptive analysis of registration data. BMJ open, 12(1).
-
Riemer, M., Kranke, P., Helf, A., Mayer, D., Popp, M., Schlesinger, T., ... & Weibel, S. (2021). Trial registration and selective outcome reporting in 585 clinical trials investigating drugs for prevention of postoperative nausea and vomiting. BMC Anesthesiology, 21(1), 1-10.
-
Roberts, I., Ker, K., Edwards, P., Beecher, D., Manno, D., & Sydenham, E. (2015). The knowledge system underpinning healthcare is not fit for purpose and must change. BMJ, 350.
-
Snider, S. H., Flume, P. A., Gentilin, S. L., Lesch, W. A., Sampson, R. R., & Sonne, S. C. (2020). Overcoming non-compliance with clinical trial registration and results reporting: One Institution's approach. Contemporary Clinical Trials Communications, 18, 100557.
-
Surian, D., Bourgeois, F. T., & Dunn, A. G. (2021). The automation of relevant trial registration screening for systematic review updates: an evaluation study on a large dataset of ClinicalTrials. gov registrations. BMC medical research methodology, 21(1).
-
Tan, A. C., Jiang, I., Askie, L., Hunter, K., Simes, R. J., & Seidler, A. L. (2019). Prevalence of trial registration varies by study characteristics and risk of bias. Journal of Clinical Epidemiology, 113, 64-74.
-
Trinquart, L., Dunn, A. G., & Bourgeois, F. T. (2018). Registration of published randomized trials: A systematic review and meta-analysis. BMC medicine, 16(1).
-
Tse, T., Fain, K. M., & Zarin, D. A. (2018). How to avoid common problems when using ClinicalTrials. gov in research: 10 issues to consider. BMJ, 361.
-
van Heteren, J. A., van Beurden, I., Peters, J. P., Smit, A. L., & Stegeman, I. (2019). Trial registration, publication rate and characteristics in the research field of otology: A cross-sectional study. Plos one, 14(7), e0219458.
-
Wager, E., & Williams, P. (2013). “Hardly worth the effort”? Medical journals’ policies and their editors’ and publishers’ views on trial registration and publication bias: quantitative and qualitative study. BMJ, 347.
-
Zarin, D. A., Crown, W. H., & Bierer, B. E. (2020). Issues in the registration of database studies. Journal of Clinical Epidemiology, 121, 29-31.

Introduction
The behavior and practices of research supervisors greatly affects the satisfaction and success of graduate researchers, such as PhD candidates. Indeed, when asked to identify their key problems or obstacles, research candidates often refer to inadequate or fractious relationships with supervisors. Yet, supervision is a challenging and multi-faceted endeavor. Supervisors need to fulfill many responsibilities and apply many skills.
After a comprehensive review of the literature, however, Fulgence (2019) concluded that many of the capabilities that supervisors need to develop can be divided into six main constellations:
-
expertise in the academic discipline, such as knowledge about the main theories and approaches, the culture and discourse of this discipline, and the settings or context
-
expertise in research skills, such as how to construct a research question, choose a research design, collect, analyze, and interpret data, identify limitations, and communicate the findings in oral presentations and written documents
-
the capacity to navigate workplaces effectively, such as skills in teamwork, resilience, problem solving, decision making, critical thinking, adaptability, and leadership
-
the ability to manage projects and resources, such as how to manage time, manage finances, and accommodate diversity
-
familiarity with digital skills, such as online tools and online networks
-
knowledge about the doctoral journey, such as university procedures, supervision roles, feedback and delivery.
Supervisors can distill these skills from a range of experiences and sources. In particular, supervisors may develop these skills if they
-
act as a supervisor, perhaps under the guidance of a principal supervisor (e.g., Halse, 2011)
-
receive mentoring and career coaching (Rogers et al., 2018)
-
seek information themselves from books, publications, or other supervisors (Gordon, 2014)
-
attend supervision training courses or workshops (e.g., Botha & Muller, 2016)
Nevertheless, to help supervisors acquire these skills, institutions are becoming increasingly reliant on training courses or workshops. Many universities oblige research supervisors to complete formal training and development, designed to enhance their supervisory practices (McCulloch & Loeser, 2016). National good practice guidelines, such as the Quality Assurance Agency for Higher Education in the UK and the Australian Council of Graduate Research, often mandate this formal training and development of supervisors. Similarly, the Salzburg Principles Section 2.3, stipulated by the European Universities Association (EUA Council for Doctoral Education, 2010), maintains that institutions are assigned the responsibility to facilitate the professional development of supervisors.
This training and development of supervisors can range from short workshops, lasting an hour or two, to comprehensive programs. An example of a comprehensive program is the postgraduate certificate in research degree supervision, introduced by the Edge Hill College of Higher Education in 1999 (Cryer, 2000).
As Wisker and Kiley (2014) underscored, these programs tend to vary along two dimensions. First, these experiences vary on the duration over which development evolves. At one pole, most of the development opportunities may be confined to one time, often induction. At other institutions, the development opportunities might continue indefinitely, sometimes called CPD or continued professional development.
Second, the breadth of content also varies across these development experiences. Some training and development are confined to topics that are directly relevant to immediate supervision needs, such as the HDR policies and procedures and some guidelines that supervisors can apply now. Other training and development programs are more encompassing and impart knowledge and skills that could be relevant to the future, such as the changing landscape of doctoral education, the pedagogy of graduate research, and reflections on case studies.
Besides these two dimensions, training programs also vary on whether they are mandatory. Many institutions grapple with the tension between mandatory programs in which participants often feel resentment or show resistance and optional programs in which participants often do not attend (Manathunga, 2005). Supervisors who are not reflective, and thus often ineffective, may be the least inclined to attend.
As McCulloch and Loeser (2016) bemoan, few studies have substantiated the benefits of these initiatives. Although institutions often evaluate this training and development of supervisors—perhaps by asking participants to complete evaluation forms, for example—this evaluation does not necessarily indicate whether the event actually improved supervisory behavior and benefited graduate researchers.
Examples of supervision training: A program at Coventry University
Although not prevalent, scholars occasionally publish journal articles that delineate and evaluate a supervision training program. To illustrate, Jara (2021) designed and commenced an evaluation of a program that facilitates the development of research supervisors. This study, in essence, revealed that supervisors greatly appreciated the discussion of case studies, the exchange of experiences, the insights of experienced supervisors, and the reflection on their own practice. Specifically, a team of staff at Coventry University reviewed the literature to determine the contents of this program, comprising a set of nine independent modules they could complete in any order, such as
-
the role of supervisors
-
how to support diverse candidates
-
how to deliver effective feedback
-
how to improve the writing and critical thinking of candidates
-
how to monitor progress
-
how to navigate the examination process
-
how to maintain effective and trusting relationships
During each module, participants read a relevant empirical article in advance and discussed key issues that were relevant to this paper, lasting about four hours. These discussions revolved around hypothetical case studies, exchange of experiences, insights from experienced supervisors, and reflection on practice. The cases were designed to reveal multifaceted complications in diverse settings. These case studies, typically fewer than 300 words, were derived from personal experiences, books, articles, and websites, such as academia.stackexchange.com. The modules were conducted online, but individuals could attend in person.
To evaluate the program, participants completed a survey at the end of each module and could participate in interviews after the program ended. Overall, the results were encouraging:
-
about 63% of participants completed all the modules, a reasonable level of attendance, suggesting this modular approach may improve participation. A previous variant of this program that was not modular did not attract this level of participation
-
the vast majority of participants—over 90%--agreed or strongly agreed with statements that indicate the program was coherent, the discussions and activities facilitated learning and reflection, the insights could be applied in practice, and the experienced improved their confidence
-
participants especially appreciated the case studies that depict the challenges of PhDs from the perspective of candidates
-
however, only 70% of the participants agreed or strongly agreed the concept or strategies were new.
These findings indicate that participants can perceive these experiences as invaluable and helpful to their supervisory practice, but even if they do not learn novel concepts or strategies.
Examples of supervision training: The supervising at UniSA workshop
Rather than comprehensive programs, some universities present shorter courses, such as induction training, that is designed to impart key insights but also inspire supervisors to develop their practices over time in the future. For example, McCulloch and Loeser (2016) outlined and evaluated the supervision program at the University of South Australia. This induction workshop introduces supervisors to doctoral education at the university and imparts some insights on the pedagogy of supervision as well as inspires supervisors to pursue more training and development in the future. The workshop is conceptualized as an opportunity to reflect upon supervision and collaborate with peers rather than as a hurdle that teach these individuals how to supervise. Hence, the workshop primarily adopts a constructivist approach to learning, in which supervisors develop their own insights, rather than merely a didactic approach—although some information is disseminated. The workshop, offered several times a year, lasts one day, including lunch, during which participants eat lunch together and engage with each other.
Before lunch, the workshop comprises two sections. First, the Dean of Graduate Studies and administrative staff from the research team briefly outline the policies, procedures, and operations at the university that relate to graduate research as well as the key trends across the nation. Second, a specialist in the field demonstrates how doctoral education has shifted over the years and highlights the key challenges that supervisors are increasingly experiencing.
After lunch, the workshop also comprises two sections. Specifically, experienced research supervisors at the university first discuss their solutions and responses to many of the challenges that supervisors are increasingly experiencing. And finally, a specialist in the field invites small teams of supervisors receive case studies, delineating common problems that candidates and supervisors experience. These teams develop strategies to resolve these problems—and then share these strategies with the other participants in the workshop.
After participants complete this workshop, they receive a digital badge, embedded in a platform that recognizes all the development activities that research candidates and supervisors complete. They also complete an evaluation sheet immediately after the workshop as well as a more extensive questionnaire months or years later, designed to assess the lasting impact of this experience. For example, in addition to demographic questions, the questions prompt the supervisors to consider which insights and practices they had applied or suggested to peers and whether they had pursued other training and development activities.
In general, the results of this questionnaire indicated the workshops were useful. Specifically
-
more than 50% of participants felt the workshop enhanced their understanding of the national trends around graduate research, how graduate research has shifted in the last two decades, the university policies and procedures, and the online support that is available
-
for example, 81% of participants felt the workshop enhanced their understanding of the national trends either to a large extent or to some extent
-
likewise, more than 50% of participants felt they had applied the knowledge and insights they learned, especially around university policies, university procedures, scholarships, online resources, and the strategies that experienced supervisors had applied
Examples of supervision training: A program directed at senior or experienced supervisors
Many supervisor training programs are primarily designed to benefit inexperienced supervisors (for an exception, revolving around conversations, see Spiller et al., 2013). In contrast, Wichmann-Hansen et al (2020) delineated a program, launched in a Danish university, lasting an extended duration, but targeted more at experienced supervisors. Thirteen senior academics participated.
Participation was mandatory and embedded within the job description. If individuals do not complete the course, some funding is withheld from the budget; otherwise, some funding is directed to the budget. To receive this incentive, participants needed to complete all homework activities and attend the sessions, consuming about 60 hours altogether.
To evaluate this program, Wichmann-Hansen et al (2020) derived data from evaluation forms that participants completed after the program ended, comprising rating scales and short answers, as well as perceptions of candidates on whether the quality of supervision had improved.
The program mainly revolved around how to deliver constructive feedback to drafts, how to maintain trusting relationships with candidates while they still monitor progress, and how to promote the independence of candidates. During the sessions,
-
the participants interacted in groups to discuss issues and cases
-
the participants completed quizzes, with MentiMeter to monitor feedback
-
the participants listened to insights about best practice
About half the 60 hours was dedicated to homework activities. The homework activities included
-
uploading their feedback on student drafts
-
writing their MoU or supervision agreement, on the roles and responsibilities of supervisors and candidates, and delivering feedback to other MoUs
-
write about a challenging case in supervision and then to share this case later in the workshop
-
writing a scholarly essay on the lessons participants gained from their observations of other supervisors; these observations were video recordings of actual meetings
The analysis suggested the program was generally effective. Specifically
-
93% of participants rated the program as effective or very effective
-
participants valued the opportunity to share practices and tools
-
participants, although skeptical initially, greatly appreciated the opportunity to receive feedback about their recorded practices
-
participants felt that, in the future, they will ask more open-ended questions to candidates, listen actively, clarify expectations, and implement better techniques to deliver feedback
-
more than 80% of candidates were more satisfied with their supervisor after this training
Challenges of candidates that could inform case studies and experiences: Liminal experiences
As these examples and many other studies demonstrate, supervisors often benefit from reflections and discussions about case studies and the challenging that peers experience. Therefore, to improve supervision training and development, coordinators need to understand some of the challenges that candidates, supervisors, or both candidates and supervisors experience.
Keefer (2015) invoked the notion of liminality to characterize some of the challenges that candidates experience. That is, during graduate research, candidates often experience sudden insights or shifts that transform the perceptions or understandings of some topic or matter. Typically, this transformation follows a period of uncertainty or confusion, called a liminal state. Liminality refers to the state after individuals have detached themselves from a previous identity or perspective but have yet to adopt the transformed identity or perspective entirely. These states are common in doctoral candidates, who often waver between their identity as students and their identity as academics, often manifesting as imposter syndrome. During most of their journey, these candidates experience obstacles and troubles that challenge their identity as academics. The role of supervisors is partly to help candidates circumvent these obstacles and assume the role of academics.
Keefer (2015) conducted a narrative inquiry to characterize these challenges and liminal experiences in 23 doctoral candidates. The aim of this study was to help supervisors appreciate the sources of this uncertainty and confusion in doctoral candidates and, thus, address these liminal experiences.
The results uncovered three key liminal experiences or challenges that many candidates, across a range of disciplines, experience. These experiences can last days, weeks, or even months and years. The first experience revolves around feelings of isolation and loneliness. Candidates referred to feeling of solitude, amplified by the unique path that each individual must travel to unearth original findings. Participants also referred to the burden they feel knowing they are entirely responsible for their research. These individuals did not expect to experience this loneliness, partly because they heard about the success of peers—success that only magnified their sense of loneliness.
The second liminal experience revolved around imposter syndrome, in which candidates questioned whether they have acquired the skills and capabilities to thrive in academia. Some participants were reluctant to disclose these concerns to their supervisor, partly because of unpleasant experiences in the past. Indeed, candidates often experience an incessant feeling their incompetence would eventually be unmasked. They felt their peers would recognize they do not belong. Explicit references to imposter syndrome helped some participants, because they felt the term normalizes the experience.
The final liminal experience revolved a sense of misalignment between the candidate and the supervisor or institution on research priorities and practices. Sometimes, this misalignment emanated from differences in the background, experience, qualifications, and personality between candidates and supervisors or other academics. But this misalignment can manifest in candidates as serious doubts about their capabilities and direction, after they receive strident, albeit, unfair criticism from supervisors or other academics.
The implication of these findings, according to Keefer (2015), is that supervisors should forewarn candidates about these potential challenges and offer to discuss these matters if they transpire. Supervisors might also suggest other avenues candidates may pursue in response to these concerns.
Challenges of supervisors that could inform case studies and experiences: Admission
Supervisors experience a range of challenges across the entire doctoral journey, from admission to examination. To illustrate challenges during admission, in study of 20 supervisors and 20 doctoral candidates, conducted by Denis et al. (2019), supervisors conceded they could not really predict which applicants are likely to thrive during their degree. Despite this uncertainty, inexperienced supervisors did not feel they were granted the luxury to reject many applicants—because they cannot as readily attract this interest.
Challenges of supervisors that could inform case studies and experiences: Remote supervision
When supervisors and candidates are not located in same region, and thus must communicate remotely, other challenges are unlikely. Nasiri and Mafakheri (2015) conducted a comprehensive review to identify, and to characterize, these challenges as well as to unearth the strategies that supervisors apply to address these challenges. As examples of challenges, the authors revealed that
-
if located in separate time zones, supervisors and candidates need to identify a time that suits both parties. This goal may be hard, because individuals often thrive on specific activities, such as activities that demand creativity, at particular times of the day (Breslin, 2019).
-
If supervisors and candidates do not meet in person, they may not feel a sense of familiarity with one another; the conversations, therefore, tend to be more formal and stilted rather than engaging and motivating (cf Sussex, 2008).
-
If located in separate time zones, supervisors might receive messages from a candidate outside their usual work hours. Responses to many emails outside hours—or even the prospect
Challenges of supervisors that could inform case studies and experiences: Changes over time
The practice of supervision has gradually, and sometimes abruptly, shifted in the last few decades. For example
-
in some previous decades, candidates were usually assigned only one supervisor; today, the vast majority of candidates are assigned two or more supervisors, usually operating as a panel (Green & Bowden, 2012)
-
in previous decades, supervisors often perceived themselves as the master, whose role was to impart knowledge to their apprentice; today, many candidates now prefer and expect their supervisor to facilitate their development collaboratively rather than merely impart knowledge or convey instructions (Fenge, 2012).
Other important content of supervision training: Technology
Although supervisors benefit if they analyze cases, share experiences, and reflect upon their practices, institutions still need to confirm that vital content is included in these workshop. Maor et al. (2016) argued that perhaps more training and opportunities around technology, such as Web 2.0 tools, could also facilitate supervision.
This recommendation was derived from the research that Maor et al conducted about Web 2.0 and online communities in doctoral research. Specifically, Maor et al conducted a comprehensive review to explore this topic. Initially, 196 papers that had been peer reviewed, 64 conference proceedings, 8 theses, and 16 reports fulfilled the selection criteria. The authors then extracted the 18 most relevant empirical articles on this topic and derived the recommendations from these sources. These articles examined or alluded to range of possible on technologies. These technologies include
-
tools that facilitate communication over distance, such as Elluminate, Wimba, and Second Life
-
tools that enable individuals to share information and insights, such as Microblogging and e-Portfolios
-
more common tools such as Skype and Microsoft Office SharePoint.
Furthermore, in two of the studies, the institutions had developed their own virtual spaces: Doctoralnet (Danby & Lee, 2012) and Form@doct (Malingre et al., 2013).
The main insight from these studies was the advent of a novel pedagogy, revolving around more sustained virtual communities of candidates and sometimes their supervisors, interested in an overlapping topic, in which individuals learned from one another and debated a range of relevant issues. Individuals might initially observe these communities. But then gradually, as they develop specialist knowledge and use tools that facilitate knowledge exchange—such as ePortfolios—they become more active participants. When these communities were most effective, rather than adopt the role as elite specialists or masters, the supervisors were often collaborative participants of these ecosystems. Doctoral candidates were granted more autonomy on when and how to learn.
Several tools, applications, platforms, and websites, such as Doctoralnet, can underpin these communities. These tools enable candidates and supervisors to discuss issues, across institutions, collaborate on writing, and share information.
Yet, in practice, many supervisors were not as familiar as their candidates in Web 2.0 technologies and, therefore, continued to adopt more traditional approaches. These supervisors did not often encourage their candidates to engage with online communities.
Even if supervisors contribute to these online communities, these collaborations may subside over time. To sustain these communities, coordinators may need to attract significant individuals, such as renowned academics or potential employers, to contribute occasionally.
Other sources of development
Formal training sessions are not the only source of development in supervisors. Indeed, Fulgence (2019) classified all the experiences and sources of this development, derived from past studies, into five clusters. First, supervisors might engage in deliberate education and training to develop these skills. They might attend workshops, either at their institution or outside their institution, both online and in person, or read articles and books on supervision. Some MOOCs, including Futurelearn and AuthorAID, impart supervision skills. Alternatively, supervisors might have even learned some of these skills during the training they received as a graduate candidate (Reguero et al., 2017).
Second, supervisors might derive these skills from their experiences in the realm of research. That is, during their doctorate or other research experiences, individuals are exposed to the practices and beliefs of many individuals, such as their supervisors or contacts the meet at conferences. These practices and beliefs shape their own knowledge and assumption about suitable research practices in general and supervision practices in particular. That is, during these experiences, they learn a range of management skills, research knowledge, supervision practices, networking abilities, and interdisciplinary skills, all of which shape their supervision practices. Indeed, according to Henderson, (2018), the practices of inexperienced supervisors are largely dependent on their practices of their supervisor during the doctoral studies.
Third, when academics are finally granted opportunities to supervise, this experience alone can shape their supervisory capabilities and tendencies (Reguero et al., 2017). They might derive insights from their observations of other supervisors on the panel, from reflections on their behavior, or from the advice of mentors they might arrange to facilitate this supervision. These experiences can help supervisors acquire more extensive knowledge about their discipline, research practices, relationship development, time management, and pedagogical practices.
Yet, these experiences, unless buttressed with more formal opportunities, is not sufficient to facilitate the development of supervisors. The fourth source of supervisor development—the policies, procedures, and guidelines of institutions—can overcome this shortfall, at least partly. These policies not only impart key information, such as the role and responsibilities of supervisors, but embed practices that can also be a source of learning, such as the feedback of examiners.
Finally, supervisors might participate in research that is designed to improve supervision. They might collate, distil, and analyze data that could be relevant to their role—such as the concerns that other candidates have raised about their supervision.
References
-
Bitzer, E. M., Trafford, V. N., & Leshem, S. (2013). 'Love it when you speak foreign': A trans-national perspective on the professional development of doctoral supervisors in South Africa. South African Journal of Higher Education, 27(4), 781-796.
-
Bøgelund, P. (2015). How supervisors perceive PhD supervision – And how they practice it. International Journal of Doctoral Studies, 10, 39-55.
-
Botha, J., & Muller, N. (Eds.) (2016). Institutional research in South African higher education. Stellenbosch: SUN PRESS
-
Breslin, D. (2019). Group creativity and the time of the day. Studies in Higher Education, 44(7), 1103-1118.
-
Danby, S., & Lee, A. (2012). Researching doctoral pedagogy close up: Design and action in two doctoral programmes. Australian Universities' Review, The, 54(1), 19-28.
-
Denis, C., Colet, N. R., & Lison, C. (2019). Doctoral supervision in North America: Perception and challenges of supervisor and supervisee. Higher Education Studies, 9(1), 30-39.
-
Dimitrova, R. (2016). Ingredients of good PhD supervision–Evidence from a student survey at Stockholm University. Utbildning & Lärande, 10(14), 40-52.
-
Fenge, L. A. (2012). Enhancing the doctoral journey: The role of group supervision in supporting collaborative learning and creativity. Studies in Higher education, 37(4), 401-414.
-
Fulgence, K. (2019). A theoretical perspective on how doctoral supervisors develop supervision skills. International Journal of Doctoral Studies, 14, 721.
-
Gordon, N. (2014). Flexible pedagogies: Technology-enhanced learning. Heslington, UK: The Higher Education Academy.
-
Green, P., & Bowden, J. (2012). Completion mindsets and contexts in doctoral supervision. Quality Assurance in Education, 20(1), 66–80.
-
Guerin, C., Kerr, H., & Green, I. (2015). Supervision pedagogies: Narratives from the field. Teaching in Higher Education, 20(1), 107-118.
-
Halse, C. (2011). Becoming a supervisor: The impact of doctoral supervision on supervisors’ learning. Studies in Higher Education, 36(5), 557-570.
-
Halse, C., & Bansel, P. (2012). The learning alliance: Ethics in doctoral supervision. Oxford Review of Education, 38(4), 377-392.
-
Henderson, E. F. (2018). Anticipating doctoral supervision: (Not) bridging the transition from supervisee to supervisor. Teaching in Higher Education, 23(4), 403–418.
-
Hill, G., & Vaughan, S. (2018). Conversations about research supervision – Enabling and accrediting a community of practice model for research degree supervisor development. Innovations in Education and Teaching International, 55(2), 153–163.
-
Jara, M. (2021). Research-based doctoral supervision development programme: Learning through peer learning, reflection and case studies. Innovations in Education and Teaching International, 58(4), 441–450.
-
Keefer, J. M. (2015). Experiencing doctoral liminality as a conceptual threshold and how supervisors can use it. Innovations in Education and Teaching International, 52(1), 17-28.
-
Lee, A. (2018). How can we develop supervisors for the modern doctorate? Studies in Higher Education, 43(5), 878-890.
-
Malingre, M. L., Serres, A., Sainsot, A., & Men, H. L. (2013). Form@ doct: Designing innovative online tutorials for PhD students in France. IFLA journal, 39(1), 45-57.
-
Manathunga, C. (2005). The development of research supervision: “Turning the light on a private space”. International Journal for Academic Development, 10(1), 17-30.
-
Maor, D., Ensor, J. D., & Fraser, B. J. (2016). Doctoral supervision in virtual spaces: A review of research of web-based tools to develop collaborative supervision. Higher Education Research & Development, 35(1), 172-188.
-
McCallin, A., & Nayar, S. (2011). Postgraduate research supervision: A critical review of current practice. Teaching in Higher Education, 17(1), 63-74.
-
McCulloch, A., & Loeser, C. (2016). Does research degree supervisor training work? The impact of a professional development induction workshop on supervision practice. Higher Education Research and Development, 35(5), 968–982.
-
Nasiri, F., & Mafakheri, F. (2015). Postgraduate research supervision at a distance: A review of challenges and strategies. Studies in Higher Education, 40(10), 1962-1969.
-
Olmos-López, P., & Sunderland, J. (2017). Doctoral supervisors’ and supervisees’ responses to co-supervision. Journal of Further and Higher Education, 41(6), 727-740.
-
Pearson, M., & Kayrooz, C. (2004). Enabling critical reflection on research supervisory practice. International Journal for Academic Development, 9(1), 99-116.
-
Reguero, M., Carvajal, J. J., García, M. E., & Valverde, M. (Eds.) (2017). Good practices in doctoral supervision: Reflections from the Tarragona Think Tank. Tarragona, Spain: Universitat Rovirai Virgili.
-
Rogers, J., Sorkness, C., Spencer, K., & Pfund, C. (2018). Increasing research mentor training among biomedical researchers at Clinical and Translational Science Award hubs: The impact of the facilitator training initiative. Journal of Clinical and Translational Science, 2(3), 118-123.
-
Saleem, T., & Mehmood, N. (2018). Assessing the quality of supervision experiences in the different research stages at postgraduate level. Journal of Education and Educational Development, 5(2), 8-27.
-
Spiller, D., Byrnes, G., & Bruce Ferguson, P. (2013). Enhancing postgraduate supervision through a process of conversational inquiry. Higher Education Research & Development, 32(5), 833-845.
-
Sussex, R. (2008). Technological options in supervising remote research students. Higher Education, 55(1), 121-137.
-
Sweitzer, V. B. (2009). Towards a theory of doctoral student professional identity development: A developmental networks approach. The Journal of Higher Education, 80(1), 1-33.
-
Turner, G. (2015). Learning to supervise: Four journeys. Innovations in Education and Teaching International, 52(1), 86-98.
-
Vilkinas, T. (2008). An exploratory study of the supervision of PhD/research students’ theses. Innovative Higher Education, 32(5), 297-311.
-
Wichmann-Hansen, G., Godskesen, M., & Kiley, M. (2020). Successful development programs for experienced doctoral supervisors – What does it take? International Journal for Academic Development, 25(2), 176-188.
-
Wisker, G., & Kiley, M. (2014). Professional learning: lessons for supervision from doctoral examining. International Journal for Academic Development, 19(2), 125-138.