Pragmatic Free Trial Meta
Pragmatic Free Trail Meta is an open data platform that enables research into pragmatic trials. It shares clean trial data and ratings using PRECIS-2 permitting multiple and varied meta-epidemiological studies that compare treatment effects estimates across trials with different levels of pragmatism and other design features.
Background
Pragmatic trials are increasingly acknowledged as providing evidence from the real world for clinical decision making. However, the use of the term "pragmatic" is not uniform and its definition as well as assessment requires further clarification. Pragmatic trials are intended to guide clinical practices and policy decisions rather than prove a physiological or clinical hypothesis. A pragmatic trial should aim to be as similar to the real-world clinical environment as possible, including in its selection of participants, setting and design as well as the execution of the intervention, and the determination and analysis of outcomes as well as primary analyses. This is a major distinction between explanation-based trials, as described by Schwartz & Lellouch1 that are designed to prove the hypothesis in a more thorough way.
The most pragmatic trials should not conceal participants or clinicians. This can lead to bias in the estimations of the effect of treatment. Pragmatic trials should also seek to enroll patients from a variety of health care settings, so that their results can be compared to the real world.
Furthermore, trials that are pragmatic must focus on outcomes that matter to patients, such as quality of life and functional recovery. This is particularly relevant in trials that require surgical procedures that are invasive or may have harmful adverse consequences. this link trial29, for example focused on the functional outcome to compare a 2-page case-report with an electronic system for the monitoring of patients in hospitals suffering from chronic heart failure. In addition, the catheter trial28 used urinary tract infections that are symptomatic of catheters as its primary outcome.
In addition to these aspects the pragmatic trial should also reduce the procedures for conducting trials and requirements for data collection to reduce costs. In the end, pragmatic trials should aim to make their findings as relevant to actual clinical practices as possible. This can be accomplished by ensuring that their analysis is based on an intention-to treat method (as described within CONSORT extensions).
Despite these guidelines, many RCTs with features that defy pragmatism have been incorrectly self-labeled pragmatic and published in journals of all kinds. This can lead to false claims of pragmaticity and the use of the term should be standardized. The creation of a PRECIS-2 tool that can provide a standardized objective evaluation of the pragmatic characteristics is the first step.
Methods
In a pragmatic research study it is the intention to inform clinical or policy decisions by demonstrating how an intervention could be integrated into routine treatment in real-world settings. This differs from explanation trials, which test hypotheses about the causal-effect relationship in idealized situations. In this way, pragmatic trials could have less internal validity than explanatory studies and be more prone to biases in their design as well as analysis and conduct. Despite their limitations, pragmatic studies can be a valuable source of information to make decisions in the healthcare context.
The PRECIS-2 tool evaluates the level of pragmatism that is present in an RCT by assessing it across 9 domains, ranging from 1 (very explicit) to 5 (very pragmatic). In this study, the areas of recruitment, organization and flexibility in delivery, flexibility in adherence, and follow-up received high scores. However, the main outcome and the method for missing data were scored below the practical limit. This suggests that a trial could be designed with well-thought-out practical features, but without harming the quality of the trial.
It is difficult to determine the level of pragmatism within a specific trial because pragmatism does not possess a specific characteristic. Certain aspects of a research study can be more pragmatic than other. Moreover, protocol or logistic changes during an experiment can alter its pragmatism score. Koppenaal and colleagues found that 36% of 89 pragmatic studies were placebo-controlled or conducted prior to the licensing. They also found that the majority were single-center. They aren't in line with the usual practice, and can only be considered pragmatic if their sponsors accept that the trials are not blinded.
Furthermore, a common feature of pragmatic trials is that the researchers try to make their results more relevant by analyzing subgroups of the trial. This can lead to unbalanced results and lower statistical power, which increases the likelihood of missing or misinterpreting differences in the primary outcome. In the case of the pragmatic studies included in this meta-analysis, this was a major issue since the secondary outcomes were not adjusted for the differences in the baseline covariates.
Furthermore the pragmatic trials may have challenges with respect to the collection and interpretation of safety data. This is because adverse events are generally reported by the participants themselves and are prone to reporting errors, delays or coding errors. It is crucial to improve the quality and accuracy of the outcomes in these trials.
Results
While the definition of pragmatism does not require that all clinical trials be 100% pragmatist There are advantages to including pragmatic components in trials. These include:
Enhancing sensitivity to issues in the real world as well as reducing study size and cost as well as allowing trial results to be faster transferred into real-world clinical practice (by including patients who are routinely treated). However, pragmatic trials can also have disadvantages. For example, the right kind of heterogeneity can allow a trial to generalise its results to different patients and settings; however, the wrong type of heterogeneity may reduce the assay's sensitivity, and thus decrease the ability of a study to detect even minor effects of treatment.
A variety of studies have attempted to classify pragmatic trials using different definitions and scoring methods. Schwartz and Lellouch1 have developed a framework that can differentiate between explanation studies that confirm a physiological or clinical hypothesis, and pragmatic studies that help inform the choice for appropriate therapies in the real-world clinical practice. The framework was comprised of nine domains assessed on a scale of 1-5 which indicated that 1 was more informative and 5 was more pragmatic. The domains included recruitment and setting, delivery of intervention with flexibility, follow-up and primary analysis.
The original PRECIS tool3 had similar domains and scales from 1 to 5. Koppenaal et al10 developed an adaptation of the assessment, called the Pragmascope that was simpler to use for systematic reviews. They found that pragmatic systematic reviews had higher average scores across all domains, but lower scores in the primary analysis domain.
This difference in the primary analysis domain could be explained by the fact that the majority of pragmatic trials analyze their data in an intention to treat method while some explanation trials do not. The overall score for pragmatic systematic reviews was lower when the domains of management, flexible delivery and following-up were combined.
It is important to remember that a pragmatic study should not mean that a trial is of poor quality. In fact, there is an increasing number of clinical trials which use the word 'pragmatic,' either in their abstracts or titles (as defined by MEDLINE, but that is neither precise nor sensitive). These terms may signal a greater awareness of pragmatism within titles and abstracts, but it's unclear whether this is evident in the content.
Conclusions
In recent years, pragmatic trials have been increasing in popularity in research because the value of real world evidence is becoming increasingly acknowledged. They are clinical trials that are randomized that evaluate real-world alternatives to care instead of experimental treatments in development, they include patient populations which are more closely resembling the ones who are treated in routine care, they use comparisons that are commonplace in practice (e.g. existing medications), and they depend on the self-reporting of participants about outcomes. This method could help overcome the limitations of observational research, such as the biases that arise from relying on volunteers, and the limited availability and coding variability in national registry systems.
Pragmatic trials offer other advantages, like the ability to leverage existing data sources and a greater chance of detecting significant differences than traditional trials. However, they may still have limitations which undermine their effectiveness and generalizability. The participation rates in certain trials could be lower than expected due to the health-promoting effect, financial incentives, or competition from other research studies. The need to recruit individuals quickly restricts the sample size and the impact of many practical trials. Some pragmatic trials also lack controls to ensure that any observed differences aren't caused by biases that occur during the trial.
The authors of the Pragmatic Free Trial Meta identified 48 RCTs that self-described themselves as pragmatic and were published from 2022. The PRECIS-2 tool was used to evaluate the pragmatism of these trials. It includes domains such as eligibility criteria, recruitment flexibility as well as adherence to interventions and follow-up. They discovered that 14 trials scored highly pragmatic or pragmatic (i.e. scoring 5 or more) in at least one of these domains.
Trials with a high pragmatism score tend to have broader eligibility criteria than traditional RCTs which have very specific criteria that are unlikely to be used in the clinical setting, and include populations from a wide range of hospitals. These characteristics, according to the authors, may make pragmatic trials more useful and applicable in the daily practice. However, they cannot ensure that a study is free of bias. The pragmatism characteristic is not a fixed characteristic and a test that doesn't have all the characteristics of an explanatory study may still yield reliable and beneficial results.