Critical Appraisal of Translational Research Models for Suitability in Performance Assessment of Cancer Centers

  1. Abinaya Rajana,b,
  2. Richard Sullivanc,
  3. Suzanne Bakkera and
  4. Wim H. van Hartena,b

  1. aDepartment of Psychosocial Research and Epidemiology, The Netherlands Cancer Institute-Antoni van Leeuwenhoek Hospital, Amsterdam;

  2. bDepartment of Health Technology and Services Research, University of Twente, The Netherlands;

  3. cInstitute of Cancer Policy, Kings Health Partners Integrated Cancer Center, King's College London, United Kingdom

Abstract:

Background. Translational research is a complex cumulative process that takes time. However, the operating environment for cancer centers engaged in translational research is now financially insecure. Centers are challenged to improve results and reduce time from discovery to practice innovations. Performance assessment can identify improvement areas that will help reduce translational delays. Currently, no standard method exists to identify models for use in performance assessment. This study aimed to critically appraise translational research models for suitability in performance assessment of cancer centers.

Methods. We conducted a systematic review to identify models and developed a set of criteria based on scientometrics, complex adaptive systems, research and development processes, and strategic evaluation. Models were assessed for linkage between research and care components, new knowledge, systems integration, performance assessment, and review of other models.

Results. Twelve models were identified; six described phases/components for translational research in different blocks (T models) and six described the process of translational research (process models). Both models view translational research as an accumulation of new knowledge. However, process models more clearly address systems integration, link research and care components, and were developed for evaluating and improving the performance of translational research. T models are more likely to review other models.

Conclusion. Process models seem to be more suitable for performance assessment of cancer centers than T models. The most suitable process models (the Process Marker Model and Lean and Six Sigma applications) must be thoroughly tested in practice.

Footnotes

  • C/A:
    Consulting/advisory relationship
    RF:
    Research funding
    E:
    Employment
    H:
    Honoraria received
    OI:
    Ownership interests
    IP:
    Intellectual property rights/inventor/patent holder
    SAB:
    scientific advisory board

Introduction

Translational research is a complex, cumulative, and often unpredictable process focused on moving a single or combination of basic research findings into clinical practice. The recent identification of and attention to this field is not just meant to raise awareness, but also to improve performance in terms of efficiency and effectiveness. A particular challenge to translational research in oncology, as in other clinical fields, are perceptions about unnecessary delays in or complete blockage of translation.

In the fiscal year 2004–2005, global spending on cancer research reached approximately €14 billion ($17.64 billion). The U.S. (dominated by the National Cancer Institute) accounted for most of the spending, with per capita spending almost three times greater than Europe. However, in terms of publications and an increasing trend towards more applied clinical outputs, relative research productivity was better in Europe [1]. Apart from effectiveness issues, translation of research into practice still takes a lot of time. There are claims that translation of only 14% of new health-related scientific discoveries to clinical practice [2] takes an average of 17 years [3]. Ioannidis et al. examined 101 promising claims of new discoveries with clear clinical potential that were reported in major basic science journals between 1979 and 1983; only five resulted in interventions with licensed clinical use by 2003 and only one had extensive clinical use [4].

Imatinib is an example of successful translation from oncology. It shows the time it took for an intervention to reach licensed clinical use based on knowledge that emerged slowly over many decades. The drug focuses on disrupting one specific protein that seems to fuel the cancer while sparing other enzymes. The initial knowledge appeared in the 1960s when scientists first noticed chromosomal abnormalities in the blood of patients with chronic myeloid leukemia. However, it was not until the 1980s that genetic mapping helped determine that chromosomal abnormality produces a cancer-causing kinase enzyme. It took 2 years to create and test 400 molecules to find one that would target this enzyme without disrupting any of the hundreds of other similar enzymes in a healthy cell. Another 8 years of safety testing and development was needed before the drug could be tested with patients, finally giving remarkable results. While clinical trials were being expanded, the U.S. Food and Drug Administration put the drug on fast track for approval in 2001 [5].

Apart from effectiveness issues, translation of research into practice still takes a lot of time. There are claims that translation of only 14% of new health-related scientific discoveries to clinical practice takes an average of 17 years. Ioannidis et al. examined 101 promising claims of new discoveries with clear clinical potential that were reported in major basic science journals between 1979 and 1983; only five resulted in interventions with licensed clinical use by 2003 and only one had extensive clinical use.

Translational research is cumulative. To improve its performance and reduce unnecessary delays, acquiring insight into the process and performance assessment can add value. This means assessing performance in cancer centers against a set of predetermined criteria of the economy, efficiency, and effectiveness of that organization in conducting translational research (adapted from the Organisation for Economic Cooperation and Development definition) [6] with the purpose of supporting continuous improvement and transparent accountability at multiple organizational levels. This would help address delays by identifying areas for improvement, including innovation transfer management, organizational administration of research projects, incentive mechanisms to motivate researchers, and communication strategies between researchers and other key stakeholder groups. These areas can promote multidisciplinary collaboration that in turn can speed the rate at which basic research discoveries eventually become clinically viable health technologies.

For performance assessment, it is essential to know what is being translated and how it is being translated. Initially, models need to be systematically identified and critically appraised before they can be tested in practice. To a large extent, the process of translational research seems to be generic, and it is not clear if a specific model should be preferred for oncology. At present, it is unknown how many models exist and which of those are suitable for performance assessment. Most recent references are based on two studies. Trochim et al. reviewed and synthesized four models to illuminate important issues to evaluate translational research [7]. Morris et al. looked at quantification of translational time lags; in that context, they offered a tentative model based on synthesis of a few models [8]. However, the studies do not specify if they conducted a systematic identification of models, nor did they use systematic criteria to appraise the identified models. Moreover, in the study by Morris et al., it is not clear how many models were used to synthesize their model.

The current study aims to identify models of translational research using a systematic literature review and critically appraise them by using common criteria that were specifically developed for this purpose. The rationale is to identify the models that are most suitable for assessing the performance of cancer centers in translational research.

Methods

A systematic literature review was carried out to identify translational research models using a combination of search terms in four databases: PubMed, Embase, Trip Database, and Scopus (supplemental online data). The first search included scientific terms and common expressions for translational research and terms associated with models and performance assessment, whereas the second search included scientific terms and common expressions for translational research and different phases of translation (Fig. 1). In addition, we tracked the references and citations for a few papers that were identified through the previous search method, which either proposed a model and/or identified other models. We did not limit our search to models specific to oncology nor to the year of their publication.

Figure
Figure 1.

Search terms used to identify models of translational research.

Criteria Development to Appraise Models

At present, there is no standard methodology to assess the suitability of translational research models for performance assessment purposes. We developed a set of criteria (CR; Table 1). The models were awarded a yes or no answer for each question, in which yes meant that the model seemed suitable for performance assessment. Model appraisal focused on how translational research was presented in terms of its main purpose, component(s) that can be evaluated, strategies to evaluate the identified components, and testing of the chosen strategies in practical settings. To validate our focus, we referred to a range of literature from both medical and nonmedical disciplines, such as organizational management.

Table 1.

Critical appraisal of translational research models

With reference to the scientometric analysis conducted by Jones et al., we deduced that translational research emerged to link the research and care components (CR1) [29]. Cancer research is a complex adaptive system in which the components must be regularly assessed to improve their performance (CR2 and CR3) [30]. Fifth-generation research and development suggests that performance assessment strategies should integrate organizational systems to link the process of translation that occurs through cross-boundary learning and knowledge flow (CR4) [31]. Using the theory of the evaluation of strategic options by Johnson and Scholes, we framed criteria for evaluating the strategies of the models for suitability and feasibility (CR5 and CR6) [32]. A seventh criterion based on acceptability (CR7) was meant to check if models have been tested or applied in practice. This last criterion is not been presented in Table 1 because we were able to assess only one model.

Results

Identified Translational Research Models

A total of 2,397 studies were identified after removing the duplicates (Fig. 2). Title screening showed that the majority of studies were related to specific biomedical discoveries focusing on basic and translational issues. Many studies referred to animal models, not conceptual models. Only 385 papers contained a description of translational research. Abstract screening led to 182 papers that contained bench-to-bedside issues; 89 studies used descriptive statements to define translational research. Only 12 studies that contained and described a model were included in the resulting appraisal. Of these, 6 studies described the main phases/components for translational research within different translational blocks (T models) [2, 9, 1214, 19]. The remaining 6 papers mapped the steps/processes for translational research (process models) [7, 2226]. Both type of models start at basic discovery; the following phases extend to clinical trials or even beyond to widespread diffusion or population impact (Fig. 3).

Figure
Figure 2.

A systematic review to identify models of translational research.

Figure
Figure 3.

An overview of translational models and process models of translational research.

Overview of T Models

The terminologies and position of the types of translations are inconsistent in all T models. Overall, the T blocks identify the specific translational areas that are also barriers for translation, but steps to overcome these barriers and improve performance are not clearly addressed.

Type 1 Translation

In the six models, descriptions of type 1 translation (T1) have similar starting points but are phrased differently. T1 encompassed “basic research to patient based research” [9], “basic science research to human clinical research” [2], “basic science research (phase 0) to early human trials (phase 1) and early clinical trials (phase 2)” [14], “basic biomedical science to clinical efficacy knowledge” [13], “basic biomedical to clinical science knowledge” [12], and “gene discovery to health applications” [19]. Because of these variations, it is hard to establish where T1 ends.

Type 2 Translation

The description of type 2 translation (T2) is also inconsistent over all models. T2 encompassed “patient oriented to population oriented research” [9], “human clinical research to practice based research” [2], “early clinical trials (phase 2) to late clinical trials (phase 3)” [14], “clinical efficacy knowledge to clinical effectiveness knowledge” [13], “clinical science knowledge to improved health” [12], and “health applications to evidence-based guidelines” [19].

Type 3 Translation

The location and extent of type 3 translation (T3) also varies in all models. T3 encompassed “population-based research to basic research” [9], “practice-based research to clinical practice” [2], “late clinical trials (phase 3) to implementation phase (phase 4)” [14], and “clinical effectiveness knowledge to improved population health” [13]. In Sung et al.'s model, there was no T3 [12]. In Khoury et al.'s model, T3 was the translation of guidelines to health practice [19].

Type 4 Translation

Only Khoury et al.'s model contained type 4 translation (T4), which was the translation of practice to population health impact [19].

Overview of Process Models

Three process models used T terminologies. The early translational pathways by Ernest et al. [23] used the T1-T2 model, but the pathways were mapped only for T1. They were developed to aid the transformation of scientific discoveries into new clinical modalities for oncology—specifically risk assessment modalities (biospecimen-based risk assessment devices and image-based risk assessment) and interventive modalities (agents, immune response modifiers, interventive devices, lifestyle alterations).

The biomedical research continuum by Drolet and Lorenzi [22] consisted of a zone of translation with three translational chasms (T1-T3): T1 was laboratory to clinical research between basic science discovery to proposed human application; T2 was safety and efficacy research between proposed human application and proven clinical application; T3 was implementation and adoption research between proven clinical applications and clinical practice. A pathway, inquiry, and action for each chasm were given.

The Lean and Six Sigma applications to clinical and translational research by Schweikhart and Dembe [24] used the T1-T4 phases by Khoury et al. [19] to improve the efficiency of translational research. Each phase consisted of business management strategies for process assessment. The Process Marker Model by Trochim et al. identified key steps of translational research, which were not represented by Ts but described as three integrated systems: basic research system, clinical trials system, and practice-based system. The model aims to evaluate the process of translational research in order to reduce the time lag [7].

The Need to Knowledge model by Lane and Flagg [25] identified unmet needs that lead to the generation of knowledge through the outputs of three activities: research discovery, prototype intervention, and product innovation. It recognized that knowledge implementation and beneficial societal impacts involve effective communication of each successive knowledge state to the relevant stakeholders. Finally, Ogilvie et al.'s model is a framework to advance translational research that identifies a pivotal role for evidence synthesis that translates knowledge of nonlinear and intersectoral interfaces to the public realm [26].

Oncology-Specific Models

It was difficult to confirm which of the appraised models are currently being used to inform translational research in cancer centers in Europe and/or the U.S. However, only one model was specifically developed for oncology: the early-stage translational pathways by Ernest et al. [23]. They used the T1-T2 model proposed by the President's Cancer Panel [18]. This was one of the first models in translational research to emerge and is also known as bench-to-bedside-to-practice. The pathways were developed in T1 phase to facilitate the process of basic discoveries in cancer to be developed into clinical modalities, but they have not been adopted in practice.

Evidence From Appraisal of T Models and Process Models

The process models were more favorable when appraised against our criteria than T models (Table 1), suggesting that they may be better suited for performance assessment in cancer centers. There is only one similarity between the two types of models: they view translational research as accumulation of new knowledge. The differences are that process models more clearly address systems integration, link research and care components, and were developed for evaluating and improving the performance of translational research. In contrast, T models tend to review other models; their purpose is to present the phases of translational research but not to assess and improve its performance.

Three process models (Lean and Six Sigma applications, the Process Maker Model, and the Need to Knowledge Model) seem to have been developed to evaluate translational research. In particular, the first two models scored highest in the appraisal (Figs. 4 and 5). They track the time between various steps of the different translational phases in order to improve translational process efficiency. Lean and Six Sigma is the only model that clearly gave evidence that it had been tested in practice in a process improvement project focused on redesign of the scheduling system at the clinical trials unit of Ohio State University [24].

Figure
Figure 4.

Examples of process marker models at three levels of scale. Reprinted from [7] with permission from Wiley.

Abbreviations: FDA, U.S. Food and Drug Administration; IRB, institutional review board.

Figure
Figure 5.

Example of a process improvement project at a clinical trial unit using Lean techniques. Reprinted from [24] with permission from Wolters Kluwer Health.

Possible Implementation of Lean and Six Sigma Techniques in Performance Assessment of Translational Research

A research process improvement project involving redesign of the scheduling system in the clinical trials unit of the Ohio State University (Fig. 5) used a five-stage intervention. The aim was to improve the efficiency of the patient scheduling process by replacing paper-based calendar system with a more coherent data-driven computerized scheduling system. It is a practical example of the applicability of Lean and Six Sigma techniques in assessing and improving the performance of translational research.

In stage 1, an environmental scan was undertaken by a research team to determine stakeholder needs, as well as to sufficiently identify and understand various steps that are involved in the patient accrual and scheduling process, including protocol requirements, the total number of trials being conducted, software requirements, inpatient bed capacity, number of available nurses and other staff per shift, examination and treatment room availability, number of expected visits and specific visit number in the sequence of protocol. The improvement strategy was to develop acuity measures to gauge resource intensity in each step.

Next, in stage 2, the team identified and mapped each process step and relationship between those steps using value stream maps or process flow maps. As an improvement strategy, they developed different scheduling algorithms based on acuity measures and other factors. In stage 3, the team identified obstacles for and inefficiencies between patient scheduling and planning of the available resources. The improvement strategy led to the development of standardized scheduling instructions for physicians and patients to improve resource utilization.

In stage 4, the team performed repeated field testing of various scheduling algorithms. As an improvement strategy, an acuity table with estimates for each activity was calculated. For example, the activity of “simple specimen collection” was given an acuity score of 5. A scheduling algorithm matched the scores with key internal and external factors (e.g., availability of a specific number of research staff per shift, room availability, protocol-related requirements) to optimize patient and staff scheduling on a given day.

Finally, in stage 5, an assessment of organizational structure and culture was done in the research unit to evaluate readiness for change. The improvement strategy led to cross-disciplinary training of research staff to make them understand and use the new patient scheduling system. The concerns and suggestions by staff regarding the practical use of the system were addressed during the training. The above stages led to the adoption of the system in daily practice [24].

Drawing on this example in more generic terms, a five-stage intervention for applying performance assessment models in translational research in cancer centers should address the following:

  1. Environmental scanning to understand key activities in translational research

  2. Elaborating different algorithms in which the identified key activities will be efficiently performed

  3. Evaluation of these algorithms by performing continuous improvement cycles to check which algorithm is most suitable

  4. Using estimates (metrics such as frequency/duration) to map the key activities identified and correlating that to key internal and external factors that may affect those estimates

  5. Training of research staff on the new system and ensuring that its implementation within the cancer center is acceptable to key stakeholders

Based on these stages, qualitative and quantitative indicators can be derived.

Discussion

This study aimed to identify models of translational research and appraise their suitability for performance assessment of cancer centers. We managed to identify 12 models of translational research: six T models and six process models.

T models contribute to our understanding of translational research by mapping its key components. However, these components vary from model to model, confirming the statement of Australia's chief scientist, Professor Ian Chubb: “If you were to ask ten people what translational research means, you're likely to get ten different answers” [33]. It is not clear whether the variations in T models reflect actual variations in practice or are related to specific objectives or circumstances of various stakeholders. These variations may also reflect models being developed for specific research and/or clinical domains. In contrast, process models identify methods to facilitate, track, and assess knowledge flows and interfaces along the continuum, including multiple starting points for innovation, pathway mapping, process markers, using strategies and tools from business management, and inclusive evidence synthesis.

Based on our appraisal, two process models seem to be most suitable for performance assessment of cancer centers: the Process Marker Model and Lean and Six Sigma applications. Process markers can help cancer centers assess the performance of translational research by tracking the time taken between markers, such as prepiloting of studies, submission of research proposals, funding of studies, the start and end of data collection for studies, and inclusion of the study in research synthesis (e.g., publications or mainstreaming of research activities) that leads to subsequent stages of translational research. Process markers can include both process steps as well as reflect the transfer process per step (known as subprocess markers). Process markers can be defined for phases in clinical trials, proposal submission, Institutional Review Board approval, funding of proposal, accrual of first subject, closed to accrual, and presentation and publishing of results etc [7]. Process markers might help to identify and possibly reduce the time between different phases of clinical trials in cancer.

Lean and Six Sigma applications are complementary to the Process Marker Model and might help cancer centers define markers more clearly. For example, in basic research, process makers could include turnaround time of toxicology results, transfer of samples in laboratory, and response to regulatory requests. In clinical trials, cancer centers could track the unnecessary time and/or added value per process step for biostatistical consultations, minimizing protocol amendments, checking if placebos are needed, patient recruitment campaigns, patient monitoring process, and elimination of early-phase design errors [24].

The fear regarding [performance] assessments among some stakeholder groups is that it might jeopardize serendipity that is characteristic for many research processes, fail to capture research excellence that might exist partially or completely outside the scope of assessment criteria, and enable bureaucrats to take control of fields they do not really comprehend. A cautious and stepwise approach is therefore advisable if cancer centers intend to use these models for performance assessment.

However, the models still have some limitations. Lean and Six Sigma applications are derived from a nonmedical field. Although their pilot results are positive, they need to be tested in other phases of translational research to fully validate their use along the continuum. The Process Marker Model lacks precisely stated operational definitions of markers and an inferential statistical analysis framework [7]. In addition, although markers primarily measure time lags, qualitative value related criteria are still lacking.

The five-stage intervention for the possible implementation of Lean and Six Sigma techniques can be adapted to different phases of the translational research continuum. It can aid performance improvement from basic science along the continuum to population impact. However, defining activities or markers for the earlier phases is relatively easier than for later phases, such as population impact. These later phases tend to be beyond the primary scope of some comprehensive cancer centers. Hence, inclusive evidence synthesis is needed to understand the later phases from a broader public health perspective [26] before performance assessment models can be implemented.

Translational research is not a simple linear process. Some may argue that its complex and unpredictable nature prohibits the use of models for performance assessment. The fear regarding such assessments among some stakeholder groups is that it might jeopardize serendipity that is characteristic for many research processes, fail to capture research excellence that might exist partially or completely outside the scope of assessment criteria, and enable bureaucrats to take control of fields they do not really comprehend. A cautious and stepwise approach is therefore advisable if cancer centers intend to use these models for performance assessment. As a first step, acquiring structured insight into the various aspects of the translational process and comparing these results between cancer centers might help centers identify improvement opportunities. For that purpose, more precise operating definitions are needed at three levels: performance dimensions, performance indicators, and sufficiently detailed metrics [34]. It is hard to say whether the cancer field has specific needs, but all stakeholders, including clinicians, should be open to the idea that models from other medical and/or nonmedical fields can also be used to assess cancer centers. These models should be thoroughly tested in practice to know their potential for actual performance assessment.

The strengths of this study are that, to our knowledge, this is the first time that a systematic review has been undertaken to identify models of translational research that were appraised using a set of criteria. These criteria were based on a range of issues for translational research identified from relevant literature. Undoubtedly, the criteria that we used can be critiqued. However, it is necessary for cancer centers to carefully select models for performance assessment and our framework provides a basis for that. The criteria can be refined with views from key stakeholder groups (e.g., basic researchers, clinical researchers, clinicians, funding agencies, senior executives, and patients).

There are two possible limitations to our study. First, we could not check if all the models had been tested and implemented in practice. One could argue that the elements of these models are supported by “findings” or evidence from academic or experiential literature. The second limitation is that, because of a lack of consensus on terminologies in translational research, it was hard to identify models. Therefore, there could be models that we did not consider in this appraisal. To increase the possibility of identifying models in future, the title, abstract, and keywords of studies should clearly use a common term and/or commonly associated terms of translational research. Substitutions such as “bench-to-bedside,” “implementation science,” and “biomedical research” should be restricted to the main content of the papers, with clear explanation of these terms that can help the reader understand the model. Addition of a specific MeSH term for models in databases (e.g., conceptual models of translational research) may be useful to ensure that models are easily listed and identified.

Conclusion

Performance assessment can help improve the process of translational research by identifying areas for improvement in its management, knowledge exchange, and engagement of multidisciplinary teams to deliver efficient and effective translational research, which would help reduce unnecessary time lag. Two models of translational research appear to be more suitable for performance assessment: the Process Marker model and the Lean and Six Sigma applications to clinical and translational science. It will be necessary to thoroughly test them in practice. Finally, cancer centers need to come to a consensus on terminologies in translational research, which will help to identify and select models for performance assessment that can improve the performance of translational research for the benefit of patients.

Author Contributions

Conception/Design: Abinaya Rajan, Richard Sullivan, Wim van Harten

Collection and/or assembly of data: Abinaya Rajan, Suzanne Bakker

Data analysis and interpretation: Abinaya Rajan, Wim van Harten

Manuscript writing: Abinaya Rajan, Wim van Harten

Final approval of manuscript: Abinaya Rajan, Richard Sullivan, Suzanne Bakker, Wim van Harten

Acknowledgments

The study was supported by the FP7 program of the European Commission as part of a Eurocan platform project (An Excellence Designation System for Comprehensive Cancer Centers in Europe). The project consists of 28 cancer institutions across Europe that currently collaborate on different work packages to help advance translational cancer research.

Footnotes

  • C/A:
    Consulting/advisory relationship
    RF:
    Research funding
    E:
    Employment
    H:
    Honoraria received
    OI:
    Ownership interests
    IP:
    Intellectual property rights/inventor/patent holder
    SAB:
    scientific advisory board

References


























Copyright © 2014 - Material published on this website is protected by copyright. You are expressly prohibited from electronically storing any material found here or republishing it by any means.