Keywords clinical decision support - genomics - clinical trial - molecular tumor board - trial matching - genetic alteration
Background and Significance
Background and Significance
Genome sequencing has become more cost-effective and feasible due to technical developments such as next-generation sequencing (NGS) methods.[1 ]
[2 ] Oncology, in particular, benefits from these developments, as sequencing of the cancer genome can provide extended molecular tumor profiling, as well as valuable information on potential personalized treatment strategies.[3 ] This consideration has led to the development of a variety of targeted therapy approaches.[4 ]
[5 ]
[6 ]
Since the same targetable mutations can occur in different tumor entities, more and more clinical trials are being conducted as so-called basket studies which include tumor entities based on its specific mutation or biomarker.[7 ]
[8 ] This leads to a much wider spread of tumor characterizations and therefore also to a reduction of matching patients and corresponding case numbers.[9 ]
To cope with the complexity of molecular-based cancer treatment, many institutions have implemented molecular tumor boards (MTBs) complementing the already existing organ-based tumor boards.[10 ] These MTBs are composed of interdisciplinary experts reviewing and discussing the complex personalized therapy options based on clinical and advanced molecular diagnostics. For treatment recommendations, it is crucial to find accessible clinical trials that match the genetic profile of the patient's tumor to provide patient the opportunity to participate in these trials, or to include findings from completed trials. Especially in the last few years, it has been possible to expand the therapy guidelines based on genetic tumor profiling, as shown for example for lung carcinoma by Lindeman et al.[11 ] However, mainly due to the increase in advanced diagnostics using next generation sequencing, there are still other unknown genetic findings where no guided line therapy is currently available.
However, the search for suitable clinical trials is often performed manually, and therefore, it is a time-consuming process.[12 ] One tool that can be employed to use bioinformatic and clinical resources to determine the clinical relevance of gene alterations for the MTB is the open-source platform cBioPortal developed by the Memorial Sloan Kettering Cancer Center.[13 ]
[14 ] It allows analysis of clinical and molecular patient data of cancer patients and can be accessed as a public instance or deployed locally enabling customization of the portal and analysis of own patient data. Buechner et al performed an extensive requirements analysis and specification of an MTB platform[15 ] in which they found cBioPortal as a suitable basis for such an application. One of the identified still missing features for the use of cBioPortal in the clinical setting of an MTB was facilitating the search of suitable clinical studies for patients.
Utilizing cBioPortal as a local clinical and genomic data warehouse, there were already efforts to integrate clinical trials matching based on the genetic profile and clinical patient data. This is achieved by the integration of MatchMiner[16 ] that matches the patient's genetic profile to clinical trials that have previously been curated by domain experts and converted into a structured proprietary format. However, the curation constitutes a time-consuming process, therefore resulting in a relatively small number of available studies and a constant effort to keep the data up to date. Other solutions to find matching studies based on genetic characteristics are either proprietary, such as My Cancer Genome,[17 ] or single-institution, such as TrialProspector,[18 ] and therefore not publicly available.
Objectives
In this work, we developed an extension to cBioPortal to enable patient-centric clinical trials search. It is based on the integration of ClinicalTrials.gov and leverages cBioPortal's structured molecular and clinical data which can be used to prefill the search fields. The main requirement was that the search for clinical trials should not require any manual curation or preprocessing of trial data and should be executed on the fly from the fronted of cBioPortal. It should aid MTB participants during the often laborious process of searching clinical trials for MTB patients, and therefore support an important one of their routine tasks. To account for the specific needs and requirements of the MTB participants, a prototype was implemented using a user-centered design approach including multiple feedback loops and evaluated in the context of a usability test.
Methods
Buechner et al[15 ] revealed the need of the MTB for the previously described extension in their requirements analysis and also already created a high-fidelity mockup for the design thereof. Adding on their findings, an in-depth requirements analysis for the specifics of the extension using open expert interviews was conducted. The interviews for this requirements analysis were performed in the course of a biweekly jour fixe about the advancement of processes and software in MTBs with over 20 regularly participating clinicians and medical informatics specialists of nine German partner sites. Based on the mockup ([Fig. 1 ]) and the resulting detailed requirements, a prototype was implemented using a user-driven development process with a total of three iterations over a period of 2 months. During this process, the prototype was further refined iteratively, taking into account the feedback of the real-world users during the aforementioned jour fixes, particularly oncologists who provide MTB therapy recommendations for patients. This feedback of the prototype was collected by allowing users to try out the prototype and report any problems, ambiguities, or missing features during the jour fixes or afterward as written feedback. The prototype used real patient data from the cBioPortal public database and real study data from ClinicalTrials.gov.
Fig. 1 The high-fidelity mockup of the frontend integration of the ClinicalTrials.gov search as proposed by Buechner et al.[15 ] It is derived from a screenshot of the patient view, adding the studies search tab using an image editing for the purpose of better demonstrating the feature. The mockup does not provide any functionality, but it served as a blueprint for the prototype.
To evaluate the extension, a formative usability test (the summative evaluation will be performed in a future study along with multiple other features extending cBioPortal for its use in MTBs) was conducted with real-world users. The call to participate in the usability test went to all potential probands, that is, 10 clinicians regularly attending MTBs and the aforementioned jour fixes. Five of these participants of the MTB were asked to perform a clinical trial search for a patient[19 ] using the newly implemented extension as the scenario for the usability test. During the test process, the participants were instructed to review the patient's genetic alterations and clinical data and select fitting data to feed into the search tool. In the first step of the search, the users were asked to prioritize studies the patient could realistically enroll in, most importantly meaning studies close to the patient's place of residence. The second step should also include worldwide studies from which the user may only derive new therapy approaches, as enrollment of own patients would not be practical due to the place of residence. During the whole process, the user was asked to verbalize all thoughts (think-aloud protocol). Afterward, the participant was interviewed using open-style and Likert's scale[20 ] questions and finally asked to complete the system usability scale (SUS)[21 ] questionnaire in German[22 ] which is the native language of all participants. The usability tests were performed as web meetings using the teleconferencing software Zoom, taking approximately 30 to 60 minutes, with screen sharing enabled which allowed the recording of the users' interactions with the user interface, as well as all feedback during the think-aloud protocol. These recordings were then systematically analyzed. All documented errors, difficulties, and hesitations from the usability perspective during the think-aloud protocol, as well as questions and suggestions for improvement from the open interview were grouped into categories and summarized to single change requests after collection.
Results
The detailed requirements analysis revealed that the search results should be displayed as a table within a new tab of the patient view of cBioPortal ([Fig. 2 ]). The adjustable search parameters should consist of the following parameters: (1) variants and copy number variations (CNVs), (2) generic free-text keywords, (3) recruiting statuses of the study, (4) trial location by country, and (5) the age and sex of the patient. To be able to review the proposed trial results as fast as possible, the table should include (1) the recruiting status, (2) found keywords, (3) the title of the study, (4) conditions/diagnoses, (5) interventions, and (6) locations of partner sites. During the iterative feedback rounds with real-world users, prior requirements were adapted and refined and the prototype could be further developed. Multiple comments were related only to the user interface, for example, the way of entering study locations, how to select relevant alterations, or improving the explanation of fields using tooltips. But there were also suggestions requiring adaptions of the general requirements and therefore changes to multiple layers including the user interface and the logical handling of user input that affects the query of clinical trials. First, it should also be possible to differentiate between required and optional keywords and an option to specify whether they should be combined with “AND” or “OR”. Second, the location search should be improved to not only enable a search by country but a proximity search using a city as input.
Fig. 2 The prototype of the newly developed tab in the cBioPortal patient view of the ClinicalTrials.gov search.
The aforementioned search parameters are then used without further modification or enrichment to build a request to query the application programming interface (API) of ClinicalTrials.gov.[23 ] Scoring of the search results was achieved by first filtering the results (if specified) by country, recruiting status, and selected alterations. Afterward, the remaining trials were ranked by a combined weighted query based on feedback of the users during the iterative feedback rounds which consisted of (importance in descending order) count of found keywords, matching tumor entity, distance to closest partner site, matching sex, and matching age.
Next, different methods of integrating the extension were considered, including integration in the cBioPortal frontend (running in the user's browser), the cBioPortal backend (running on a dedicated server), and as a standalone service, running on the server in addition to the cBioPortal backend (for more information about the architecture of cBioPortal, refer to reference Unberath et al[24 ]). Since cBioPortal does not provide a plugin concept,[24 ] the extension should be coupled as loosely as possible to the codebase to ensure better compatibility with future updates. This indicates an implementation in the frontend which also promotes the feature of retrieving search results on the fly. Since results are not cached, curated, or preprocessed in advance by the server, the display of results should be focused on ranking trials according to the number of matching criteria, listing results with less matches at the end rather than excluding them completely. This leaves the final assessment of the relevance of all found studies to the user. Another key feature of the extension stems from its usage in a clinical setting with real patients, the proximity search, which considers the patient's place of residence and ranks the studies with matching geographical sites higher accordingly.
Finally, the concluding usability evaluation was performed in October 2020 with five clinicians with different backgrounds (oncology, systems medicine, or bioinformatics), who regularly participate in MTBs. Several additional shortcomings were identified that were not identified during the feedback rounds and needed to be addressed for finalizing the tool. Most importantly, it was not clear to the users that by design the tumor entity was invariably part of the ranking of the tool in the background, as it is always present in cBioPortal samples. All users therefore demanded that ranking parameters should be made selectable and customizable for the query. Other important feedback included improving the explanation of how search results were filtered and ranked and how to enter the optional and required keywords (each mentioned by three participants). The participants also mentioned additional nice to have features, such as the ability to search for the phase of the study, for interventions or alterations based on pathways (each mentioned by one participant). The latter can be considered as a suggestion for improvement for further developments.
The results of the quantifiable questions of the usability evaluation can be found in [Table 1 ]. The evaluation of the SUS questionnaire resulted in an overall SUS score of 83.5 (number of participants was five, standard deviation = 11.58, individual scores were 92.5, 95, 90, 65, and 75, respectively). The SUS score of 83.5 is in the middle of the “acceptable” range and represents a “good” on the adjective rating scale.[25 ]
Table 1
The results of the quantifiable questions of the usability evaluation
Question
Mean
SD
The function of each field of the search mask was comprehensible.
3.80
0.75
The information provided in the search results helped to identify suitable or unsuitable studies.
3.80
0.75
The order in which the studies are displayed was helpful.
3.25
1.01
The application overall facilitates the search for suitable studies.
4.00
0.71
Abbreviation: SD, standard deviation.
Note: Scores range from 1 (strongly disagree) to 5 (strongly agree).
Discussion
Clinical trials are a major driver in the advancement of cancer treatment and while the willingness of patients to participate in trials is estimated as high as 70%,[26 ] less than 5% of cancer patients enroll in clinical trials.[27 ]
[28 ] Unger et al[27 ] identified multiple barriers in different categories from which this work focuses on the structural barrier of identifying available clinical trials. Although being resource intensive,[29 ] the process of searching and identifying appropriate trials is a crucial part of the preparation of patients for the MTB, as these patients usually already have exhausted all guideline-based treatment options or have rare tumors.[30 ]
[31 ]
While there are other works regarding software solutions for MTBs, these developments are still in early stages. Halfmann et al[32 ] developed an MTB support tool also using a user-driven development process. Their tool focuses on the presentation part of an MTB and not on the preparation including the search for clinical trials for MTB patients and furthermore, the study did not include a usability evaluation. Fegeler et al[33 ] are currently developing a solution focusing on the administrative part including communication methods for performing virtual tumor boards. Studies in the field of recruitment support often tackle different aspects than searching fitting trials for a specific patients, for example, optimize eligibility criteria[34 ] or assess trial population representativeness,[35 ] supporting data collection[36 ] or helping to find patients for screening.[37 ]
[38 ]
A key factor in enrollment of MTB patients in clinical trials are matching genomic alterations of the patient's tumor and the study's inclusion criteria, yet these alterations are not specified in a well-structured matter and often only found study's descriptive text.[39 ] Therefore, several attempts have been made to curate precision cancer studies from trial registries like ClinicalTrials.gov or institution-specific registries to generate a structured database to query. The drawback of these approaches generally is the high maintenance of curating trials and keeping the data up to date.[40 ] Additionally, institution-specific systems, like MatchMiner[16 ] or the Phase One Spot Tracker (POST),[41 ] do not publish curated datasets. While those data would not directly benefit the enrollment at other institutions (depending on the proximity), they would constitute a valuable resource for benchmarking or validating the results of new matching tools. However, such a validation was not in the scope of this work and was therefore not performed. The focus was on streamlining the process of the manual search by integrating ClinicalTrials.gov into cBioPortal and prefilling data already available for the patient.
In contrast to aforementioned tools, this work focuses on finding studies by directly using the necessary information for the search from the patient data already available in cBioPortal. It is obvious that such an approach can never achieve sensitivity and specificity levels of tools that require upfront efforts from domain experts. Therefore, this tool focuses on a high usability to enable physicians to query up to date and publicly available trial registries (currently solely ClinicalTrials.gov) on the fly as fast and straightforward as possible. The tool does have technical limitations, some of which originating from the API of ClinicalTrials.gov. For instance, while the advanced search of the ClinicalTrials.gov web interface features a practical distance search, the API lacks this feature. Thus, the extension implements an own distance search using a set of predefined locations which needs to be further expanded. For the search of synonyms rather than exact string matches, the API provides the same functionality as the web interface, utilizing the Medical Subject Headings (MeSH)[42 ] to search for all synonyms of the given term. Currently, the extension uses the HUGO Gene Symbol from cBioPortal for querying the API, but in the context of molecular profiling, it could be beneficial to also include more specialized vocabularies, such as Gene Ontology,[43 ] for a broader synonym coverage, and using gene identifiers as HUGO symbols may change over time.[44 ] Future developments could also consider pathways of the patient's genes affected by mutations.
One approach to improve the fit of our tool's search results would have been to preprocess or automatically curate trials from ClinicalTrials.gov in advance. Previous studies shown that even sophisticated curation pipelines require manual posttreatment.[39 ] This fact also discouraged the implementation of an extract, transform, load (ETL)-pipeline from ClinicalTrials.gov to the MatchMiner trial format Clinical Trial Markup Language (CTML). Preprocessing trials in the form of building a custom registry would have also required a solution that is server-, for example either cBioPortal backend- order standalone-tool-based. As one of our main goals was to contribute our extension to the public cBioPortal open-source project, solutions requiring additional computing resources server-sided or manual data updates were not feasible. With the chosen frontend integration of the search tool, all computing is performed in the browser of the end user.
While developing our extension, we focused on a user-centered design[45 ] with multiple feedback loops to be able to analyze and then support the workflow of the physicians as much as possible. This greatly improved the design and development process as most of the times, translating the requirements of the real-world users to technical features implemented in software is not trivial and should be reviewed and revised when necessary. In summary, the findings of the evaluation including the SUS score of 83.5 showed overall, the tool facilitates the search for matching trials ([Table 1 ]).
The extension described in this work is only one of the needed functionalities identified by Buechner et al[15 ] to use cBioPortal as an MTB platform. The search for clinical trials will be refined using the results of the usability test and gradually other key features will be additionally implemented to supply MTB participants with a comprehensive solution for their routine tasks. The completed prototype for the MTB platform will then be extensively evaluated in a future study.
Limitations
The final usability evaluation was intended to be the last, more detailed round of feedback in the development process, rather than a summative evaluation of a final product. However, the five participants match the often cited threshold to identify the most serious usability issues[46 ] and provided valuable feedback from real-world users regularly participating in MTBs. The evaluation provides an indication of the usability of the application but needs to be confirmed after the tool has been finalized during the aforementioned extensive evaluation of the completed MTB platform. In particular, the large deviation of SUS scores indicates usability issues that may only apply to a subgroup of users and should be further investigated.
Conclusion
Using a user-centered design process, we developed an integration of the ClinicalTrials.gov registry with cBioPortal. The architecture of the integration is as lightweight as possible, only being coupled with the cBioPortal frontend and requiring no additional server-sided resources. The final evaluation showed a good overall usability and that the tool can assist physicians to find appropriate studies for individual patients in the preparation of MTBs with less manual effort.
Clinical Relevance Statement
Clinical Relevance Statement
This work proposes a tool for searching clinical trials based on molecular alterations in tumors of cancer patients. This facilitates the process of preparing patient data and discussing therapy recommendations in the clinical setting during Molecular Tumor Boards.
Multiple Choice Questions
Multiple Choice Questions
1. How was the search tool integrated?
In the backend of cBioPortal
In the frontend of cBioPortal
As a standalone service
As a combination of frontend and backend integration
Correct Answer: The correct answer is option b. The integration of the tool in cBioPortal is frontend-based. As cBioPortal does not provide a plugin concept, implementing the tool in the frontend achieves a loose coupling to the codebase and better compatibility with future updates.
2. What was the main requirement of application?
Transforming trials to a genomics-optimized data format
Facilitate the curation of trial data by experts
Searching trials without the need for manual curation in advance
Enabling patients to search trials in which they could participate
Correct Answer: The correct answer is option c. The main requirement was that the search for clinical trials should not require any manual curation or preprocessing of trial data and should be executed on the fly. The application should be used by experts in or during the preparation of an MTB.
