Keywords
training - educational program - simulation - guidelines and protocols - emergency
Background and Significance
Background and Significance
Trauma is an unpredictable medical emergency that requires a rapid response. Reducing the time to treatment to reduce the number of preventable deaths that occur during the first peak of trauma deaths remains a challenge.[1]
[2] To provide this treatment, trauma specialists must respond immediately without hesitation. Therefore, proper training is required.[3]
[4]
[5]
[6]
As noted by Wurmb et al,[7] severely injured patients are those who need treatment that best avoids preventable errors because they have a direct impact on mortality and morbidity. Therefore, a standardized educational program is required for these patients. Advanced Trauma Training and Life Support (ATLS), developed by the American College of Surgeons Commission on Trauma, is recognized and practiced worldwide.[8] Additionally, there is evidence that this training provides techniques and skills for the rapid and efficient treatment of trauma patients, but the principles need to be translated into local contexts.[9]
[10]
[11]
Given the potential for unpredictable situations, there is a debate about the adequacy of trauma management guidelines and protocols.[12]
[13]
[14]
[15] Nevertheless, there are studies that show clear improvements in trauma management after following trauma guidelines, clinical pathways, or protocols.[16]
[17]
[18]
[19]
[20]
[21]
[22]
[23] These are three levels of standardized procedures, from more general guidelines first, to more detailed clinical pathways, then to protocols.[12]
[24]
[25]
[26]
[27]
[28] However, what has not been discussed is the ability to study deviations from defined protocols to improve training, build flexibility in protocols, and constantly evaluate whether changes in analyzed protocols have been considered.[29]
[30]
[31]
[32]
[33]
[34]
[35]
[36] How to develop and implement these standardized procedures is an important aspect that should be addressed[37]
[38]
[39]
[40]
[41]
[42]
[43]
[44]; nevertheless, only a few studies have focused on their evaluation.[27]
[45]
[46]
[47]
[48]
[49]
Objectives
Since trauma management includes simulation as an important part of training, the inclusion of an objective assessment system in simulation is a clear need.[50] The use of simulators to learn and more specifically practice standard trauma procedures is already a reality. New technologies make it possible by incorporating new modalities, as the current simulation environment is quite diverse. Therefore, different simulation modalities would allow different medical aspects to be learned and practiced.[51]
[52]
[53]
[54]
[55]
[56] A web-based trauma simulator shows flexibility and a good possibility of implementing standard procedures that can be practiced by a large number of participants simultaneously, and which would allow the incorporation of an objective assessment system, which is a clear need in simulation.[57]
[58]
[59] To do this, it is important to consider all possible treatments for one patient and keep in mind that some actions may be equivalent, similar, or completely different.
Current assessment methods available within simulation consider written assessment checklists in which simulation information is not automatically linked. Thus, the objective of this article is to develop and implement an objective evaluation system within a web-based trauma simulator to obtain objective information about how the simulation is performed in real time.
Methods
Several aspects should be considered when evaluating how a sequence of actions is performed in a simulation environment. On one hand, the number of correct and incorrect actions is important, but on the other hand, the order of performing different actions is important. Therefore, a distance metric should be defined to measure the distance between the sequence of actions performed by the trainee and the sequence of actions that should be performed according to the trauma protocol. There are several distance measures: edit distance, token-based distance, and sequence-based distance.[60] Edit compares the distance between two strings by counting the minimum number of operations required to convert one string to another. Token-based distance compares two strings by checking string units (tokens). The sequence-based distance of two strings compared them by examining different sequences of the strings. From these criteria, it was decided to focus on the Needleman–Wunsch algorithm.[60] This is because it is an edit distance algorithm that allows for different considerations: if there are matches, mismatches, or gaps between two sequences. This gives us the flexibility to tailor the algorithm to compare two trauma management sequences. In addition, two other metrics were established. One is the diagonal score (DS), which provides information about the correct sequence of actions, and the other is the subsequence score (SS), which provides information about the number of subsequent sequences of actions performed during the simulation.
Modified Needleman–Wunsch algorithm
Initially, the Needleman–Wunsch algorithm appeared as a dynamic programming algorithm that would provide a global solution to the problem of comparing two amino acid sequences.[61] This global alignment (GA) would allow the distance between two sequences to be measured. This algorithm consists of three steps: the first is to initialize the score matrix, the second is to calculate the scores to construct the traceback matrix, and finally, the third step is to derive the best alignment from the traceback matrix.
Following this algorithm, a modification has been included considering that some actions may have the same impact on the patient's vital signs, some others may have a similar impact, some others may be completely opposite, and some others may just be different. All these options are included in the algorithm to create a traceback matrix that would then allow finding the best alignment. The way the score is calculated to obtain the traceback matrix, D (i, j), is as follows: (Eq. 1)
being s (xi, yi) different scores with respect to four different possibilities: actions that match, S
match
, actions that are equivalent and therefore could be swapped, S
swap
, actions that are opposed, S
contrary, or actions that are just different and, therefore, they are considered a mismatch, S
mismatch. The score provided to a gap is S
g.
Assuming that the sequence of actions to be performed is: “5 7 8 6 3 2 4 1 9 6” and that a trainee performed the following sequence: “7 4 1 3 2 8 6 9” as shown in [Fig. 1]; the traceback matrix is built ([Fig. 2]) considering the values of the scores mentioned above and the best alignment obtained is shown in [Fig. 3].
Fig. 1 Example of the sequence to accomplish: “5 7 8 6 3 2 4 1 9 6” and the sequence of actions performed by the trainee “7 4 1 3 2 8 6 9.”
Fig. 2 Example of the traceback matrix calculation and the best alignment obtained for two sequences: “5 7 8 6 3 2 4 1 9 6,” which is the sequence that should be done and “7 4 1 3 2 8 6 9,” which is the sequence of actions performed by the trainee.
Fig. 3 Best sequence alignment obtained after applying the modified Needleman–Wunsch algorithm for two sequences: “5 7 8 6 3 2 4 1 9 6,” the ideal one and “7 4 1 3 2 8 6 9,” the one performed by the trainee.
Therefore, the best sequence alignment is [Fig. 3]. The blue color in [Fig. 3] means that a gap has been introduced in the sequence, which means that nothing has been done or should not be done to better fulfill the sequence of actions that the student should take. The green color in [Fig. 3] means that the actions between the two sequences match, and the dark red color means that there is a mismatch. In the best alignment path shown in [Fig. 2], the vertical arrows indicate gaps entered in the upper level of the matrix, diagonal arrows indicate that there is a match or mismatch, and horizontal arrows indicate gaps introduced in the sequence on the left side of the matrix. Then in this example, the best sequence alignment has four matches, one mismatch, and eight gaps. The maximum score is given to the match and then the swap. Later, if there is a gap, the score provided is higher than the contrary score and the mismatch score as when dealing with trauma patients, doing something that should not be done is worse than doing nothing. Ultimately, the worst-case scenario is taking an action that has the opposite impact to what the patient should have. A total of 15 trauma experts were asked to define the four categories of actions above. For example, actions that are considered equivalent are “oxygenate the patient with an oxygen mask,” “oxygenate with a self-inflating bag,” and “use an oropharyngeal airway.”
Therefore, all scores set to fulfill the traceback matrix shown in [Fig. 2] follow the criteria: S
match
> S
swap
> S
gap
> S
contrary
> S
mismatch.
Then, once the traceback matrix is created and the best sequence for comparison is obtained, a score is provided for that comparison. This score is called the GA score and is calculated as follows: (Eq. 2)
Eq. 2:
GA = n
match
S
match
+ n
swap
S
swap
+ n
gap
S
gap
+ n
mismatch
S
mismatch
+ n
contrary
S
contrary (2)
where n
match, n
swap, n
gap, n
mismatch, and n
contrary are the number of matches, swaps, gaps, mismatches, and contrary actions that exist within the best alignment. Given the maximum and the minimum possible punctuation values, this score is normalized to GA ϵ [−1, 1]. A negative value of this score means that the two sequences are different, and a positive value means that they are similar. The higher the value, the more different and the more alike are the two sequences. Therefore, this score provides information on how the trainee performed the simulation, considering all the options above.
Diagonal Score
This score is created to provide information about the correct actions taken during the simulation. If an action is executed at the correct time, it will receive more points than if it is executed at any other time during the simulation. Therefore, a score matrix, S (i, j), is built representing the actions performed by the trainee as rows and the actions to be performed as columns. Then, if the two actions match, a score of 1 is entered into the matrix, otherwise zero is entered as shown in Eq. 3.
(Eq. 3)
seq1 is the sequence of actions performed by the trainee and seq2 is the sequence of actions that should have been performed. [Fig. 4] shows an example of how to construct a score matrix. The sequence “7 4 1 3 2 8 6 9” is performed by the trainee and the sequence “5 7 8 6 3 2 4 1 9 6” is the one that should have been performed ([Fig. 4]).
Fig. 4 Example of a score matrix for two sequences: “5 7 8 6 3 2 4 1 9 6” and “7 4 1 3 2 8 6 9,” in which S (i, j) is 1 when the two sequences are identical and 0 when they are not.
To calculate the DS, the values in the score matrix called individual scores are summed along the diagonal of the matrix, diagi
, being i the number of the diagonals. Following this same example, the values of the diagonals are shown in [Fig. 5] with the subsequent values: diag−7
= 0, diag−6
= 0, diag−5
= 0, diag−4
= 0, diag−3
= 2, diag−2
= 0, diag−1
= 0, diag0
= 0, diag1
= 3, diag2
= 0, diag3
= 1, diag4
= 0, diag5
= 2, diag6
= 0, diag7
= 0, diag8
= 0 diag9
= 0 ([Fig. 5]).
Fig. 5 Diagonals used to sum the individual values of the score matrix, S (i, j), for two sequences: “5 7 8 6 3 2 4 1 9 6” and “7 4 1 3 2 8 6 9.”
Then, they are all squared and summed up together according to Eq. 4 to obtain the DS:
(Eq. 4)
DS = ∑
n
i
=0 diag
i
2
i
2 (4)
being i the number of diagonals from 0 to n. Considering that the maximum DS is obtained when the sequences are identical, this score is normalized. The values of this score: DS ϵ [0, 1]. As the values provided are always positive, the maximum value obtained when two sequences are identical is 1 and the value 0 means that they are completely different.
Subsequences Score
This score focuses on identifying the correct subsequences performed by the trainee and the length of each subsequence. As mentioned for DS, a matrix S (i, j) is built with both sequences containing a 1 when two actions match and a 0 when they do not match. This SS identifies the number of actions that are executed in order and join them together until an action that should not be executed is found. This is done by comparing the value of S (i, j) and the value of S (i + 1, j + 1) according to Eq. 5.
(Eq. 5)
The actions that fulfill the first condition are included into a subsequence vector until the value of S (i + 1, j + 1) = 0, which means that the values of two sequences are not the same. When this is the case, the subsequence ends, and the algorithm tries to find other subsequences that might appear. By applying this algorithm in the example presented, the subsequences identified are: [4, 1], [3, 2], and [8, 6]. Therefore, as shown in [Fig. 6], there are three subsequences with a length of two ([Fig. 6]).
Fig. 6 The sequence of actions performed by the trainee: “7 4 1 3 2 8 6 9” finds a first subsequence comparing it with the actions that should have been performed: “5 7 8 6 3 2 4 1 8 6” when he or she performs the actions [4,1], then a second one [3,2] and finally, the actions [8,6].
Then, SS is calculated as follows: (Eq. 6)
being i the number of subsequences from 0 to n. The value of this score is also normalized to: SS ϵ [0, 1]. As for the DS, considering the values provided to the score matrix and that the length of the sequences is always positive, a 1 means that there is a unique subsequence which matches entirely with the sequence of actions that should have been done, and 0 means that there is not a single subsequence in the actions performed by the trainee.
Pilot Study
Once these scores are set, a pilot study is accomplished using the web-based simulator developed in Larraga-García et al.[62] This pilot study was accomplished with 24 participants: 14 final-year medical students from Universidad Autónoma de Madrid and 10 first-year residents from Hospital Universitario La Paz. On the first day, they were asked to manage four different trauma scenarios. Then, for the next 2 weeks, the participants trained with the simulator at least one simulation per day. After each simulation, the trainee could download a pdf document with all the steps taken as well as their impact along the simulation. Finally, the first four different trauma scenarios were repeated. Within this pilot study, 91 simulations were performed before training with the web-based simulator and 66 simulations were repeated after the training. This difference is due to the fact that not all the trainees performed all the scenarios after the training period. Then, the scores previously defined are obtained for the simulations pre- and post-training. To compare the evolution, a Wilcoxon's rank-sum test was used, and statistical significance is obtained when the p-value is lower than 0.05.[63]
Results
Once the pilot study was accomplished, all the data were gathered and analyzed. The four trauma scenarios trained were: a prehospital pelvic trauma, a hospital pelvic trauma, a prehospital lower limb trauma, and a hospital lower limb trauma. All the results are shown comparing the pre-training simulation, which is shown in blue, and the post-training simulation, which is in orange. From the 91 simulations performed pre-training, 26 are prehospital pelvic, 22 are hospital pelvic, 23 are prehospital lower limb, and 20 are hospital lower limb scenarios. From the 66 simulations performed post-training, 17 are prehospital pelvic, 17 are hospital pelvic, 16 are prehospital lower limb, and 16 are hospital lower limb scenarios.
Modified Needleman–Wunsch Algorithm
The evolution of the GA score obtained with the modified Needleman–Wunsch algorithm, pre- and post-training, is shown in [Fig. 7].
Fig. 7 Global alignment scores pre- and post-training for different trauma scenarios: (A) prehospital pelvic trauma scenario, (B) hospital pelvic trauma scenario, (C) prehospital lower limb trauma scenario, and (D) hospital lower limb trauma scenario.
This score remains practically the same in [Fig. 7(a], c), having the same median value in the first case and a slightly lower value in the second one. In [Fig. 7a], the median values are zero, whereas in [Fig. 7c], the median values are negative. Nevertheless, in [Fig. 7b], a drop is shown in the GA score being in both, pre- and post-training, a negative value. Finally, in [Fig. 7d], a clear improvement is shown pre- and post-training showing statistical significance with a p-value of 0.025. All the relevant values are shown in [Table 1], in which the median values, the interquartile ranges, the statistics, and the p-values are presented.
Table 1
Main results of the global alignment scores for all the trauma scenarios
|
Median
|
Q1
|
Q3
|
Stats
|
p-Value
|
Pre/post
|
|
Prehospital pelvic
|
0.0
|
−0.06578
|
0.10045
|
0.36343
|
0.14660
|
Pretraining
|
|
0.0
|
−0.13544
|
0.04574
|
|
|
Posttraining
|
|
Hospital pelvic
|
−0.02542
|
−0.15477
|
0.10541
|
1.61158
|
0.20426
|
Pretraining
|
|
−0.18758
|
−0.20147
|
−0.01478
|
|
|
Posttraining
|
|
Prehospital lower limb
|
−0.10445
|
−0.14978
|
0.01445
|
1.88409
|
0.69861
|
Pretraining
|
|
−0.11254
|
−0.15007
|
−0.03014
|
|
|
Posttraining
|
|
Hospital lower limb
|
−0.17585
|
−0.20144
|
−0.02504
|
2.88435
|
0.02500
|
Pretraining
|
|
−0.09887
|
−0.11578
|
0.02355
|
|
|
Posttraining
|
Diagonal Score
The DS improves in all scenarios except in the prehospital lower limb scenario in which it slightly drops. In [Fig. 8(a], b, d), the DS improves showing statistical significance only in the hospital lower limb scenario ([Fig. 8d]), with a p-value of 0.03. In the case in which the DS decreases the median value, it drops only from 0.0468 to 0.0443. All the relevant values are shown in [Table 2], in which the median values, the interquartile ranges, the statistics, and the p-values are presented.
Table 2
Main results of the diagonal scores for all the trauma scenarios
|
Median
|
Q1
|
Q3
|
Stats
|
p-Value
|
Pre/post
|
|
Prehospital pelvic
|
0.04687
|
0.03472
|
0.05769
|
0.36343
|
0.54660
|
Pretraining
|
|
0.05228
|
0.04273
|
0.06923
|
|
|
Posttraining
|
|
Hospital pelvic
|
0.04769
|
0.04127
|
0.06201
|
0.01747
|
0.89483
|
Pretraining
|
|
0.05128
|
0.03921
|
0.06070
|
|
|
Posttraining
|
|
Prehospital lower limb
|
0.04687
|
0.03125
|
0.05555
|
0.02078
|
0.88537
|
Pretraining
|
|
0.04437
|
0.03375
|
0.05236
|
|
|
Posttraining
|
|
Hospital lower limb
|
0.03703
|
0.03047
|
0.04887
|
4.69810
|
0.03019
|
Pretraining
|
|
0.05428
|
0.03928
|
0.07164
|
|
|
Posttraining
|
Fig. 8 Diagonal scores pre- and post-training for different trauma scenarios: (A) prehospital pelvic trauma scenario, (B) hospital pelvic trauma scenario, (C) prehospital lower limb trauma scenario, and (D) hospital lower limb trauma scenario.
Subsequences Score
For the subsequences score, the median value improves in all cases in the post-training simulation. However, [Fig. 9a] shows, in some simulations, that the subsequences score values are better in the pre-training simulation than in the post-training one. Nevertheless, the median value increases from 0 to 0.125 in the post-training simulation. . All the relevant values are shown in [Table 3], in which the median values, the interquartile ranges, the statistics, and the p-values are presented.
Table 3
Main results of the subsequences scores for all the trauma scenarios
|
Median
|
Q1
|
Q3
|
Stats
|
p-Value
|
Pre/post
|
|
Prehospital pelvic
|
0.0
|
0.0
|
0.25
|
0.36343
|
0.54660
|
Pretraining
|
|
0.125
|
0.0
|
0.13333
|
|
|
Posttraining
|
|
Hospital pelvic
|
0.0
|
0.0
|
0.09090
|
1.99754
|
0.15755
|
Pretraining
|
|
0.125
|
0.0
|
0.16666
|
|
|
Posttraining
|
|
Prehospital lower limb
|
0.0
|
0.0
|
0.0
|
1.17371
|
0.27863
|
Pretraining
|
|
0.0
|
0.0
|
0.1357
|
|
|
Posttraining
|
|
Hospital lower limb
|
0.039
|
0.035
|
0.0498
|
0.0
|
1
|
Pretraining
|
|
0.04
|
0.0
|
0.1357
|
|
|
Posttraining
|
Fig. 9 Subsequences scores pre- and posttraining for different trauma scenarios: (A) prehospital pelvic trauma scenario, (B) hospital pelvic trauma scenario, (C) prehospital lower limb trauma scenario, and (D) hospital lower limb trauma scenario.
Discussion
The obtained results show that the simulations improve after training with the web-based simulator. Nevertheless, improvements are clearly seen in the diagonal and SSs, whereas in the GA scores these improvements are not so clear. This may be because the diagonal and SSs consider only a unique aspect. The DS analyzes the correct sequence of actions, and the SS examines the number of correct sequences and their length. However, the GA score takes into account several aspects. This score provides additional information that can be provided as real-time feedback to trainees such as when and in what order to perform the actions. The web-based simulator used for this pilot study did not provide this real-time feedback to trainees during the simulation, which may explain the poor improvement achieved for this score.
Hospital Pelvic Trauma Scenario
Interestingly, for hospital pelvic trauma, the DS increases, meaning more actions are performed correctly at the right time, but the GA score decreases. This is because even if the right actions and the right moments are improved, there is still a significant number of mismatches, gaps, or contrary actions. It is important to emphasize that mismatches, gaps, or contrary actions have negative score values in the GA score. Thus, obtaining a negative GA score provides information about the balance between positive actions, similar and equivalent actions taken, and negative actions, opposite actions, mismatches, and gaps. Indeed, the Needleman–Wunsch literature provides different score values for match, mismatch, and gap penalties, but there are no guidelines that provide recommendations on which criteria should be used. In general, a negative value for the GA score means that the two sequences are different, and a positive value means that they are similar. In addition, the inclusion of different scores for matches, swaps, gaps, contrary, and mismatches in the GA score provides more comprehensive information for trauma management. Furthermore, it is important to include this information in web-based simulators. This provides real-time information about student performance, highlights errors, and suggests what to do next. Integrating this information into the simulator supports the learning rate of the trauma protocol. The DS and SS provide information not included in the GA score, providing a more comprehensive assessment of the simulation performance according to predefined clinical trauma guidelines.
Prehospital Pelvic Trauma Scenario
Considering the prehospital pelvic trauma scenario, there are improvements on the diagonal and the SSs, indicating better trainee performance in the post-training simulations. However, no statistical significance is obtained. Furthermore, the GA score has not improved. This improvement is less clear compared with the hospital pelvic trauma scenario, as there is a small improvement in the DS and the GA score worsens. Nevertheless, there is an improvement on the SS, but it does not show statistical significance. Then, an analysis of these simulations was performed with trauma specialists considering that there was an improvement on how well the trainees performed after the training. Nevertheless, incorporating all the information provided by these new metrics would support the trauma training process. This allows us to identify areas that need further reinforcement to enhance learning of trauma management protocols.
Prehospital Lower Limb Trauma Scenario
The prehospital lower limb trauma scenario decreases on the GA and DSs, but slightly improves SSs. This contrasts with the apparent improvement in hospital lower limb trauma scenario.
Hospital Lower Limb Trauma Scenario
In this scenario, the improvement is statistically significant for global and diagonal alignment scores, with p-values of 0.014 and 0.03, respectively. This is the only trauma scenario where a statistical difference is observed. This means that hospital management for lower limb injuries may be better learned whereas the prehospital setting must be improved.
Conclusion
The goal of this article was to develop and implement an objective evaluation system obtaining real-time objective information. To do so, new metrics have been developed. This supports the need to include, in clinical simulation, an objective evaluation system and, additionally, to provide valuable and objective information about how the simulation has been performed.
These metrics have been successfully built and implemented in a web-based simulator to evaluate trauma protocols to treat two specific trauma injuries: pelvic and lower limb lesions and they could be easily adapted for other types of traumas.
Additionally, these metrics could be used to provide real-time information to trainees during the learning process, providing information with respect to the actions that are equivalents or the ones that are not done in order, amongst other. This would make the simulation a self-explanatory learning tool that could be adapted to different levels of expertise. Moreover, these metrics are a powerful tool for trainers as they could objectively evaluate a simulation with all the data coming directly from the simulator.
Nonetheless, this pilot study should be extended to a larger community and to other trauma injuries to continue analyzing the evolution of the trauma management learning process. Additionally, with a larger sample of studies, a deeper statistical analysis should be performed.
Clinical Relevance Statement
Clinical Relevance Statement
Trauma management training is of key importance considering that trauma is one of the leading death causes worldwide. Trauma deaths have followed a classical trimodal distribution and even though the epidemiology of these deaths has changed since the year 2000 toward a bimodal distribution, the number of immediate deaths is still quite high. Therefore, trauma training remains to be a necessary task. There are several trauma trainings that have been created since the ATLS started in 1978 which use different simulation modalities to achieve the goal to train on trauma management skills. Considering that not so many studies in the literature were found incorporating an objective evaluation system, this research presents metrics to evaluate trauma management learning. This allows us to, objectively and automatically, evaluate knowledge acquisition. Therefore, incorporating this information in real time would help encourage trainees and help them acquire the skills more efficiently.
Multiple Choice Questions
Multiple Choice Questions
-
Which is the third step of the Needleman–Wunsch algorithm?
-
The initialization of the score matrix
-
The calculation of the individual scores
-
The traceback matrix
-
To find the best alignment
Correct Answer: The correct answer is option d. The calculation of the individual scores can be changed by together with the traceback matrix is the second step as to di one, implies to accomplish the other one. Therefore, the third step is to find the best alignment.
-
Which type of action evaluates if the trainee did not perform an action that should have been performed?
-
A mismatch action
-
A contrary action
-
A gap action
-
A swap action
Correct Answer: The correct answer is option c. A gap means that either nothing has been done in the sequence of actions performed by the trainee when he or she should be doing something according to the trauma management protocol to evaluate.
-
Which is the score that provides information about how well the actions are accomplished with respect to the correct timing in which they should be done?
-
Subsequence score
-
Diagonal score
-
Match score
-
Swap score
Correct Answer: The correct answer is option b. The diagonal score is built to provide information with respect to the correct actions accomplished along the simulation considering that, if they are done at the correct timing, the score will be higher than if they are done in a different moment.
