Keywords
data quality - electronic health record - secondary use - real-world data
Background and Significance
Background and Significance
Medical real-world data (RWD) is increasingly used for research studies. Electronic medical records (EMRs) and claims data are major sources, with more focused data repositories such as tumor registries and clinical departmental systems also being mined. It is important that users of RWD are aware of the limitations on a dataset that may not be explicitly apparent. For instance, given the various sources of patient medications—inpatient and outpatient hospital-based dispensing, community pharmacies—the completeness of a dataset's medication data may not be evident.[1]
Other RWD limitations arise from patient privacy regulations.[2] Access to these data is governed by the Health Insurance Portability and Accountability Act (HIPAA), which requires either that research access be approved by an institutional review board (IRB) or that the data be deidentified per the HIPAA privacy rule.[3]
[4]
[5] To be deidentified, all protected health information (PHI) must be removed from a dataset. PHI includes direct identifiers such as name, address, social security number, and medical record number. Also, the dates can only be specified by a year. Compliance with this restriction on dates poses both technical and analytical problems. Database date formats typically require specifying a month and day as well as the year. If this technical problem is addressed by setting all dates to the same arbitrary day and month, such as January 1, then all temporal relationships among observations within a given year are destroyed (as they would be if only the year was specified).
As an alternative to deidentified data, HIPAA also provides for “limited datasets” (LDS) in which direct identifiers (such as name and address) cannot be included, but actual complete dates for patient encounters may be used. IRB approval must be obtained for deployment of LDS, and justification for complete dates be provided. Many institutions employ LDS in their research clinical data repositories. However, even though LDS allow full actual dates, many health care organizations (HCO) still alter these dates as an additional measure to increase the protection of patient privacy, retaining temporal relationships but not actual temporal values.
Shifting of these encounter dates became a common method employed by these institutions. This approach aims to preserve the temporal integrity of a patient's chart by consistently shifting all the dates in the chart by the same randomly chosen number of days.[6]
[7] While the number used to shift data is random among patients, it is constant within a single patient's data. So, for a specific patient, a date-shifted prostate biopsy precedes an initial radiation treatment by the same number of days as in the actual data, whereas the deidentified dates differ from the actual dates. Typically, dates are shifted randomly anywhere from ± 7 to ± 365 days. This algorithm of shifting dates by a constant random value for a given patient, but varying the constant value between patients, up to some maximum number of days, is the common method used across multiple institutions. Only one variation was observed in which a site only selected random multiples of 7 days to shift dates. This preserved the day of the week, which is potentially of value for certain studies. No evidence of other algorithms for shifting were observed in the sites we evaluated.
The primary motivation for date shifting in the United States is HIPAA's base requirement that dates have only the year specified. Adherence to this for encounter dates is clearly unsuitable for many studis. Researchers and IRB panels may view limited datasets, in which encounter dates can have their actual values, as posing too much of a risk to patient privacy. It may be that some researchers and panel members may not even be aware of the possibility of using limited datasets. And in some instances, date shifting may be seen as added assurance of maintaining patient privacy, although the costs to studis where actual dates are necessary are likely not considered.
It should be noted that the ability of date shifting to protect patient privacy has been questioned.[8]
[9] Dates can be inferred, and patients identified by observing clinical content or family relationships. A malevolent attempt to uncover patient identities, or even one motivated by curiosity, is now punishable by law and is grounds for job dismissal at most institutions. Of course, it can be argued that obscuring dates certainly protects against technically unsophisticated individuals attempting to identify patients and is definitely of value when a data breach has occurred.
While the efficacy of date shifting to protect patient privacy is defensible, there are situations in which date shifting significantly detracts from the usefulness of RWD, even making it altogether unusable for particular research purposes. These are instances where having access to actual dates is more important than a temporal relationship among a patient's clinical encounters. An example of a need for actual dates is a study requiring an accurate number of patients initially diagnosed with liver cancer in the calendar year 2017. It cannot be assumed that random date shifting results in an equal number of patients being shifted in and out of the date range being studied. Another example are studies that may need to know the number of coronavirus disease 2019 (COVID-19) patients diagnosed within a specific time period given the peculiarities of timelines associated with viral variant emergence and approvals of treatments and vaccines.[10]
[11] The fact that within a patient's data, temporal relationships have been preserved is irrelevant. These studies will not yield valid results if the dates within the real-world dataset have been shifted.
Objectives
Regardless of whether date shifted RWD may or may not be appropriate for a given study, researchers should be aware of whether the dates of a dataset being used have been altered. Perhaps surprisingly, the authors experience with datasets from multiple institution has been that this information is not easy to obtain. Not only is it not always readily available, but also it may not even be known by the current owners of the dataset. Because it is not uncommon for reported date shifting status to be unreliable or difficult to obtain, in this study we set out to develop analytic metrics to detect the presence of date shifting and estimate its maximum magnitude in a given dataset. We have defined and evaluated the reliability of several such metrics. We propose that this methodology be used by data analysts to ascertain whether a dataset they are working with contains actual or altered dates. Knowing this is of crucial importance when a dataset is being used for such use cases as infectious disease research. This methodology is meant to inform an analyst of the presence as well as an approximate magnitude of a date shift. It is not intended to, nor is it capable of, deriving actual dates in date-shifted data.
Methods
Feasibility of Detection
Given that certain studies require actual dates, it is crucial to know whether a particular dataset contains actual or shifted dates before utilizing it. This might be learned from a dataset's provenance, if available. But even if it is, it should be verified. A reliable method for detecting possible shifted dates, therefore, is required prior to any analysis of data.
Our first approach to creating a date shifting detection method was checking a dataset to see whether a specific occurrence of medical significance occurred on the expected date. A recent event presents a useful example: on March 19, 2020, then President Trump declared hydroxychloroquine—an anti-malarial drug treatment—a “game changer” in the fight against COVID-19.[12] Many datasets in the United States subsequently showed a pronounced spike in the volume of hydroxychloroquine use ([Fig. 1]). This suggested that “sharp” temporal events—hydroxychloroquine spike and others like it—can serve as markers to detect the presence of date shifting. In data where dates have been shifted, a sharp spike may disappear, or appear in the wrong month, indicating that date shifting has occurred.
Fig. 1 Patients treated with hydroxychloroquine show a spike at a large academic institution in 2020.
However, the hydroxychloroquine spike, while a great illustration of a useful temporal marker, has a limited applicability; only datasets with medication information and preferably well-represented outpatient coverage would be expected to demonstrate this feature. Of course, the dataset also must cover early 2020 to observe this event.
Using other possible sentinel dates entail similar potential pitfalls. The date of the chosen event may not be necessarily as unique as thought. For example, if one found the earliest instance of International Classification of Diseases (ICD)-10-CM coding as an indication of the start of the use of ICD-10 as opposed to ICD-9, it ignores institutions that delayed implementation of the new coding after its official start. Also, it would not work at all for datasets in which all prior ICD-9 coding was translated to their ICD-10 equivalent.
In addition to a singular event indicating date shifting, several other temporal features which could give an indication of date shifting were studied. An obvious candidate was the occurrence of seasonal medical events, such as influenza diagnosis or heat stroke. One would expect surges in these diagnoses during appropriate months (fall and winter or summer, respectively, for these examples). This approach of looking for increased numbers of certain medical diagnoses appearing in expected months led to yet another method: the day-of-the-week differences in the occurrence of medical procedures that would not typically occur on weekends, such as routine physicals and elective medical procedures.
Synthetic Models
To detect potential shifts of different magnitudes, we wanted to simulate how various temporal “markers” would appear when dates were shifted by various amounts. We hypothesized that small magnitude date shifts would obscure high-frequency (e.g., weekly) temporal events, whereas large magnitude shifts would be necessary to obscure events that happen only infrequently (e.g., once a year). We studied three date patterns:
-
A pattern that occurs with high frequency is the drop in volume of observations during weekends. For example, in the United States, it is very unlikely that an annual physical would be scheduled on a Sunday.
-
A seasonal pattern where a disease is more prominent in the summer or winter.
-
A one-time drop, as in the case of patients postponing elective care at the beginning of the COVID-19 pandemic.
For each of these scenarios, we modeled the behavior of these patterns when shifting dates by various amplitude (specifically, the maximum amplitude in the common date shifting algorithm as described above) using synthetic data designed to exhibit the given pattern. This allowed us to calibrate and understand, for example, how much shift is needed to obscure a weekday/weekend or a seasonal or yearly pattern. We later used this information to compare to the observed patterns in real-world dataset and judged the possible presence of each degree of shift scenario. In [Fig. 2] we see the pattern observed for an encounter, such as a checkup, which normally occurs only Monday through Friday. When the synthetic data are not shifted (i.e., number of days shifted = 0) we see almost no counts on Sunday and Saturday, with any small counts attributable most likely to data entry error or special screening events. As the number of days shifted assume greater values (± 3, ± 7, ± 30, ± 90, ± 365), we see a smoothening of the encounter counts over the days of the week such that there is little difference between the weekend and weekdays. To an analyst ignoring a potential date shift, this would erroneously imply that the checkups occur as likely on the weekend as during the week.
Fig. 2 Weekly pattern using synthetic data. The number above each graph indicates the number of days shifted.
[Fig. 3] shows the distortion seen when data with a one-time drop caused by a sentinel event in certain encounters are date shifted. In the synthetic dataset an abrupt drop in months 16 through 18 is totally obscured as shifting values are increased.
Fig. 3 One-time drop caused by a sentinel event using synthetic data. The number above each graph indicates the number of days shifted.
In [Fig. 4] we see the effect of date shifting on synthetic data for seasonal encounters such as influenza. Once again, increasing the number of days shifted obscures peaks actually observed during certain seasons.
Fig. 4 Yearly pattern of seasonal events using synthetic data. The number above each graph indicates the number of days shifted.
Our experimentation with synthetic models confirmed that date shifting affects the appearance of expected temporal patterns such as volume fluctuations between weekdays and weekends, significant events, such as the COVID-19 pandemic, and seasonal disease patterns. Furthermore, they demonstrated that the maximum magnitude of date shift affects these patterns in a predictable manner, obscuring them proportionally to the magnitude of the shift.
Real-World Data
The TriNetX Global Network of data from health care organizations were utilized for this study.[13] The patient data available is from EMR systems, tumor registries, and departmental systems (e.g., pathology). We hypothesized that observations of the following medical encounters for the health care organizations in our study would exhibit seasonal patterns when date-shifting was not employed. The ICD-10-CM codes for these encounters are noted in parentheses:
Sunburn, heatstroke, URI, and influenza all should have a strong yearly pattern, meaning a few months of very high volume, a few months of a shoulder season, and otherwise relatively low volume. If this pattern is not observed, it can be assumed the HCO is shifting, perhaps by as many as 365 days. To evaluate whether this pattern was present we considered only sites with at least 500 observations of each of these specific diagnoses and determined the distribution of these encounters by computing monthly sums (ignoring the year) and then the median of the monthly sums. If a site exhibited at least 2 months with the number of encounters for a diagnosis exceeding 1.5 times the median value, then we concluded the data were not shifted. Correspondingly, if the monthly encounter numbers were more uniformly distributed, the site was deemed to have provided date-shifted data.
We further hypothesized that nonacute visits, such as routine physical encounters or encounters for dermatitis, would follow a Monday–Friday occurrence, with infrequent weekend occurrences in unshifted data:
-
Routine physical (Z00)
-
Dermatitis (L20–L30)
Routine checkups and treatment for dermatitis would have likely been postponed during March and April 2020 due to COVID-19. Once again, only sites with at least 500 observations for routine checkups and/or dermatitis treatment were considered. We computed the median of monthly encounter sums and concluded that any site having a decrease in such encounters of at least 80% in March 2020 and/or at least 50% in April 2020 had not date shifted their data. Otherwise, a shift of 30 days or more would be assumed. We selected these thresholds because we hypothesized that these values would capture substantial changes and not minor variation.
In addition, a previous study[10] by our group showed decreases in breast and colorectal cancer screenings of 89% and 84% for April 2020.
Similarly, we expect checkups and dermatitis to occur mainly on weekdays and not weekends. The weekday distribution of routine physicals and dermatitis treatments were computed and then the median of the weekday encounters sums obtained. If the sums for Saturday or Sunday exceeded 0.25 times the median, we concluded a shift of at least 7 days. [Fig. 5] summarizes the decision tree applied to the encounter data from the sites in our study.
Fig. 5 Outline of the procedure used to detect the presence and degree of date shifting in the datasets studied.
Results
We applied our date shifting detection methods—looking at sentinel events (COVID-19 pandemic), seasonal patterns for certain diagnoses, and weekday patterns for elective encounters—to 76 sites in the United States. Twenty-two sites exhibited a conflict, an inconsistency between our preexisting records of the presence of data shifting and comparison of the observed data patterns with our synthetic models. We contacted those 22 institutions in an attempt to reconcile these conflicts. We established a dialog with 17 of 22 sites where we asked the sites to double-check their date shifting status and we reevaluated our interpretation of the model to arrive at a consensus. The remaining five sites did not respond to our inquiries, and we excluded these sites from further analyses.
These conflict resolution attempts allowed us to fine-tune our date shifting detection methodology. Most significantly, we made the decision to remove seasonal URI from consideration as it did not appear as predictive as anticipated. Respiratory diseases do not appear to be as sharply seasonal as we originally hypothesized. When we compared URI with the other seasonal predictors, we found that it was more likely to give false positives, potentially due to differences in onset and duration of URI waves by location within the United States, or due to differences in how the URI diagnoses are captured.
Seasonal predictors also are dependent on geographic location. At a top level, whether the data source was in the northern or southern hemisphere. Our decision to confine this study to U.S. data providers was primarily based on removing geographic considerations from seasonal predictors, although north–south location within the United States is still a factor on the onset and degree of seasonal predictors.
For the 71 HCOs in this study, [Fig. 6] shows the observed presence of date shifting. Thirty-nine organizations, or 55%, displayed no date shifting by our methodology. This conclusion was confirmed by individual data providers. Our methodology concluded that another 28 organizations showed some amount of date shifting, which was also confirmed by the providers. For four organizations our methodology's conclusions differed from the provider's description and we were not able to resolve the conflict to a mutual satisfaction.
Fig. 6 Observed presence of date shifting by study methodology, which was in agreement with data provider's description of their dataset. Datasets where there was disagreement between the study's methodology and the provider on the presence of date shifting are shown as “conflict.”
[Fig. 7] shows the distribution of the number of days for which dates were shifted among the 28 HCOs in the study for which the observed date shifting was confirmed by the data provider. A quarter of the shifting institutions do so by 7 days, and a fifth by 365, with the rest in the middle.
Fig. 7 Distribution of the magnitude of the date shift (in days) for the 28 health care organizations with confirmed observed date shifting.
Discussion
Prediction of Date Shifting Presence
The most reliable encounter observation for predicting date shifting is the routine medical checkup (ICD-10-CM Z00) when its distribution over day-of-week is extracted ([Table 1]). In the United States these encounters should rarely if ever occur over the weekend (Saturday or Sunday). This conclusion is applicable only to countries like the United States where these exams are normally scheduled for Mondays through Fridays exclusively. Other medical encounters, such as for sunburn, are not reliably tracked in datasets. Encounters for upper respiratory infections or influenza, while having seasonal increases, are not sufficiently distinct during certain months.
Table 1
Correlation between observed encounter and date shift detection
|
Measure
|
Yearly/day-of-week
|
Correlation
|
Number of HCOs
|
|
Sunburn
|
Yearly
|
0.71
|
30
|
|
Influenza
|
Yearly
|
0.69
|
70
|
|
URI
|
Yearly
|
0.20
|
70
|
|
Checkup
|
Yearly
|
0.25
|
61
|
|
Dermatitis
|
Yearly
|
0.53
|
70
|
|
Checkup
|
Day-of-week
|
0.92
|
61
|
|
Dermatitis
|
Day-of-week
|
0.79
|
70
|
Abbreviation: HCO, health care organization; URI, upper respiratory infection.
Note: The strength of the correlation demonstrates the measure's ability to correctly predict date shifting.
[Fig. 8] shows the distribution by day of week of routine medical checkup encounters for the HCOs studied. The left plot shows the expected pattern with almost no encounters occurring on Sunday or Saturday. The data provider confirmed that no date shifting was applied to this dataset. The right plot shows an almost uniform distribution of these encounters over every day of the week. The degree of date shifting for this dataset, confirmed by the data provider, was ± 7. Thus, date shifting of any magnitude from a week to a year will obliterate the expected weekday-only occurrence of routine medical checkups.
Fig. 8 Distribution by day of week of routine medical checkup encounters for the health care organizations studied.
Another limitation of using day of week occurrence of routine encounters was observed for the single data provider who chose to shift their data by a multiple of 7 days, thus preserving the day of week of the encounter. This form of date shifting can be exposed by the unexpected volume of routine encounters on holidays such as Christmas day, New Year's Day, Memorial Day, Independence Day, Labor Day, and Thanksgiving.
No Potential for Date Reidentification
Concern that this methodology may be improperly used to restore (reidentify) actual encounter dates is understandable. It should be kept in mind that use of actual encounter dates is permissible if IRB approval has been obtained for use of an LDS. As mentioned in the Background section above, whenever individual patient data are loaded or refreshed in a dataset, a random patient-specific date offset value within the range chosen by the provider is applied to that patient's dates. For example, if the dataset offset range is ± 365, the first patient's data may randomly be assigned an offset value of −34. All dates in that patient's data in the dataset has 34 days subtracted. The second patient in the dataset may be randomly assigned a date offset of +182. All dates in that patient's data in the dataset has 182 days added. Determining that the entire dataset is date shifted by up to 365 days using our methodology, it is not evident how one would know what offset value to apply to the dates in each patient record in the dataset. (In this example, adding 34 days for the first patient while subtracting 182 days for the second patient.) Possibly one could do multiple attempts adjusting each patients' dates by all possible combinations of date offset values up to 365 to find the adjusted dataset with the minimal deviance from nonurgent encounters occurring on weekdays. But there is no guarantee that this minimum deviance dataset is correct or unique. To reiterate, the methodology to detect the presence and maximum amplitude of a date shift we are proposing is neither meant to circumvent the deidentification in any way, nor can the authors see how it could be used for this purpose.
Our study shows that shifting by a varying random number of days for each patient's chart provides a signal that the dates have been shifted. This signal is desirable as it indicates the dataset is not suitable for studies in which actual dates are necessary. However, if the intent is to confuse or hide the date shifting from the research data consumer, then this can be achieved by using nonrandom numbers of days for shifting, particularly multiples of 7. We do not recommend this obscuring of the application of date shifting.
Conclusion
The obscuring effect of date shifting diminishes the usefulness of RWD for studies when actual dates are required. Dataset provenance is not always reliable even when available. Knowledge of whether date shifting has occurred is necessary when using real-world datasets. The objective of this study was to develop methods for detecting date shifting of encounter data in datasets but not for date reidentification.
We found a simple test to detect the existence of date shifting in an EMR dataset from U.S. HCOs, observing whether almost all routine medical exams or similar nonurgent encounters were performed during weekdays is predictive of a date shift of at least 7 days. Other measures such as the patterns of seasonal illnesses or sentinel temporal events allowed us to detect the magnitude of the shift.
For non-U.S. data sources, or datasets not containing routine physical encounters, a comparable test could be developed using other nonurgent encounters having a customary distribution over certain days of the week.
Slightly over one-half (39 of 71, 55%) of institutions in our U.S. sample do not shift dates. Of those that shift, a quarter do so by ± 7 days and about one-fifth by ± 365 days, with the rest shifting by various amounts in between. The heterogeneity of the shift magnitude is telling; there does not seem to be a widely accepted agreement on the application of date shifting among HCOs.
Clinical Relevance Statement
Clinical Relevance Statement
This study provides methodology for users of RWD to ascertain whether the dates in the data being used has been shifted in value and to what degree if it had. This contributes to providing valid results of studies utilizing RWD.
Multiple-Choice Questions
Multiple-Choice Questions
-
What is the most reliable encounter type for detecting date shifting?
-
Emergency surgeries
-
Routine physicals
-
CT scans
-
Phlebotomy
Correct Answer: The correct answer is option b. These encounters are scheduled and do not occur unexpectedly. They are usually scheduled only on certain days of the week. (For example, routine physicals usually occur only on weekdays Monday through Friday in the United States.) Hence, they should display an appropriate day-of-week pattern.
-
Can real-world datasets ever contain actual complete dates and be HIPAA Safe Harbor-compliant?
-
No, never
-
Yes, if dates are at least 5 years ago
-
Yes, if it is an LDS
-
Yes, if leap years are not included
Correct Answer: The correct answer is option c. Under HIPAA, an LDS is health information that excludes certain direct identifiers (such as patient's name) but may include certain dates with all elements. Use of an LDS requires IRB approval and is not considered deidentified data under the privacy rule.
