Part 6. Are M&S outputs real?
TL;DR
Once validated, a model predicts a therapy's efficacy. The prediction rests on knowledge that science takes to be real, applied to virtual data profiling the virtual patients. The way virtual patients are designed and validated makes them stand for the real patients who could benefit from the therapy.
Efficacy predicted with M&S is no less real than efficacy estimated with an RCT. The main differences:
sample size is unlimited in silico
running an RCT is a complicated process with many challenges
the main limits of an RCT are external generalizability and the difficulty of detecting adverse events, which then surface in phase IV and increasingly lead to market withdrawal
The first thing people ask when they meet the M&S approach is whether the simulation results are real.
Reality and real things have been debated for as long as humanity and philosophy, in an attempt to reconcile ordinary perception with what philosophers and scientists think. Rather than enter that debate, we compare two ways of approaching the same reality in our sphere of interest: the efficacy of a new therapy.
An example from history introduces the topic. Almost 23 centuries ago, Eratosthenes combined an indisputable sense of observation with a model of reality built from the astronomy and geometry of his time, and calculated a value for the Earth's circumference that we can consider roughly real (see Box 9).
Box 9: an old demarche: reasoning, use of knowledge, mathematical modeling, and observation to approach reality
Eratosthenes deduced the circumference of the Earth in a purely geometric way around 230 BC. He considered the sun's light rays to be parallel at any point on the earth, because the sun was very large compared to the earth and very far away. He compared the observation he made on the shadow of two objects located in two places, Syene and Alexandria, considered to be on the same meridian, which was approximately correct, on June 21 (summer solstice) at noon. It is at this precise time of the year that in the northern hemisphere the sun occupies the highest position above the horizon. Now, in a previous observation, Eratosthenes had noticed that there was no shadow, at that time, in a well in Syene (a town located roughly on the Tropic of Cancer). So at this time and at this precise place, the sun was vertical and its light directly illuminated the bottom of the well. Eratosthenes noticed, however, that on the same day at the same time a year later, a gnomon located in Alexandria had a shadow. The sun was therefore not there vertically. The shadow and the gnomon formed the sides of a right triangle whose solar radius was the hypotenuse. Eratosthenes therefore deduced by geometry that the angle between the solar ray and the vertical there was 7.2 degrees (our degrees). He estimated then the distance between Syene and Alexandria to be about 5,000 stadia. It is said that he used for this evaluation the average duration of the journey by the caravans of camels between the two towns and the observation that the speed of a camel is constant. Eratosthenes proposed a simple model of what he had observed: a circle (section of the earth along the meridian joining Alexandria to Syene, the earth was said to be spherical since at least Aristotle's observations on eclipses and Strabon’s one on the horizon which prevents navigators from seeing the distant lights), the angle of the two rays joining the center of the circle to, on the one hand, the well of Syene and on the other hand the gnomon of Alexandria was the same 7.2 degrees because of the geometric theory of congruent alternate-interior angles. To this angle therefore corresponded 1/50 of the circumference of the earth, the arc connecting Syene in Alexandria, an arc measuring 5,000 stadia. The circumference of the Earth was estimated at 250,000 stadia, or about 39,600 km, very close to the value accepted today.
The story shows that a reality can be the result of a calculation, and that by calculating we build a new layer of reality.
A model of disease and therapy is more complicated than Eratosthenes' model, but in essence it applies the same reasoning. Once validated, it predicts the therapy's efficacy. The prediction rests on knowledge that science takes to be real, applied to virtual data profiling the virtual patients. The way virtual patients are designed and validated makes them stand for the real patients who could benefit from the therapy if it proves effective.
Effectiveness measured by applying the model to the virtual population may differ from effectiveness measured by the gold standard, the randomized controlled clinical trial (RCT), applied to a real population.
RCT methodology is rigorous, and it is designed to avoid bias. Even strictly applied, though, its results carry uncertainty, from false positives, rare if the statistical analysis is done correctly, and from false negatives, when the magnitude of efficacy was overestimated at the start. Sample size and imbalance between arms contribute too, and are more or less well controlled: randomization makes the arms comparable on average, never identical. Add the questionable representativity of trial participants, and this uncertainty weighs heavily on how far the efficacy measured in the trial generalizes to patients who were not in it, including prospective patients and patients from another race or cultural environment. Heraclitus captures the point: "You could not step twice into the same river."
Efficiency measures for an RCT are calculations on data collected electronically after a human investigator recorded it. The information those data carry has passed through several steps from what it is supposed to represent. How does it differ, in uncertainty and in distance from reality, from the data describing the virtual population the model is applied to?
The efficiency measures the model produces are not affected by the same sources of uncertainty. Since the model and the virtual population are validated, uncertainty in the efficacy measures comes primarily from the validation data and the accepted deviation. Sources of measurement uncertainty specific to a clinical trial on real patients, such as sample size or imbalance between arms, do not apply in silico. Adapting the measurement to other situations follows the same process, the same model and virtual population, so the uncertainty stays stable.
After this excursion into what is real, the conclusion is that neither measure is much more real than the other. Why should they not both be equally real? Joseph Fourier (French mathematician, 1768 - 1830) offers a line that supports both the randomized clinical trial and M&S: "... mathematical analysis can still grasp the laws of these phenomena / It makes them present and measurable to us and seems to be a faculty of human reason intended to make up for the brevity of life and the imperfection of the senses" [1].
Table 5 summarizes the comparison between RCT and M&S as tools for measuring and predicting the clinical efficacy of a new therapy.
Table 5: Efficacy prediction with M&S is not less “real” than efficacy estimation with a RCT.
Uncertainty aside, real RCT and in silico CT outputs differ in other ways:
- Unlimited sample size is both a strength and a weakness of the in silico CT. It can detect significant differences that are not clinically meaningful.
- Running a real RCT brings its own challenges: unblinding, cross-over, poor compliance, drop-outs, and missing data can push efficacy estimates up or down, not to mention deliberate circumvention of the rigors of an RCT.
- The main limits of an RCT are external generalizability and the difficulty of detecting adverse events, which then surface in phase IV and increasingly lead to market withdrawal. The in silico CT handles the generalizability issue, and in theory could handle the toxicity issue too, though experience there is still thin.
1. Fourier J. Théorie analytique de la chaleur. Paris: Firmin Didot Père et Fils. 1822 ↩︎