Related: Models in Science - CCC lecture

Models, as well as the explanations and predictions they produce, are on everyone's minds these days, due to the climate crisis and the Corona pandemic. But how do these models work? How do they relate to experiments and data? Why and how can we trust them and what are their limitations? As part of the omega tau podcast, I have asked these questions of dozens of scientists and engineers. Using examples from medicine, meteorology and climate science, experimental physics and engineering, this talk explains important properties of scientific models, as well as approaches to assess their relevance, correctness and limitations.

For more than twelve years I have been interviewing scientists and engineers for my podcast omega tau. In many of the conversations, the pivotal importance of models for science and engineering becomes clear. Due to the pandemic and the climate crisis, the meaningfulness, correctness and reliability of models and their predictions is ever present in the media. And because most of us don't have a lot of experience with building and using models, all we can do is to "believe". This is unsatisfactory. I think that, in the same way as we must become media literate to cope with the flood of (fake) news, we must also acquire a certain degree of "model literacy": we should at least understand the basics how such models are developed, what they can do, and what their limitations are.

With this talk my goal is to teach a degree of model literacy. I discuss validity ranges, analytical versus numerical models, degrees of precision, parametric abstraction, hierarchical integration of models, prediction versus explanation, validation and testing of models, parameter space exploration and sensitivity analysis, backcasting, black swans as well as agents and emergent behavior. The examples are taker from meteorology and climate science, from epidemiology, particle physics, fusion research and socio-technical systems, but also from engineering sciences, for example the control of airplanes or the or the construction of cranes.

I am far less optimistic about the models than the speaker, but he does a great overview.

I think that most models are oversimplified, the errors in the models are then corrected with hidden parameters.
So when the models do not work, the parameters are adjusted to keep the model working. But now the model makes false predictions by default, only corrected by the parameters.

Sometimes the models create the data themselves hidden in slightly wrong maths or slightly wrong data.
Or they hide problematic data that should be visible, like side-effects of drugs.
Often this is combined with some P-hacking.
For example: I think such a combined process created the "black-hole image", but that is now so locked in group-thinking, that it will be hard to change opinions about it.

I think that we need to create a better standard for the correctness of models. Not based on what we like to be correct, but based on what raw experimental data tells us.
Like: what does it mean when we get unexplained signals, or how much did our instruments and maths influence the parameters in the model.

Some standard questions:
How much noise did we accidentally convert to signal? Can we get a similar signal from a different source? Can someone have created the signal artificially? Is there a selection-process that can cause p-hacking?
Are there unexplained parts in the signal, or things that are hidden or noisy? What possible or weird errors/risks are involved? Can we test those errors/risks?

What are the edge-conditions of the model? How does the model stand up against a null-hypothesis? (the hypothesis that the model/idea is wrong). How does the model stand up against an evolving-hypothesis? (the hypothesis that the model needs to evolve further)

These questions can tell us a bit about how much we can trust a certain experiment and the tested model.