Reality-based assessment

Overview

This unit discusses the idea of assessing a model based on expert knowledge.

Learning Objectives

• Understand the importance of reality checks for model assessment.

Introduction

Subject-matter expertise is critical at every step of the modeling and analysis process. Your models need to make a-priori sense. For biomedical applications, the model needs to agree with known biology. More broadly speaking, the model needs to make sense, given what we know about reality.

There are several aspects that you should keep in mind when assessing how overall reasonable a model is.

Model choice

There are lots of different types of models available. You will have to make specific choices for your model. That is ok and part of the process of building models. Every model contains some abstraction/simplification of the real world. You should choose the type of model that is best suited for your situation and the question you want to address.

In this paper, we used maps as an analogy for computational/statistical/mathematical models¹. Maps are models of the real world. They serve specific purposes, and it is important that a given map be useful for the intended purpose. Consider the three maps (models) of the fictional country of antibodia (see the figure).²

Three different maps of the same fictional country. They all have their distinct strengths and weaknesses.

If you want to know where this country is located, the left map is useful. If instead you want to know how to drive from T-town to Dendriticella, the middle map would be the most useful. if you want to know where most people live in this country, the right map is most useful. it is the same system under consideration (the country of antibodia), but depending on the question, different maps (models) are needed. Analogously, for the same system under study, different types of models might be needed, depending on the question you want to answer. The usefulness of maps (and models) is that they capture the information that is needed for a specific situation, while ignoring details that are not important for a given question, thus producing the right level of complexity.

Fortunately, often more than one type of model can be suitable for a given system. Just ensure you always remember that results are contingent on the choices you made, and it is not clear how well results might generalize.³ One way to figure out what impact the modeling choices have on outcomes is to explore a lot of different model variants. This quickly gets time-consuming and is rarely done. Occasionally, someone compares models of the same system done by different individuals/groups.⁴

The more common and easier to do thing is to clearly explain your model choice, the assumptions you make, and how that might impact what you find and how you can interpret your findings. Being clear about these components helps you build a better model, and it helps your readers understand the strengths and limitations of your model.

Model complexity

Model complexity can be considered a part of the model choice. Though it is possible to first choose a class of models with certain structures, and within that class look at complexity. For instance you might decide to use a linear model (that would be model structure) and then you need to decide how many variables to include (that would be model complexity).

Going back to the map example, once you decided on the overall model structure, you need to decide what details to include in your map/model to make it useful for your purpose. You don’t want it to be too complex, otherwise you’ll need a lot of data and your model and results might be hard to interpret and likely too specific for your situation and not very generalizable. On the other hand, you obviously also don’t want to omit any important aspect of your system.

There are no general rules dictating what simplifications are reasonable to make and which ones are not. It depends on the specifics of the system and the question. Even for the same system, different questions will lead to different models.

When you assess models, you need to ask yourself if you have the right level of detail to answer the question you are trying to answer.

Model results/predictions

It is important to look at model results and ensure they make sense. Recall the example from the introduction: If you used a linear model to look at correlations between children’s age and height, and then use the model to predict that a 30 year old person will be 20 feet tall, you know something isn’t right. In this case, you extrapolated outside the range of where the model might be useful - a common occurrence.⁵ For this example, the solution is to either only use the model in the range where it was calibrated, or if you want to predict the height of a 30 year old person, you need to change your model.

More common and harder to spot are situations in which your model seems to work well on your data - sometimes too well. This can be a sign of over-fitting, we’ll talk about that shortly.

Summary

Always perform critical comparisons of your models with reality. If something doesn’t look right or makes sense, it probably isn’t and you need to fix your model.

Further Resources

None that I can think of ATM. Let me know if you have any suitable ones.

Test yourself

Which of the following is NOT an important factor when considering model complexity?

Amount of available data Ability to capture data patterns Lines of code one needs to write Risk of overfitting

A more detailed model is always better than one that contains less details. TRUEFALSE

There is a single right type of model for every problem, and if you don’t get it exactly right, you are screwed. TRUEFALSE

Practice

• Revisit the papers you found for the previous unit’s exercise. Go through them again and specifically focus on the model structure and complexity, and if results make sense. Try, as best as you can, to critically evaluate if the authors made suitable choices and explained their choices well.

Three different maps of the same fictional country. They all have their distinct strengths and weaknesses.

Footnotes

1. I’m not the first one to come up with the map analogy. I’m not sure where I heard it first, but I know I didn’t invent it. Only the specific example shown here is from our previous paper.

2. This uses immunology as an example, but of course this could be any topic to make the same analogy.

3. This is not unique to mathematical/statistical models. If you do an experiment in a BALB/c mouse, you are using a model system. It doesn’t mean the results would be the same in another type of mouse, let alone in humans.

4. This paper shows a nice example in which the authors compared several influenza models. By comparing the models to each other and to data, they were able to discuss how models differed and which ones agreed with the data (spoiler alert: there was no one single best model, most of them were good at reproducing some data and not good at reproducing other data).

5. You might think that nobody would do something silly like that. You would be wrong. You can find a lot of extrapolations gone wrong in the literature. Often, it is subtle, but sometimes it is blatant. This is a good example of the latter.