Lung cancer is a severe disease and often only diagnosed at an advanced stage, meaning the cancer has locally spread or can be found in other organs (e.g. the liver, bone, brain…). At an early stage, lung cancer can be removed surgically, or treated with a combination of radiotherapy and chemotherapy, followed by immunotherapy. These are treatments with curative intent. In more advanced stages of the diseases, patients are treated with chemotherapy and immunotherapy or targeted therapy. When the cancer is diagnosed at this advanced stage, no cure is possible. In this situation, it is hard to determine for a clinician which (combination of) therapy is optimal as the prognosis is difficult to estimate, and treatments might impact the quality of life of the patients in a negative way. At AZ Delta in Roeselare, Belgium, a team of engineers developed a prognostic tool to predict the likelihood a patient with advanced stage lung cancer will die in the next 6 weeks, 3 months and 6 months. They developed this tool using machine learning, meaning they trained a computer algorithm to make these predictions based on a cohort of advanced stage lung cancer patients diagnosed at the hospital between 2018 and 2021. The engineers selected a number of variables (e.g. comorbidities, lab results…) based on which this tool has to make a prediction, but the computer itself had to learn which of these variables are important ones and which are not. They also trained a second version of this tool, but only gave the tool approximately a third of the variables to work with. Our paper has the goal to compare the outcomes as predicted by the tool, to the real world outcomes. For use in clinical practice, the tool should have the highest possible accuracy, meaning that the predictions of the tool are representative for the real world. This accuracy also depends on the cut-off used for calculating the probability. We solved this by applying different cut-off values for probability of mortality, meaning that every prediction above this cut-off is regarded as if a patients will be deceased and vice versa. For example, at a cut-off value of 50%, every patient with a prediction higher than 50% was seen as deceased, whereas every patient with a prediction lower than 50% was seen as alive. In this paper, we investigated a cohort with patients with advanced stage lung cancer diagnosed and treated in the AZ Delta Hospital in 2022 (which were 100 patients) to compare the outcomes as predicted by the tool to what happened in real life. Using some statistics, we demonstrated that the model can make significant predictions at every end point and even with the selected amount of variables. It is important to note that this significance could be proven at different cut-offs and that these vary between the different models. Furthermore, we argue in this article that we prefer to use the highest cut-off for probability of mortality with significance for each model. A very high cut-off for probability of mortality, means that the tool would predict with a very high likelihood that a patient won’t be alive at the given timepoint. If this is a short term timepoint (e.g. 6 weeks after diagnosis), treating physicians might decide not to give potentially toxic treatments (with negative impact on quality of life), but rather focus on supportive and palliative care. Another important note is that currently there is no clear jurisdiction about this type of diagnostic tool. Does a patient have the right to know what odds the tool predicted? And if so, is it obliged that this information is given by a doctor, so it is framed correctly? And what about patients who do not receive treatment but still survive? Can a patient’s family use the prediction of the tool in court if the doctor made a decision against it? A lot of questions are to be made regarding this topic as it is very new in our society and its place in it is rather unclear. The way we look at the use of artificial intelligence by doctors and in medicine in general, as well as the things we expect from artificial intelligence applications will drastically change once we let computers make these kinds of (assistance in) decision making and it is our job as clinicians to make sure it is in an appropriate and well validated way.