dr-ai-radiologist

Click the buttons to move through each training waypoint:

Training completion: 0%
Images seen: 0
Seconds Elapsed:0

In this picture, the white dot circled in red is a small nodule. None of the other dots are nodules.

Bot

Nodule detection: 0.00% @Recall1
Malignancy detection: 0.00% @Recall1

Radiologist

The AI system will need to learn two different skills: how to detect nodules in the lungs, and how to identify which nodules are likely to be malignant.

Detection:

A nodule is a small lump of tissue that is not normally present in the lungs. Finding nodules is a classic "needle-in-a-haystack" problem. This is because nodules are usually tiny, and many other structures like blood vessels and scars can look like them on first glance.

In this picture, the white dot that is circled in red is a small nodule. None of the other dots are nodules.

Malignancy:

There are a range of nodule features that radiologists use to determine which nodules might be cancerous. The most commonly used system is the Fleischner criteria, which only consider the size and number of nodules, and the presence of risk factors like smoking. Other systems are more complicated, and include the shape and location of the nodules.

Training completion: 25%
Images seen: 47,583
Seconds Elapsed:1064

A doctor would know for certain the area circled in red is not a nodule, but at this point, the AI still doesn’t get it right.

Bot

Nodule detection: 46.40% @Recall1
Malignancy detection: 46.32% @Recall1

Radiologist

The AI at this point doesn't really know anything about nodules. Quite often it will highlight blood vessels, saying that it isn't certain they’re nodules but is suspicious they are cancer. Since cancer only occurs in nodules, this makes no sense—it’s an impossible contradiction.

A human, on the other hand, would know this is not a nodule—and therefore, is not cancer, because it’s branching. That’s a clear sign that it’s a blood vessel.

Training completion: 50%
Images seen: 95,166
Seconds Elapsed:2199

The half-trained AI system thinks this fold in the wall of the large bowel in the abdomen might be a lung nodule.

Bot

Nodule detection: 59.80% @Recall1
Malignancy detection: 68.71% @Recall1

Radiologist

At this stage, the AI system has very high confidence when detecting large nodules (over 1 cm in diameter). It still hasn't learned some very simple lessons though, and is moderately confident that there are lung nodules in areas of completely normal lung, or even in the bowel. This shows the complete lack of common sense that characterizes modern AI. Even a three-year-old knows the difference between the chest and the belly, but this AI only learns to look for "nodule-like" patterns. It doesn't know what the abdomen is.

Training completion: 75%
Images seen: 142,749
Seconds Elapsed:3329

This tiny lump on the outer edge of the lung is almost certainly not cancer, as the AI system has started to learn.

Bot

Nodule detection: 64.30% @Recall1
Malignancy detection: 76.38% @Recall1

Radiologist

The AI is really starting to get the hang of this now. Almost all of the really confident nodule detections are nodules, and are even the lower-confidence detections are "nodule-like." A difficult task, even for humans, is differentiating between nodules and scarring from previous infections. As a general rule, tiny nodules that are at the very edge of the lung are almost always benign and can be ignored. The AI system is getting quite good at this by now. It isn't as convinced that they are actually nodules (which they are not), and it is fairly sure they are not cancer.

Training completion: 100%
Images seen: 190,332
Seconds Elapsed:4472

The AI thinks the region circled in red is a nodule, but not cancer; it believes the large nodule at the top of the picture, circled in white, is almost certainly a cancer.

Bot

Nodule detection: 67.90% @Recall1
Malignancy detection: 82.82% @Recall1

Radiologist

Once it has finished training, the model is doing pretty well. If we take only the very high confidence predictions, we find it is identifying most of the large and suspicious nodules.

As we might hope, the model is almost 100% certain that these large nodules are malignant. Radiologists would definitely treat them as cancer until proven otherwise, and the AI system is just as worried.

Once we get into lower-confidence predictions, we see an interesting pattern: The AI system is less certain that the "nodule-like" things in the lungs are actually nodules, but much more importantly, it is almost certain they are not cancer. In many cases the AI is moderately confident there is a nodule, but nearly 100% certain it is benign. While a human would probably not call them "nodules" in the first place, the answer for the patient is the same.

The AI system thinks the region circled in red is a nodule with fairly high confidence, but is almost certainly not cancer. A radiologist would agree; in particular, the fact that it is attached to the edge of the lung, and that it is located behind the aorta is reassuring. On the other hand, the large nodule at the top of the picture (circled in white) is almost certainly a cancer, and the AI system agrees.

An honorable mention goes to a tricky problem that the AI system learns to deal with. Calcified nodules are almost certainly benign, but there are not many of them in the dataset, so it will be hard for the AI system to learn them. This is trivial for humans, because we only need to recognize they are calcified. Once we have been told what to look for, we don't even need to practice. We can get the right answer on our first go.

The AI system can't understand "calcified" as a concept, it can only learn that a certain image pattern is benign. In this case, calcified nodules are much denser (brighter) than non-calcified nodules. Once it has seen many nodules like this, that never turned out to be cancer, it can learn that the pattern is relevant.

One particular nodule was tricky, because it was calcified but also large (which is a strong indicator of cancer). Initially the AI thought it was definitely cancer, even after training on 50,000 images. But it learned that the density was relevant, and by 100,000 images it was 100% certain that this nodule was benign.

The nodule circled in red is very dense (bright), compared to the nodule circled in white, which is more grey. The former is a benign calcified nodule, the latter is almost certainly cancerous.

Conclusion

The AI performs very well, although it still isn't at the level of a radiologist. This is mostly because the dataset isn't big enough, containing fewer than 1,000 CT scans. Similar systems developed by tech companies or hospitals would try to use of tens or even hundreds of thousands. We’re working with a free dataset, and there’s a reason why companies like IBM pay billions of dollars to acquire companies and their training data. The system would definitely benefit from more variety in training, and we could expect the performance to improve to near- human levels with enough data.

This project highlights the major differences between humans and modern AI systems. Humans are very efficient at learning, because we use our pre-existing knowledge as a scaffold. For example, consider calcified nodules, which are almost certainly benign. These are simple for humans to learn to recognize, because calcified nodules are much denser (brighter) than non-calcified nodules. We can simply be told “a very dense nodule is calcified and therefore benign” and we can use that knowledge to get the right answer, even without practice.

However, the AI system has no way to incorporate knowledge; it can't understand "calcified" as a concept. Since the AI has no concept of what calcium looks like, or that calcified nodules are benign, it needs to build that understanding from many examples. In this case, it took over 50,000 images to learn what a human could learn from a description in a book.

On the other hand, AI systems gain experience fast. Those 50,000 images were reviewed in under 20 minutes, a task that would take a human years. So for the more complicated decisions, where experience is more valuable than knowledge, the AI systems may even have an advantage.

The following people contributed to this report:

All photos courtesy of the Lung Image Database Consortium and the Image Database Resource Initiative
Additional assistance from Jeremy Howard, founder of fast.ai
Edited by Elijah Wolfson