Is it a birthmark or perhaps a melanoma? The trained dermatologist's eye sees a lot, of course, but not everything. Studies show that even with dermatoscopes, i.e. simple mobile reflected-light microscopes, diagnostic sensitivity is currently only around 85 percent. Artificial intelligence (AI) is opening up new opportunities to offer doctors even better support. Five Baden-Württemberg research institutions have combined their expertise to jointly develop a novel diagnostic system based on AI-supported imaging. These institutions are: the FZI Research Centre for Information Technology, the ILM Institute for Laser Technologies in Medicine and Measurement Technology at the University of Ulm, the Hahn-Schickard Institutes in Villingen-Schwenningen and Stuttgart, and the NMI Natural and Medical Sciences Institute at the University of Tübingen. All five belong to the innBW research alliance (Innovationsallianz Baden-Württemberg) and have been networking for a long time.

The joint concept was so convincing that it received funding from the Baden-Württemberg government from 2019 to 2021. The results of the first funding period were promising enough for the project partners to receive follow-up funding of two million euros for 18 months. The researchers want to use this money to further advance their project towards application. Essentially, their approach is to use sophisticated measurements to generate image-based data sets with additional information in order to train artificial intelligence models in such a way that they can distinguish benign from malignant skin changes. This approach is partly based on the knowledge that changes in skin tissue cause light directed at it to scatter and be absorbed differently and partly on structured irradiation and different penetration depths of varying wavelengths – even beyond visible light – to create a profile of skin changes and acquire data from deeper skin layers.

"The approach pursued in the project is based on multispectral imaging under structured irradiation. Although we work with cameras, the data sets generated are not comparable with conventional RGB images," says Christoph Becker, who heads the project for the FZI and coordinates the joint project. The idea behind this is that each wavelength investigated is projected onto the skin area to be examined with different fringe patterns. The reflections of the irradiation patterns are then recorded by a camera system. As the rays penetrate to different depths, it is possible to obtain information from skin layers at different depths. "The measuring system uses nine different wavelengths up to the near-infrared range and thus penetrates several millimetres deep into the skin. It can generate more than 40 images per picture," Becker specifies. With the help of structured irradiation, the image data can also be used to create a three-dimensional topography of the skin change under examination.

The optical measurement system was developed by the ILM and the required optics were improved in close cooperation with the Hahn-Schickard Institute in Stuttgart. At the moment, the project team is working with two cameras in order to eliminate interfering artefacts, such as those caused by surface scattering. In the second phase of the project, the process is expected to become more robust, which would also make it easier to build a compact device. "We may be able to do without the second camera if the effect of surface scattering turns out to be irrelevant. That is one of the issues we want to clarify in the second funding phase," says Becker.

An AI model is fed with images that have already been generated - more will be added in the next phase. This model then undertakes hundreds of learning cycles to determine which image data correspond to a benign or malignant tumour. As early as October 2020, the partners started training AI models with their data, and the accuracy is getting better and better. Results are constantly being verified with the help of laboratory analyses. "However, there is not yet sufficient data to conclusively evaluate the quality of the approach, so we are continuing to collect training data," says Becker, adding: "It is already quite challenging to collect training data, because very comprehensive data sets are needed for the neural network." This is where the expertise of the Hahn-Schickard Institute in Villingen-Schwenningen comes into play: the project partners there focus on aspects of distributed learning. "They work with measurement and simulated data," says Becker.

The ILM initially generated synthetic data, i.e. data based only on simulations. They subsequently worked with the Department of Dermatology at Tübingen University Hospital to carry out measurements on patients in order to obtain data from practical application. The goal now is to collect data at more hospitals, and possibly also doctors’ practices, and to have an AI model learn with the data on site at each of the hospitals and practices involved. The partners will then share their learned AI models, or merge them at a suitable point as well as providing feedback to the hospitals involved on what they have learned. "How exactly the communication will take place and how it can be automated constitute further development goals," says Becker.

Becker and his team specialise in business informatics and have an integrating function in the project: the data and AI management platform was set up at the FZI so the different systems used by the various partners work together and can be adapted when changes occur. The FZI's tasks also include the development of suitable visualisation for doctors.

The NMI Reutlingen has taken on another special task: its researchers are analysing which protein and mRNA-based molecular biomarkers enable underlying melanomas to be classified more precisely, for example, are able to indicate an aggressive disease course or a recurrence. The results are correlated with the image data and thus also find their way into the AI learning models.

The next phase of the project will also focus on the very practical aspects of manufacturing the devices. It is hoped to be able to manufacture the complex optics cheaply and flexibly. In addition, the optics need to be miniaturised, for example by using suitable micro-optical components. The ILM and the Hahn-Schickard-Institute in Stuttgart are working closely together on this. "The design and prototypes of the optics come from the ILM and are produced using two-photon lithography. The Hahn-Schickard researchers in Stuttgart are developing, among other things, a process to reproduce the optical prototypes easily and cost-effectively using injection moulding technology," Becker explains. All in all, he and his colleagues still have a long way to go before the system is ready for application: "At the moment, we are still fine-tuning the prerequisites for commercial product development. Once this is in place, we will be looking for suitable partners for the commercialisation of the product."