?

EndoPlan-3D is developing an AI-assisted, interactive 3D environment to improve the planning of endoscopic procedures for gastrointestinal early neoplasia. By fusing multimodal data from white-light endoscopy and endoscopic ultrasound using AI algorithms, individualized, dynamic 3D models are generated. These models are designed to support physicians in a VR environment by enabling accurate identification of tumor boundaries, critical hotspots, and optimal resection strategies. The University of Augsburg leads the project’s participatory design process. Through interviews and co-design workshops with endoscopy experts, a user-centered 3D environment is being developed to support realistic and effective surgical planning. Speech data generated during the “think-aloud” planning process is analyzed using large language models to enable automated, speech-based annotation. In addition, the NovA annotation tool, developed at the University of Augsburg, is used to systematically capture surgical planning workflows - including time-stamped critical events relevant to both planning and execution. The University of Augsburg is also responsible for investigating the ethical, legal, and social aspects (ELSA) of clinical integration and for developing corresponding guidelines.