Automated Polyp Detection System in Colonoscopy Using Deep Learning and Image Processing Techniques

Introduction:Undetected colonic polyps are considered a major cause of interval cancer of the colon. The Automatic Polyp Detection System (APDS) (Magentiq Eye LTD, Haifa, Israel) was developed to enhance the ability of endoscopists to detect polyps during screening colonoscopy. It is designed to be used both in real-time and offline. APDS runs directly on the video output of the endoscopic camera and highlights the polyp on the screen. APDS utilizes the power of Deep Learning and Computer Vision in order to improve polyp detection rates thus improving the performance of the endoscopist.

Aims and Methods:This study measured the performance of the prototype using video sequences of about half a minute each: 75 sequences to train the system, 10 sequences to validate the training, and 35 sequences to test the trained system. The total database of training, validation and testing included 121,500 video frames featuring 120 different polyps. Free Response Operating Characteristics (FROC) function, which represented the system performance, was measured. True Positive Rate (TPR) expressed system sensitivity and False Positive Per Frame (FPPF) signified the complement of the system’s specificity. The FROC was produced by changing the threshold parameter values of the system. For each threshold value, the number of polyps detected in at least 3 different consecutive frames was calculated. Dividing this number by the total number of polyps produced the System Polyp Detection Rate (SPDR).

Results:At the optimal point of the FROC, the TPR (i.e. sensitivity) was 89% and FPPF was 0.016 (equivalent to 98.4% specificity). SPDR was 86% (for this optimal working point).

Conclusion:The APDS prototype results, as a decision support system, are promising. The goal of high automated colonoscopic imaging polyp detection rate (≥ 95%) with high specificity (≥ 98%) will be achieved by exposing the system to greater amounts of data during the training phase and adding more computer vision and logic capabilities to the system

Keywords: Polyp Detection; Endoscopy; Imaging; Colonoscopy; Artificial Intelligence; Diagnostics

Colonoscopy is the gold standard for detection of colonic polyps. However, adenomas are missed in up to 20% of cases and cancers are missed in about 0.6%, as evidenced by ultimate detection of these missed lesions at interval colonoscopy [1,2]. Adenoma detection rate (ADR) is variable, and depends on the patient’s risk factors, physician’s performance and instrumental limitations.

Patient’s individual anatomy and the quality of bowel preparation are important determinants of quality colonoscopy. The performance of quality colonoscopy by the physician depends on factors such as successful cecal intubation, careful inspection during extended withdrawal time and overall endoscopic experience. Endoscopist fatigue and inattention are risk factors that can cause the physician to miss polyps whereas earlier procedure start time in a session correlates with better outcomes [3]. Noteworthy is that the ADR increased in sites that underwent a quality improvement program, whereas awareness of monitoring or simply being observed positively influenced the ADR for the better [4]. Innovative technological advances have been introduced to improve the performance of endoscopists. Some of these advances include optics that achieve a wider field of view and improve picture resolution [5-8]. Distal colonoscope attachments such as balloons caps or rings to improve visualization behind mucosal folds [9-12]. Although these advances are shown to improve ADR, there is still a need to further improve ADR. In this study, we present our experience with the Automatic Polyp Detection System (APDS) which aims to increase ADR by helping to avoid missed polyps. Its intended use is to support the endoscopist by serving as an additional detective tool beside the human eye, standardize endoscopists’ competence and assist as a learning and feedback system for endoscopists in training.

The APDS comprises, among other components, Computer Image Processing, which signifies the computer’s ability to understand a picture and make an appropriate respond or recommendation (like face or pattern recognition). It relies on a defined software algorithm, as well as on Deep Machine Learning (DML), where the computer is exposed to a variety of examples and then taught the correct answer (supervised learning algorithms) [13]. The main innovative part is the DML; The regular Machine Learning (ML) algorithms enable the replacement of some of the components of the conventional algorithms (such as object detection or classification algorithms) but not the whole algorithmic process. On the contrary, the DML enables the replacement of almost all the components of such conventional algorithms. The DML allows the system to achieve much better performance when exposed to a large amount of such pairs (Big Data), and it has higher accuracy in solving the issue in question, than the former ML algorithms (Figure 1).

One of the popular implementations of DML algorithms is Convolutional Neural Network (CNN) which gets images or videos as an input (An example is displayed in Figure 2).

Presently, technological improvements in computer’s hardware and software enable solving complexed challenges. Combined image processing and DML are already being utilized in medicine, as in the case of diagnostic medical imaging. They improve diagnosis, facilitate identification of findings and support the expert’s workflow [14]. The use of DML algorithms for diagnostics shows that the algorithms get better, in the same way as the human physician, with experience. The algorithms diagnostic capabilities improve as they are exposed to more relevant data during the learning phase.

DML algorithms for medical diagnostic modalities such as Computerized Tomography and Magnetic Resonance Imaging have already started to become practical and systems based on such algorithms already have obtained regulatory approvals [15]. To date, in endoscopy, there have been no regulatory approvals granted for diagnostic DML algorithms either offline or in real-time modality during the procedure itself. In this sense the APDS represents an innovative approach.

The APDS prototype was designed to detect polyps in recorded colonoscopy videos and to mark their location on each frame. It uses a DML (CNN) algorithm which is based on the Visual Geometry Group (VGG) net [16].

A VGG type of CNN consists of blocks, each of which comprises a number of 3×3 convolutional layers configured to preserve the spatial resolution of the data within a block. Max-pooling operators reduce the spatial resolution between blocks. This type of CNN has been widely used as a plug-and-play replacement of CNN’s that were developed before, such as the ANet, due to its simplicity and superior performance [17]. The APDS uses the VGG with some modifications, and analysis of patches of the image to combine detection approach with following segmentation of the polyp area, and also temporal considerations to enhance the detection rate and to reduce the false positive rate. It is based on learning from examples of time sequences of pairs of input (colonoscopy video frame) and output (whether there are polyps in the frame and the location of the polyps in frame). We measured the performance of the prototype using video sequences of about half a minute each. For this, we used 75 sequences to train the system, 10 sequences to repeatedly validate the training process while the system is being trained and 35 sequences to test the trained system. The recorded video sequences were collected from 12 physicians in 6 gastrointestinal endoscopy centers during the years 2014 to 2018. The total database of training, validation and testing included 121,500 video frames featuring 120 different polyps (16% over 1cm, 66% 0.5 to 1cm, 14% less than 5mm, and 4% flat polyps). The data included 70% standard definition and 30% high-definition videos, and all the frames were resized to a standard size of 640x360 keeping aspect ratio of 16:9 in order to fit CNN prevalent architectures and prevalent computational platforms. Highly blurred frames were filtered by the system automatically. The comparators (the ground truth data) for the polyp location (in which frame it appears and in what location in the frame) and the polyp size are the procedure reports and the pathology reports, and a thorough analysis of the recorded videos to correlate each polyp with the reports. While analyzing the recorded videos the apparent polyps were also manually tagged, in each frame, by a delineating contour. In the test videos, the Free Response Operating Characteristics (FROC) function, which represents the system performance, was measured. The FROC was produced based on changing the free parameter of the system (the threshold of a blob region average probability to be a polyp) and for each selected value of this parameter allowing the system to predict on all the test frames. Detection was considered positive if the ADPS final decision of a polyp in the image achieved a value of Intersection over Union (IoU) of more than 20% when compared with the ground truth manual tagging of the polyp. The True Positive Rate (TPR) expressed the system’s sensitivity and the False Positive Per Frame (FPPF) signified the complement of the system’s specificity. The FROC was produced by changing the threshold parameter values in the system. For each threshold value the number of polyps, each detected in at least 3 different consecutive frames, was calculated. Dividing this number by the total number of polyps produced the System Polyp Detection Rate (SPDR). As a supporting mechanism, the system enables also to produce for each tested frame a heating map which shows the probability for each pixel in the image to be a polyp (as the pixel has more yellowish color in the heatmap the higher probability that it can be a part of a polyp region). Based on this heatmap, and some additional parameters and logics, the system takes its final automatic decision about the existence and the location of the polyps in the video frames.

The APDS main display prototype includes a physician screen that demonstrates the polyp with its computerized automatic signature, with delineating contour, bounding box, or both (Figure 3). An additional screen is optional: it is a technician screen that represents the heatmap, and based on this screen the system internal confidence threshold (a probability value above which a pixel or a blob of pixels is considered to be a polyp) can be configured.

The results are designed to appear on the screen immediately with no delay since the system uses special optimization techniques as well as a rapid tracking algorithm to compensate for a delay that might, nevertheless, appear since the main algorithmic computation. A sound alert is also an option for the user.

The FROC was produced by allowing the APDS to predict the presence of a polyp in images from the testing database, and by changing the system confidence threshold (the system internal free parameter) between 10 different values. An optimal working point was found at a TPR (i.e. sensitivity) of 89% and an FPPF of 0.016 (equivalent to 98.4% of specificity). The SPDR was 86% for this optimal working point (Figure 4).

When the SPDR reached value of 86%, the automatically detected polyps included the different types of polyps in the testing database, such as more easy-to-identify sessile polyps morphology and more difficult-to-identify flat polyps (Figure 5 and 6).

The APDS works through DML that involves a multifactorial supervised learning algorithm. After the data is learned and checked and the software applied, there is a continuous process of quality control and the software changes and adjusts according to the flux of new data.

This system which aims to reduce polyp miss rates was found to be feasible and effective in detecting colonic polyps in its offline version, where it was tested on recorded videos. Its performance is expected to improve as the learning process continues, thus resulting in higher sensitivity and specificity on full data, but even the current results on the still limited size of data (89% of sensitivity), when comparing to the studies on miss rate during the colonoscopy procedure and especially to study on miss rate when using technologies to improve vision behind the colonic folds, Show the tendency of the APDS to improve the detection rate when it will be used as a decision support system. Classifying the system results according to the polyp size and type (sessile or not) and comparing it to the physicians performances according to previous studies and to our analysis on the recorded procedure videos that we have is a work that is now being done[18].

As a decision support system, the APDS fulfills the current needs of practicing endoscopists. Its sensitivity and specificity values are promising. From the clinical point of view, keeping the physician aware of the existence of every new polyp is the main current purpose of the APDS; therefore, the promising values of the SPDR of the APDS seem to be a practical predictor, even more than its sensitivity and specificity values, for the prospective clinical performance of the system.

The system has its online version for real-time reassurance and quality measurements and offline version for second look, quality control measurements and educational purposes. For both versions, and especially for the offline version, it is planned to increase soon the resolution of the processed frames from 640x360 to 720x405 (and later on to larger resolutions) and thus to enable the exploitation of better available endoscopic technologies (especially the HD technology), expecting it to improve the APDS results. On these days the APDS is tested with some additional CNN architectures, which were presented after the VGG architecture was presented, and the initial results show tendency to improve the performances. For the real-time version the expected processing time per frame is less than 20ms on the target computer and thus it shall cope with most of the video types without any latency. In addition, the system includes a rapid algorithm which tracks the endoscope movements in the colon and thus even if a latency occurs in the detection engine, it will be compensated by extrapolating the detected region to its actual location in the most updated frame which is presented on the screen. It is expected that as the trust of the endoscopist in the system will increase, the use of the system will reduce the withdrawal time. In addition, the APDS is trained to be expert on sessile and flat polyps, by exposing it to large number of polyps of these types, and this is since sessile and flat polyps are characterized by higher miss-rate [19]. Thus, a main effort is performed on these days to increase the number of collected videos which include flat polyps in order to increase the number of such polyps to which the system is exposed during its testing and training phase. At this level the system is planned to help the endoscopist to increase the Polyp Detection Rate (PDR) and derived from that also the Adenoma Detection Rate (ADR).

In the future versions of the system it is planned to train the system to classify in real-time between adenoma and non-adenoma polyps when using ultra-high optical zoom of the scope, and thus to increase further the ADR and to reduce the miss-rate at the right colon where adenoma polyps are more prevalent [20].

Compared to other diagnostic DML systems that have been studied and have received regulatory approval (ww.arterys.com), the APDS takes the DML technique, CNN based, to a new level (the endoscopy) and to a new type of data (video) which also includes consideration of a temporal aspect when being analyzed [21,22]. With its promising results, the APDS should meet the requirements as a clinically helpful tool. Comparing to another updated work that was published in 2018 on using artificial intelligence system for automatic polyp detection, it seems that the APDS has similar sensitivity but higher specificity. In addition, it should be noted that under specific circumstances, such as an unclean colon or rapid movements of the endoscope in the colon, it is expected that also the APDS could miss polyps [23]. Therefore, the endoscopist will still have to remain alert, but will have the support of additional expert observer with a trained technological “eye” watching and analyzing the video. The endoscopist will have the option to ignore the system recommendations, but the system can register, using its tracking algorithm, the general location of untreated polyp to be reminded later on if needed. Finally, as more technologies are introduced to improve vision behind the colonic folds, more polyps can be expected to appear in the video, and thus the APDS is expected to be even more effective

In the future we may expect the use of APDS with a fully autonomous system that consists of automated video pictures acquisition and interpretation.

The APDS prototype results, as a decision support system, are promising. The goal of high automated colonoscopic imaging polyp detection rate (≥ 95%) with high specificity (≥ 98%) will be achieved by exposing the system to greater amounts of data during the training phase and adding more computer vision and logic capabilities to the system.