Optical imaging technology in colonoscopy:Is there a role for photometric stereo?

Benjamin M Shandro,Khemraj Emrith,Gregory Slabaugh,Andrew Poullis,Melvyn L Smith

Benjamin M Shandro,Andrew Poullis, Department of Gastroenterology,St George's University Hospitals NHS Foundation Trust,London SW17 0QT,United Kingdom

Khemraj Emrith,Melvyn L Smith, Centre for Machine Vision,University of the West of England,Bristol BS16 1QY,United Kingdom

Gregory Slabaugh, Department of Computer Science,City,University of London,London EC1V 0HB,United Kingdom

Abstract

Colonoscopy screening for the detection and removal of colonic adenomas is central to efforts to reduce the morbidity and mortality of colorectal cancer.However,up to a third of adenomas may be missed at colonoscopy,and the majority of post-colonoscopy colorectal cancers are thought to arise from these.Adenomas have three-dimensional surface topographic features that differentiate them from adjacent normal mucosa.However,these topographic features are not enhanced by white light colonoscopy,and the endoscopist must infer these from two-dimensional cues.This may contribute to the number of missed lesions.A variety of optical imaging technologies have been developed commercially to enhance surface topography.However,existing techniques enhance surface topography indirectly,and in two dimensions,and the evidence does not wholly support their use in routine clinical practice.In this narrative review,co-authored by gastroenterologists and engineers,we summarise the evidence for the impact of established optical imaging technologies on adenoma detection rate,and review the development of photometric stereo(PS)for colonoscopy.PS is a machine vision technique able to capture a dense array of surface normals to render three-dimensional reconstructions of surface topography.This imaging technique has several potential clinical applications in colonoscopy,including adenoma detection,polyp classification,and facilitating polypectomy,an inherently three-dimensional task.However,the development of PS for colonoscopy is at an early stage.We consider the progress that has been made with PS to date and identify the obstacles that need to be overcome prior to clinical application.

Key words:Photometric stereo;Colonoscopy;Colonic polyps;Adenomas;Image enhancement;Machine vision

INTRODUCTION

Colorectal cancer(CRC)is the third most common cancer in the world[1].The detection and removal of colonic polyps during colonoscopy is central to efforts to reduce CRC mortality,through its earlier detection,and the detection and removal of its major precursor lesion,the adenoma.

Adenoma detection rate(ADR)has emerged as one of the most important measures of colonoscopy quality.It is used as a surrogate marker for post-colonoscopy CRC,which is the ultimate aim of colonoscopy screening.Post-colonoscopy CRC can arise from lesions missed at index colonoscopy or due to inadequately resected adenomas.A Dutch population study suggests that the majority of post-colonoscopy CRC is due to the former[2].The miss rate for adenomas is estimated to be 22%-30%,and small(＜ 1 cm),flat,and proximal lesions are more likely to be missed[2-4].

Recent evidence has proven ADR to be an appropriate surrogate for postcolonoscopy CRC.Compared to patients examined by endoscopists with an ADR of 20% or greater,those examined by endoscopists with an ADR of less than 20% have a ten-fold increase in the hazard ratio of interval CRC and an absolute risk 0.12%[5].Another study found that for every 1% increase in ADR,there is a 3% decrease in the risk of post-colonoscopy CRC[6].

ADR is determined by multiple service,patient,endoscopist and technical factors,which are displayed in Table 1,although the weight of evidence supporting the impact of each factor varies widely.There has also been a great deal of interest in harnessing new technologies to realize improvements in colonoscopy quality.In this paper we summarize the evidence for the impact of commercially available optical imaging technologies on ADR in the average-risk population,and review photometric stereo(PS),a machine vision technique with potential clinical applications in colonoscopy.

ESTABLISHED OPTICAL IMAGING TECHNOLOGIES

Colonic adenomas have pronounced topographic features,such as elevations,recessions and pit patterns,which differentiate them from normal mucosa.In white light colonoscopy,the lighting illuminates the field of view and enhances coloration,but not topographic contrast.This may contribute to the number of missed lesions.Several optical imaging technologies have been introduced that enhance topographic contrast to facilitate the detection of adenomas.

High definition white light

High definition white light(HD-WL)colonoscopes and monitors produce higher resolution images and display more images per second than a standard definition white light(SD-WL)colonoscope.Although early studies did not report a significant improvement in ADR,more recent observational studies(4093 patients)demonstrated an increase in ADR of 4.5%-12.6% when HD-WL was compared to SD-WL colonoscopy[7-9].However,patient characteristics and adjustment for potential confounders were not standardized across these studies.This might explain why the increase in ADR was more marked than that reported in a 2011 meta-analysis of five studies(4422 patients)comparing HD-WL to SD-WL colonoscopy,which showed a more modest 3.5% increase[10].However,higher resolution images confer other benefits outwith ADR,and HD-WL colonoscopes are now in widespread use.

Table 1 Factors considered to impact on adenoma detection rate

Dye-based chromoendoscopy

Chromoendoscopy is a technique where contrast dyes,such as indigo-carmine or methylene blue,are sprayed onto the colonic mucosa during the withdrawal of the colonoscope.The contrast dyes pool in recessions,thereby accentuating surface topography when viewed in HD- or SD-WL.A 2016 Cochrane review of seven randomized controlled trials(RCTs)(2727 patients)found that chromoendoscopy increased the odds of an average-risk patient having one or more neoplastic lesion detected by approximately 50%(pooled OR = 1.53,95%CI:1.31-1.79)[11].However,some of the included trials compared chromoendoscopy to SD-WL colonoscopy,which has since been superseded by HD-WL in clinical practice,so the gains might now be smaller.A more recent large RCT(1065 patients)found a small increase in ADR when comparing chromoendoscopy to HD-WL colonoscopy,but this did not reach statistical significance(OR = 1.13,95%CI:0.87-1.48)[12].

Virtual chromoendoscopy

In addition to dye-based chromoendoscopy,various optical imaging technologies have been developed commercially.These are commonly termed “virtual chromoendoscopy”,and include narrow band imaging(NBI),i-scan digital contrast(i-scan),flexible spectral imaging colour enhancement(FICE),blue light imaging(BLI),linked colour imaging(LCI)and autofluorescence imaging(AFI).Of these,NBI is the most established.

Narrow Band Imaging

NBI uses red,green and blue light filters to enhance the superficial mucosa and vasculature.A 2012 Cochrane review of eight RCTs(3673 patients)found no difference in ADR between white light colonoscopy and NBI(RR = 0.94,95%CI:0.87-1.02)[13].However,a 2019 meta-analysis of RCTs from which individual patient data was available(4491 patients)demonstrated a modest but statistically significant increase in ADR when NBI was compared to white light colonoscopy,but only when second generation NBI was used(OR = 1.28,95%CI:1.05-1.56)or bowel preparation was excellent(OR = 1.30,95%CI:1.05-1.56)[14].

I-scan digital contrast and flexible spectral imaging colour enhancement

The i-scan digital image processing system offers surface enhancement,contrast enhancement and tone enhancement.FICE uses a computed spectral estimation system to narrow the bandwidth of light in order to enhance the visibility of mucosal and vascular details.A 2014 meta-analysis of five RCTs(3032 patients)compared both i-scan and FICE to HD-WL colonoscopy and found no increase in ADR(RR = 1.09,95%CI:0.97-1.23)[15].However,a recent large RCT(740 patients)comparing i-scan to HD-WL colonoscopy found a significant increase in ADR in the intention-to-treat analysis(47.2%vs37.7%;P= 0.01)[16].The observed increase in ADR was largely due to enhanced detection of small,flat,proximal adenomas,which are the lesions most likely to be missed at screening colonoscopy.

Blue light imaging and linked colour imaging

BLI and LCI are more recent techniques for virtual chromoendoscopy,based on narrow-band observation of mucosa illuminated by a laser light source.In a large RCT(963 patients)comparing BLI to white light colonoscopy,no increase in ADR was observed,although this was not the primary outcome measure[17].One RCT(141 patients)comparing LCI to white light colonoscopy demonstrated a significantly increased per-patient ADR(37%vs28%)[18].

Autofluorescence imaging

In AFI a rotating filter produces short-wave light bursts that excite different tissue types.The excited tissues emit fluorescent light that is detected and reconstructed in two dimensions(2D).A 2015 meta-analysis of six RCTs(1199 patients)showed no significant difference in ADR between AFI and white light colonoscopy(OR = 1.01,95%CI:0.74-1.37)[19].A subsequent RCT(802 patients)confirmed no increase in ADR using updated AFI instead of white light colonoscopy,but did demonstrate a significant increase in the detection of proximal flat lesions[20].

Summary

It is widely accepted that dye-based chromoendoscopy improves ADR.However,this has not been demonstrated consistently in RCTs,and no studies have demonstrated an increase in the detection of advanced neoplasia compared to white light colonoscopy.In addition,dye-based chromoendoscopy is cumbersome to perform,and any increase in ADR must be balanced against the financial and opportunity costs of the additional time required to perform each procedure.As a result,this technique is generally only recommended for high-risk populations,such as those with inflammatory bowel disease or hereditary polyposis syndromes[21].Even in these populations,national bodies have drawn different conclusions from the same evidence base[21,22].

Virtual chromoendoscopy does have advantages over dye-based chromoendoscopy,not least the ease with which it can be performed.However,additional training and experience are required to interpret the enhanced images correctly[23],and the evidence that these technologies increase ADR,in a clinically meaningful and repeatable manner,is lacking.

It is clear that there is still a need for research into less established imaging technologies that have the potential to enhance surface topography during colonoscopy,and might thereby increase ADR.One such technology is PS.

PHOTOMETRIC STEREO

PS is a machine vision technique introduced by Woodham[24]in 1980.In PS,a series of images of an object are obtained from a single fixed viewpoint,with the object illuminated from multiple light sources orientated in different directions[24](Figure 1).In addition to generating 2D colour images,the surface normals of the object can be estimated at each pixel location and the surface topography reconstructed in three dimensions(3D)[25](Figure 2).

PS functions best in geometrically controlled situations,where the position of the camera and light sources relative to the object are known.Potential applications are wide-ranging,and PS has been demonstrated to provide accurate 3D surface topographic data for objects as diverse as particles and tablets in pharmaceutical manufacturing[26,27]through to nuclear reactors[28].In plants,the application of PS for the in-field analysis of crops has been studied[29],in addition to early disease detection in asymptomatic plants,in combination with multispectral imaging[30].

In humans,PS has been used to perform contactless biometric identification using 3D handprints or finger knuckle patterns[31,32],3D facial reconstruction and recognition[33,34],in vivomeasurement of intravascular blood flow[35],exandin vivocharacterization of skin lesions[36,37],and 3D reconstruction of a phantom human tongue[38].The latter has potential applications in traditional Chinese medicine,where visual inspection of the tongue surface is used to diagnose syndromes and diseases affecting distant organs[39].

Figure 1 Principles of photometric stereo.A single fixed viewpoint captures multiple images of a surface illuminated by differently orientated light sources.The known properties of the viewpoint and light sources can be used to derive the surface orientation,which is not known,from the image series.(Courtesy of Smith ML,co-author).

PHOTOMETRIC STEREO IN COLONOSCOPY

Unlike commercially available virtual chromoendoscopy techniques,the study of PS in colonoscopy is at a nascent stage.A Boston-Madrid consortium has carried out a number of proof-of-concept studies using the PS technique.Firstly,they imaged a silicon phantom colon,using a bench top PS imager with cross-polarizers placed in front of the light sources and camera lens to reduce specular reflections at image acquisition[40].They demonstrated accurate 3D reconstruction of the haustra and three 0.5-1 mm sessile elevations in the phantom colon[40].

Following on from this work,they imaged three humanex vivogastrointestinal tissues using the same bench top PS imager:A colonic post-polypectomy site,a benign sessile colonic polyp,and a small bowel melanoma metastasis[41].These specimens were wet,and therefore represent a better model for the reflective properties of colonic mucosa encountered in colonoscopy.Each pathological finding was identifiable in the 3D reconstructions[40].However,these results were obtained under conditions dissimilar to those encountered during colonoscopy,particularly in terms of the distance between the camera,light sources and mucosa.

The consortium subsequently modified a commercially available gastroscope by adding four additional light sources orientated equally around the gastroscope tip,and synchronizing the additional light sources with the video signal[40].The resulting system had a total diameter of 14 mm,similar to commercial colonoscopes.The software modifications enabled real-time white light imaging,and topographical reconstructions every four frames.Unfortunately,the dimensions of the gastroscope tip precluded the use of cross-polarizers.Using a non-specular 3D-printed phantom colon,they compared the images obtained using the modified gastroscope to those obtained using their previously described bench top PS imager.The elevations and depressions in the phantom colon were accurately reconstructed in 3D by both imaging systems[40].

The consortium then imaged threeex vivoporcine colons,which had been dissected and laid flat,using the modified PS gastroscope fixed above the tissue[42].They carried out dye-based chromoendoscopy on the samples using indigo-carmine,and compared these images with images obtained by virtual chromoendoscopy by colourequalisation,and with images obtained by virtual chromoendoscopy combined with PS.They detected statistically significant image improvement when virtual chromoendoscopy and PS were combined,compared to virtual chromoendoscopy alone[42].However,it should be noted that this study compared still images,rather than real-time video,and the working distance was fixed.Both challenges need to be overcome prior to clinical application.

Finally,they evaluated the capability of PS to capture topographic data in the human rectumin vivo,using their modified gastroscope on eight human subjects[43].The white light images obtained through the modified gastroscope were displayed in real-time,but the 3D topography could only be reconstructed in post-processing.When imaging obliquely to the mucosa,elevations from blood vessels and diminutive lesions were reconstructed appropriately,relative to qualitative inspection of the white light images.However when imaging perpendicular to the rectal mucosa,specular reflections caused insurmountable topographical artifacts[43].

The Boston-Madrid consortium has not published any new data since 2014,and is presumably no longer active.However,one researcher from the consortium has continued to study PS imaging.In 2019 conference proceedings they reported the successful imaging of a synthetic phantom colon using a multimodal system that combined white light,PS and speckle contrast flow imaging[44];and described a deep learning method for depth estimation using computer-generated PS images[45].

Outside of the Boston-Madrid consortium,very little has been published about PS in the gastrointestinal tract.Our group in the United Kingdom has applied PS toex vivoporcine gut,using a handheld PS imaging system to capture topographic data in the porcine colon,duodenum,oesophagus,and gastro-oesophageal junction[25,46](Figure 2).Phantom polyps were created by raising small areas of the mucosa by submucosal injection of saline solution.A least squares approximation method was used to adjust for specular reflections,and a 3D reconstruction generated.We demonstrated that the Shape Index differentiated locally spherical phantom polyps from the adjacent normal mucosa[25](Figure 3).Such an approach could potentially be applied to the computer-aided detection of pedunculated or sessile,though not flat,polyps.

A South Korean group constructed a bench top multimodal endoscopic system that combined white light imaging,multispectral imaging and PS[47].They tested its function using a 3D-printed polyp-mimicking phantom of the human colon and fourex vivomouse colons that had been inoculated with human CRC cell lines.They demonstrated that the PS component could detect and reconstruct phantom sessile polyps with diameters as small as 0.5 mm[47].In the mouse colons,the PS-derived 3D reconstructions demonstrated a polypoid surface distinct from the adjacent normal mucosa,the locations of which aligned with the spectral-classified tumour sites.Formal histological analysis of the multimodally-detected tumour sites demonstrated CRC,the margins of which correlated with the polypoid surface delineated in the PSgenerated 3D image[47].

Figure 2 Porcine colonic mucosa captured with photometric stereo imaging.A:One of six captured white light images;B:Reconstructed surface normal map;C:Reconstructed height map;D:Shape Index plot.(Courtesy of Poullis A,co-author).

POTENTIAL CLINICAL APPLICATIONS OF PHOTOMETRIC STEREO IN COLONOSCOPY

These early studies have demonstrated that PS can differentiate abnormal tissue from the surrounding normal mucosa,which has potential applications in colonoscopy.

The primary clinical application of PS would be to increase ADR by emphasising the surface topography of adenomas in the field of view - essentially as a novel method of virtual chromoendoscopy.The evidence generally supports dye-based chromoendoscopy as a technique to increase ADR,whilst that supporting existing virtual chromoendoscopy technologies is less compelling,at least in average risk populations.This may be because the origin of the enhanced surface definition in dyebased chromoendoscopy is the surface topography itself(i.e.,the dye accumulating in pits and crevices in the mucosa),whereas the enhancement derived from commercial virtual chromoendoscopy is instead based on the optical properties of the mucosa.In this respect,PS has more in common with conventional chromoendoscopy than with established virtual chromoendoscopy techniques.However,PS has an advantage over dye-based chromoendoscopy in that it could be readily integrated into computeraided adenoma detection systems.

When a polyp is detected,the type of lesion must be diagnosed to determine future CRC risk[48].Optical diagnosis using existing optical imaging technologies can achieve acceptable sensitivities and specificities in expert hands[49].However these results have not been replicated in routine clinical practice[50],and adequate training is not widely available.PS has been shown to differentiate between benign and malignant skin lesions[36,37],and could improve optical diagnosis by generating 3D data for interpretation by the endoscopist,given sufficient training,or by a computer-aided diagnostic algorithm.However,no published studies have applied PS to polyp diagnosis rather than detection.

With no proven methods of optical diagnosis in widespread use,most detected polyps are subsequently removed by polypectomy and sent for histological analysis.Polypectomy is a challenging task,particularly more advanced techniques such as endoscopic mucosal resection or endoscopic submucosal dissection.It is a procedure that is inherently performed in 3D,but the endoscopist must infer depth information from indirect cues from a 2D video monitor.In laparoscopy,3D systems that provide binocular depth perception have been shown to reduce procedure time and error rates in experimental settings[51].It is conceivable that 3D colonoscopy,such as could be rendered by PS in the future,could reduce procedure times and complications in polypectomy as well.

Figure 3 Phantom polyp differentiation using the Shape Index.A spherical phantom polyp is differentiated from adjacent normal tissue by applying a hysteresis thresholding technique to the Shape Index.(Courtesy of Poullis A,co-author).

OBSTACLES TO THE CLINICAL APPLICATION OF PHOTOMETRIC STEREO

Only small proof-of-concept studies have been carried out to date,and there are multiple obstacles that must be overcome prior to clinical application.Firstly,PS assumes Lambertian reflectance,and the moist colonic mucosa is an innately non-Lambertian surface.This gives rise to specular reflections,which can cause artifactual distortion in the 3D reconstruction.This issue has been reported in all studies of PS in the gastrointestinal tract,and may be exaggerated when the camera lens is perpendicular to the mucosa[43].

A variety of post-processing approaches have been used to try to compensate for these specular reflections,including least squares approximation[25],exclusion of the reflections using spectral or directional cues[52],combining perspective projection and the Blinn-Phong reflectance model[53],and simultaneous mesh-based computation of surface normals and reflectance[54].However,the optimal approach to take in the application of PS to colonoscopy is unknown.

In the multimodal imaging study by Kimet al[47],the total time for image acquisition and reconstruction was 9 s,during which the lesion and image plane had to be fixed.This acquisition time is impractical for colonoscopy,as the distance between the colonoscope tip and the colonic mucosa is constantly changing due to movement of the colonoscope tip and the colon itself.However,the majority of this processing time was attributable to the multispectral component,with the PS acquisition and 3D reconstruction taking approximately 1 s,which is more promising.In addition,PS imaging has previously been applied to fast-moving surfaces in other fields,such as quality control in manufacturing[55].A similar technique could potentially be applied to colonoscopy.

In theirin vivostudy of human rectums,the Boston-Madrid consortium was able to display white light images in real-time using their PS-modified gastroscope,with the PS data extracted in post-processing[43].With further advances in computer processing,it is anticipated that real-time PS topographic data could be made available to endoscopists in the future,either as a visual representation orviaa computer-aided detection system.However,even when this is achieved,the most effective way to convey 3D information to the endoscopist remains unknown.

Finally,although the technology is relatively inexpensive and unsophisticated compared to some other approaches,the hardware has not yet been miniaturized for integration into commercial colonoscopes.However,colonoscope tips already house a camera and multiple light sources,and the Boston-Madrid consortium documented their conversion of a commercial gastroscope to obtain PS images using a bespoke end-of-scope device.With commercial input,integrating PS into the next generation of colonoscopes should not be an insurmountable task.

CONCLUSION

PS can derive accurate 3D surface topographic data from colon phantoms,animal models and human colonic tissue.However,research into the application of PS to colonoscopy is at a very early stage.In humans,PS imaging has only been performed on a singleex vivocolonic polyp and eightin vivorectums to date.Several obstacles have been identified and incompletely resolved,particularly how to deal with specular reflections and an unfixed field of view.Furthermore,whilst miniaturization of the existing technology to permit integration into the next generation of commercial colonoscopes is certainly possible,it has yet to be attempted.Although PS imaging could have a significant impact on colonoscopy in the future,that future remains distant.