Drishti Paint 3.2: 一種開(kāi)源的綜合二維與三維分割的新工具

Drishti Paint 3.2： a new open-source tool for both 2D and 3D segmentation

Abstract X-ray computed tomography （CT） has been an important technology in paleontologyfor several decades. It helps researchers to acquire detailed anatomical structures of fossils nondestructively.Despite its widespread application， developing an efficient and user-friendlymethod for segmenting CT data continues to be a formidable challenge in the field. Most CT datasegmentation software operates on 2D interfaces， which limits flexibility for real-time adjustmentsin 3D segmentation. Here， we introduce Curves Mode in Drishti Paint 3.2， an open-source tool forCT data segmentation. Drishti Paint 3.2 allows users to manually or semi-automatically segmentthe CT data in both 2D and 3D environments， providing a novel solution for revisualizing CT datain paleontological studies.

Key words X-ray computed tomography （CT）， 2D and 3D segmentation， 3D reconstruction，Drishti Paint

Citation Wang M J， Limaye A， Lu J， 2024. Drishti Paint 3.2： a new open-source tool for both 2D and 3D segmentation. Vertebrata PalAsiatica， 62（4）： 313–320

1 Introduction

The past few decades have witnessed the transformation and application of industrial X-raycomputed tomography （CT） in the research of paleontology （Du et al.， 1997; Sutton et al.， 2014，2016; Kouraiss et al.， 2019; Yin and Lu， 2019）. CT data， generated from CT， micro-CT andsynchrotron radiation X-ray phase-contrast microtomography， enables researchers to acquireboth external and internal morphological information of fossils precisely and non-destructively（Sutton et al.， 2014; Rahman and Smith， 2015）. The morphological structures from CT data arevital in the present paleontological research， revealing a lot of anatomical details and helpingresearchers to have a better understanding of the development and preservation of the fossils（Cunningham et al.， 2014; Sutton et al.， 2014; Lautenschlager， 2016）. However， the CT data are a series of two-dimensional （2D） sections or slices （usually presented in image formatssuch as TIFF） and not able to show straightforward morphological structures （Buzug， 2011;Cunningham et al.， 2014; Sutton et al.， 2014）. Consequently， segmenting and reconstructingthe CT data forms the foundation for further morphological study. CT data segmentationestablishes Region of Interest （ROI） and separates the desired structure from the rest of thedataset （Lakare and Kaufman， 2000）. The accumulation of such work enables researchers tounderstand and diagnose the internal anatomical characters of 3D-preserved fossils. The 3Dmodels produced from fossil CT data segmentation play an important role in paleontologicalstudies， science communication and education （Cunningham et al.， 2014; Lautenschlagerand Rücklin， 2014; Lautenschlager， 2016）.

A large amount of commercial software has been developed and put into use to processCT data in paleontological research， covering the procedure of 3D reconstruction， segmentationand rendering， such as Mimics （materialize.com）， VG studio max （www.volumegraphics.com） and AVIZO （www.thermofisher.cn）. Nonetheless， there is a lack of interchangeabilityamong them， which creates barriers between researchers and 3D revisualization （Hu et al.，2020， 2024）. Many open-source software has been designed to deal with 3D segmentation.Among them， Drishti provides the most systematical service for the whole process of 3Drevisualization.

Drishti includes three major modules – Drishti Import， Drishti Render and DrishtiPaint， which cover the entire workflow in dealing with CT data， from data importing，segmentation， visualization， animation producing， etc. On its debut， Drishti presentedthe modules Render and Import （Limaye， 2012）. The professional segmentation moduleDrishti Paint was first released in 2013. During the past decade， lots of improvements andamendments have been implemented for a better and easier user experience （Hu et al.， 2020，2021， 2024）.

Drishti Paint is characterized by its real-time combination of 2D and 3D segmentation，and its flexibility in converting between these two scenarios. After successful segmentation，the volume data can be exported and applied to further surface or volume rendering in DrishtiRender.

We here introduce a new segmentation tool in the new version Drishti 3.2 – the CurvesMode used in segmenting 3D data in Drishti Paint， which simultaneously integrates 2D and3D segmentation of CT data. It is a practical， free and more efficient segmentation methodfor all kinds of CT data and has become a new choice for paleontological researchers.

2 Material and method

The CT data used here belongs to the skeleton specimen of an anterior proportion froma polypterus， Erpetoichthys， obtained by Micro 225 CT in 2010 in the Institute of Vertebrate Paleontology and Paleoanthropology， Chinese Academy of Sciences， Beijing， China （IVPP）.The data are available at ADMorph （http：//admorph.ivpp.ac.cn/fossilInfo.html？menuId= 49bce6f530444460b02118e87788ee62amp;id=ff80808182fda48f0182fdaae3ac0000）. Drishti Paint3.2 and the former releases of this set can be downloaded at GitHub （https：//github.com/nci/drishti）.

Installation of Drishti requires computers equipped with GPUs with at least OpenGL 3.3capability and a dedicated graphics card with at least 2 GB memory. Installation tutorials canbe found in the supplementary materials.

3 Tutorial for Drishti Paint 3.2

3.1 Import

Drishti Import supports 15 file formats， involving most of the mainstream CT dataformats so far， such as DICOM， NIFTI， RAW and standard image files like TIFF， PNG， etc. （seeTable S1）. Drishti Import provides preliminary cropping and adjusting functions （Fig. S3） andcan finally export files in Drishti format （.pvl.nc） （Fig. S4）， which can be smoothly loaded intoDrishti Render and Drishti Paint for further operation. Users can import Drishti file （.pvl.nc）into Drishti Paint through a conventional import process （Figs. 1， S8） as well as drag-and-dropaction. A detailed introduction to Drishti Import can be found in the supplementary materials.

3.2 Curves Mode

3.2.1 Core Features of Curves Mode

（1） Curves Mode in Drishti Paint 3.2 provides a real-time segmentation tool integrating"2D and 3D scenarios.

（2） The segmentation using Curves Mode can be either manual or semi-automatic.

（3） Two interpolation modes perfectly serve different interpolating conditions.

（4） The checkpoint function enables users to retrieve former working results if the"segmentation is not satisfying.

（5） This tool is open-source and continuously updated， users are welcome to provide"feedback to developers whenever coming up with new suggestions.

3.2.2 Tutorial of Curves Mode

After the files are successfully loaded， users can switch to the Curves Mode on the top ofthe working interface， with Graph Cut Mode in default （Figs. 1， S6）. Under this mode， userscan mark the region of interest utilizing creating new curves， interpolating the curves and thenbaking curves （connecting the region with specific tag colors） （Figs. 1， S5， S15）. The region ofinterest being tagged is successfully segmented from the other part of the CT data in this way（Fig. 2）.

Before creating new curves， it is advisable to figure out the appropriate appearance of a specific slice. Users can adjust the histogram and gradients on the right panel （Fig. S6）， whichare aligned with the threshold and contrast on the 2D working interfaces. It helps to reach asatisfactory segmentation more quickly.

Each time when creating new curves， users can click on the [New Curve] buttonon the left panel or press [C] on the keyboard （Figs. 1， S6）. Then users can mark theinterested regions using either regular mode （hand drawn curves） or semi-automaticlivewire mode （by turning on the livewire button） （Figs. 1， S6， S15）. The latter enablesusers to create new curves that follow high gradient ridges， which is usually the marginof a region to be segmented. After pressing [C]， users can click on the interface with theleft mouse to set a start point wherever suitable and then create new tracks along with themovement of the mouse （Fig. S16）. If the track is not suitable， the former operation can bewithdrawn by clicking the right mouse and new points can be set to provide amendment.To remove the whole curve through pressing [Delete] or [Backspace] is recommended ifthe curve is hard to amend. If the area of the curves cannot cover the suitable region orcover too much， users can enlarge or reduce the area by pressing [D] （meaning dilation）or [E] （meaning erosion）. If holding on to the shift key at the same time， the erosion ordilation actions would be employed on all existing curves. After users bake the curves favorably， all the curves can be deleted by clicking on the [Remove All] button （Figs. 1，S6）.

There are two ways of curve interpolation provided in Drishti Paint， DT （using signeddistance transform） and WM （using weighted means of strings method） （Figs. 1， S6）. Theformer one works when there are multiple curves in a slice. However， it does not work wellwhen the curves are farther apart and it is recommended to adopt the latter interpolate modein this case， but it performs better for single curves far apart. If the interpolated curves arenot satisfactory， they can be deleted when hovering one of them and then pressing [Delete].After that， users can draw a new curve for in-between slices to generate a better set ofinterpolated slices.

When the region of interest is confirmed， users can tag it through [Bake Curves] （Figs. 1，S15）. By choosing corresponding tagged color， different parts of the data are classified into onesection and by changing the visibility of different tagged colors， the segmentation is realized（Figs. 1， 2）.

Besides creating new curves in the 2D interfaces， users can tag the 3D model directly inthe Curves Mode， just as in Graph Cut Mode （Fig. S17）. Users should click at the tagged colorpanel to choose a wanted tagged color and then draw with the movement of the left mousewhen holding the shift key. Erasing can be carried out if users draw with the right mousebutton in the same situation. The 3D segmentation parameters can be modified in the 3DPreview panel （Figs. S6， S14， S17）.

There is a panel on the bottom left that can change parameters of 3D preview interface.When users want to confirm the location of a specific point， put a tick on the box before the[position]. Edges and shadows can also be changed for better performance （Figs. S8， S15）.

On the right side of the interface， the TF block is used for different segmentation workof the same set of data， enabling users to tag the same region into different parts， which iscomparable to setting a new filter （Figs. S6， S17）.

3.3 Export

After all the segmentation works have been finished， the tagged region can be extractedand exported as Drishti file （.pvl.nc） or as surface mesh， which can be used for furtherrendering and animation producing （Figs. S18， S19）.

If users have any questions about the operational panel and hotkeys when working withDrishti Paint 3.2， they can click on the [？] button to get brief instructions， further and moredetailed tutorials are accessible in the supplementary materials of this article.

4 Conclusion

As open-source software， Drishti provides a whole procedure of CT data editingand revisualization， offering great potential in low-cost and user-friendly choices of fossilrevisualization. Besides segmentation， Drishti can also be implemented into file formattransformation and surface mesh rendering， which are also important in the process of 3Ddata revisualization. In the new version of Drishti Paint 3.2， the integration of 2D and 3Dsegmentation has made it much easier to process CT data segmentation therefore reducingthe time and fee consumption in the related paleontological study. Except for paleontologicalresearch， Drishti Paint is also of great use in many other fields in need of CT scanning andsegmentation to reveal the external and internal structures， such as biological， medical andindustrial research.

Acknowledgement We are grateful to HOU Y.-M. for the CT scanning. This work wassupported by National Natural Science Foundation of China （42130209）.

摘要：計(jì)算機(jī)斷層掃描（CT）技術(shù)在近幾十年來(lái)被廣泛應(yīng)用于古生物學(xué)相關(guān)研究領(lǐng)域。研究者借助CT掃描技術(shù)可以無(wú)損獲得化石內(nèi)部精細(xì)信息，進(jìn)一步對(duì)材料進(jìn)行深入研究。但是，如何對(duì)CT數(shù)據(jù)進(jìn)行高效的三維分割與重建仍是研究者們?cè)谑褂萌S數(shù)據(jù)時(shí)面臨的主要問(wèn)題。大多數(shù)CT數(shù)據(jù)分割軟件都采用二維分割的傳統(tǒng)方法，難以直接對(duì)三維結(jié)果進(jìn)行實(shí)時(shí)修正。介紹了三維可視化開(kāi)源軟件Drishti Paint 3.2版本中的曲線模式（Curves Mode）及其使用方法。該軟件為研究者提供了同時(shí)在二維切片及三維窗口手動(dòng)或半自動(dòng)分割CT數(shù)據(jù)的新工具，為如何對(duì)古生物學(xué)研究中化石CT數(shù)據(jù)進(jìn)行高效的三維分割與重建提供了新思路。

關(guān)鍵詞：計(jì)算機(jī)斷層掃描（CT）， 二維與三維分割，三維重建，Drishti Paint

中圖法分類號(hào)：TP391.4 文獻(xiàn)標(biāo)識(shí)碼：A 文章編號(hào)：2096–9899（2024）04–0313–08

References

Buzug T M， 2011. Computed tomography. In： Kramme R， Hoffmann K P， Pozos R S eds. Springer Handbook of Medical"Technology. Berlin， Heidelberg： Springer Berlin Heidelberg. 311?342

Cunningham J A， Rahman I A， Lautenschlager S et al.， 2014. A virtual world of paleontology. Trends Ecol Evol， 29：347?357

Du X K， Zhang F K， Lu J C et al.， 1997. Computed tomography in research on paleontology. J Beijing Med Univ， 29：46?47

Hu Y Z， Limaye A， Lu J， 2020. Three-dimensional segmentation of computed tomography data using Drishti Paint： new tools and developments. R Soc Open Sci， 7： 201033

Hu Y Z， Limaye A， Lu J， 2021. A 3D segmentation amp; reconstruction method in paleontology using Drishti. Bio-101：e1010668

Hu Y Z， Limaye A， Lu J， 2024. 3D revisualisation： a new method to revisit segmented data. R Soc Open Sci， 11：240375

Kouraiss K， El Hariri K， El Albani A et al.， 2019. Digitization of fossils from the Fezouata Biota （Lower Ordovician，Morocco）： evaluating computed tomography and photogrammetry in collection enhancement. Geoheritage， 11：1889?1901

Lakare S， Kaufman A， 2000. 3D segmentation techniques for medical volumes. Center for Visual Computing， Department of"Computer Science， State University of New York， 2000： 59?68

Lautenschlager S， 2016. Reconstructing the past： methods and techniques for the digital restoration of fossils. R Soc Open"Sci， 3： 160342

Lautenschlager S， Rücklin M， 2014. Beyond the print—virtual paleontology in science publishing， outreach， and education.J Paleontol， 88： 727?734

Limaye A， 2012. Drishti： a volume exploration and presentation tool. SPIE Proc， 8506： 191?199

Rahman I A， Smith S Y， 2015. Virtual paleontology： computer-aided analysis of fossil form and function. J Paleontol， 88：633?635

Sutton M， Rahman I， Garwood R， 2014. Techniques for Virtual Palaeontology. New Jersey： John Wiley amp; Sons， Inc.1?200

Sutton M， Rahman I， Garwood R， 2016. Virtual paleontology—an overview. Paleontol Soc Pap， 22： 1?20

Yin Z J， Lu J， 2019. Virtual palaeontology： when fossils illuminated by X-ray. Palaeoworld， 28： 425?428

國(guó)家自然科學(xué)基金（批準(zhǔn)號(hào)：42130209）資助。