MedEye3d.jl

MedEye3d

Main goal of the package is conviniently visualize 3d medical imaging to make segmentation simpler

Some oficial introduction - you can skip it

Image segmentation in the medical domain has mul-tiple use cases. Most importantly it enables delin-eation of physiological and pathological structures,in order to confirm or reject some diagnostic hy-pothesis. In case of all segmentation problems, avery important step in evaluation of the segmen-tation algorithm output is visual inspection. Suchinspection enables researchers that are responsiblefor creating and or evaluating developed algorithmsto easily spot problems, and compare different algo-rithms in more detailed ways than can be achievedby usage of segmentation metrics alone. Howeverin order for such in development visual evaluationto be useful it needs to meet some usage criteria.It needs to be easily integrable to the program-ming language and libraries used by researchers.Performance of the tool must be adequate inorder to suit the iterative process of algorithm de-velopment and refinement.Representation accuracy must be sufficient forthe task at hand. It should not require an exces-sive amount of computational resources, in order tominimize its influence on running algorithms.Support for in memory data structures (arrays)should be convenient. Needs to provide possibility of simple manualannotations, on given mask and ability to controlvisibility and visual representation of given mask.Should provide also the possibility to display somemetadata in text format like segmentation metricsfor example DICE score.Ideally it should be also open source and welldocumented in order to enable users to modify itaccording to the task at hand.In order to address all of those issues in themedical domain and Julia language ecosystem thedescribed below package was developed.

Image below just represents limitless possibilities of color ranges, and that thanks to OpenGl even theorethically complex data to display will render nearly instantenously.

image

Below the functionality of the package will be described on the basis of some examples In case of any questions, feature requests, or propositions of cooperation post them here on Github or contact me via LinkedIn linkedin.com/in/jakub-mitura-7b2013151

Defining Helper Functions and imports

#I use Simple ITK as most robust
using NuclearMedEye, Conda,PyCall,Pkg

Conda.pip_interop(true)
Conda.pip("install", "SimpleITK")
Conda.pip("install", "h5py")
sitk = pyimport("SimpleITK")
np= pyimport("numpy")

import NuclearMedEye
import NuclearMedEye.ForDisplayStructs
import NuclearMedEye.ForDisplayStructs.TextureSpec
using ColorTypes
import NuclearMedEye.SegmentationDisplay

import NuclearMedEye.DataStructs.ThreeDimRawDat
import NuclearMedEye.DataStructs.DataToScrollDims
import NuclearMedEye.DataStructs.FullScrollableDat
import NuclearMedEye.ForDisplayStructs.KeyboardStruct
import NuclearMedEye.ForDisplayStructs.MouseStruct
import NuclearMedEye.ForDisplayStructs.ActorWithOpenGlObjects
import NuclearMedEye.OpenGLDisplayUtils

Helper functions used to upload data - those will be enclosed (with many more) in a package that Is currently in development - 3dMedPipe

"""
given directory (dirString) to file/files it will return the simple ITK image for futher processing
isMHD when true - data in form of folder with dicom files
isMHD  when true - we deal with MHD data
"""
function getImageFromDirectory(dirString,isMHD::Bool, isDicomList::Bool)
    #simpleITK object used to read from disk 
    reader = sitk.ImageSeriesReader()
    if(isDicomList)# data in form of folder with dicom files
        dicom_names = reader.GetGDCMSeriesFileNames(dirString)
        reader.SetFileNames(dicom_names)
        return reader.Execute()
    elseif(isMHD) #mhd file
        return sitk.ReadImage(dirString)
    end
end#getPixelDataAndSpacing

"""
becouse Julia arrays is column wise contiguus in memory and open GL expects row wise we need to rotate and flip images 
pixels - 3 dimensional array of pixel data 
"""
function permuteAndReverse(pixels)
    pixels=  permutedims(pixels, (3,2,1))
    sizz=size(pixels)
    for i in 1:sizz[1]
        pixels[i,:,:] =  reverse(pixels[i,:,:])
    end# 
  
    for i in 1:sizz[2]
    pixels[:,i,:] =  reverse(pixels[:,i,:])
    end# 
    return pixels
  end#permuteAndReverse

"""
given simple ITK image it reads associated pixel data - and transforms it by permuteAndReverse functions
it will also return voxel spacing associated with the image
"""
function getPixelsAndSpacing(image)
    pixelsArr = np.array(sitk.GetArrayViewFromImage(image))# we need numpy in order for pycall to automatically change it into julia array
    spacings = image.GetSpacing()
    return ( permuteAndReverse(pixelsArr), spacings  )
end#getPixelsAndSpacing

Directories - obviously you need to provide path to place where it is stored on your disk. You can download PET/CT data from. You can download example data from https://wwsi365-my.sharepoint.com/:f:/g/personal/s9956jmmswwsiedupl/EstYmEuRHqZNlFIlPBzhbQIBvMwQBJks2lUcCSWgwCYSOg?e=nfW95Q

# directories of PET/CT Data - from https://wiki.cancerimagingarchive.net/display/Public/Head-Neck-PET-CT
dirOfExample ="C:\\GitHub\\JuliaMedPipe\\data\\PETphd\\slicerExp\\all17\\bad17NL-bad17NL\\20150518-PET^1_PET_CT_WholeBody_140_70_Recon (Adult)\\4-CT AC WB  1.5  B30f"
dirOfExamplePET ="C:\\GitHub\\JuliaMedPipe\\data\\PETphd\\slicerExp\\all17\\bad17NL-bad17NL\\20150518-PET^1_PET_CT_WholeBody_140_70_Recon (Adult)\\3-PET WB"

# in most cases dimensions of PET and CT data arrays will be diffrent in order to make possible to display them we need to resample and make dimensions equal
imagePET= getImageFromDirectory(dirOfExamplePET,false,true)
ctImage= getImageFromDirectory(dirOfExample,false,true)
pet_image_resampled = sitk.Resample(imagePET, ctImage)

ctPixels, ctSpacing = getPixelsAndSpacing(ctImage)
# In my case PET data holds 64 bit floats what is unsupported by Opengl
petPixels, petSpacing =getPixelsAndSpacing(pet_image_resampled)
petPixels = Float32.(petPixels)

# we need to pass some metadata about image array size and voxel dimensions to enable proper display
datToScrollDimsB= NuclearMedEye.ForDisplayStructs.DataToScrollDims(imageSize=  size(ctPixels) ,voxelSize=ctSpacing, dimensionToScroll = 3 );
# example of texture specification used - we need to describe all arrays we want to display, to see all possible configurations look into TextureSpec struct docs .
textureSpecificationsPETCT = [
  TextureSpec{Float32}(
      name = "PET",
      isNuclearMask=true,
      # we point out that we will supply multiple colors
      isContinuusMask=true,
      #by the number 1 we will reference this data by for example making it visible or not
      numb= Int32(1),
      colorSet = [RGB(0.0,0.0,0.0),RGB(1.0,1.0,0.0),RGB(1.0,0.5,0.0),RGB(1.0,0.0,0.0) ,RGB(1.0,0.0,0.0)]
      #display cutoff all values below 200 will be set 2000 and above 8000 to 8000 but only in display - source array will not be modified
      ,minAndMaxValue= Float32.([200,8000])
     ),
  TextureSpec{UInt8}(
      name = "manualModif",
      numb= Int32(2),
      color = RGB(0.0,1.0,0.0)
      ,minAndMaxValue= UInt8.([0,1])
      ,isEditable = true
     ),

     TextureSpec{Int16}(
      name= "CTIm",
      numb= Int32(3),
      isMainImage = true,
      minAndMaxValue= Int16.([0,100]))  
];
# We need also to specify how big part of the screen should be occupied by the main image and how much by text fractionOfMainIm= Float32(0.8);
fractionOfMainIm= Float32(0.8);
"""
If we want to display some text we need to pass it as a vector of SimpleLineTextStructs - utility function to achieve this is 
textLinesFromStrings() where we pass resies of strings, if we want we can also edit those structs to controll size of text and space to next line look into SimpleLineTextStruct doc
mainLines - will be displayed over all slices
supplLines - will be displayed over each slice where is defined - below just dummy data
"""
import NuclearMedEye.DisplayWords.textLinesFromStrings

mainLines= textLinesFromStrings(["main Line1", "main Line 2"]);
supplLines=map(x->  textLinesFromStrings(["sub  Line 1 in $(x)", "sub  Line 2 in $(x)"]), 1:size(ctPixels)[3] );

If we want to pass 3 dimensional array of scrollable data we need to supply it via vector ThreeDimRawDat's struct utility function to make creation of those easier is getThreeDims which creates series of ThreeDimRawDat from list of tuples where first entry is String and second entry is 3 dimensional array with data strings needs to be the same as we defined in texture specifications at the bagining data arrays needs to be o the same size and be of the same type we specified in texture specification

import NuclearMedEye.StructsManag.getThreeDims

tupleVect = [("PET",petPixels) ,("CTIm",ctPixels),("manualModif",zeros(UInt8,size(petPixels)) ) ]
slicesDat= getThreeDims(tupleVect )
"""
Holds data necessary to display scrollable data
"""
mainScrollDat = FullScrollableDat(dataToScrollDims =datToScrollDimsB
                                 ,dimensionToScroll=1 # what is the dimension of plane we will look into at the beginning for example transverse, coronal ...
                                 ,dataToScroll= slicesDat
                                 ,mainTextToDisp= mainLines
                                 ,sliceTextToDisp=supplLines );

This function prepares all for display; 1000 in the end is responsible for setting window width for more look into SegmentationDisplay.coordinateDisplay

SegmentationDisplay.coordinateDisplay(textureSpecificationsPETCT ,fractionOfMainIm ,datToScrollDimsB ,1000);

As all is ready we can finally display image

Main.SegmentationDisplay.passDataForScrolling(mainScrollDat);

So after invoking this function one should see image sth like below

image

Interactions

Next all Interactions are done either by mouse or by keyboard shortcuts

left click and drag - will mark active texture (look below - set with alt ...) if it is set to be modifiable in the texture specifications, to the set value and size (by tab...) right click and drag - sets remembered position - when we will change plane of crossection for example from tranverse to coonal this point will be also visible on new plane

all keyboard shortcuts will be activated on RELEASE of keys or by pressing enter while still pressing other; +,- and z keys acts also like enter

shift + number - make mask associated with given number visible

ctrl + number - make mask associated with given number invisible

alt + number - make mask associated with given number active for mouse interaction

tab + number - sets the number that will be used as an input to masks modified by mouse

when tab plus (and then no number) will be pressed it will increase stroke width

when tab minus (and then no number) will be pressed it will increase stroke width

shift + numberA + "-"(minus sign) +numberB - display diffrence between masks associated with numberA and numberB - also it makes automaticall mask A and B invisible

ctrl + numberA + "-"(minus sign) +numberB - stops displaying diffrence between masks associated with numberA and numberB - also it makes automaticall mask A and B visible

space + 1 or 2 or 3 - change the plane of view (transverse, coronal, sagittal)

ctrl + z - undo last action

tab +/- increase or decrease stroke width

F1 - will display wide window for bone Int32(1000),Int32(-1000)

F2 - will display window for soft tissues Int32(400),Int32(-200)

F3 - will display wide window for lung viewing Int32(0),Int32(-1000)

F4, F5 sets minimum (F4) and maximum (KEY_F5) value for display (with combination of + and minus signs - to increase or decrease given treshold) -

In case of continuus colors it will clamp values - so all above max will be equaled to max ; and min if smallert than min

In case of main CT mask - it will controll min shown white and max shown black

In case of maks with single color associated we will step data so if data is outside the rande it will return 0 - so will not affect display F6 - controlls contribution of given mask to the overall image - maximum value is 1 minimum 0 if we have 3 masks and all control contribution is set to 1 and all are visible their corresponding influence to pixel color is 33% if plus is pressed it will increse contribution by 0.1 if minus is pressed it will decrease contribution by 0.1

Benchmark PET/CT

For transparency I include Below code used to benchark PET/CT data

window = Main.SegmentationDisplay.mainActor.actor.mainForDisplayObjects.window
syncActor = Main.SegmentationDisplay.mainActor

using GLFW,DataTypesBasic, ModernGL,Setfield
using BenchmarkTools

BenchmarkTools.DEFAULT_PARAMETERS.samples = 100
BenchmarkTools.DEFAULT_PARAMETERS.seconds =5000
BenchmarkTools.DEFAULT_PARAMETERS.gcsample = true


function toBenchmarkScroll(toSc) 
    NuclearMedEye.ReactToScroll.reactToScroll(toSc ,syncActor, false)
end


function toBenchmarkPaint(carts)
    NuclearMedEye.ReactOnMouseClickAndDrag.reactToMouseDrag(MouseStruct(true,false, carts),syncActor )
end


function toBenchmarkPlaneTranslation(toScroll)
    NuclearMedEye.ReactOnKeyboard.processKeysInfo(Option(toScroll),syncActor,KeyboardStruct(),false    )
    OpenGLDisplayUtils.basicRender(syncActor.actor.mainForDisplayObjects.window)
    glFinish()
end


function prepareRAndomCart(randInt) 
    return [CartesianIndex(12+randInt,13+randInt),CartesianIndex(12+randInt,15+randInt),CartesianIndex(12+randInt,18+randInt),CartesianIndex(2+randInt,10+randInt),CartesianIndex(2+randInt,14+randInt)]
end

#we want some integers but not 0
sc = @benchmarkable toBenchmarkScroll(y) setup=(y = filter(it->it!=0, rand(-5:5,20))[1]  )

paint =  @benchmarkable toBenchmarkPaint(y) setup=(y =  prepareRAndomCart(rand(1:40,1)[1]  ))  

translations =  @benchmarkable toBenchmarkPlaneTranslation(y) setup=(y = setproperties(syncActor.actor.onScrollData.dataToScrollDims,  (dimensionToScroll=rand(1:3,2)[1])) )  

using BenchmarkPlots, StatsPlots
# Define a parent BenchmarkGroup to contain our suite

scrollingPETCT = run(sc)
mouseInteractionPETCT = run(paint)
translationsPETCT = run(translations)


plot(scrollingPETCT)

If all will be ready you should see sth like on the image below

PURE CT image exaple , MHD file

Files taken from https://sliver07.grand-challenge.org/ As previosly adjust path to your case

exampleLabel = "C:\\GitHub\\JuliaMedPipe\\data\\liverPrimData\\training-labels\\label\\liver-seg002.mhd"
exampleCTscann = "C:\\GitHub\\JuliaMedPipe\\data\\liverPrimData\\training-scans\\scan\\liver-orig002.mhd"

Loading data

imagePureCT= getImageFromDirectory(exampleCTscann,true,false)
imageMask= getImageFromDirectory(exampleLabel,true,false)

ctPixelsPure, ctSpacingPure = getPixelsAndSpacing(imagePureCT)
maskPixels, maskSpacing =getPixelsAndSpacing(imageMask)

We need to pass some metadata about image array size and voxel dimensions to enable proper display

datToScrollDimsB= NuclearMedEye.ForDisplayStructs.DataToScrollDims(imageSize=  size(ctPixelsPure) ,voxelSize=ctSpacingPure, dimensionToScroll = 3 );
# example of texture specification used - we need to describe all arrays we want to display
listOfTexturesSpec = [
    TextureSpec{UInt8}(
        name = "goldStandardLiver",
        numb= Int32(1),
        color = RGB(1.0,0.0,0.0)
        ,minAndMaxValue= Int8.([0,1])
       ),
    TextureSpec{UInt8}(
        name = "manualModif",
        numb= Int32(2),
        color = RGB(0.0,1.0,0.0)
        ,minAndMaxValue= UInt8.([0,1])
        ,isEditable = true
       ),

       TextureSpec{Int16}(
        name= "CTIm",
        numb= Int32(3),
        isMainImage = true,
        minAndMaxValue= Int16.([0,100]))  
 ];

We need also to specify how big part of the screen should be occupied by the main image and how much by text fractionOfMainIm= Float32(0.8);

fractionOfMainIm= Float32(0.8);
"""
If we want to display some text we need to pass it as a vector of SimpleLineTextStructs 
"""
import NuclearMedEye.DisplayWords.textLinesFromStrings

mainLines= textLinesFromStrings(["main Line1", "main Line 2"]);
supplLines=map(x->  textLinesFromStrings(["sub  Line 1 in $(x)", "sub  Line 2 in $(x)"]), 1:size(ctPixelsPure)[3] );

"""
If we want to pass 3 dimensional array of scrollable data"""
import NuclearMedEye.StructsManag.getThreeDims

tupleVect = [("goldStandardLiver",maskPixels) ,("CTIm",ctPixelsPure),("manualModif",zeros(UInt8,size(ctPixelsPure)) ) ]
slicesDat= getThreeDims(tupleVect )

"""
Holds data necessary to display scrollable data
"""
mainScrollDat = FullScrollableDat(dataToScrollDims =datToScrollDimsB
                                 ,dimensionToScroll=1 # what is the dimension of plane we will look into at the beginning for example transverse, coronal ...
                                 ,dataToScroll= slicesDat
                                 ,mainTextToDisp= mainLines
                                 ,sliceTextToDisp=supplLines );

This function prepares all for display; 1000 in the end is responsible for setting window width for more look into SegmentationDisplay.coordinateDisplay

SegmentationDisplay.coordinateDisplay(listOfTexturesSpec ,fractionOfMainIm ,datToScrollDimsB ,1000);

As all is ready we can finally display image

Main.SegmentationDisplay.passDataForScrolling(mainScrollDat);

Next part of benchmark for pure CT

#we want some integers but not 0
scPureCt = @benchmarkable toBenchmarkScroll(y) setup=(y = filter(it->it!=0, rand(-5:5,20))[1]  )

paintPureCt =  @benchmarkable toBenchmarkPaint(y) setup=(y =  prepareRAndomCart(rand(1:40,1)[1]  ))  

translationsPureCt =  @benchmarkable toBenchmarkPlaneTranslation(y) setup=(y = setproperties(syncActor.actor.onScrollData.dataToScrollDims,  (dimensionToScroll=rand(1:3,2)[1])) )  

using BenchmarkPlots, StatsPlots



scB = @benchmarkable toBenchmarkScroll(y) setup=(y = filter(it->it!=0, rand(-5:5,20))[1]  )

paintB =  @benchmarkable toBenchmarkPaint(y) setup=(y =  prepareRAndomCart(rand(1:40,1)[1]  ))  

translationsB =  @benchmarkable toBenchmarkPlaneTranslation(y) setup=(y = setproperties(syncActor.actor.onScrollData.dataToScrollDims,  (dimensionToScroll=rand(1:3,2)[1])) )  

using BenchmarkPlots, StatsPlots
# Define a parent BenchmarkGroup to contain our suite

scrollingPureCT = run(scB)
mouseInteractionPureCT = run(paintB)
translationsPureCT = run(translationsB)

When all will be ok and you will scroll up you should see sth like below

soft_liv

MedEye3d.SegmentationDisplay.cleanUpMethod

In order to properly close displayer we need to : remove buffers that wer use remove shaders remove all textures unsubscibe all of the subscriptions to the mainActor finalize main actor and reinstantiate it close GLFW window

source
MedEye3d.SegmentationDisplay.coordinateDisplayFunction

coordinating displaying - sets needed constants that are storeds in forDisplayConstants; and configures interactions from GLFW events listOfTextSpecs - holds required data needed to initialize textures keeps also references to needed ..Uniforms etc. windowWidth::Int,windowHeight::Int - GLFW window dimensions fractionOfMainIm - how much of width should be taken by the main image heightToWithRatio - needed for proper display of main texture - so it would not be stretched ...

source
MedEye3d.SegmentationDisplay.passDataForScrollingMethod

is used to pass into the actor data that will be used for scrolling onScrollData - struct holding between others list of tuples where first is the name of the texture that we provided and second is associated data (3 dimensional array of appropriate type)

source
MedEye3d.SegmentationDisplay.prepareForDispStructFunction

Preparing ForWordsDispStruct that will be needed for proper displaying of texts numberOfActiveTextUnits - number of textures already used - so we we will know what is still free fragmentshaderwords - reference to fragment shader used to display text vbowords - vertex buffer object used to display words shaderprogram_words - shader program associated with displaying text widthh, heightt - size of the texture - the bigger the higher resolution, but higher computation cost

return prepared for displayStruct

source
MedEye3d.SegmentationDisplay.updateSingleImagesDisplayedMethod

enables updating just a single slice that is displayed - do not change what will happen after scrolling one need to pass data to actor in listOfDataAndImageNames - struct holding tuples where first entry in tuple is name of texture given in the setup and second is 2 dimensional aray of appropriate type with image data sliceNumber - the number to which we set slice in order to later start scrolling the scroll data from this point

source
MedEye3d.ReactingToInput.setUpForScrollDataMethod

adding the data about 3 dimensional arrays that will be source of data used for scrolling behaviour onScroll Data - list of tuples where first is the name of the texture that we provided and second is associated data (3 dimensional array of appropriate type)

source
MedEye3d.ReactingToInput.subscribeGLFWtoActorMethod

when GLFW context is ready we need to use this function in order to register GLFW events to Rocket actor - we use subscription for this actor - Roctet actor that holds objects needed for display like window etc... return list of subscriptions so if we will need it we can unsubscribe

source
MedEye3d.ReactingToInput.updateSingleImagesDisplayedSetUpMethod

enables updating just a single slice that is displayed - do not change what will happen after scrolling one need to pass data to actor in struct that holds tuple where first entry is -vector of tuples whee first entry in tuple is name of texture given in the setup and second is 2 dimensional aray of appropriate type with image data

  • Int - second is Int64 - that is marking the screen number to which we wan to set the actor state
source
MedEye3d.ReactOnKeyboard.findTextureBasedOnNumbMethod

given number from keyboard input it return array With texture that holds the texture specification we are looking for listOfTextSpecifications - list with all registered Texture specifications numb - string that may represent number - if it does not function will return empty option return Option - either Texture specification or empty Option

source
MedEye3d.ReactOnKeyboard.parseStringMethod

Given string it parser it to given object on the basis of with and multiple dispatch futher actions will be done it checks each character weather is numeric - gets substring of all numeric characters and parses it into integer listOfTextSpecifications - list with all registered Texture specifications return option of diffrent type depending on input

source
MedEye3d.ReactOnKeyboard.reactToKeyboardMethod

Given keyInfo struct wit information about pressed keys it can process them to make some actions - generally activating keyboard shortcuts shift + number - make mask associated with given number visible ctrl + number - make mask associated with given number invisible alt + number - make mask associated with given number active for mouse interaction tab + number - sets the number that will be used as an input to masks modified by mouse shift + numberA + "m" +numberB - display diffrence between masks associated with numberA and numberB - also it makes automaticall mask A and B invisible ctrl + numberA + "m" +numberB - stops displaying diffrence between masks associated with numberA and numberB - also it makes automaticall mask A and B visible space + 1 or 2 or 3 - change the plane of view (transverse, coronal, sagittal) ctrl + z - undo last action tab +/- increase or decrease stroke width F1, F2 ... - switch between defined window display characteristics - like min shown white and mx shown black ...

source
MedEye3d.ForDisplayStructs.KeyboardCallbackSubscribableMethod

given pressed keys lik 1-9 and all letters resulting key is encoded as string and will be passed here handler object responsible for capturing action str - name of key lik 1,5 f,.j ... but not ctrl shift etc action - for example key press or release scancode - if key do not have short name like ctrl ... it has scancode

source
MedEye3d.ReactOnMouseClickAndDragModule

module code adapted from https://discourse.julialang.org/t/custom-subject-in-rocket-jl-for-mouse-events-from-glfw/65133/3 it is design to help processing data from -GLFW.SetCursorPosCallback(window, (, x, y) -> println("cursor: x, y")) and for example : cursor: 29.0, 469.0 types Float64 Float64 -GLFW.SetMouseButtonCallback(window, (, button, action, mods) -> println("button action")) for example types MOUSEBUTTON1 PRESS GLFW.MouseButton GLFW.Action The main function is to mark the interaction of the mouse to be saved in appropriate mask and be rendered onto the screen so we modify the data that is the basis of the mouse interaction mask and we pass the data on so appropriate part of the texture would be modified to be displayed on screen

source
MedEye3d.ReactOnMouseClickAndDrag.reactToMouseDragMethod

we use mouse coordinate to modify the texture that is currently active for modifications - we take information about texture currently active for modifications from variables stored in actor from texture specification we take also its id and its properties ...

source
MedEye3d.ReactOnMouseClickAndDrag.registerMouseClickFunctionsMethod

we pass coordinate of cursor only when isLeftButtonDown is true and we make it true if left button is presed down - we make it true if the left button is pressed over image and false if mouse get out of the window or we get information about button release imageWidth adn imageHeight are the dimensions of textures that we use to display

source
MedEye3d.ReactOnMouseClickAndDrag.translateMouseToTextureMethod

given list of cartesian coordinates and some window/ image characteristics - it translates mouse positions to cartesian coordinates of the texture strokeWidth - the property connected to the texture marking how thick should be the brush mouseCoords - list of coordinates of mouse positions while left button remains pressed calcDims - set of values usefull for calculating mouse position return vector of translated cartesian coordinates

source
MedEye3d.ForDisplayStructs.MouseCallbackSubscribableMethod

we define how handler should act on the subject - observable so it will pass event onto subject - here we have 2 events that we want to be ready for - mouse button press example of possible inputs that we would be intrested in for example : cursor: 29.0, 469.0 types Float64 Float64 for example MOUSEBUTTON1 PRESS types GLFW.MouseButton GLFW.Action MOUSEBUTTON1 RELEASE types GLFW.MouseButton GLFW.Action We get two overloads so we will be able to respond with single handler to both mouse click and mouse position Enum GLFW.Action: RELEASE = 0 PRESS = 1 REPEAT = 2 Enum GLFW.MouseButton: MOUSEBUTTON1 = 0 MOUSEBUTTON2 = 1

experiments show that max x,y in window is both 600 if window width and height is 600 so in order to specify weather we are over aour quad we need to know how big is primary quad - defaoul it is occupying 100% of y axis and first left 80% of x axis hence we can calculate max height to equal the height of the window

source
MedEye3d.ReactToScrollModule

module that holds functions needed to react to scrolling Generally first we need to pass the GLFW callback to the Rocket obeservable code adapted from https://discourse.julialang.org/t/custom-subject-in-rocket-jl-for-mouse-events-from-glfw/65133/3

source
MedEye3d.ReactToScroll.reactToScrollFunction

in case of the scroll p true will be send in case of down - false in response to it it sets new screen int variable and changes displayed screen toBeSavedForBack - just marks weather we wat to save the info how to undo latest action

  • false if we invoke it from undoing
source
MedEye3d.ReactToScroll.registerMouseScrollFunctionsMethod

uploading data to given texture; of given types associated returns subscription in order to enable unsubscribing in the end window - GLFW window stopListening - atomic boolean able to stop the event listening cycle return scrollback - that holds boolean subject (observable) to which we can react by subscribing appropriate actor

source
Rocket.on_subscribe!Method

configuting Rocket on Subscribe so we get custom handler of input as we see we still need to define actor

source
MedEye3d.PrepareWindow.displayAllMethod

preparing all for displaying the images and responding to mouse and keyboard input listOfTexturesToCreate- list of texture specifications needed to for example create optimal shader calcDimsStruct - holds important data about verticies, textures dimensions etc.

source
MedEye3d.TextureManag.activateTexturesMethod

activating textures that were already initialized in order to be able to use them with diffrent shader program shader_program- regference to OpenGL program so we will be able to activate textures listOfTextSpecs - list of TextureSpec structs that holds data needed to bind textures to shader program (Hovewer this new shader program have to keep the same ..Uniforms) return unmodified textures

source
MedEye3d.TextureManag.addTextToTextureMethod

Given vector of SimpleLineTextStructs it will return matrix of data that will be used to display text lines - data about text to be displayed calcDimStruct - struct holding important data about size of textures etc. wordsDispObj - object wit needed constants to display text

source
MedEye3d.TextureManag.assignUniformsAndTypesToMasksMethod

on the basis of the type supplied in texture characteristic it supplies given set of ..Uniforms to it It would also assign proper openGl types to given julia data type, and pass data from texture specification to opengl context textSpecs - list of texture specificaton that we want to enrich by adding information about ..Uniforms return list of texture specifications enriched by information about ..Uniforms

source
MedEye3d.TextureManag.createTextureMethod

creating texture that is storing values like integer, uint, float values that are representing main image or mask data and which will be used by a shader to draw appropriate colors juliaDataType- data type defined as a Julia datatype of data we are dealing with width,height - dimensions of the texture that we need GL_RType,OpGlType - Open Gl types needed to properly specify the texture they need to be compatible with juliaDataType

source
MedEye3d.TextureManag.initializeTexturesMethod

initializing textures - so we basically execute specification for configuration and bind all to OpenGL context listOfTextSpecs - list of TextureSpec structs that holds data needed to calcDimStruct - struct holding necessery data about taxture dimensions, quad propertiess etc.

source
MedEye3d.TextureManag.setProperOpenGlTypesMethod

On the basis of the type associated to texture we set proper open Gl types associated based on https://www.khronos.org/registry/OpenGL-Refpages/gl4/html/glTexImage2D.xhtml and https://www.khronos.org/opengl/wiki/OpenGL_Type

source
MedEye3d.TextureManag.setuniformsMethod

helper for assign..UniformsToMasks On the basis of the name of the Texture it will assign the informs referencs to it

  • ..Uniforms for main image will be set separately
source
MedEye3d.TextureManag.updateImagesDisplayedMethod

coordinating updating all of the images, masks... singleSliceDat - holds data we want to use for update forDisplayObjects - stores all needed constants that holds reference to GLFW and OpenGL

source
MedEye3d.TextureManag.updateTextureMethod

uploading data to given texture; of given types associated - specified in TextureSpec if we want to update only part of the texture we need to specify offset and size of texture we use Just for reference openGL function definition void glTextureSubImage2D( GLuint texture, GLint level, GLint xoffset, GLint yoffset, GLsizei width, GLsizei height, GLenum format, GLenum type, const void *pixels);

source
MedEye3d.DisplayWordsModule

Module controlling displaying of the text associated with the segmentation

  • either text releted to all slices or just a single one currently displayed or both
source
MedEye3d.DisplayWords.activateForTextDispMethod

In order to be able to display texture with text we need to activate main shader program and vbo shaderprogram- reference to shader program fragmentshader_words - reference to shader associated with text displaying calcDim - holds necessery constants holding for example window dimensions, texture sizes etc.

source
MedEye3d.DisplayWords.bindAndActivateForTextMethod

First We need to bind fragment shader created to deal with text and supply the vertex shader with data for quad where this text needs to be displayed shader_program- reference to shader program this function is intended to be invoked only once

source
MedEye3d.DisplayWords.createTextureForWordsFunction

Creates and initialize texture that will be used for displaying text !!!! important we need to first bind shader program for text display before we will invoke this function numberOfActiveTextUnits - number of textures already used - so we we will know what is still free widthh, heightt - size of the texture - the bigger the higher resolution, but higher computation cost actTextrureNumb -proper OpenGL active texture return fully initialized texture; also it assigne texture to appropriate sampler

source
MedEye3d.DisplayWords.reactivateMainObjMethod

Finally in order to enable later proper display of the images we need to reactivate main quad and shaders shaderprogram- reference to shader program fragmentshader_main- reference to shader associated with main images

source
MedEye3d.DisplayWords.renderSingleLineOfTextMethod

Given single SimpleLineTextStruct it will return matrix of data that will be used by addTextToTexture function to display text texLine - source od data textureWidth - available width for a line fontFace - font we use

source
MedEye3d.Uniforms.changeTextureContributionMethod

controlls contribution of given mask to the overall image - maximum value is 1 minimum 0 if we have 3 masks and all control contribution is set to 1 and all are visible their corresponding influence to pixel color is 33% if plus is pressed it will increse contribution by 0.1 if minus is pressed it will decrease contribution by 0.1 it also modifies given TextureSpec change - how should the texture spec be modified

source
MedEye3d.Uniforms.coontrolMinMaxUniformValsMethod

sets minimum and maximum value for display - in case of continuus colors it will clamp values - so all above max will be equaled to max ; and min if smallert than min in case of main CT mask - it will controll min shown white and max shown black in case of maks with single color associated we will step data so if data is outside the rande it will return 0 - so will not affect display

source
MedEye3d.Uniforms.setCTWindowMethod

function cotrolling the window for displaying CT scan - min white and max maxshownblack uniformsStore - instantiated object holding references to uniforms controlling displayed window

source
MedEye3d.Uniforms.uniform!Function
    uniform!(location, values...) -> Nothing

Set the uniform vec variable at location.

Examples

myuniform = getuniform(program, "someUniform")
# set float vec3
uniform!(myuniform, 1.0, 1.0, 0.5)
# set float vec4
uniform!(myuniform, Cfloat[1, 2, 3, 4])
# set a 4x4 matrix
uniform!(myuniform, rand(Cfloat, 4, 4))
source
MedEye3d.CustomFragShadModule

functions that will enable creation of long String that will be the code for custom fragment shader that will be suited for defined textures

source
MedEye3d.CustomFragShad.createCustomFramgentShaderMethod

We will in couple steps create code for fragment shader that will be based on the texture definitions we gave listOfTexturesToCreate - list of textures on the basis of which we will create custom fragment shader code maskToSubtrastFrom,maskWeAreSubtracting - texture specifications used in order to generate code needed to diplay diffrence between those two masks - every time we want diffrent diffrence we will need to recreate shader invoking this function

source
MedEye3d.CustomFragShad.divideTexteuresToMainAndRestMethod

We divide textures into main image texture and the rest listOfTexturesToCreate - list of textures on the basis of which we will create custom fragment shader code returns tuple where fist entr is the main image texture specification and second is rest of textures

source
MedEye3d.CustomFragShad.getMasksSubtractionFunctionMethod

used in order to enable subtracting one mask from the other - hence displaying pixels where value of mask a is present but mask b not (order is important) automatically both masks will be set to be invisible and only the diffrence displayed

In order to provide maximum performance and avoid branching inside shader multiple shader programs will be attached and one choosed that will use diffrence needed maskToSubtrastFrom,maskWeAreSubtracting - specifications o textures we are operating on

source
MedEye3d.CustomFragShad.getNuclearMaskFunctionsMethod

Enable displaying for example nuclear medicine data by applying smoothly changing colors to floating point data 1)in confguration phase we need to have minimum, maximum and range of possible values associated with nuclear mask 2)as uniform we would need to have set of vec4's - those will be used to display colors 3)colors will be set with algorithm presented below a)range will be divided into sections (value ranges) wich numbers will equal length of colors vector - 1 b)in each section the output color will be mix of 2 colors one associated with this section and with next one - contribution of the color associated with given section will vary from 100% at the begining of the section to 0% in the end where 100% of color will be associated with color of next section

source
MedEye3d.CustomFragShad.mainFuncStringMethod

controlling main function - basically we need to return proper FragColor which represents pixel color in given spot we generete separately r,g and b values by adding contributions from all textures

source
MedEye3d.PrepareWindowHelpersModule

It stores set of functions that need to be composed in order to prepare GLFW window and display verticies needed for texture display

source
MedEye3d.PrepareWindowHelpers.createDAtaBufferMethod

data is loaded into a buffer which passes it into thw GPU for futher processing - here the data is just passing the positions of verticies GLSTREAMDRAW the data is set only once and used by the GPU at most a few times. GLSTATICDRAW the data is set only once and used many times. GLDYNAMICDRAW the data is changed a lot and used many times.

source
MedEye3d.PrepareWindowHelpers.glVertexAttribSettingMethod

showing how openGL should read data from buffer in GPU in case of code like below it would mean:

first parameter specifies which vertex attribute we want to configure Remember that we specified the location of the position vertex attribute in the vertex shader next argument specifies the size of the vertex attribute. The vertex attribute is a vec2 so it is composed of 2 values. The third argument specifies the type of the data which is GLFLOAT next argument specifies if we want the data to be normalized. If we’re inputting integer data types like int, byte and we’ve set this to GLTRUE The fifth argument is known as the stride and tells us the space between consecutive vertex attributes. Since the next set of position data is located exactly 2 times the size of a float we could’ve also specified the stride as 0 to let OpenGL determine the stride he last parameter is of type void* and thus requires that weird cast. This is the offset of where the position data begins in the buffer.

glVertexAttribPointer positionAttribute, 2, GLFLOAT, false, 0, CNULL

The position data is stored as 32-bit  so 4 byte floating point values. 
Each position is composed of 2 of those values.
source
MedEye3d.StructsManag.modifySliceFull!Method

modifies given slice in given coordinates of given data - queried by name data - full data we work on and modify coords - coordinates in a plane of chosen slice to modify (so list of x and y coords) value - value to set for given points return reference to modified slice

source
MedEye3d.ForDisplayStructs.ActorWithOpenGlObjectsType

Actor that is able to store a state to keep needed data for proper display

currentDisplayedSlice::Int=1 # stores information what slice number we are currently displaying mainForDisplayObjects:: forDisplayObjects=forDisplayObjects() # stores objects needed to display using OpenGL and GLFW onScrollData::FullScrollableDat = FullScrollableDat() textureToModifyVec::Vector{TextureSpec}=[] # texture that we want currently to modify - if list is empty it means that we do not intend to modify any texture isSliceChanged::Bool= false # set to true when slice is changed set to false when we start interacting with this slice - thanks to this we know that when we start drawing on one slice and change the slice the line would star a new on new slice textDispObj::ForWordsDispStruct =ForWordsDispStruct()# set of objects and constants needed for text diplay currentlyDispDat::SingleSliceDat =SingleSliceDat() # holds the data displayed or in case of scrollable data view for accessing it calcDimsStruct::CalcDimsStruct=CalcDimsStruct() #data for calculations of necessary constants needed to calculate window size , mouse position ... valueForMasToSet::valueForMasToSetStruct=valueForMasToSetStruct() # value that will be used to set pixels where we would interact with mouse lastRecordedMousePosition::CartesianIndex{3} = CartesianIndex(1,1,1) # last position of the mouse related to right click - usefull to know onto which slice to change when dimensions of scroll change forUndoVector::AbstractArray=[] # holds lambda functions that when invoked will undo last operations maxLengthOfForUndoVector::Int64 = 10 # number controls how many step at maximum we can get back isBusy::Base.Threads.Atomic{Bool}= Threads.Atomic{Bool}(0) # used to indicate by some functions that actor is busy and some interactions should be ceased

source
MedEye3d.ForDisplayStructs.KeyboardCallbackSubscribableType

Object that enables managing input from keyboard - it stores the information also about needed keys wheather they are kept pressed examples of keyboard input (raw GLFW input we process below) action RELEASE GLFW.Action key s StringPRESS key s String action PRESS GLFW.Action key s StringRELEASE key s String action RELEASE GLFW.Action

source
MedEye3d.ForDisplayStructs.KeyboardStructType

Holding necessery data to controll keyboard shortcuts

isCtrlPressed::Bool = false# left - scancode 37 right 105 - Int32 isShiftPressed::Bool = false # left - scancode 50 right 62- Int32 isAltPressed::Bool= false# left - scancode 64 right 108- Int32 isEnterPressed::Bool= false# scancode 36 isTAbPressed::Bool= false# isSpacePressed::Bool= false# isF1Pressed::Bool= false isF2Pressed::Bool= false isF3Pressed::Bool= false

lastKeysPressed::Vector{String}=[] # last pressed keys - it listenes to keys only if ctrl/shift or alt is pressed- it clears when we release those case or when we press enter #informations about what triggered sending this particular struct to the actor mostRecentScanCode ::GLFW.Key=GLFW.KEYKP4 mostRecentKeyName ::String="" mostRecentAction ::GLFW.Action= GLFW.RELEASE

source
MedEye3d.ForDisplayStructs.MaskType

data needed for definition of mask - data that will be displayed over main image this struct is parametarized by type of 3 dimensional array that will be used to store data

source
MedEye3d.ForDisplayStructs.MaskTextureUniformsType

hold reference numbers that will be used to access and modify given uniform value In order to have easy fast access to the values set the most recent values will also be stored inside In order to improve usability we will also save with what data type this mask is associated for example Int, uint, float etc

source
MedEye3d.ForDisplayStructs.TextureSpecType

Holding the data needed to create and later reference the textures

name::String="" #human readable name by which we can reference texture numb::Int32 =-1 #needed to enable swithing between textures generally convinient when between 0-9; needed only if texture is to be modified by mouse input whichCreated::Int32 =-1 #marks which one this texture was when created - so first in list second ... - needed for convinient accessing ..Uniforms in shaders isMainImage ::Bool = false #true if this texture represents main image isNuclearMask ::Bool = false # used for example in case of nuclear imagiing studies isContinuusMask ::Bool = false # in case of masks if mask is continuus color display we set multiple colors in a vector color::RGB = RGB(0.0,0.0,0.0) #needed in case for the masks in order to establish the range of colors we are intrested in in case of binary mask there is no point to supply more than one color (supply Vector with length = 1) colorSet::Vector{RGB}=[] #set of colors that can be used for mask with continous values strokeWidth::Int32 =Int32(3)#marking how thick should be the line that is left after acting with the mouse ... isEditable::Bool =false #if true we can modify given texture using mouse interaction GLRtype::UInt32 =UInt32(0) #GlRtype - for example GLR8UI or GLR16I OpGlType ::UInt32 =UInt32(0) #open gl type - for example GLUNSIGNEDBYTE or GLSHORT actTextrureNumb ::UInt32 =UInt32(0) #usefull to be able to activate the texture using GLActivetexture - with proper open GL constant associatedActiveNumer ::Int64 =Int64(0) #usefull to be able to activate the texture using GLActivetexture - with proper open GL constant ID::Base.RefValue{UInt32} = Ref(UInt32(0)) #id of Texture isVisible::Bool= true #if false it should be invisible uniforms::TextureUniforms=MaskTextureUniforms()# holds values needed to control ..Uniforms in a shader minAndMaxValue::Vector{T} = []#entry one is minimum possible value for this mask, and second entry is maximum possible value for this mask

source
MedEye3d.ForDisplayStructs.forDisplayObjectsType

Defined in order to hold constant objects needed to display images listOfTextSpecifications::Vector{TextureSpec} = [TextureSpec()] window = [] vertexshader::UInt32 =1 fragmentshader::UInt32=1 shader_program::UInt32=1 stopListening::Base.Threads.Atomic{Bool}= Threads.Atomic{Bool}(0)# enables unlocking GLFW context for futher actions vbo::UInt32 =1 #vertex buffer object id ebo::UInt32 =1 #element buffer object id mainImageUniforms::MainImageUniforms = MainImageUniforms()# struct with references to main image TextureIndexes::Dictionary{String, Int64}=Dictionary{String, Int64}() #gives a way of efficient querying by supplying dictionary where key is a name we are intrested in and a key is index where it is located in our array numIndexes::Dictionary{Int32, Int64} =Dictionary{Int32, Int64}() # a way for fast query using assigned numbers gslsStr::String="" # string giving information about used openg gl gsls version windowControlStruct::WindowControlStruct=WindowControlStruct()# holding data usefull to controll display window

source
MedEye3d.DataStructs.getLocationDictMethod

given Vector of tuples where first is string and second is RawDataToDisp it creates dictionary where keys are those strings - names and values are indicies where they are found

source
MedEye3d.DataStructs.CalcDimsStructType

struct holding data needed for calculating proper mouse position , getting proper size for the texture depending on image dimensions getting into account proportions of diffrent parts of display usefull stats for proper text display

source
MedEye3d.MaskDiffrence.displayMaskDiffrenceMethod

SUBTRACTING MASKS used in order to enable subtracting one mask from the other - hence displaying pixels where value of mask a is present but mask b not (order is important) automatically both masks will be set to be invisible and only the diffrence displayed

In order to achieve this we need to have all of the samplers references stored in a list

  1. we need to set both masks to invisible - it will be done from outside the shader
  2. we set also from outside uniform marking visibility of diffrence to true
  3. also from outside we need to set which texture to subtract from which we will achieve this by setting maskAtoSubtr and maskBtoSubtr int ..Uniforms those integers will mark which samplers function will use
  4. in shader function will be treated as any other mask and will give contribution to output color multiplied by its visibility(0 or 1)
  5. inside the function color will be defined as multiplication of two colors of mask A and mask B - colors will be acessed similarly to samplers
  6. color will be returned only if value associated with maskA is greater than mask B and proportional to this difffrence

In order to provide maximum performance and avoid branching inside shader multiple shader programs will be attached and one choosed that will use diffrence needed maskToSubtrastFrom,maskWeAreSubtracting - specifications o textures we are operating on

source
MedEye3d.WindowControll.processKeysInfoMethod

KEYF1 - will display wide window for bone Int32(1000),Int32(-1000) KEYF2 - will display window for soft tissues Int32(400),Int32(-200) KEYF3 - will display wide window for lung viewing Int32(0),Int32(-1000) KEYF4, KEYF5 - sets minimum (F4) and maximum (KEYF5) value for display (with combination of + and minus signs - to increase or decrease given treshold) - in case of continuus colors it will clamp values - so all above max will be equaled to max ; and min if smallert than min in case of main CT mask - it will controll min shown white and max shown black in case of maks with single color associated we will step data so if data is outside the rande it will return 0 - so will not affect display KEY_F6 - controlls contribution of given mask to the overall image - maximum value is 1 minimum 0 if we have 3 masks and all control contribution is set to 1 and all are visible their corresponding influence to pixel color is 33% if plus is pressed it will increse contribution by 0.1 if minus is pressed it will decrease contribution by 0.1

source
MedEye3d.WindowControll.setTextureWindowMethod

sets minimum and maximum value for display - in case of continuus colors it will clamp values - so all above max will be equaled to max ; and min if smallert than min in case of main CT mask - it will controll min shown white and max shown black in case of maks with single color associated we will step data so if data is outside the rande it will return 0 - so will not affect display

source