1
Spatially Constrained Segmentation
of Dermoscopy Images
Howard Zhou1
, Mei Chen2
, Le Zou2
, Richard Gass2
,
Laura Ferri...
2
Skin cancer and melanoma
 Skin cancer : most common of all cancers
[ Image courtesy of “An Atlas of Surface Microscopy ...
3
Skin cancer and melanoma
 Skin cancer : most common of all cancers
 Melanoma : leading cause of mortality (75%)
[ Imag...
4
Skin cancer and melanoma
 Skin cancer : most common of all cancers
 Melanoma : leading cause of mortality (75%)
 Earl...
5
[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]
Clinical ViewDermoscopy view
6
Dermoscopy
 Improve diagnostic accuracy by 30% in the hands
of trained physicians
 May require as much as 5 year exper...
7
First step of analysis:
Segmentation
 Separating lesions from surrounding skin
 Resulting border
 Gives lesion size a...
8
Domain specific constraints
 Spatial constraints
 Four corners are skin (Melli et al.2006, Celebi et al. 2007)
 Impli...
9
Domain specific constraints
 Spatial constraints
 Four corners are skin (Melli et al.2006, Celebi et al. 2007)
 Impli...
10
We explore …
 Spatial constraints arise from the growth
pattern of pigmented skin lesions
Meanshift (c = 32, s = 8)
11
We explore …
 Spatial constraints arise from the growth
pattern of pigmented skin lesions –
radiating pattern
Meanshif...
12
Embedding constraints
Meanshift (c = 32, s = 8) Polar (k = 6)
 Radiating pattern from lesion growth
 Embedding constr...
13Polar (k = 6)Meanshift Polar
Embedding constraints
 Radiating pattern from lesion growth
 Embedding constraints as pol...
14Meanshift Polar
Comparison to the Doctors
 Radiating pattern from lesion growth
 Embedding constraints as polar coords...
15
Dermoscopy images
Common radiating appearance
16
Growth pattern of pigmented
skin lesions
 lesions grow in both radial and vertical direction
 Skin absorbs and scatte...
17
Radiating growth pattern on
skin surface
 Difference in appearance: more significant
along the radial direction than a...
18
Radiating growth pattern on
skin surface
 Difference in appearance: more significant
along the radial direction than a...
19
 Each pixel  feature vector in R4
 3D: R,G,B or L, a, b in the color space
 1D: polar radius measured from the cent...
20
 Each pixel  feature vector in R4
 Clustering pixels in the feature space
 Replace pixels with mean for compact
rep...
21
Radiating pattern
Dermoscopy vs. natural images
Derm dataset (216)
… …
BSD dataset (300)
22
 Mean per-pixel residue:
average per-pixel color
difference of each pair
Embedding spatial constraints
Grouping featur...
23
Dermoscopy vs. natural images
Polar vs. Cartesion
BSD dataset (300)
Residue (polar)
Residue (Cartesian)
Derm dataset (2...
24
Dermoscopy vs. natural images
Polar vs. Cartesion
 Mean per-pixel residue (k-means++, k = 30)
25
Polar vs. Cartesian
 The regions appear more blocky in the
Cartesian case
Polar (k = 30) Cartesian (k = 30)
26
Six super-regions
 30 clusters  6 super clusters (K-means++)
Polar (k = 6) Cartesian (k = 6)
27
Final segmentation
Polar Cartesian
28
Polar vs. Meanshift
 The regions appear more blocky in the
Meanshift case
Polar (k = 6) Meanshift (c = 32, s = 8)
29
Final segmentation
Polar Meanshift
30
 Given a dermoscopy image
Algorithm overview
31
 Given a dermoscopy image
Algorithm overview
original
32
1. First round clustering: K-means++ (k = 30)
Algorithm overview
original 30 clusters
33
2. Second round: clusters(30) super-regions(6)
Algorithm overview
original 30 clusters 6 Super-regions
34
3. Apply texture gradient filter (Martin, et al. 2004)
Algorithm overview
original 30 clusters 6 Super-regions
Texture ...
35
4. Find optimal boundary (color+texture)
Algorithm overview
original 30 clusters 6 Super-regions
Texture edge map Final...
36
 First round clustering: K-means++ (k = 30)
 Reduce noise
 Groups pixels into homogenous regions – a
more compact re...
37
 First round clustering: K-means++ (k = 30)
 Reduce noise
 Groups pixels into homogenous regions – a
more compact re...
38
 K = 6 : clusters(30) super-regions(6)
 Account for intra-skin and intra-lesion variations
 Avoid a large k
 Super...
39
 K = 6 : clusters(30) super-regions(6)
 Account for intra-skin and intra-lesion variations
 Avoid a large k
 Super...
40
3. Color-texture integration
 Incorporating texture information can
improve segmentation performance.
 Severely sun d...
41
3. Color-texture integration
 Incorporating texture information can
improve segmentation performance.
 Severely sun d...
42
3. Color-texture integration
 Incorporating texture information can
improve segmentation performance.
 Severely sun d...
43
 Optimal skin-lesion boundary
 Color: Earth Mover’s Distance (EMD) between every
pair of super-regions
4. Optimal bou...
44
 Optimal skin-lesion boundary
 Color: Earth Mover’s Distance (EMD) between every pair
of super-regions
 Texture: Tex...
45
 Optimal skin-lesion boundary
 Color: Earth Mover’s Distance (EMD) between every pair
of super-regions
 Texture: Tex...
46
 Our collaborating dermatologist Dr. Ferris manually
outline the lesions in 67 dermoscopy images
 The border error is...
47
Typical segmentation result
Error = 12.96%
White: Dr. Ferris
Red : Dr. Zhang
Blue : computer
48
Comparison
Compared to ground-truth outlined by Dr. Ferris
11.32
20.64 21.41
16.92
19.49 20.13
15.91 14.93
0
5
10
15
20...
49
Additional results
Error = 5.80%
White: Dr. Ferris
Red : Dr. Zhang
Blue : computer
50
Additional results
Error = 13.61%
White: Dr. Ferris
Red : Dr. Zhang
Blue : computer
51
Additional results
Error = 16.60%
White: Dr. Ferris
Red : Dr. Zhang
Blue : computer
52
Additional results
Error = 34.09%
White: Dr. Ferris
Red : Dr. Zhang
Blue : computer
53
Limitation
 Assumption that lesions appear relatively
near the center may not hold
 Fairly low number of super region...
54
Conclusion
 Growth pattern of pigmented skin lesions can be used to
improve lesion segmentation accuracy in dermoscopy...
55
Future work
 Extend to meanshift?
56
Comparison to other methods
Compared to ground-truth outlined by Dr. Ferris
26.74
20.43 20.77 20.13
14.93
11.32
0
5
10
...
57
Color and texture cue integration
 Apply texture gradient filter (Martin, et al. 2004)
 Pseudo-likelihood map - edge ...
Upcoming SlideShare
Loading in …5
×

talk

483 views

Published on

Published in: Health & Medicine, Technology
  • Be the first to comment

talk

  1. 1. 1 Spatially Constrained Segmentation of Dermoscopy Images Howard Zhou1 , Mei Chen2 , Le Zou2 , Richard Gass2 , Laura Ferris3 , Laura Drogowski3 , James M. Rehg1 1 School of Interactive Computing, Georgia Tech 2 Intel Research Pittsburgh 3 Department of Dermatology, University of Pittsburgh
  2. 2. 2 Skin cancer and melanoma  Skin cancer : most common of all cancers [ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]
  3. 3. 3 Skin cancer and melanoma  Skin cancer : most common of all cancers  Melanoma : leading cause of mortality (75%) [ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]
  4. 4. 4 Skin cancer and melanoma  Skin cancer : most common of all cancers  Melanoma : leading cause of mortality (75%)  Early detection significantly reduces mortality [ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]
  5. 5. 5 [ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ] Clinical ViewDermoscopy view
  6. 6. 6 Dermoscopy  Improve diagnostic accuracy by 30% in the hands of trained physicians  May require as much as 5 year experience to have the necessary training  Motivation for Computer-aided diagnosis (CAD) in this area Clinical view Dermoscopy view
  7. 7. 7 First step of analysis: Segmentation  Separating lesions from surrounding skin  Resulting border  Gives lesion size and border irregularity  Crucial to the extraction of dermoscopic features for diagnosis  Previous Work :  PDE approach – Erkol et al. 2005, …  Histogram thresholding – Hintz-Madsen et al. 2001, …  Clustering – Schmid 1999, Melli et al. 2006…  Statistical region merging – Celebi et al. 2007, …
  8. 8. 8 Domain specific constraints  Spatial constraints  Four corners are skin (Melli et al.2006, Celebi et al. 2007)  Implicitly enforcing Local neighborhood constraints on image Cartesian coordinates (Meanshift)
  9. 9. 9 Domain specific constraints  Spatial constraints  Four corners are skin (Melli et al.2006, Celebi et al. 2007)  Implicitly enforcing Local neighborhood constraints on image Cartesian coordinates (Meanshift) Meanshift (c = 32, s = 8)
  10. 10. 10 We explore …  Spatial constraints arise from the growth pattern of pigmented skin lesions Meanshift (c = 32, s = 8)
  11. 11. 11 We explore …  Spatial constraints arise from the growth pattern of pigmented skin lesions – radiating pattern Meanshift (c = 32, s = 8)
  12. 12. 12 Embedding constraints Meanshift (c = 32, s = 8) Polar (k = 6)  Radiating pattern from lesion growth  Embedding constraints as polar coords improves segmentation performance
  13. 13. 13Polar (k = 6)Meanshift Polar Embedding constraints  Radiating pattern from lesion growth  Embedding constraints as polar coords improves segmentation performance
  14. 14. 14Meanshift Polar Comparison to the Doctors  Radiating pattern from lesion growth  Embedding constraints as polar coords improves segmentation performance White: Dr. Ferris Red : Dr. Zhang Blue : computer
  15. 15. 15 Dermoscopy images Common radiating appearance
  16. 16. 16 Growth pattern of pigmented skin lesions  lesions grow in both radial and vertical direction  Skin absorbs and scatters light.  Appearance of pigmented cells varies with depth  Dark brown  tan  blue-gray  Common radiating appearance pattern on skin surface [ Image courtesy of “Dermoscopy : An Atlas of Surface Microscopy of Pigmented Skin Lesions]
  17. 17. 17 Radiating growth pattern on skin surface  Difference in appearance: more significant along the radial direction than any other direction.
  18. 18. 18 Radiating growth pattern on skin surface  Difference in appearance: more significant along the radial direction than any other direction.
  19. 19. 19  Each pixel  feature vector in R4  3D: R,G,B or L, a, b in the color space  1D: polar radius measured from the center of the image (normalized by w) Embedding spatial constraints Feature vectors original r {R, G, B}
  20. 20. 20  Each pixel  feature vector in R4  Clustering pixels in the feature space  Replace pixels with mean for compact representation Embedding spatial constraints Grouping features filteredoriginal r {R, G, B}
  21. 21. 21 Radiating pattern Dermoscopy vs. natural images Derm dataset (216) … … BSD dataset (300)
  22. 22. 22  Mean per-pixel residue: average per-pixel color difference of each pair Embedding spatial constraints Grouping features original {Ro, Go, Bo} polar {Rp, Gp, Bp} Cartesian {Rc, Gc, Bc}
  23. 23. 23 Dermoscopy vs. natural images Polar vs. Cartesion BSD dataset (300) Residue (polar) Residue (Cartesian) Derm dataset (216) Residue (Cartesian) Residue (polar)  Mean per-pixel residue (k-means++, k = 30)
  24. 24. 24 Dermoscopy vs. natural images Polar vs. Cartesion  Mean per-pixel residue (k-means++, k = 30)
  25. 25. 25 Polar vs. Cartesian  The regions appear more blocky in the Cartesian case Polar (k = 30) Cartesian (k = 30)
  26. 26. 26 Six super-regions  30 clusters  6 super clusters (K-means++) Polar (k = 6) Cartesian (k = 6)
  27. 27. 27 Final segmentation Polar Cartesian
  28. 28. 28 Polar vs. Meanshift  The regions appear more blocky in the Meanshift case Polar (k = 6) Meanshift (c = 32, s = 8)
  29. 29. 29 Final segmentation Polar Meanshift
  30. 30. 30  Given a dermoscopy image Algorithm overview
  31. 31. 31  Given a dermoscopy image Algorithm overview original
  32. 32. 32 1. First round clustering: K-means++ (k = 30) Algorithm overview original 30 clusters
  33. 33. 33 2. Second round: clusters(30) super-regions(6) Algorithm overview original 30 clusters 6 Super-regions
  34. 34. 34 3. Apply texture gradient filter (Martin, et al. 2004) Algorithm overview original 30 clusters 6 Super-regions Texture edge map
  35. 35. 35 4. Find optimal boundary (color+texture) Algorithm overview original 30 clusters 6 Super-regions Texture edge map Final segmentation
  36. 36. 36  First round clustering: K-means++ (k = 30)  Reduce noise  Groups pixels into homogenous regions – a more compact representation of the image  Artuhur and Vassilvitskii, 2007  R4 : {L*a*b* (3D), w * polar radius (1D)} 1. First round clustering original
  37. 37. 37  First round clustering: K-means++ (k = 30)  Reduce noise  Groups pixels into homogenous regions – a more compact representation of the image  Artuhur and Vassilvitskii, 2007  R4 : {L*a*b* (3D), w * polar radius (1D)} 1. First round clustering original 30 clusters
  38. 38. 38  K = 6 : clusters(30) super-regions(6)  Account for intra-skin and intra-lesion variations  Avoid a large k  Super-regions correspond to meaningful regions such as skin, skin-lesion transition, and inner lesion, etc. 2. Second round clustering original 30 clusters
  39. 39. 39  K = 6 : clusters(30) super-regions(6)  Account for intra-skin and intra-lesion variations  Avoid a large k  Super-regions correspond to meaningful regions such as skin, skin-lesion transition, and inner lesion, etc. 2. Second round clustering original 30 clusters 6 super-regions
  40. 40. 40 3. Color-texture integration  Incorporating texture information can improve segmentation performance.  Severely sun damaged skin; texture variations at boundaries in addition to color variations original
  41. 41. 41 3. Color-texture integration  Incorporating texture information can improve segmentation performance.  Severely sun damaged skin; texture variations at boundaries in addition to color variations  Apply texture gradient filter (Martin, et al. 2004) original
  42. 42. 42 3. Color-texture integration  Incorporating texture information can improve segmentation performance.  Severely sun damaged skin; texture variations at boundaries in addition to color variations  Apply texture gradient filter (Martin, et al. 2004)  Texture edge map: pseudo-likelihood original Texture edge map
  43. 43. 43  Optimal skin-lesion boundary  Color: Earth Mover’s Distance (EMD) between every pair of super-regions 4. Optimal boundary 6 super-regions
  44. 44. 44  Optimal skin-lesion boundary  Color: Earth Mover’s Distance (EMD) between every pair of super-regions  Texture: Texture edge map 4. Optimal boundary Texture edge map6 super-regions
  45. 45. 45  Optimal skin-lesion boundary  Color: Earth Mover’s Distance (EMD) between every pair of super-regions  Texture: Texture edge map  Minimizing the integrated color-texture measure 4. Optimal boundary Texture edge map6 super-regions
  46. 46. 46  Our collaborating dermatologist Dr. Ferris manually outline the lesions in 67 dermoscopy images  The border error is given by  Computer : binary image obtained by filling the automatic detected border  ground-truth : obtained by filling in the boundaries outlined by Dr. Ferris Validation and results
  47. 47. 47 Typical segmentation result Error = 12.96% White: Dr. Ferris Red : Dr. Zhang Blue : computer
  48. 48. 48 Comparison Compared to ground-truth outlined by Dr. Ferris 11.32 20.64 21.41 16.92 19.49 20.13 15.91 14.93 0 5 10 15 20 25 30 35 none Cartesian polar Dr. Zhang Spatial constraints Percentageerror Dr. Zhang RGB CIELAB Color + texture To account for inter-operator variation, we also asked Dr. Alex Zhang to manually outline boundaries on the same dataset
  49. 49. 49 Additional results Error = 5.80% White: Dr. Ferris Red : Dr. Zhang Blue : computer
  50. 50. 50 Additional results Error = 13.61% White: Dr. Ferris Red : Dr. Zhang Blue : computer
  51. 51. 51 Additional results Error = 16.60% White: Dr. Ferris Red : Dr. Zhang Blue : computer
  52. 52. 52 Additional results Error = 34.09% White: Dr. Ferris Red : Dr. Zhang Blue : computer
  53. 53. 53 Limitation  Assumption that lesions appear relatively near the center may not hold  Fairly low number of super regions (6) may limit the algorithm to perform well on lesions with more colors
  54. 54. 54 Conclusion  Growth pattern of pigmented skin lesions can be used to improve lesion segmentation accuracy in dermoscopy images.  An unsupervised segmentation algorithm incorporating these spatial constraints  We demonstrate its efficacy by comparing the segmentation results to ground-truth segmentations determined by an expert.
  55. 55. 55 Future work  Extend to meanshift?
  56. 56. 56 Comparison to other methods Compared to ground-truth outlined by Dr. Ferris 26.74 20.43 20.77 20.13 14.93 11.32 0 5 10 15 20 25 30 Meanshift JSEG (Celebi 2006) SRM (Celebi 2007) SCS Cartesian SCS polar Dr. Zhang Segmentation methods Percentageerror
  57. 57. 57 Color and texture cue integration  Apply texture gradient filter (Martin, et al. 2004)  Pseudo-likelihood map - edge caused by texture variation is present at a certain location

×