Arsalan

Adaptive Shape from Focus with an Error Estimation in Light Microscopy Stefan Scherer Institut for Computer Graphics & Vision Graz University of Technology A-8010, Inffeldgasse 16/11 e-mail:-graz.ac.at Franz Stephan Helmli Institut for Computer Graphics & Vision Graz University of Technology A-80 10, Inffeldgasse 16/11 e-mail:-graz.ac.at In optics this problem was discussed by Helmholtz [3]. In computer vision measuring of the sharpness of an image was introduced by A.P. Pentland. In [ 131 and [ 13 the authors improve the robustness of Pentlands work. A comprehensive discussion about a complete shape from focus system was done by [ 6 ] . In [7] a real time shape from focus system was presented. Finding the optimal focus measure and the rating of focus measure vectors, was presented in [ 111 and [ 121. Recently [ 151 has invented a new focus measure which is using moments and gives an absolute result of the sharpness. The differences between those classic approaches and the presented system are: Abstract Light microscopy enlarges the viewing angle while decreasing the depth of focus. This leads to mainly blurred images i f the specimen being observed consists of significant high changes. In computer vision solving this problem is known as 'shapefrom focus'. Algorithms exist that perform both, the calculationof a sharp image and the recovery of the three dimensional structure of the specimen. In this paper three classic approaches for detecting sharp image regoins are evaluated. Three new so called focus measures are introducedand are compared to the classic approaches. A new adaptive reconstruction schemefor calculating range images as well as sharp images is presented. Experimental results on synthetic and real data demonstrate the performance of the proposed algorithm. 0 0 1 Introduction In light microscopy conditional on the small depth of focus the acquired images are mainly blurred. A variation of the specimen-objective distance now results in an imagestack, where each image consists of regions in focus and out of focus. The problem formulated in terms of computer vision is the calculation of the sharpness of a certain image region, used to detect the sharpest region in that imagestack. With a calibrated setup the shape from focus algorithm provides both, a sharp image and the three dimensional structure of the observed specimen. The classic shape from focus algorithms found in [2], [ll], [lo], [6], [14], [9] work as follows. For each of the pixels in the imagestack of the individual planes the contrast of the pixel neighborhood can be calculated. The maximum of these contrast values is correspondent to the plane where the pixel is in focus. This plane again corresponds to a distance towards the object point. A sharp image as well as the range image is now obtained by applying this procedure to every point in the imagestack. New focus measures are invented and an adaptive automatic evaluation scheme for determining the optimal focus measure is used. A new calculation method for the focus measure vector and an accurate approximation method of this vector for depth recovery is presented. 1.1 Definitions and Abbreviations A pixel at the coordinates (s,y) will be denoted as p ( z ,y). The grey value in an image at the coordinates (z, y) will be g(s, y). The local neighborhood at the coordinates (s, y) will denoted as U ( s ,y). A focus measure of the type T at the coordinates (5, y) will be F h f ~ ( y). z , For an image stack the coordinates are (s, y, n),n is the plane of the imagestack. The calculated depth value at the coordinates ( 5 ,y) will be denoted d(s,y). 2 Focus Measures proaches - Classic and New Ap- A focus measure is a quantity to evaluate the sharpness of a pixel. It takes a local neighborhood and calculates the sharpness of the chosen center pixel. Since different objects 188 Authorized licensed use limited to: University Kassel. Downloaded on April 01,2021 at 13:45:28 UTC from IEEE Xplore. Restrictions apply. have different surface textures only a comparison between the same object point and different focus settings is applicable. The following focus measures can only compare the sharpness of corresponding pixels. 2.1 Classic Focus Measures The most common focus are the the Tenengrad and the variance measure which are described in the following. With pu(xo,yo)the mean of the grey values in the neighborhood U(z0, yo). 2.2 New Focus Measures Three new focus measures are presented. The motivation to find new focus measures is to obtain more robust reconstruction results than with the previously discussed classic approaches.. . 2.2.1 Mean Method Focus Measure 2.1.1 Laplace Focus Measure If the image is becoming sharper the variance Of the grey Of that Scene is getting higher. The ratio Of the mean grey value to the center grey value in the neighborhood can be as new focus In order to detect regions of higher contrast the Laplace operator can be applied. Modified versions ofthis operator are found in [6], [7], [lo] and [SI. Using the standard Laplace operator yields no response of the filter when fix = -fyy . The squared second derivation should be used instead of this. For the discreet model the self convolution of the Sobel Operator is applied. In order to increase the robustness the summation of the responses of the two filter masks (horizontal and vertical) in a local window is calculated as the Laplace focus measure F ML. FML(X0,YO) P(U(Z,Y)) > S(X, Y) (4) The ratio results in one if there is a constant grey value or there is no texture present. If high variations are present the ratio is different from one. This factor needs to be inverted if it is found in the interval (0,l). Like the Laplace and Tenengrad focus measure this focus measure F h f ~is summed in a local window. = (1) 2.1.2 Tenengrad Focus Measure The Tenengrad Focus measure FMT, which is also discussed in [4], [5] and [ 141 measures the sum of the squared responses of a horizontal and a vertical Sobel mask. For enhancement of the robustness this result is exactly like the Laplace focus measure ’a summation in a local window’. 2.2.2 Curvature Focus Measure As already discussed in the last section a sharper image region implies a higher grey value variance. Therefore if the grey values are treated as a 3D surface (x, y, g(X, y)), the curvature in a sharp image region is expected to be higher than in an unsharp region. The first step of calculating this curvature based focus measure is to approximate the surface f ( ~y), = pox ply p2z2 p3y2. The coefficients P = (PO,pl ,~ 2 ~ are3 found ) ~ using a least squares approximation (7) with go and g2. + 2.1.3 Variance Focus Measure In the case of a sharp image region the variance of the grey values is higher than in the case of being unsharp. Therefore FMv can be used as a focus measure. For example this method was applied in [5] and [lo]. -1 go= ( - 1 -1 + 0 1 0 1 ) 0 1 + 1 0 1 g2= ( 1 0 1 ) 1 0 1 (6) 189 Authorized licensed use limited to: University Kassel. Downloaded on April 01,2021 at 13:45:28 UTC from IEEE Xplore. Restrictions apply. The next step is to combine these coefficients in order to obtain a sufficient focus measure. An experimental evaluation (see 4.1) shows that the simple sum of the absolute values results in an adequate focus measure Flllc. 2.2.3 Point Focus Measure If the image plane is out of focus the grey value in an image point is the combination ofthe lightrays of its object point 0, and from object points close to o. Therefore the grey value has an extremum if being in focus. This leads to a very fast new focus measure F h l p for shape from focus variation without the use of grey values in a pixel neighborhood. Figure 2. The basic scheme for our shape from focus procedure. By approximating the focus measure vector by a polynomial of degree 4 and an estimation of its extremum reasonable results can be achieved. Figure 1 shows the range image of a synthetic image stack. As the calculation of the extremum is very likely ambiguous the reconstruction scheme has to take care of this ambiguity. This effect mainly occurs in regions which are almost sharp at the top or bottom levels of the imagestack. This is clearly seen in Figure 1. As a result of this problem outliers fi-equentlyoccur. Therefore this focus measure is not subject to firther discussions. vector with the best rating must be selected (section 3.2). With this focus measure vector a corresponding depth value for each pixel can be calculated. The resulting range image can now be corrected with the method described in 3.3.1. From the corrected range image a texture image can be reconstructed by indexing the depth in the imagestack. 3.1 Calculation of the Focus Measure Vector For a pixel in an imagestack the focus measure in all planes of the imagestack is calculated. This vector of the focus measures is now called the focus measure vector. The classic approach to find the entries of the focus measure vector is to calculate the focus measure in the local neighborhood within one plane. We extend the local neighborhood approach to a local cube including nearby planes. 3.2 Rating of the Focus Measure Vector Figure I.Range images obtained by variance (left) and point (right) focus measure. The quality (i.e. the robustness) of a focus measure cannot be directly evaluated. For example a decrease of the variance cannot be directly linked to a reduced sharpness within a single plane. This is only valid if the related values corresponding to different planes are regarded. Therefore a quality of the focus measure is rated via the quality of the focus measure vector. In the following a rating for the focus measure vector based on the assumption that it's shape can be approximated by a bell-shaped curve ([GI and [14]) is introduced. 3 Adaptive-Reconstruction Scheme Once a robust focus measure is found the next step is to apply this focus measure to a reconstruction scheme in order to obtain a range image. In the following our approach for an adaptive reconstruction scheme will be described. First the calculation of the focus measure vector (section 3.1) and its rating (section 3.2) is shown. Next the reconstruction of the range image (section 3.3) and the texture image (section 3.4) is discussed. Figure 2 outlines this scheme. For every focus measure the smoothed focus measure vector is calculated. For fiuther usage the focus measure (10) represents the mathematical formulation of that curve. In order to solve the mathematical problem of fitting the three coefficients p , 1% and cr the steepest descent 190 Authorized licensed use limited to: University Kassel. Downloaded on April 01,2021 at 13:45:28 UTC from IEEE Xplore. Restrictions apply. approach along with the sum of the squared errors for the error function is applied. The necessary initial values polho and 00 are found through the maximum of the focus measure vector and the searching of the point (po- cr, 0.60 * ho) in the focus measure vector. The increment for the steepest descent algorithm is defined by the well known Fibonacci search. For the rating B ( F h f V )itself we propose to apply the correlation coefficient CC between the focus measure vector and the fitted data. floating point numbers, the texture information can be linearly combined from the grey values of the adjacent planes in the imagestack. 4 Experimental Results Experiments are performed using synthetic and real images. The performance of focus measures, calculation and approximation of focus measure vectors, focus measure vector rating and the whole scheme is evaluated. The synthetic dataset is generated from a texture image and a range image. In order to simulate the blurring of the images in the imagestack a Gaussian filter is applied. The cr value of the kernel is proportional to the difference of the depth value d ( s , y ) in the range image and the actual plane 72 of the pixel p ( s ,y I 1 2 ) . The resolution of the images are 128 x 128 pixel with consisting of a 50 planes imagestack. The dataset Chip is a real imagestack of an EPROM microchip taken with the system illustrated in Figure 4. The resolution of the imagestack is 768x576 pixel with 48 planes. The physical dimension of the image is 286 x 215pm, with a spacing of 3pnz between each plane. The dataset Plane is a real imagestack from a plate of glass. The physical dimension is 286 x 215pm, with a spacing of l p m between each plane. B(FA4V) = CC(FhIV(i),G p / , h / , o f ( i ) ) (11) For the rating of the range image the sum of the ratings of the focus measure vectors are used. 3.3 Reconstruction of the Range Image Each plane index is corresponding to a certain depth value. The maximum of the focus measure vector is obtained by setting the first derivation of the bell-shaped curve to zero. This inherently yields sub plane accuracy depth indices. A calibration of the setup where each plane index corresponds to a certain depth value allows the direct computation of a range image. 3.3.1 Error Correction of Range Images All shape &om focus methods are based on the assumption that the grey level variance has an extremum if being in focus. Due to noise and other effects the results are errorprone. A preprocessing of the final range image is therefore highly recommended. As the smoothing of depth values with a median filter smooths already correct depth information, its application is not advisable in shape from focus methods. Our approach is to detect spikes in the range image and then to replace them with the response of the median filter. In order to detect spikes in the range image the embedding criterion (12) is introduced. Figure 3. 3D models of the synthetic dataset and the two real datasets. In the following the depth error of the reconstruction is measured using the sum of the absolute errors in the depth values of the range image. In case of the synthetic data the depth error is evaluated through the given range image. In the case ofthe real objects the a-priory knowledge ofhaving a plane shaped object is used to establish a quasi-ground truth. Figure 4 shows the shape kom focus system. The main parts are the light microscope with its motorized stage and the 3 CCD-chip color camera. d ( s , y) E [median(U(z,y)) f 5 * stddewM(U(z,y))] (12) Where median(U(z,y ) ) is the median filter response at position (s,y) in the range image and s t d d e v M ( U ( s ,y ) ) is the median deviation of the local neighborhood. 5 denotes a smoothing parameter. 3.4 Reconstruction of the Sharp Image 4.1 After calculating of the range image the texture information of the object can be reconstructed from the imagestack. The depth values give an index of the sharpest texture information in the imagestack. Because the depth values are Evaluation of the Focus Measures In order to evaluate the performance of the presented focus measures a window size of 11 x 11 was applied to all 191 Authorized licensed use limited to: University Kassel. Downloaded on April 01,2021 at 13:45:28 UTC from IEEE Xplore. Restrictions apply. focus measure vector smoothing 3 2.5 g E 2 1.5 t i 0.5 0 0 2 4 6 8 1 0 sigma value Figure 4. The shape from focus system. A: computer, B: 3 CCD-chip color camera, C: light microscopewith motorized xyz-stage, D: power supply for illumination, E&F: steppermotor controll unit. Figure 5. Depth error versus focus measure vector smoothing. focus measure Chip 1.817 0.787 Table 3. The depth errors of the classic and our approach using the three sample objects. Table 1. Normalized errors of the reconstruction for each dataset by using different focus measures. Lower values indicate a higher precision. 4.3 Evaluation of the Focus Measure Vector Calculation The proposed calculation method from 3.1 of the focus measure vector within a cubic neighborhood can be achieved by smoothing the focus measure vector. In Figure 5 the depth error versus the amount of smoothing is shown. For each ofthe three objects there is a minimum depth error. This minimum depth error is in each case smaller than the depth error without smoothing (a= 0). objects. In Table 1 the normalized depth errors of the reconstruction of the three objects are shown. The means of these scaled depth errors indicate that the focus measure curvature performs best. 4.2 Evaluation of the Rating Method 4.4 In order to evaluate the performance of the rating of the focus measure vector a reconstruction of the three objects with the focus measures Laplace, Tenengrad, variance, curvature and mean method at the local window sizes 5 x 5 and 7 x 7, are calculated. From each of these reconstructions an absolute error evaluated through the ground truth and the sum of the ratings can be given. From this a correlation coefficient is calculated (Table 2). objects: correl. coeff.: I synthetic I -0.9864 Chip -0.9021 Evaluation of the Whole Scheme In this section the overall method is evaluated. In Table 3 it can be seen that our enhanced method always performs better than the classic calculation scheme. The depth error is between 30% and 55% smaller in the case of the proposed scheme. 4.5 Reconstruction Examples Figure 6 shows the 3D model of a reconstructed ski surface at a region size of 573~43Opin.The reconstruction was performs using 55 planes with an image size of 768x576 pixels. The second sample object shows a snow crystal reconstructed fi-om 60 planes. In Figure 7 the resulting sharp image can be seen. The acquiring of the imagestack was Plane -0.7602 Table 2. Correlation coefficients between absolute error and rating of the reconstruction. 192 Authorized licensed use limited to: University Kassel. Downloaded on April 01,2021 at 13:45:28 UTC from IEEE Xplore. Restrictions apply. References S. Chang, S. H. Lai. and C.-W. Fu. A generalized depth estimation algorithm with a single image. IEEE Transactionson Pattern Analysis and Machine Intelligence, 14(4):405-4 1 I , 1992. P. GeiRler and T. Dierig. Handbook of Computer Vision and Applications, Volume 2, Signal Processing and Pattern Recognition, Kapitel: Depth from Focus. Academic Press, 1999. H. L. F. V. Helmholtz. Helmholtz’s treatise on physiological optics. Optical Society ofAmerica, 1924. A. Horii. The focusing mechanism in the KTH head eye system. Computational Vision and Active Perception Laporatoy, June 1992. J. Martinez-Baena and J. F.-V. J. A. Garcia. A multi channel autofocusing scheme for gray level shape scale detection. Pattern Recognition, 30( 10):-, 1996. S. K. Nayar and Y.Nakagawa. Shape from focus. IEEE Transactionson Pattern Analysis and Machine Intelligence, 16(8):82483 I, 1994. S. K. Nayar, M. Noguchi, M. Watanabe, and Y. Nakagawa. Real-time focus range sensors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(12):1186--1198, 1996. M. Noguchi and S. K. Nayar. Microscopic shape from focus using active illumination. Interational Conference on Pattern Recognition, pages 147-152, 1994. I. R. Nourbakhsh, D. Andre, C. Tomasi, and M. R. Genesereth. Mobile robot obstacle avoidance via depth from focus. robotics and autonomous Systems, 22:151-158, 1997. M. Subbarao and T. Choi. Accurate recovery of threedimensional shape from image focus. IEEE Transactionson Pattern Analysis and Machine Intelligence, 17(3):266-274, March 1995. M. Subbarao and Y.-F. Liu. Accurate reconstruction of three-dimensional shape and focused image from a sequence of noisy defocused images. Technical report, Department of Electrical Engineering, State University of New York at Stony Brook, 1995. M. Subbarao and J.-K. Tyan. Selecting the optimal focus measure for autofocusing and depth-from-focus. IEEE Transactionson Pattern Analysis and Machine Intelligence, 20(8):864-870, 1998. N. G. M. Subbarao. Depth recovery from blurred edges. IEEE International Conference for Computer Vision and Pattern Recognition, pages 498-503, 1988. Y. Xiong and S. A. Schafer. Depth from focusing and defocusiong. IEEE Computer Vision and Pattern Recognition (CVPR), pages 68-73, 1993. Y. Zhang, Y. Zhang, and C. Wen. A new focus measure methode using moments. Image and Vision Computing,-,2000. Figure 6.3D model of the reconstruction of a ski profile. performed using a Peltier element to cool the crystal down to -5°C. Figure 7. The reconstructed sharp image of a snow crystal condensed on the surface of a peltier element. 5 Conclusion and Outlook The present work is addressed to the shape fi-om focus problem. Three new focus measures for determining the sharpness of an image region, the so called focus measures, are invented. A comparison with well-known techniques shows a superior behaviour in the case of the novel curvature measure. A robust scheme for calculating the pixel in focus fi-om an imagestack is discussed. The robustness of the final range image is enhanced by the introduction of an automatic focus measure vector rating. Not discussed in the present work is the utilization of the additional information provided by color. Further work is therefore focused on this topic. One direction could be the combination of the imagestacks of the color channels into one imagestack with a higher radiometric resolution. 193 Authorized licensed use limited to: University Kassel. Downloaded on April 01,2021 at 13:45:28 UTC from IEEE Xplore. Restrictions apply.