New Trends in Surface Reconstruction Using Space-Time Cameras |
15 |
|
|
Figure 5. Stereo triangulation and uncertanity in depth. The true point can lie anywhere inside the shaded uncertanity region. Uncertanity in depth ΔΗ is reduced by increasing the length of base line b (In the formulation, it is assumed that the maximum error of matching is one pixel in every camera image). In practice, the right configuration is employed to make efficient use of camera plane.
2.3. Reconstruction of Unique Space Curves
Here, we describe the different steps involved in the curve matching process to extract 3D position of unique feature points by forcing the constraint of space curves as global object descriptions. The unique points are defined as the points in three-dimensional space that are matchless after forcing all the constraints, i.e. edge positioning, epipolar line, capability of curve forming with proper length by joining the neighboring points, curvature and torsion consistency of relevant space curve in two or more consecutive frames, and the uniqueness of such space curve. The unique space curves are also composed from adequate number of continuous adjacent unique points. As illustrated in figure 6, to check the uniqueness of any edge point in the left camera, all potential matches are labeled in the right camera by intersecting all existence curves with the epipolar line. Then, the associated curves to test point and each labeled point are projected to space through their
16 |
Hossein Ebrahimnezhad and Hassan Ghassemian |
|
|
camera centers. Each intersection generates a potential space curve with different shape, which should be checked for curvature and torsion consistency during the object motion. This process is done by intersecting the moved version of curves in the next frame as illustrated in below part of figure 6. The consistent space curve in torsion and curvature is selected as valid space curve if there were only one solution. The 3D point on this curve that corresponds to the test point is labeled as unique point. Details of the presented algorithm are given at the following:
Algorithm:
Step1- At any instance of time, the moving object is captured by two calibrated fixed cameras. The edge curves of image are extracted and thinned by the canny method. Edges are then linked into chains, jumping up to one pixel gap. The small size edge regions are removed to reduce the effect of noise and get the more descriptive curves. The extracted image curves are never perfect in practice: there will be missing segments, erroneous additional segments, etc. Therefore, to improve robustness to these imperfections we begin with one such segment in the left image and seek confirming evidence in the right one.
Step2- To extract the unique points, the resulted left curve image in step 1 is scanned and all edge points are checked one by one for uniqueness. For each examined edge point cL (s0 ), which is considered as initial point, the corresponding
epipolar line in the right curve-image is computed. Intersection of the epipolar line with edge curves are labeled as candidate match points. One or more match
candidates c(Ri)(s0′), i =1,2,... may be found in this step (see figure 6). Only one of the candidate points is the true match and the other points are outliers.
Step3- The points cL (s0 ) and c(Ri)(s0′), i =1,2,... are grown n points from two
sides to form the curves. The curves with smaller length than 2n and the branched curves are discarded.
Step4- To distinguish the true match from other points, the next sequence of stereo images is also considered. It is assumed that the frame rate is adjusted as well to capture consecutive images with small amount of the object motion. The neighborhood of cL (s0 )is inspected in the next sequence of left camera to find the
shift of the curve cL (s) as cLm (s). The key property of the curve with small
movement is its proximity and similarity to the main curve. Therefore, we define a correlation function as a combination of curve distance and curvature difference along the curves:
New Trends in Surface Reconstruction Using Space-Time Cameras |
17 |
|||||||||||||
|
|
|
|
|
|
|
|
|
|
|
|
|
||
cor(cL (s),cLm (s))=max cor(cL (s0 ),cLm (sj )) ; |
cor(cL (s0 ),cLm (sj ))= |
|
|
|
|
1 |
|
|
||||||
|
|
|
|
n |
|
|
||||||||
|
j |
|
|
|
|
|
cL (s0 )−cLm (sj ) |
+α ∑(κcL (sk )−κcLm (sk + j )) |
||||||
|
|
|
|
|
|
|
|
|||||||
|
|
|
|
|
|
|
|
|
|
|
|
k =−n |
(18) |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
cL (s0 )−cLm (sj ) |
|
n |
|
2 |
|
|
|
2 |
|
|
|||
where: |
= ∑ (xcL (sk )−xcLm (sk + j ))+(ycL (sk )−ycLm (sk + j )) |
(19) |
||||||||||||
|
|
|
k =−n |
κ = |
xy −yx |
|
|
|
|
|
(20) |
|||
|
|
|
|
|
|
|
|
|
||||||
|
|
|
|
(x 2 +y 2 )3 2 |
|
|
|
|
|
|
||||
The shift of cL (s) is selected |
as |
the argument |
cLm(s) |
which maximizes the |
||||||||||
correlation function cor c |
(s),c |
(s) |
. The center of c |
Lm |
(s) |
is also determined as an |
||||||||
|
( L |
Lm |
) |
|
|
|
|
|
|
|
|
|
||
argument sj , which maximizes the cor(cL(s0),cLm(sj )).
Step 5- The epipolar line of cLm(sj )is computed in next sequence of the right image. Intersections of the curves with the epipolar line are labeled as c(Rmi) (s′j ), i =1,2,... according to their proximity to c(Ri)(s0′), i =1,2,... .
Step 6- The Space curves SC |
(i ) |
(s |
|
), i =1,2,... corresponding to |
c |
(s |
),c |
(i) |
(s |
′ |
) |
|||
|
|
|
|
|
0 |
{ |
{ L |
0 |
|
R |
} |
0 |
} |
|
and the Space curves |
(i) |
(sj ), i =1,2,... corresponding to cLm |
|
(i) |
|
′ |
|
are |
||||||
SCm |
(s j ),cRm (s j ) |
|||||||||||||
established by projecting the two dimensional curves in to the space and intersecting the rays. For each space curve, the curvature and torsion are computed from Eq. 5 and 6. The correlation between two space curves before and after motion is computed from Eq. 21 for i=1, 2, …
cor(SC(i ),SC(i )m )= |
|
|
|
|
|
1 |
|
|
|
|
|
|
|
(21) |
n |
|
|
|
|
|
n |
|
|
|
|
|
|
||
|
|
|
|
|
|
|
|
|
||||||
|
∑ |
κ (i ) |
(sk ) |
−κ (i ) |
(sk + j ) |
+ ∑ |
τ (i ) |
(sk ) |
−τ (i ) |
(sk + j ) |
|
|
|
|
|
k =−n |
SC |
SCm |
|
k =−n |
SC |
SCm |
|
|
|
||||
|
|
|
|
|
|
|
|
|
|
|
|
|||
The space curve i=q that maximizes the correlation function, is selected as the consistent space curve and the pair cL (s0 )and c(Rq)(s0′) are selected as the unique
points with determined depth value. If there were more than one solution because of having the close values of correlation function, the third sequence is also inspected to find the more confident answer. If there were only one solution, the
18 |
Hossein Ebrahimnezhad and Hassan Ghassemian |
|
|
point cL (s0 )would be selected as known depth value. Otherwise, it would be labeled as non-unique point and rejected.
Figure 6. Curve stereo matching to find the unique points of the object.
Step 7- Going back to step 2 and repeat the procedure for all edge points to find adequate number of unique points. At the end, the unique space curves are composed from continuous adjacent unique points.
Shape descriptivity, branching, and occlusion are considered as three factors to choose the proper length of curves in matching process. The short curves are less descriptive in shape and result in low confidence matching, as the number of detected similar curves will be increased. Hence, the uniqueness-checking process will fail to find the best match. On the contrary, the long curves are more descriptive in shape and result in high confidence matching, as the number of detected similar curves will be decreased. Occlusion and branching are the other factors that restrict lengthening of the curves, so that the number of appropriate curves reduces by increasing the length of curves.
Our experiments show that the curve length between 20 to 40 points (for 480×640 image size) provides good result. Of course, the proposed length is a representative value. Depending on texture of the object, number of the similar