CLC number: TP391.41
On-line Access: 2010-09-07
Received: 2010-03-19
Revision Accepted: 2010-06-04
Crosschecked: 2010-08-02
Cited: 3
Clicked: 9239
Rui Wang, Wei-feng Chen, Ming-hao Pan, Hu-jun Bao. Harmonic coordinates for real-time image cloning[J]. Journal of Zhejiang University Science C, 2010, 11(9): 690-698.
@article{title="Harmonic coordinates for real-time image cloning",
author="Rui Wang, Wei-feng Chen, Ming-hao Pan, Hu-jun Bao",
journal="Journal of Zhejiang University Science C",
volume="11",
number="9",
pages="690-698",
year="2010",
publisher="Zhejiang University Press & Springer",
doi="10.1631/jzus.C1000067"
}
%0 Journal Article
%T Harmonic coordinates for real-time image cloning
%A Rui Wang
%A Wei-feng Chen
%A Ming-hao Pan
%A Hu-jun Bao
%J Journal of Zhejiang University SCIENCE C
%V 11
%N 9
%P 690-698
%@ 1869-1951
%D 2010
%I Zhejiang University Press & Springer
%DOI 10.1631/jzus.C1000067
TY - JOUR
T1 - Harmonic coordinates for real-time image cloning
A1 - Rui Wang
A1 - Wei-feng Chen
A1 - Ming-hao Pan
A1 - Hu-jun Bao
J0 - Journal of Zhejiang University Science C
VL - 11
IS - 9
SP - 690
EP - 698
%@ 1869-1951
Y1 - 2010
PB - Zhejiang University Press & Springer
ER -
DOI - 10.1631/jzus.C1000067
Abstract: Traditional gradient domain seamless image cloning is a time consuming task, requiring the solving of Poisson’s equations whenever the shape or position of the cloned region changes. Recently, a more efficient alternative, the mean-value coordinates (MVCs) based approach, was proposed to interpolate interior pixels by a weighted combination of values along the boundary. However, this approach cannot faithfully preserve the gradient in the cloning region. In this paper, we introduce harmonic cloning, which uses harmonic coordinates (HCs) instead of MVCs in image cloning. Benefiting from the non-negativity and interior locality of HCs, our interpolation generates a more accurate harmonic field across the cloned region, to preserve the results with as high a quality as with Poisson cloning. Furthermore, with optimizations and implementation on a graphic processing unit (GPU), we demonstrate that, compared with the method using MVCs, our harmonic cloning gains better quality while retaining real-time performance.
[1]Agarwala, A., 2007. Efficient gradient-domain compositing using quadtrees. ACM Trans. Graph., 26(3), Article No. 94, p.1-5.
[2]DeRose, T., Meyer, M., 2006. Harmonic Coordinates. Pixar Technical Memo 06–02. Pixar Animation Studios. Available from http://graphics.pixar.com/HarmonicCoordinates/
[3]Farbman, Z., Hoffer, G., Lipman, Y., Cohen-Or, D., Lischinski, D., 2009. Coordinates for instant image cloning. ACM Trans. Graph., 28(3), Article No. 67, p.1-9.
[4]Floater, M.S., 2003. Mean value coordinates. Comput. Aided Geom. Des., 20(1):19-27.
[5]Georgiev, T., 2004. Photoshop Healing Brush: a Tool for Seamless Cloning. Workshop on Applications of Computer Vision, p.1-8.
[6]Iserles, A., 1996. A First Course in Numerical Analysis of Differential Equations. Cambridge University Press, New York, USA.
[7]Jeschke, S., Cline, D., Wonka, P., 2009. A GPU Laplacian solver for diffusion curves and Poisson image editing. ACM Trans. Graph., 28(5), Article No. 116, p.1-8.
[8]Jia, J., Sun, J., Tang, C.K., Shum, H.Y., 2006. Drag-and-drop pasting. ACM Trans. Graph., 25(3):631-637.
[9]Joshi, P., Meyer, M., DeRose, T., Green, B., Sanocki, T., 2007. Harmonic coordinates for character articulation. ACM Trans. Graph., 26(3), Article No. 71, p.1-9.
[10]Kazhdan, M., Hoppe, H., 2008. Streaming multigrid for gradient-domain operations on large images. ACM Trans. Graph., 27(3), Article No. 21, p.1-10.
[11]Levin, A., Zomet, A., Peleg, S., Weiss, Y., 2004. Seamless Image Stitching in the Gradient Domain. 8th European Conf. on Computer Vision, p.377-389.
[12]McCann, J., Pollard, N.S., 2008. Real-time gradient-domain painting. ACM Trans. Graph., 27(3), Article No. 93, p.1-7.
[13]Orzan, A., Bousseau, A., Winnemöller, H., Barla, P., Thollot, J., Salesin, D., 2008. Diffusion curves: a vector representation for smooth-shaded images. ACM Trans. Graph., 27(3), Article No. 92, p.1-8.
[14]Pérez, P., Gangnet, M., Blake, A., 2003. Poisson image editing. ACM Trans. Graph., 22(3):313-318.
[15]Shewchuk, J.R., 2005. A Two-Dimensional Quality Mesh Generator and Delaunay Triangulator. Available from http://www.cs.cmu.edu/~quake/triangle.html [Accessed on Mar. 22, 2010].
[16]Szeliski, R., 2006. Locally adapted hierarchical basis preconditioning. ACM Trans. Graph., 25:1135-1143.
Open peer comments: Debate/Discuss/Question/Opinion
<1>
Weifeng Chen@Zhejiang University<chenweifeng@cad.zju.edu.cn>
2010-09-14 15:06:13
Thank you for the comments on our paper. But we totally disagree that our work is a copy of "Coordinates for Instant Image Cloning" paper, which uses Mean-Value Coordinates (MVCs) for image cloning.
First, our method solves one main limitation of the MVCs work at cloning concave regions. Please refer to examples in Fig. 5. We compare our method with the MVC and visualize the difference. It can be observed that our method produces more smooth and natural cloning results. It is because, in mathematics, our Harmonic coordinates has better capability for concave cases that the MVCs is less successful. We only provide one comparison is because we think one image is able to clarify the difference of two methods. If it requires, we’d like to put more comparison on the web. The analysis of advantages of our HCs than MVCs is presented in section 3.2. The author of the MVC based cloning paper argues that “cloning with highly concave regions (when MVC performs badly) is hardly ever needed”. But, in an interactive image cloning system, nobody is able to expect what is the next shape or region user wants to clone. It is worthy to exploring a new method to overcome the shortage of the MVCs based cloning and provide better solution.
We also noticed another image cloning paper (“Improved coordinate-based image and video cloning algorithm”, Lee, S. and Lee, I., ACM SIGGRAPH ASIA 2009 Posters) aiming at solving the same problem. Different from our method that employs a new coordinate, it uses a new scheme to compute the coordinate weights. Both of our works make steps on improving the MVCs. Although we all agree the "Coordinates for Instant Image Cloning" is a great paper, there is no reason to stop the paces to explore better coordinates and solutions for image cloning.
Second, our main contribution is the Harmonic coordinates for image cloning. We provide our solution to compute the coordinates and present a GPU-based Harmonic coordinates solver. In section 4 we propose the equations to generate the coordinates and in section 5 we give the GPU-based implementation to solve the Poisson equation. Based on these contributions, we are able to achieve better quality of results and still retain real-time or interactive image cloning. As an extension and improvement of MVCs, our method requires including some algorithms in MVCs to make the representation of our paper clear and complete. But, to clarify the originality and novelty hence avoid the confusion and misleading, in our paper, we explicitly cited the MVCs paper and clearly claimed in necessary places which parts are from the MVCs paper and which parts are our adaption. We do not regard or intend to regard algorithms proposed in MVCs paper as our contributions. The adaptive mesh, hierarchical boundary sampling and the temporal smoothing for video cloning are all marked out from the MVCs paper. It is unfair and unjust to make censures but ignoring our contributions and explicit citations.
Third, we presented our work as well as the MVCs paper in the same session on “The 1st China-Israel Bi-National Conference on Graphics and Geometric Computing”, 2009. (Please refer to http://cg.cs.tsinghua.edu.cn/ci2009/?page_id=4 for the conference schedule.) We exchanged comments and opinions with many academic peers including one author of the MVCs paper in the conference. However, we did not receive any comments on regarding our work as a copy of the MVCs paper in the conference or after it. It is surprising for us to read such a comment here. We hope our work can be evaluated justly.
In summary, our paper provides an alternative coordinates method for image cloning. Although our work is based on the MVCs paper, our new coordinates and fast GPU solver enables a better image cloning especially for cloning concave shapes or regions.
Gil Hoffer@Tel Aviv Uinversity<gilhoffe@cs.tau.ac.il>
2010-09-11 16:01:33
This paper is almost an exact copy of our "Coordinates for Instant Image Cloning" (Farbman et. al.) paper.
There are complete paragraphs which are almost identical, and many many algorithmic details which are exactly the same (using an adaptive and hierarchic scheme, the temporal smoothing for video cloning, etc...)
Also, you don't really give a good example for why your HC based system performs better than easy-to-implement MVC based cloning. From our experiments, cloning with highly concave regions (when MVC performs bad) is hardly ever needed (for instance in your example in figure 5, the more sensible choice would be to use a convex disc-like shape for the concealment)