Recently, I have implemented two fragment shaders for interactive Poisson Blending (seamless blending).
Here is my real-time demo and code on ShaderToy:
Press 1 for normal mode, press 2 for mixed gradients, press space to clear the canvas.
Technical Brief
I followed a simplified routine of the Poisson Image Editing paper [P. Pérez, M. Gangnet, A. Blake. Poisson image editing. ACM Transactions on Graphics (SIGGRAPH’03)]:
- Let’s name the bottom image as “BASE”, and the image to overlay as “SRC”, the final image as “RESULT”, then the color in the boundary should be the same
- RESULT(u, v) = BASE(u, v) ∀(u, v) ∈ ∂ SRC
- Inside the mask region, just add the gradient of “SRC” (or the bigger gradient of “SRC” and “BASE”, if we want to mix the gradients) into the current result image:
- RESULT(u, v) = RESULT(u, v) + ∇ SRC(u, v);
- alternatively,
- RESULT(u, v) = RESULT(u, v) + max{ ∇ SRC(u, v), ∇ BASE(u, v) );
We can use jump flood algorithm to speed this procedure up: http://www.comp.nus.edu.sg/~tants/jfa.html
Implementation Details
I used one frame buffer to store the drawing of the user.
- iChannel0 stores the frame buffer itself
- iChannel1 stores the keyboard response for interaction
This is a very simple drawing shader which can also be adapted for other drawing applications.
Update on 4/5/2017
It’s best to stop the Poisson blending when the delta modification abs(before – after) is small enough, or after a certain amount of iterations.
We can use the mask buffer to store the times of iterations in the second channel or rgb.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 |
// Mask Image, white indicates foreground // Ruofei Du (http://duruofei.com) // License Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License. #define BRUSH_SIZE 0.1 #define INITIAL_CIRCLE_SIZE 0.4 const float KEY_1 = 49.5; const float KEY_2 = 50.5; const float KEY_SPACE = 32.5; const float KEY_ALL = 256.0; bool getKeyDown(float key) { return texture2D(iChannel1, vec2(key / KEY_ALL, 0.5)).x > 0.1; } bool getMouseDown() { return iMouse.z > 0.0; } bool isInitialization() { vec2 lastResolution = texture2D(iChannel0, vec2(0.0) / iResolution.xy).yz; return any(notEqual(lastResolution, iResolution.xy)); } void mainImage( out vec4 fragColor, in vec2 fragCoord ) { vec2 uv = fragCoord.xy / iResolution.xy; vec2 p = 2.0 * (fragCoord.xy - 0.5 * iResolution.xy) / iResolution.y; float mixingGradients = texture2D(iChannel0, vec2(1.5) / iResolution.xy).y; float frameReset = texture2D(iChannel0, vec2(1.5) / iResolution.xy).z; float mask = 0.0; bool resetBlending = (getKeyDown(KEY_1) && mixingGradients > 0.5) || (getKeyDown(KEY_2) && mixingGradients < 0.5); if (getKeyDown(KEY_1)) mixingGradients = 0.0; if (getKeyDown(KEY_2)) mixingGradients = 1.0; if (isInitialization() || getKeyDown(KEY_SPACE)) { // reset canvas vec2 q = vec2(-0.7, 0.5); if (distance(p, q) < INITIAL_CIRCLE_SIZE) mask = 1.0; resetBlending = true; } else if (getMouseDown()) { // draw on canvas vec2 mouse = 2.0 * (iMouse.xy - 0.5 * iResolution.xy) / iResolution.y; mask = (distance(mouse, p) < BRUSH_SIZE) ? 1.0 : texture2D(iChannel0, uv).x; } else { mask = texture2D(iChannel0, uv).x; } if (fragCoord.x < 1.0) { fragColor = vec4(mask, iResolution.xy, 1.0); } else if (fragCoord.x < 2.0) { if (resetBlending) frameReset = float(iFrame); fragColor = vec4(mask, mixingGradients, frameReset, 1.0); } else { fragColor = vec4(vec3(mask), 1.0); } } |
The second frame buffer to used to iterate the Poisson blending process:
- iChannel0 stores the previous frame buffer itself
- iChannel1 stores the mask buffer
- iChannel2 stores the base image
- iChannel3 stores the source image to blend
Sometimes, the texture is not loaded in the first few frames in ShaderToy, so I sample the last pixel of this framebuffer to test whether it is initialized with correct image.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 |
// Poisson Blending // Ruofei Du (http://duruofei.com) // License Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License. #define NUM_NEIGHBORS 4 float mixingGradients; vec2 neighbors[NUM_NEIGHBORS]; #define RES(UV) (tap(iChannel0, vec2(UV))) #define MASK(UV) (tap(iChannel1, vec2(UV))) #define BASE(UV) (tap(iChannel2, vec2(UV))) #define SRC(UV) (tap(iChannel3, vec2(UV))) vec3 tap(sampler2D tex, vec2 uv) { return texture2D(tex, uv).rgb; } bool isInitialization() { vec2 lastResolution = texture2D(iChannel1, vec2(0.5) / iResolution.xy).yz; return any(notEqual(lastResolution, iResolution.xy)) || iFrame < 4; } bool isMasked(vec2 uv) { return texture2D(iChannel1, uv).x > 0.5; } void mainImage( out vec4 fragColor, in vec2 fragCoord ) { vec2 uv = fragCoord.xy / iResolution.xy; fragColor.a = 1.0; mixingGradients = texture2D(iChannel1, vec2(1.5) / iResolution.xy).y; float resetBlending = texture2D(iChannel1, vec2(1.5) / iResolution.xy).z; // init if (isInitialization() || resetBlending > 0.5) { fragColor.rgb = BASE(uv); return; } vec2 p = uv; if (isMasked(p)) { vec3 col = vec3(0.0); float convergence = 0.0; neighbors[0] = uv + vec2(-1.0 / iChannelResolution[3].x, 0.0); neighbors[1] = uv + vec2( 1.0 / iChannelResolution[3].x, 0.0); neighbors[2] = uv + vec2(0.0, -1.0 / iChannelResolution[3].y); neighbors[3] = uv + vec2(0.0, 1.0 / iChannelResolution[3].y); for (int i = 0; i < NUM_NEIGHBORS; ++i) { vec2 q = neighbors[i]; col += isMasked(q) ? RES(q) : BASE(q); vec3 srcGrad = SRC(p) - SRC(q); if (mixingGradients > 0.5) { vec3 baseGrad = BASE(p) - BASE(q); col.r += (abs(baseGrad.r) > abs(srcGrad.r)) ? baseGrad.r : srcGrad.r; col.g += (abs(baseGrad.g) > abs(srcGrad.g)) ? baseGrad.g : srcGrad.g; col.b += (abs(baseGrad.b) > abs(srcGrad.b)) ? baseGrad.b : srcGrad.b; } else { col += srcGrad; } } col /= float(NUM_NEIGHBORS); convergence += distance(col, RES(p)); // TODO: converge fragColor.rgb = col; return; } fragColor.rgb = RES(uv); } |
The main fragment shader is for showing the result:
- iChannel0 stores the previous frame buffer
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
// Interactive Poisson Blending // Ruofei Du (http://duruofei.com) // License Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License. // Reference: P. Pérez, M. Gangnet, A. Blake. Poisson image editing. ACM Transactions on Graphics (SIGGRAPH'03), 22(3):313-318, 2003. const int NUM_NEIGHBORS = 4; vec2 neighbors[NUM_NEIGHBORS]; bool isMasked(vec2 uv) { return texture2D(iChannel1, uv).r > 0.5; } void mainImage( out vec4 fragColor, in vec2 fragCoord ) { vec2 uv = fragCoord.xy / iResolution.xy; fragColor = texture2D(iChannel0, uv); } |
Here is another result:
Misc
Finally, here is a mysterious result with wrong mixing of gradients 🙂
