Bir süredir kendi kız kardeşime kapıyı açmıyorum. Ne aramalar, ne ziyaretler, ne de en ufak bir ilgi—tamamen görmezden geliyorum. Kulağa belki acımasız geliyor. Ama sade# Background Removal with Structured Light
[Paper’s Link](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/BackgroundRemovalEurographics2004.pdf)
## Summary
We present a real-time system for creating seamless composite videos and photos where the foreground consists of the objects in front of a planar background and the new background is an arbitrary image or video. The key idea of our approach is to use structured light for capturing alpha mattes in real time (15 frames per second). For background removal we project a pattern of vertical stripes onto the background surface and then analyze the deformation of the pattern observed by the camera. The approach is based on the observation that the deformation of the projected pattern by foreground objects is significantly more pronounced than effects due to surface reflectance or curvature and thus can be used to reliably distinguish foreground from background. Since our method is purely geometric (as opposed to color-based), it can handle arbitrary backgrounds and foregrounds as long as they do not occlude each other. Our system uses PC graphics hardware to implement real-time structured light acquisition as well as for real-time compositing of the extracted foreground with new backgrounds. Additionally, we present a fully automatic calibration procedure that computes the homography between camera and projector in a few seconds. The resulting system is easy to set up and can be used by casual users.
## Keywords
image matting, structured light, real-time, projective textures, video compositing, calibration
## 1. Introduction
image matting은 컴퓨터 비전과 그래픽스에서 중요한 문제이다. `Image matting is an important problem in computer vision and graphics.` 이 문제는 일반적으로 foreground, background, alpha 로 분리하여 새 background와 합성하는 것이다. `It usually involves separating out the foreground from the background with an alpha channel (the matte) and then optionally combining the foreground with a new background (compositing).` 이것은 영화나 비디오 프로덕션에서 종종 사용된다. `This operation is commonly used in film and video production.`
지금까지 대부분의 matting system은 chroma keying와 같은 color-based methods에 의존해왔다. `Most matting systems to date rely on color-based methods, such as chroma keying.` 이러한 방법들은 특정한 색상(ex. blue or green)을 사용하여 background를 구성하는 것을 요구한다. `These methods require that the background consist of a special color (usually blue or green).` 이것은 foreground에 background color가 사용되면 문제가 된다. `This leads to problems if the foreground contains the background color.` Structured light가 이러한 color-based methosd를 보완하는 방법으로 개발되었다. `Structured light has been developed as a way to complement color-based methods (Chuang et al. 2002).`
기존 선행연구의 structured light approach는 input image sequence에서 background를 추출하기 위해 까다로운 multi-color pattern을 사용한다. `The structured light approach (Chuang et al. 2002) uses elaborate multi-color patterns to extract the background from an input image sequence.` 이 방법은 fully automatic robust matting algorithm이지만 복잡한 pattern과 선형의 optimization을 필요로한다. `While being a fully automatic robust matting algorithm, it requires complex patterns and the solution of linear optimization problems.` 그 결과로 실시간 동작이 불가능하다. `As a result, it does not operate in real-time.`
우리는 실시간으로 structured light를 이용하여 alpha matte를 추출할 수 있는 방법에 대해 제안한다. `We describe a structured light approach to extracting alpha mattes in real-time.` 우리는 까다로운 color pattern 대신 단순한 binary stripe pattern을 사용했다. `Instead of elaborate color patterns, we use simple binary stripe patterns.` foreground object에 의해 발생된 stripe deformation 만을 관찰하여 foreground를 추출하는 방식이다. `We analyze the deformation of these stripes (as caused by foreground objects) to extract the foreground.`
특히 background에 프린팅된 binary vertical stripe pattern을 관찰하여 vertical stripe pattern과 foreground object에 의해 변형된 stripe pattern과의 차이를 분석하는 방식이다. `In particular, we project a binary vertical stripe pattern onto the background, and analyze the difference between the observed stripe pattern with foreground objects and the known background pattern.`
우리 알고리즘은 structured light pattern을 이용하여 영상에서 foreground와 background를 분할한다. `Our algorithm segments foreground from background in an image by analyzing the deformation of a structured light pattern.` 이 방식은 background reflectance or curvature에 의한 효과보다 foreground object에 의한 stripe 변형이 월등히 크다는 관측에 기반을 두고 있다. `The approach is based on the observation that the deformation of the projected pattern by foreground objects is significantly more pronounced than effects due to surface reflectance or curvature.` 그러므로, foreground는 structured light input을 분석하여 추출될 수 있다. `Thus, the foreground can be extracted by analyzing the structured light input.`
## 2. Related Work
구조광을 이용한 image matting은 computer vision에서 연구된 바 있다. `The use of structured light for image matting has been studied in computer vision (Toyama et al. 1999).` Toyama et al. 은 배경에서 object를 detection하기 위해 시간에 따른 intensity difference를 이용했다. `Toyama et al. used temporal intensity differences to detect objects moving in front of a background.` 그리고 binary vertical stripe pattern을 이용하여 occlusion detection을 수행했다. `Additionally, they used a binary vertical stripe pattern to help detect occlusions in the background.`
Smith and Blinn은 blue screening 방법을 개선하기 위해 structured light를 사용하였다. `Smith and Blinn (1996) used structured light to improve the blue screening method.` 하지만, 이 방법은 color information만 사용한 blue screen method로 real-time 동작이 불가능하다. `However, their blue screen method is not real-time and relies solely on color information.`
Chuang et al. (2002) 는 structured light를 이용한 automatically compute alpha mattes 방식에 대해 연구했다. `Chuang et al. (2002) explored how to automatically compute alpha mattes using structured light.` 그들의 접근 방식은 foreground/background layer를 복원하기 위해 multi-frame multi-color patterns를 사용한다. `Their approach uses multi-frame, multi-color patterns to recover foreground/background layers.` 하지만, 선형의 optimization problem과 복잡한 color pattern을 필요로 하며 실시간 처리가 불가능하다. `However, it requires solving a linear optimization problem and complex color patterns, and is not real-time.`
우리의 방법 또한 structured light를 이용하지만, 실시간 처리가 가능하다. `Our method also uses structured light, but performs real-time acquisition and processing of alpha mattes.` 우리는 PCA와 같은 복잡한 방법 대신, vertical stripe의 변형을 관찰하는 간단한 방법을 이용하였다. `Furthermore, instead of using complicated approaches like PCA, our method relies on a simple analysis of vertical stripe deformations.`
## 3. System Overview
우리의 시스템은 camera, projector, PC로 구성되어 있다. `Our system consists of a camera, a projector, and a PC.` camera는 scene을 capture하고, projector는 structured light pattern을 background에 투영한다. `The camera captures the scene, while the projector projects a structured light pattern onto the background.` PC는 camera input을 처리하여 alpha matte를 생성하고, 새로운 background와 합성하는 역할을 수행한다. `The PC processes the camera input to generate an alpha matte and composite it with a new background.`
Structured light pattern은 binary vertical stripe이다. `The structured light pattern is a binary vertical stripe pattern.` stripe의 폭은 시스템의 resolution과 scene을 cover할 수 있는 영역에 따라 결정된다. `The width of the stripes determines the resolution of the system as well as the area that can be covered by the scene.` 일반적으로 stripe의 폭은 16~32 pixels로 설정된다. `In general, we use stripe widths between 16 and 32 pixels.`
## 4. Alpha Matte Extraction
### 4.1 Structured Light Pattern Analysis
카메라 영상에서 foreground object가 존재하지 않으면, 투영된 vertical stripe pattern은 왜곡되지 않은 형태로 관찰된다. `In the camera image, when there is no foreground object, the projected vertical stripe pattern is observed without any distortion.` foreground object가 존재할 경우, 객체에 의해서 stripe pattern이 왜곡되게 된다. `When foreground objects are present, the objects distort the stripe pattern.`
foreground object의 경계에서는 stripe pattern이 shift, stretch, compress되면서 stripe pattern의 edge가 흐려지게 된다. `At the boundaries of foreground objects, the stripe patterns are shifted, stretched, or compressed, causing blurring in the edges of the stripe patterns.` 우리는 이러한 stripe edge의 흐릿함(bluriness)를 관측하여 alpha matte를 생성한다. `We analyze this blurriness of the stripe edges to create the alpha matte.`
### 4.2 Alpha Calculation
alpha 값Artık kapıyı açmıyorum çünkü artık kendime ve çocuklarıma odaklanmam gerekiyor.
