Projects

• Home
• Research
  • User-centric Interfaces
  • Human Gesture Analysis
  • Smart Environments
  • Earlier Projects
• People
• Publications
• Courses
• Contact

Smart Environments

• User-centric Environment Discovery

  Automatic environment discovery facilitates deployment of camera networks in smart homes. In this project, objects in a smart home are identified based on observed user interactions and a data integration and reasoning technique. This approach is complementary to the traditional appearance-based object recognition methods, which often demand large training sets for detecting each object of interest. In our approach, object recognition is achieved in a semantic way by linking object behaviors to the pose and activity of the person using them. The complex relations between objects and user activities are modeled with a Markov logic network (MLN). A Markov logic network effectively encrypts commonsense knowledge of complex relations associated with the object in the syntax of first-order logic. As a probabilistic graphical model, it also handles ambiguities in the relations as well as uncertainties in the outputs of vision processing. We present a system consisting of feature extraction and activity analysis based on a camera network, and object inference with MLN. Each camera in the network produces local description of the user’s location and activity, which are fused in a collaborative processing module to calculate the user’s location in 3D space and estimate activity. Activity and location information are sent to the MLN for the inference of object probability. Differences of the inference methods between MLN and general graphical models are discussed. An embodiment of the proposed approach in a multi-camera smart home environment is developed.

• Energy-Efficient User-centric Light Control in Smart Homes

  Smart homes are conceptualized as environments responsive to user's presence and actions and adaptive to user preferences and behavior models. Visual information plays an enabling role in smart home applications such as interfaces and gesture control. This project is based on the use of cameras and a distributed processing method for automated control of lights in a smart home. The implemented optimization formulations maintain the user's comfort while reducing the energy cost of lights. Information from camera sensors provides occupancy reasoning and human activity analysis. By employing the user's positions, activities, and preference as constraints, the system optimizes the light setting for the user's satisfaction in the occupied area.

• Assisted Living

  In this project we develop a distributed vision-based smart home care system aimed for monitoring elderly persons and patients remotely. The cameras can continuously monitor the user or be triggered by a broadcast from a user badge when an accidental fall or other conditions are sensed by the badge. Tracking the approximate position of the user by the camera network allows for triggering cameras with the best views. Distributed scene analysis modules analyze the user’s posture and head location; these information are merged through a collaborative reasoning module, which makes a final decision about the type of the report that needs to be prepared. The developed prototype also allows for a voice channel to be established between the user badge and a call center over the phone line. By making use of different camera views, the number of false alarms is reduced, making the system more reliable and more efficient.

• Gesture Placement in Gaming and Virtual Reality

  Human pose estimation is a complex problem which can involve estimation of a large number of degrees of freedom for the model. In this project we explore a variety of multi-camera approaches based on feature and data fusion to develop a skeleton model replicating the pose and gestures of the user. The skeleton model contains the necessary information for placing the actions of the user in a virtual reality or gaming environment.

• Smart Surface

  In this project we develop a multi-touch surface using multiple cameras. With an overhead camera and side-mounted cameras we can determine the x-y coordinates of the fingertips and detect touch events or measure hovering patterns, which are interpreted as commands for manipulating objects in computer presentations or applications.

• Context-based Reasoning

  Until recently, computer vision has drawn heavily from signal theory. In general, classifiers have relied on local, low-level image statistics without considering surrounding objects. Natural images (i.e., those without content constraints) have a great deal of context which can be used to infer particular characteristics of objects in the scene. When multiple classifiers are run sequentially on the same image, they operate independently of one another. Intuitively however, we know that certain objects tend to be present together (e.g., spoons and forks, or tables and chairs), and are linked by well-defined spatial relationships.
  
  This project seeks to improve the performance of object classifiers by incorporating contextual information. For example, probability maps are used to guide classifiers to image regions likely to contain the object in question, based on the object's past positions and the positions of surrounding objects. Interactions between the user and the objects in the environment are used to establish behavior models, which can for example be used in assisted living applications to detect abnormal activity patterns.