PrintSense: A Versatile Sensing Technique to Support Multimodal Flexible Surface Interaction
We present a multimodal on-surface and near-surface sensing technique for planar, curved and flexible surfaces. This technique leverages temporal multiplexing of signals coming from a universal interdigitated electrode design, which is printed as a single conductive layer on a flexible substrate. It supports sensing of touch and proximity input, and moreover is capable of capturing several levels of pressure and flexing. We leverage recent developments in conductive inkjet printing as a practical way to prototype electrode patterns, and combine this with our custom-designed hardware module for supporting the full range of sensing methods. As the technique is relatively easy to implement and inexpensive, it is particularly well-suited for prototyping touch- and hover-based user interfaces, including curved and deformable ones. This paper will be presented in CHI 2014, April 26–May 1, 2014, Toronto, Ontario, Canada. | publication | video (short) (long) | slides | website |
Inkjet-printed Conductive Patterns for Physical Manipulation of Audio Signals
A major part of my research is about utilizing conductive inkjet printing to create interactive sensing surfaces. Since inkjet printing enables a new way of fast sensing prototyping in a graphic way, it enables us to design sensors that not only serve as a part of the circuit, but a fully integrated part in an object. In this collaborative work with Dr. Amit Zoran, we present the realization of a completely aesthetically driven conductive image as a multi-modal music controller. Combining two emerging technologies – rapid prototyping with an off-the-shelf inkjet printer using conductive ink and parametric graphic design, we are able to create an interactive surface that is thin, flat, and flexible. This sensate surface can be conformally wrapped around a simple curved surface, and unlike touch screens, can accommodate complex structures and shapes such as holes on a surface. We present the design and manufacturing flow and discuss the technology behind this multi-modal sensing design. Our work seeks to offer a new dimension of designing sonic interaction with graphic tools, playing and learning music from a visual perspective and performing with expressive physical manipulation.The paper is published and demoed in UIST 2013.
A Cuttable Multi-touch Sensor
As part of the development of physical manipulation of sensate surfaces, I started a collaboration with Dr. Jürgen Steimle and Simon Olberding on developing a cuttable multi-touch sensor. We propose cutting as a novel paradigm for ad-hoc customization of printed electronic components. As a first instantiation, we contribute a printed capacitive multi-touch sensor, which can be cut by the end-user to modify its size and shape. This very direct manipulation allows the end-user to easily make real-world objects and surfaces touch-interactive, to augment physical prototypes and to enhance paper craft. We contribute a set of technical principles for the design of printable circuitry that makes the sensor more robust against cuts, damages and removed areas. This includes novel physical topologies and printed forward error correction. A technical evaluation compares different topologies and shows that the sensor remains functional when cut to a different shape. The paper is published in UIST 2013 and was covered in many technology blogs and news articles including Techcrunch, and Discovery News . | website | publication | video |
Mime: Compact, Low Power 3D Gesture Sensing for Interaction with Head Mounted Displays
I collaborated with the Speech and Mobility group at the Media lab and designed the hardware for Mime, a compact, low-power 3D sensor for short-range gestural control of small display devices. The sensor’s performance is based on a novel signal processing pipeline that combines low-power time-of-flight (TOF) sensing for 3D hand-motion tracking with RGB image-based computer vision algorithms for finer gestural control. Mime is an addition to a growing number of input devices developed around the engineering design philosophy of sacrificing generality for battery-friendly and accurate performance to retain the portability advantages of our smart devices. We demonstrate the utility of Mime for Head Mounted Display control and smart phones with a variety of application scenarios, including 3D spatial input using close range gestures, gaming, on-the-move interaction, and operation in cluttered environments and in broad daylight conditions. The paper is published in UIST 2013. | publication | video |
A Customizable Sensate Surface for Music Control
We developed a novel music control sensate surface, which enables integration between any musical instruments with a versatile, customizable, and essentially cost-effective user interface. This sensate surface is based on conductive inkjet printing technology which allows capacitive sensor electrodes and connections between electronics components to be printed onto a large roll of flexible substrate that is unrestricted in length. The high dynamic range capacitive sensing electrodes can not only infer touch, but near-range, non-contact gestural nuance in a music performance. With this sensate surface, users can “cut” out their desired shapes, “paste” the number of inputs, and customize their controller interface, which can then send signals wirelessly to effects or software synthesizers. Our preliminary design and concept was published at NIME 2012 and was featured in CNET and the Financial Times. | publication | video |
Leveraging Conductive Inkjet Technology to Build a Scalable and Versatile Surface for Ubiquitous Sensing -This floor sensing system was my project at Microsoft Research Cambridge. I worked with Dr. Steve Hodges and Prof. Joseph Paradiso on developing a versatile, scalable and cost-effective sensate surface based on flexible printed electronics. The system is based on a new conductive inkjet technology, which allows capacitive sensor electrodes and different types of RF antennas to be cheaply printed onto a roll of flexible substrate that may be many meters long. By deploying this surface on (or under) a floor it is possible to detect the presence and whereabouts of users through both passive and active capacitive coupling schemes. We have also incorporated GSM and NFC electromagnetic radiation sensing and piezoelectric pressure and vibration detection.
We report on a number of experiments which evaluate sensing performance based on a 2.5m x 0.3m hardware testbed. Our first pilot study was published in Ubicomp 2011, Beijing. | publication | slides |
Dense, Low-Power Environmental Monitoring for Smart Energy Profiling- I worked with Schneider Electric to design and deploy a dense, low-power wireless sensor network aimed at environmental monitoring for smart energy profiling. This distributed sensor system (~100 sensor nodes) measures temperature, humidity, and 3D airflow, and transmits this information through a wireless Zigbee protocol. These sensing units are currently deployed in the lower atrium of E14, MIT Media Lab. The data is being used to inform CFD models of airflow in buildings, explore and retrieve valuable information regarding the efficiency of commercial building HVAC systems, energy efficiency of different building materials, and lighting choices in novel architectural designs. Our work was accepted as a lecture paper at IEEE SENSORS conference 2012, and a journal paper in the Energy and Buildings Journal.
SPINNER – I worked with Dr. Mathew Laibowitz on his PhD thesis – SPINNER and developed several applications and user studies on this platform. Here’s my presentation at CMCVR 2010 – [pdf]. The SPINNER project is a first-of-its-kind research platform designed to investigate the world of ubiquitous video devices in order to confront inherent problems and create new media applications. This system takes a novel approach to the creation of user-generated, documentary video by augmenting a network of video cameras integrated into the environment with on-body sensing. The distributed video camera network can record the entire life of anyone within its coverage range and it will be shown that it, almost instantly, records more audio and video than can be viewed without prohibitive human resource cost. This drives the need to develop a mechanism to automatically understand the raw audio-visual information in order to create a cohesive video output that is understandable, informative, and/or enjoyable to its human audience. | project website | publications |
NONO Badge – I developed this badge during my MS program as a platform for studying the privacy concern and control for users in a pervasive sensor network. The badge can talk to the Ubiquitous Sensor Portals through IRDA, one of the infrared protocol and Zigbee radio, a low-cost, low- power, wireless mesh networking standard. Through sending a unique ID, the badge can be used for tagging sensor data in order to claim ownership for further editing. Also, it can send out an opting in or opting out signal to control the ubiquitous awareness portals. With this device, users can have on-site control of their privacy and the immediate feedback of the privacy levels in different scenarios. | publications |
MusicGrip is a real-time writing instrument for music control and education. The sensors can capture the subtle dynamics of the user’s grip while writing or drawing and map this to musical control sonic outputs. My project discusses this conversion of the common motor motion of handwriting into an innovative form of music expression. We seek to create an instrument that can be used to integrate the composing aspect of music with painting and writing, creating a new art form from the resultant aural and visual representation of the collaborative performing process. [pdf]
Au(Si)-filled β-Ga2O3 nanotubes as wide range high temperature nanothermometers – Back in my Materials Sciences days, I worked with Prof. Lih J. Chen on developing nano-scaled sensors and the dynamics of low-dimensional nano-materials. My master thesis was published and selected as the cover of Applied Physics Letters. Here’s the abstract of our paper: Au(Si)-filled β-Ga2O3 nanotubes were fabricated by an effective one-step chemical vapor deposition method. The Au(Si) interior was introduced by capillarity. Linear thermal expansion of Au(Si) with a coefficient of thermal expansion (CTE) as high as 1.5×10−4(1/K) within single crystal Ga2O3 shell up to 800° C was observed by in situ transmission electron microscopy. The high CTE is correlated to partial melting of Au(Si). As Ga2O3 possesses excellent thermal and chemical stability, the structure can be used as a wide range high-temperature nanothermometer within localized regions of nanosystems. [link]
High-Sensitivity Solid-State Pb(Core)/ZnO(Shell) Nanothermometers Fabricated by a Facile Galvanic Displacement Method – I worked with Dr. Chiu-Yen Wang on nano-thermometers during my master’s program. She worked on creating nano structures with galvanic displacement methods which requires only test tubs in room temperature to fabricate. This work is about a fast and convenient method to fabricate Pb(core)/ZnO(shell) nanostructures and their application for high-sensitivity thermal sensing. The Pb/ZnO nanocables were spontaneously reduced to nanostructures grown on zinc substrate at room temperature. The sensitivity of the temperature is as high as 50 nm per 100 K. The work was published in Advanced Materials. [pdf]