The Poly-sensing Environment and Object Based Emergent Intention Matrix: Toward an integrated Physical / Augmented Reality Space. Principal Investigators: Dr. Ingrid Verbauwhede - Department of Electrical Engineering, University of California, Los Angeles Dr. Bill Seaman - Head, Graduate Digital Media Department, Rhode Island School of Design Dr. Mark Hansen - Department of Statistics, University of California, Los Angeles Statement of Objectives There are many exciting potentials related to the development of integrated sensing, networking and perception augmenting technologies at this time. In particular we can augment thought, research, education, learning, and creative expression by developing integrated sensing/augmented reality spaces that through networking link multiple environments for shared knowledge; allied physical/data-oriented information schemes and processes; and dynamic engagement with search engines that call forth digital content "associated" with sensed behavior in the physical/data space. We are developing an interactive informational and/or expressive IT system that seeks to augment human teamwork and/or interaction to promote scientific/cultural production through the creation of new collaborative multi-modal sensor devices that enable a form of "machinic perception". This "perception" and /or "associated" media is displayed and interacted with through a dynamically programmable visualization/sonification system. Methods to be Employed The top-down vision is to create an advanced integrated system - a collaborating field of bundled sensors; an object based authoring environment to focus the multi-modal employment of this embedded sensor technology (the Emergent Intention Matrix or EIM); an integrated DSP and server-based parsing environment of multi-modal data and associated media; and an augmented reality visualization/sonification environment that is networked for international collaboration. In realizing each component, the team will work together to form the set of bottom up studies to "organically grow" this environment. The members of the team best suited to exploring a particular aspect of the research will work on "chipping away" at the set of problems to be explored. The results of one experiment will be shared via networked technology with the other disciplinary teams. Through intercommunication all team members will, in an ongoing process-oriented fashion, determine how the research from one team impacts the other questions under study. Research will depend on the synergy of the groups and the constant sharing of knowledge. Methods to be employed include the setting of a communication network to link researchers; the bottom up testing of various sensors added one at a time in a modular manner in relation to the longer-term goal of the generation of a form of "Machinic perception", including experiments related to the active observation of parallel data flows and new methodologies of parsing such information; testing related to the creation, clarity, and ease of use of an object based programming language and interface for authoring "associated" media relations to sensed activity in the physical space; testing related to shifting the "attention" of the sensing nodes promoting differing qualities of observation; the researching of speed and latency questions of the sensors; the exploration of the intelligent use of energy on the chip as well as energy scavenging; the optimization of the sensors via intelligent DSP programming working in conjunction with server programming; the generation of a "machinic perception" that arises through the intelligent integrated parsing of chosen time-based data streams; the visualization of sensed data providing multiple data-oriented "pictures" of the physical environment that later become cross-referenced, informing a multi-modal "understanding" of the space; the active linking of multi-media databases and information retrieval algorithms; research related to the multi-modal "observation" of specific objects and/or behaviors to be "sensed" in a given space; ongoing work related to the integration of the various streams of research; the final shrinking and creation of a new embedded chip of bundled sensors; optimization for internet networking and higher bandwidth internet implementation. Building the poly-sensing environment will necessitate a high degree of cooperation and knowledge sharing between the different research teams. Enhancements to the expressiveness of the poly-sensing interface will initiate a cross-team discussion about new sensors that could be introduced into the environment. To add a new sensor into the network requires basic knowledge about its operating characteristics, its power consumption and the feasibility of integration into an existing node. Exploratory data analysis will establish a coarse representation of the signal obtainable from the sensor and evaluate what basic data transformations should be linked and applied at the node-level. More detailed statistical work will examine how much information the processed signal carries about activities in the poly-sensing environment, and its power to discriminate between various user activities. Some kind of training must then take place. In the final evaluation, new behaviors and new sensors might suggest themselves, expanding the EIM, requiring new statistical tools and, inevitably, new system requirements. Obviously, articulating a hierarchical ordering of such research proves to be problematic in that many aspects of the research inform other aspects in a non-linear manner. Never the less, the different disciplinary teams will work in an orderly, modular, process-oriented manner to slowly "grow" the potentials of the system through dynamic collaboration and intercommunication. This provides a unique multi-disciplinary learning environment where the research methodology of this project fosters a new model of integrated transdisciplinary scientific, engineering and media inquiry. Intellectual Merit Scientific and technological understanding will be enhanced via a series of published papers, and exhibitions in scientific, IT and cultural/artistic realms. The initial funding underscores the probable success of the research. Year 1 was funded by the Chancelor's Special Fund for Academic Border Crossing, UCLA; Year 2 was funded by the Langlois Foundation for Art, Science and Technology (both highly competitive interdisciplinary funding sources). This unique team brings together expertise in advanced chip production, statistical tools for data analysis and modeling of highly complex data streams, and novel multimedia environments. Broader Impact The significance of the system as a new integrated interface paradigm with multiple potential uses is unparalleled in its ability to integrate research and education by advancing discovery and understanding across a set of differing yet interrelated disciplinary domains. The long-term goal is the creation of an advanced tool for scientific, cultural and artistic production. Central, is the functional ability of the system to enhance the infrastructure for research, education, the creation of new kinds of machinic sensing research facilities; new forms of instrumentation that arises through visualization and sonification of the sensed environment; the networked potential of sharing this information or cultural production, and the potential partnerships arising through the networking capabilities of the environment. Clearly the long-term broader impact of this IT system is immense in that it advances knowledge across a series of disciplines, and becomes a tool that can later be used by each of the researchers for furthering their individual inquiry and practice. Yet the potential benefits for society at large cannot be underestimated in relation to educational, training, learning, cultural and scientific production. Thus, a connected physical space and associated augmented reality environment can be networked, linking individuals from around the globe. Networking capability will potentially broaden the participation of underrepresented groups that geographic proximity might normally exclude. The potential of the system is to enable any object, behavior or "sensed activity" in a given space to become an interface to an interactive augmented reality media space and/or the internet. This also suggests potential application for those with disabilities through new adaptive interface design approaches. back to menu