How Should We Evaluate Systems Contributions to Ubicomp?

Panelists:
Keith Edwards, Palo Alto Research Center
Armando Fox, Stanford University (moderator)
Anthony LaMarca, Intel Research
Brian Noble, University of Michigan
Yi-Min Wang, Microsoft Research

Ubiquitous computing is inherently multidisciplinary: it comprises systems, human-computer interaction, AI, industrial design, mechanical engineering, and a host of other disciplines. From the point of view of computer science, ubicomp is also fundamentally about systems building: we are architecting and implementing systems that will interact directly with the end user. For this very reason, some of the "traditional" criteria by which "classical" systems work has been evaluated are not easily applied to ubicomp. As a result, it is often difficult to discuss the merit of systems-centric ubicomp papers, especially for forums that solicit interdisciplinary work such as MobiSys, UbiComp, and PerComm. For example, performance may not be critical when system response time requirements are dominated by human-scale latencies. Scalability may not be a concern if a system is designed for localized deployment, such as an auditorium or a building rather than an entire campus. On the other hand, failure resilience is still important, but we must consider the effects of a failure on the user experience as well as on the behavior of the system. Similarly, extensibility, programmability and maintainability become critical if we expect these systems to survive a long time and be subject to ongoing evolution.

Do we need a new look at evaluation criteria for systems contributions to ubicomp? Should systems researchers be working much more closely with HCI and other practitioners, and if so, precisely how should we expect such collaborations to change the tactics or research foci of systems folks? Even if we know what systems properties we want to measure, are we using the right _metrics_ for measuring them in a ubicomp environment?

We will hear about this topic from academic researchers and industry practitioners who have architected, designed, implemented, and deployed mobile or ubiquitous computing systems to actual users.

Keith Edwards is a Senior Member of Research Staff at the Palo Alto Research Center (PARC), where he leads the Speakeasy project on evolvable interoperation in ad hoc networks. Keith's research interests sit at the intersection of distributed systems and human computer interaction--how systems concerns "show through" to the user experience and, in turn, how the requirements of the user experience manifest themselves in the system. Recently, he has been working to understand how to apply user-centered design and evaluation techniques to the creation of infrastructure technologies.

Anthony LaMarca is a member of the research staff at Intel Research Seattle. His research interests include ubiquitous computing, distributed systems and human-centered design. He is currently developing techniques for making ubiquitous computing systems easier to deploy and maintain. He has a BS in computer science from the University of California at Berkeley and an MS and PhD is computer science from the University of Washington. He can be contacted at lamarca@intel-research.net.

Brian Noble is the Morris Wellman Faculty Development Assistant Professor in the Electrical Engineering and Computer Science department at the University of Michigan. His research centers on software supporting mobile devices and their users. He completed the PhD in computer science at Carnegie Mellon University in 1998, and is a recipient of the NSF CAREER award.

Yi-Min Wang led the Aladdin team to build a remote home automation system. The system uses soft-state-based lookup services to integrate diverse powerline and phoneline devices, and makes the devices accessible to homeowners when they are away from home. They can send an email to close the garage door, receive a cell phone message when the refrigerator is leaking, or get notified by an IM when the kids come back home from school. The Aladdin system has been deployed in an actual house for three years and used by the homeowners on a daily basis.