SenSay: A Context-Aware Mobile Phone
`
` Daniel Siewiorek, Asim Smailagic, Junichi Furukawa, Andreas Krause, Neema Moraveji,
`Kathryn Reiger, Jeremy Shaffer, and Fei Lung Wong
`Human Computer Interaction Institute and Institute for Complex Engineered Systems
`Carnegie Mellon University, Pittsburgh, PA 15213, USA
`{dps, asim}@cs.cmu.edu, {junichif, krausea, nmoravej, kr, jshaffer, flw}@andrew.cmu.edu
`
`Abstract
`
`SenSay is a context-aware mobile phone that adapts to
`dynamically changing environmental and physiological
`states.
`In addition
`to manipulating ringer volume,
`vibration, and phone alerts, SenSay can provide remote
`callers with the ability to communicate the urgency of their
`calls, make call suggestions to users when they are idle,
`and provide the caller with feedback on the current status
`of the SenSay user. A number of sensors including
`accelerometers, light, and microphones are mounted at
`various points on the body to provide data about the user’s
`context. A decision module uses a set of rules to analyze
`the sensor data and manage a state machine composed of
`uninterruptible, idle, active and normal states. Results from
`our threshold analyses show a clear delineation can be
`made among several user states by examining sensor data
`trends. SenSay augments its contextual knowledge by
`tapping into applications such as electronic calendars,
`address books, and task lists.
`
`1. Introduction
`
`SenSay (sensing & saying) is a context-aware mobile
`phone that modifies its behavior based on its user's state
`and surroundings. It adapts to dynamically changing
`environmental and physiological states and also provides
`the remote caller information on the current context of the
`phone user. To provide context information SenSay uses
`light, motion, and microphone sensors. The sensors are
`placed on various parts of the human body with a central
`hub, called the sensor box, mounted on the waist (see
`Figure 1).
`
`states:
`four
`following
`the
`introduces
`SenSay
`Uninterruptible, Idle, Active, and the default state, Normal.
`A number of phone actions are associated with each state.
`For example, in the Uninterruptible state, the ringer is
`turned off.
`
`In a much more limited context the idea of smart
`appliances and phones was explored in [1], [3], [4] and [5].
`In [2] concepts of context-aware computing and wearable
`devices have been described.
`
`Figure 1. SenSay: sensor box mounted on the hip (left),
`the mobile phone (center), and voice and ambient
`microphones mounted on the user (right).
`
`2. SenSay Architecture
`
`A three-tier architecture was adopted: the sensor box,
`decision module, and phone. The following components
`are shown in Figure 2. The sensor module, located in the
`bottom left, collects physical sensor data, which are then
`sent to the notebook computer (henceforth called the
`platform) through the serial port. The decision module at
`the top is then notified of data arrival, and a series of
`preprocessing steps are done to the incoming data before
`the data is acted upon. Finally, the decision module
`instructs the phone to act based on the current user context.
`The decision module utilizes another serial port to
`communicate with the phone.
`
`3. SenSay Logic
`
`The decision module inspects the gathered data and
`determines the state that the phone should enter. To
`prevent bouncing between states too quickly, up to ten
`minutes of recent sensor data are stored and examined.
`Running averages are computed to give reasonable weight
`to previous data and phone state. Furthermore, four states
`are identified by the system, representing descending levels
`of uninterruptibility.
`
`The system enters uninterruptible state when the user is
`involved in a conversation or has scheduled an important
`event in the electronic calendar. Once the phone is in this
`state, all incoming calls are automatically responded to
`with SMS messages. The ringer is disabled; vibrate is
`enabled only when the light level is low. The caller has an
`
`Proceedings of the Seventh IEEE International Symposium on Wearable Computers (ISWC’03)
`1530-0811/03 $ 17.00 © 2003 IEEE
`
`Page 1 of 2
`
`

`

`abdomen. The physical activity test was run using the 3-
`axis accelerometer. The maximum of the three absolute
`component values was used as the movement data. The
`data is split into three ranges. Low Activity includes sitting,
`sleeping, etc. Short, intense movements are averaged out.
`Medium Activity represents walking or other comparable
`activity. Medium movement indicates that the user is not
`idle. High Activity includes movements such as running.
`
`To find generic threshold values, eleven subjects were
`asked to perform a test: After walking for 40 sec., they
`were asked to sit down for 10 sec. Then they were required
`to run for 30 sec. and afterwards walk again. After walking
`for 20 more sec., they sat down again for an additional 25
`sec. Figure 3 shows the resulting values. From this
`experiment, thresholds for differentiating between low,
`medium and high activity were found and annotated on the
`diagram. Other tests were conducted for the microphones
`and light sensors.
`
`5. Conclusions
`
`SenSay combines sensory data, user information and
`history information to create a context-aware phone that
`improves the overall usability of the cell phone. Our
`threshold analyses on sensor data indicate that it is possible
`for a mobile phone to derive particular characteristics of
`user state.
`
`Acknowledgements
`
` Other
`Susan Finger was another project advisor.
`participants were James Casazza, Patrick Choi, Mehmet
`Gerceker, James Grace, Gerard Hamel, Kenneth Herman,
`C. Kampman Lasater, Anna Li, Daniel Patterson, Seng Tek
`Sing, Chee Wan Teng, Louis Trebaol, Brandon Weber and
`Fei Lung Wong. This material is based upon work
`supported by the National Science Foundation under Grant
`No. 0205266 and
`the Pennsylvania
`Infrastructure
`Technology Alliance grant.
`
`References
`
`[1] A. Schmidt and K.Van Laerhoven. “How to Build
`Smart Appliances?” IEEE Personal Communications, Aug.
`2001, pp. 66 - 71.
`[2] A. Smailagic and D. Siewiorek. “Application Design
`for Wearable and Context-Aware Computers,” IEEE
`Pervasive Computing, Vol 1, No. 4, December 2002, pp.
`20 – 29.
`[3] H. Lieberman and T. Selker, “Out of Context:
`Computer Systems That Adapt To, and Learn From
`Context,” IBM Systems Journal, No. 3 & 4, 617632, 2000.
`[4] A. Schmidt, et al, “Advanced interaction in context,”
`Proc. of Intl. Workshop on Handheld and Ubiquitous
`Computing, Num 1707 LNCS, Heidelberg, Germany, 1999.
`[5] S. Hudson et al., “Modeling user behavior: Predicting
`human interruptibility with sensors: a Wizard of Oz
`feasibility study,” Proc. Conference on Human Factors in
`Computing Systems, ACM Press, April 2003, pp. 257-264.
`
`DECISION LOGIC
`
`P H O N E
`
`Decision module
`Action module
`Sensor
`module
`
`Serial
`
`Serial
`
`Scheduler
`
`SenSay App.
`
`GSM
`
`Caller
`
`Preprocessing
`
`Serial
`
`USB
`
`Serial
`
`USB
`
`X-Axis
`accelerometer
`
`Y-Axis
`accelerometer
`
`Z-Axis
`accelerometer
`
`Bluetooth
`microphone
`
`Ambient
`microphone
`
`Light
`
`S E N S O R S
`
`Figure 2. System architecture
`
`option to override this in case of emergency by calling
`again within three minutes. High physical activity or high
`ambient noise level puts the system into active state. The
`ringer is set to high and vibrate is enabled. The system
`goes into idle state when there is very little movement and
`low ambient level. The system reminds the user of missed
`calls and provides suggestions to the user. In normal state,
`the ringer and vibrate modes are set to the phone’s default
`values.
`
`4. Experiment
`
`A series of threshold analyses tests were run while
`recording sensor values and noting trends over time.
`Microphones and accelerometer values were recorded from
`eleven subjects. In addition to the raw sensor values,
`average sensor values were also observed over various
`periods of time.
`
`As an example of threshold experiments, consider user
`activity. The sensor board was taped to the user’s
`
`walk sit run walk sit
`
`Figure 3. Motion state thresholds
`
`Proceedings of the Seventh IEEE International Symposium on Wearable Computers (ISWC’03)
`1530-0811/03 $ 17.00 © 2003 IEEE
`
`Page 2 of 2
`
`