Using Salience Differentials to Making Visual Cues Noticeable

 

Andreas Heiner

Nokia Research Center, Helsinki, Finland

N. Asokan

Nokia Research Center, Helsinki, Finland

 

Abstract

 

There is a large body of research on how to design user interfaces for systems so as to ensure that important messages from the systems are presented to the user to maximize the probability that the user notices, and acts on, the messages.  Nevertheless in security-related applications on general-purpose computer systems in use today, there is significant evidence that users routinely ignore system messages.   We discuss possible reasons for this state of affairs, and propose a potential approach to improve it.  Noticeability of messages depends on the salience with which they are displayed.  We propose that given a message, the required salience differential for its display can be calculated as a function of the relevance of the message and the confidence with which the relevance was determined.  This approach has wide applicability and can be used for making visual cues more noticeable in a variety of applications such as security indicators and recommendations in automated recommendation.

 

1. Introduction

Conveying visual information to the human user is an integral part of modern computer systems and end-user applications that run on them. There is a large body of work discussing how to optimally interact with users in the execution of a primary task ([WH04]); considerable less work has been devoted to mechanisms of how to interact with users such that they switch to another (initially secondary) task.

Examples of messages targeted at changing the users’ task are advertisements and security messages. The most intrusive method is using pop-ups windows interrupting the primary task; the user has to act to return to the primary task. Another method to advertising is displaying the advertisements when a new domain is visited for the first time in a browser session. The duration of these message is short (typically 10-15 sec), and the user has the opportunity to skip them. Yet another approach is to occupy the top-bar of the screen. The least intrusive approach to advertisement is used by Google: selected relevant advertisements are presented in the right-most column in a style similar to that of the website itself; however, the user may miss the ads altogether. The dilemma is obvious: an intrusive message (shown with high salience) will irritate the user who will not pay attention to the content of the message and may look for ways of avoiding them in the future; a subtler message (shown with low salience) may not be noticed by the user at all.

 

 

Figure 1 Relevance – salience mapping. A very cautious security policy (red) will flag all messages and interrupt the primary task. A very liberal security policy (black) requires a search for the message[1].

 

Security-related messages are different in that they warn against a potential risk. When the approach is too conservative the user will learn to ignore the messages; with a too liberal approach (no warnings at all) the user is exposed to potentially serious threats. To improve the quality of decision making context-sensitive guidance can be used to assist the user in making the appropriate security decision ([BV07]). This reduces the possibility of habitual ignoring the message. However, the trigger to initiating the user guidance is still the same as in the most conservative approach; the main difference is in the quality of the decision. The method to provide a more secure environment compared to the “laissez faire” is to provide visual cues to the user. The effectiveness of this has not been satisfactory. Dhamija et al show that visual cues in browsers about the security of web sites visited are largely ignored by users  ([SDOF07], [WMG06]).

In this position paper, we first draw from research in visual perception to understand ways in which messages can be made noticeable to users.  We then discuss possible reasons why visual messages in various applications and systems today are not noticeable.  We then propose a new approach for making visual messages more noticeable and discuss its applicability. 

This paper does not discuss the source of a message; we assume that sufficiently accurate information is available, e.g., from portal owner or software security vendor. The actions following the user becoming aware of a message is also out of scope; we do not discuss the most appropriate text or icon to be used for a message.

2. Visual Perception

 

Visual salience is “the distinct subjective perceptual quality which makes some items in the world stand out from their neighbors and immediately grab our attention.”[SCH07]. Visual perception starts with a pre-attentive phase in which the observer identifies the most noticeable features (bottom-up salience map) of an image. Elements to the bottom-up salience are e.g., color, shape, luminescence, relative positioning etc. of objects in the image. The bottom-up salience is modulated by objective and task of the observer. The end result is a number of salient points in relation to the current task (task-dependent salience map). These maps can be numerically modeled with great accuracy ([RBE06], [PI07]). The most salient features can also be tracked with good accuracy in time-dependent tasks such as video games.

New elements that are detected go through the same process of pre-attentive discovery. A user notices the new object if the salience of it is larger than that the task-related features. Once aware of the new feature cognitive processing of it will commence ([TAK00], [SC03]). Whether or not a new visual element can trigger a secondary task depends therefore on the cognitive load to execute the primary task (focus). An interesting consequence is that the same object (message box) is not noticed when reading a difficult part of a document, but is noticed when reading a less complex part of the same document. The message representation did not change, but the reducing focus made it noticeable.

To prevent messages unduly affecting the primary task they should be shown in the peripheral zone. There is extensive evidence from user studies that salience in this area depends on motion and changes in luminescence of the icons, not to color or object detail ([BWC01], [BWC03]). This is consistent with the anatomy of the eye: color sensitive receptors are located in the center, those insensitive to color at the edges. However, these color insensitive cells are very sensitive towards luminescence. Moreover, the receptor density (irrespective of type) decreases with increasing distance from the center of the retina. The result is that detail recognition is good at the center, and low at the edges. However, sensitivity towards motion and luminescence changes is still high at the edges of the retina. The sensitivity towards these features is enhanced by the different chemical composition of the color-insensitive cells ([KFN05]).

3. Challenges in making messages noticeable

 

Security indicators in web browsers, such as the lock icon and anti-phishing indicators, compete with a large amount of other functions in the toolbar such as preferred search engine, email accounts, navigation buttons, URLs etc. This cluttering of information reduces the salience of each indicator; it becomes very difficult to find the correct indicator, even when the focus is on the toolbar.

When the user is focused on his actual task (reading the web page) toolbars are in the region of peripheral vision rather than in the foveal or parafoveal area. Indicating status changes by changes in color or shape of toolbar elements has little effect; peripheral vision is not sensitive to these object characteristics. Training the user to regularly inspect the status of indicators is also doomed; it will be done only when the focus on the primary task is low. The probability of developing habitual behavior is high; by definition a change in security status is an rare and unexpected event, and the user may notice the change, but does not pay due attention to it  [NWQ06]. Messages displayed in the peripheral zone have to be shown by changing object luminescence and position.

The natural question then is why visual security indicators are relegated to the peripheral region.  There are two possible explanations. First, the competition for pixel real-estate among various applications is fierce. It is impractical to reserve prime locations in the display for security; security never is the primary task. Second, application designers are wary of annoying users by distracting their attention too frequently from their primary task.  This tendency is evident in other contexts as well. For example, one reason for the popularity of Google is that the advertisements it serves up are in the periphery of the user’s vision and is therefore less intrusive, i.e. less noticeable. However, if messages have too low a salience, it defeats the purpose of showing them in the first place. Worse, in the case of security message it may be outright dangerous.

The above is also valid for handheld devices. One could argue that the display size is that small that there effectively is no peripheral area. However, mobile devices also have a large amount of status indicators: messages, connectivity icons (Bluetooth, WLAN, cellular), battery status, etc.

4. Salience Differential

 

We propose that given a visual message, an application should calculate its salience differential as a function of two parameters:

·         The relevance of the message

·         The confidence with which the relevance has been determined

The relevance metric represents how important or interesting a message is to the user; this is clearly context-dependent. Confidence expresses the certainty that the message is indeed as interesting or important as suggested by the relevance metric. The approach is illustrated in Figure 2. The action associated with a message of high relevance but low confidence is shown with medium salience (black dashed). Messages with high confidence and better than medium confidence are shown with high salience (black, solid). The user is notified of messages of medium relevance and confidence. Most other combinations become notifications. The overall result is that, even with a relative cautious security policy, the user interacts less with the system. In the case of advertisements the hit rate of advertisement is likely to become higher.  

In the case of security messages relevance may be determined by the severity of the consequences; installation of ransomware results in larger damages than installation of adware. The relevance metric could include the aspect of urgency: an ongoing fishing attack requires immediate action, indicating a site with potentially harmful software does not. The confidence of the message being correct is higher if security experts have made the assessment, rather than that it is based on rumors of non-experts.

Figure 2 Relevance – confidence map determining the message salience and the associated actions[2].

 

The relevance in rating systems typically is the average community rating. In this case the standard deviation of the (weighted) rating distribution is a logical choice for confidence parameter. For advertisements relevance can be defined by e.g. similarity between advertisement and current task description. The confidence could be based on the number of matching keywords compared to the total number of keywords.

The salience differential has similarities with the “arousal strength” used by [AM06], but also clear differences. The main difference lies in the fact that the salience differential is about controlling the noticeability of a message; the arousal strength refers to the association people make between words and icons, and the message. In their approach the primary task flow is interrupted irrespective of the importance of the underlying process that results in the message; they find that this may lead to habitual behavior. In our model the user becomes aware of the message; only in case of sufficiently high importance (mostly security threats) the primary task flow is interrupted.

Our approach draws on the approach by [BWC01], [BWC03]; it extends it by controlling the salience, and thereby controlling the moment the message is noticed by the user. We also use the concept of relevance and confidence of the message.

5. Discussion

 

5.1 Calculating the Salience Differential

 

The key to making a visual message noticeable is to ensure that that it has sufficient visual salience given its current context. The message competes with the other visual information, and, in order to be noticed, must change the salience map sufficiently that the new salience map cannot be anticipated from the previous salience map.

The salience differential is calculated as the difference between the task-modulated salience map with and without the message representation. Attributes that contribute to image saliency such as object location, dynamics, luminescence etc. are modified until the computed salience differential equals that required by the (user-defined) policies.

5.2 Advantages of using differential salience

 

The goal of conveying a visual message is to guide user action. Thus efforts to make visual messages more noticeable go hand-in-hand with ways of guiding the user towards the “correct” reaction. However, the current approach is “all or nothing”; either a message is not noticed, or the task flow is interrupted. Both [AM06] and [BV07] interrupt the primary task flow and propose methods to improve on the decision. The first author does so by selecting the most appropriate wording and icon, the second author by introducing a context-dependent “conversation” between user and system.

Salience control is very useful in the context of advertisements. If the advertisement strategy is conservative (low salience) they may not be identified as advertisement, or not noticed at all. By calculating the differential salience the advertisement can be shown with a good chance of the user actually reading it. Also, two advertisements with similar relevance and confidence (differential salience) are noticed at the same time. Note that the approach by [AM06] and [BV07] cannot be used for advertisements: their methods rely on interruption of the primary task. The advantage of controlling the salience differential is that the user (through the policies) can determine when he notices a message. In situations of high cognitive load the user will not notice less important messages, when the focus on the primary task reduces (“the user has time”) he becomes aware of the message, and can act according the salience – action policy (Figure 3)

Figure 3 Focus – salience comparison. If the focus is larger than the salience the message will not be noticed; the reverse also holds. The action is determined by the salience, the focus by the primary task[3]

 

If the message is security-related the approach can be used to indicate different degrees of risk such that they are better noticed, and the primary task flow is not interrupted needlessly. In related work on increasing the usability and security of software installation, we are taking this approach to deal with software deemed to be suspicious.  In addition to warning the user, we also make it difficult for the user to go ahead and install the software by breaking the workflow. The degree of difficulty could be a function of the relevance of and confidence in the message, similar to the way the salience differential was calculated.

Since salience differentials are calculated with respect to present the background it is easy to make messages noticeable irrespective of the background. This is important as personalized skins and rich, dynamic content of modern websites where traditional color schemes for messages may not work as well (Figure 4).

5.3 Limitations and future work

 

The validity of the proposal can be challenged in a number of ways. Most models of attention capture and saliency are based on experiments utilizing large screens; some authors suggest that larger screens are needed to better understand the mechanisms. We target small screens as well, and extrapolation is not straightforward. Further, most studies focus on attention capture and awareness of elements in the primary task using visual search as experimental model. Those are cognitive processes using information from the central area of the retina. Our objective is to initiate a secondary task, i.e. new cognitive processes should follow the surprise.

We plan to test our ideas in the context of usable and secure software installation. In this context software attributes such as “contains adware”, or “socially less acceptable” are used as relevance metric. The user is alerted of an imminent risk via a polymorphic dialog in which the complexity of installation is higher for more risk-carrying. Alerting the user is a binary operation; we intend to change this into a coarse-grained method using the approach described above. We will verify the novel approach with user studies.

6. Conclusions

 

We have proposed a novel approach to displaying messages in a pixel-efficient manner. Messages are displayed taking into account the relevance of the message in the present context, and the confidence that it is indeed relevant. The method can be used to indicate the most interesting or important sub-task or related content in the current primary task, and is very suitable to indicating important or interesting secondary tasks in a granular such as inspecting advertisements.

Acknowledgement

 

We thank the two anonymous reviewers for their helpful comments and suggestions.

References

 [AM06] Signal Words and Signal Icons in Application Control and Information Technology Exception Messages – Hazard Matching and Habituation Effects. T.S. Amer, J.B. Maris (2006) Working Paper. Northern Arizona University, CBA Working Paper Series 2006-2003, ref. 06‑05

[BV07] Improving Security Decisions with Polymorphic and Audited Dialogs Jose Carlos Brustoloni, Ricardo Villamarin-Salomon. (2007) ACM International Conference Proceeding Series; Vol. 229, pp76 – 85. Proceedings of the 3rd symposium on Usable privacy and security.

[BWC01] Moving icons: detection, distraction and task. Lyn Bartram, Colin Ware, Tom Calvert (2001). In: Hirose, M. (Ed.), Proceedings of the IFIP International Conference on Human-Computer Interaction (INTERACT) '01, IOS Press, Tokyo, pp. 157-165.

[BWC03] Moticons: detection, distraction and task. Lyn Bartram, Colin Ware, Tom Calvert (2003). International Journal of Human-Computer Studies, Volume 58, Issue 5, May 2003, pp 515‑554

[KFN05] Simple Anatomy of the eye. Helga Kolb, Eduardo Frenandez, Ralph Nelson: In. Webvision: the Organization of the retina and visual system, part 1. http://webvision.med.utah.edu/index.html.

[NWQ06] Habits: A Repeat Performance. David T. Neal, Wendy Wood, Jeffrey M. Quinn. Current Directions in Psychological Science 15 (4) (2006), 198–202

 [PI07] Congruence between model and human attention reveals unique signatures of critical visual events (2008). Robert J. Peters, Laurent Itti. To appear in “Advances in Neural Information Processing Systems, Vol. 21 (NIPS*2007), pp. 1-8, Cambridge, MA: MIT Press, Jun 2008

[RBE06] An attentional system combining top-down and bottom-up influences. Babak Rasolzadeh, Mårten Björkman and Jan-Olof Eklundh (2006). International Cognitive Vision Workshop (ICVW06) at ECCV2006, Graz, Austria, May 2006

[SC03] Controlled & automatic processing: behavior, theory, and biological mechanisms. Walter Schneider, Jason M. Chein (2003) Cognitive Science 27 (2003) 525–559

[SCH07] http://www.scholarpedia.org/article/Visual_salience

[SDOF07] The Emperor’s New Security Indicators: An evaluation of website authentication and the effect of role playing on usability studies. Stuart E. Schechter, Rachna Dhamija, Andy Ozment and Ian Fischer. The 2007 IEEE Symposium on Security and Privacy May 20-23, 2007 in Oakland, California.

[TAK00] On the time course of top-down and bottom-up control of visual attention .Jan Theeuwes, Paul. Atchley, Arthur F. Kramer. In. Attention & Performance Vol 18, p.105-125, S. Monsell & J.Driver (Eds.). Cambridge: MIT Press (2000)

 [WH04] What attributes guide the deployment of visual attention and how do they do it? J. M. Wolfe ,  T. S. Horowitz (2004) Nature Reviews Neuroscience 5:1-7

[WMG06] Do Security Toolbars Acually Prevent Phishing Attacks? Min Wu, Robert C. Miller, Simson L. Garfinkel. Conference on Human Factors in Computing Systems Montréal, Québec, Canada (2006). Proceedings of the SIGCHI conference on Human Factors in computing systems Pages: 601 – 610.

Appendix 1. Peripheral vision in mobile devices

In mobile devices focus on the primary task is obtained via highlighting the primary function (Figure 4). Status indicators in the top segment of the display (3G signal strength, unread message, battery status, WLAN) and indicators for other folders are in the peripheral area. Status indicators in the peripheral area are even more obscured by personalized skins (right).

Figure 4 Peripheral area for small displays

 

Appendix 2. Message salience in games

 

The game System Rush Evolution is a 3D Action arcade with high dynamics. The user focus is on space ship in the center of the image; the remainder of the display is background. To show an informational message to the user the standard “i” icon is hardly visible (Figure 5). Changing icon color is not useful as the background color changes rapidly. The salience of the message increases if the icon is placed in a white oval (Figure 6). The oval can be stationary, or may wiggle around a central point. The icon may be used to indicate security notifications, or presence of relevant advertisements.

Figure 5 Notification message with low salience.

 

 

Figure 6 Notification message with high salience.