Master’s Thesis: Designing for user awareness and usability

August 22, 2017

During the first half of 2017 I conducted research for the CREDENTIAL project focusing on privacy enhancing technologies (in this case the CREDENTIAL Wallet). I will try to summarize my research in this post. However, considering this research was running for approximately 8-9 months there is a lot of ground to cover. I recommend, if interested, reading the thesis as I won’t be going too in-depth here.


Introduction & basis of research

Personal data is often disclosed with every registration, sharing, or request of an online service. With the increased usage of things connected to the Internet, users’ information being collected and stored, the risks related to unknowingly sharing personal data increases. Sharing of personal information is a sensitive subject and can hurt people’s assets, dignity, personal integrity and other social aspects.

In general, users’ concerns have grown regarding protecting their personal information which has led to the development of multiple privacy-oriented systems. In scenarios where users are logging onto a website or system, they rarely notice, understand or have desire to read the conditions to which they are implicitly agreeing. These systems are often referred to as identity management systems or single sign-on systems. Recent studies have shown that users are not aware of what data transactions take place by using various authentication solutions. It is critical for these types of system dealing with privacy that researchers examine users’ understanding of the concepts through interface design.

Multiple sources have addressed that there is apparent user confusion by the user not having the control to limit disclosure (of personal information) in consenting situations because the systems are being uninformative (Ronen 2013; Robinson & Bonneau 2014; Wang et al. 2013; Xu et al. 2012). Consequently, research tackling awareness of single sign-on services have been conducted which presents that users are mostly unaware of the data transactions (Bauer et al. 2013; Ronen et al. 2013; Egelman 2013).  It seems to be a lack of investigations of what design patterns functions well in terms of usability and user awareness of data transactions in authentication scenarios.

By the end of 2016 there were already a few constructed prototypes of an authentication system within the CREDENTIAL project, called CREDENTIAL Wallet. Therefore, I was asked to help during the exploratory study of the CREDENTIAL Wallet in order to uncover usability issues (and/or hindrances in user understanding). The pilot study is presented below as it shows the authorization flow I used throughout the thesis work. The concept of the CREDENTIAL Wallet is that users can select what data types they disclose to the service provider; similar to choosing what to disclose when downloading applications from, e.g., Google Play. It became apparent when looking at the result that the usability was in decent shape (in terms of the system usability scale scores) but the users were confused of what data types they had shared.



Designing the prototypes

Moving forward, I wanted to try different concepts of the CREDENTIAL Wallet in order to see if I could enhance user awareness of data transaction between identity and service provider. While simultaneously measuring the usability. Considering I wanted to design several prototypes I wanted them to be consistent so I wanted to lay out the foundation. One of the things to reach consistency was to create a visual hierarchy. Patrick and Kenny (2003) address that the layout of items is an important aspect to facilitate user comprehension. Cooper et al. (2007) explain that it is important to develop a hierarchy of information and determine what should be displayed where. Dix (2009) explains that humans tend to be better at using an application by constructing a hierarchy as long as it does not go deep. Meyer (2015) explains that Google’s Material Design is a good example of helping users interpret interfaces and visual hierarchies. Material Design (n.d.) has created cards which is a container (using shadows creating a box-like state) with the purpose of displaying more detailed information and contains interactive content – this component can potentially be used for creating a visual hierarchy for the users while still presenting them with clear actions. Cards should be used when displaying content that involves a collection of data types, images, movies, and text; it is often used as highly contrasted with the background using shadows with a sufficient contrast ratio to the text. Dix (2009) explains that white space can be used to separate blocks and thus highlighting structures. Cards were implemented to create a hierarchy of information by distinguishing information. The order of presentation was following the pilot study prototype by presenting the service provider to the user first for clarification of authentication alert. Secondly, the user is presented by required information, followed by optional information, to make their choices of disclosing information. According to Dix’s (2009) recommendation: “if things logically belong together then we should normally visually group the together”.

An early wireframe of the CREDENTIAL Wallet presenting the visual hierarchy of the application.


For the different design concepts, there were different types of patterns/gestures I wanted to explore. Three of which used a new look whereas one prototypes were applied to the already existing pilot prototype. The reason for that was to see if there could be any radical improvements to the foundation that was already established. All of the designs had a scientific basis.


Confirmation screen prototype

Javed and Shebab (2016) constructed a summary page which included permission descriptions, personal information example, and a permission type (required or optional to disclose the information). The study used animations and showed that better user understanding of the disclosed information. Cooper et al. (2007) state it is important to provide users with a comparison state by completing an action (before-and-after scenario). Patrick and Kenny (2003) explain that the user should be given the opportunity to review the terms before consenting to the agreement, an opportunity to correct errors by, as an example, presenting a final confirmation before accepting or rejecting the agreement. For those reasons a confirmation screen was constructed to see if it could impact the user understanding of disclosing information.



Swiping prototype

In the authorization flow in the pilot study the users selected information by filling small checkboxes. However, these checkboxes were only displayed as a very small portion of the screen, they were not contrasted very well with the background, and potentially did not give the users enough feedback or a realization of completion. Cooper et al. (2007) address the importance of providing a mean for users to compare the state of an action (before-and-after scenarios). A comparison will provide context that makes information more valuable and more comprehensible to users. Norman and Nielsen (2010) explain that checking of boxes and selections can be difficult to use at times as it might take you to new locations and that it needs to be a clear directive of how to undo the action that was made. Hoekman (2011) addresses that smartphones open up a new medium and thus users learn to expect different things (e.g. tapping and swiping) from computers. Murray (2011) reports that swiping mimics the gesture of flicking something off a physical surface. Therefore, a swiping design was constructed (i.e. a design utilizing a swiping gesture) for creating a sense of completion and showing a clear example of a before-and-after scenario.

One of the important things to think about when creating the swiping functionality was that create an obvious mapping between the swiping functionality and the selection. The interface should be made obvious thus the relationship between control and outcome needs to be understandable (Norman 2013). The CREDENTIAL Wallet swiping design was made so a swipe needs to be made for every selection of information. By swiping right, the user selects the information as swiping right intuitively feels appropriate because of the left to right orientation is the progression of western text which was a way to delete an email on an iPhone (Murray 2011). However, as pointed out by Norman and Nielsen (2010), swiping functionality does not necessarily mean that things are obvious to the user as it has to be visually clear. Signifiers (e.g. arrows and icons) can provide signals about the operation (Norman 2013). Norman (2013) exemplifies signifiers by mentioning arrows of representing a swiping functionality. To try to explain the functionality of selecting information, other than using text, a swiping icon was implemented; i.e. trying to use the icon in order to enhance the comprehension of what the user can control in the privacy system (Patrick & Kenny 2003). The swiping design allows block to slide to the right and thus displaying a change being made.



Drag-and-drop prototype

As the pilot study displayed confusion of selecting items the swiping design was created to evaluate the impact of a design within the mobile medium which could potentially make the users feel a realization of completion. Pettersson et al. (2005) explain that users can, in situations such as dialog boxes displaying two alternatives (e.g. OK and cancel) react in automatized ways. A way of solving this, as showed in the PRIME project, is to use drag-and-drop actions (i.e. a design utilizing a drag-and-drop gesture) to avoid automation of behaviour (Pettersson et al. 2005). The drag-and-drop approach has to be tested on a mobile device. The design was constructed in a similar approach to the swiping design (i.e. by dragging the icon to a close drop zone to the right). Therefore, a design similar to the swiping design was made in order to see if it could increase the consciousness (Patrick & Kenny 2003).



Checkboxes prototype

The pilot study featured a checkbox prototype which used a set height for all pages. This meant that the prototype was compressed as much information had to fit in a limited amount of space; the prototype did not utilize the concept of minimalistic design whereas the new prototypes try to accomplish being a system of “less is more”. Also, as this study tries to compare the prototypes it is important to consider the impact of the icons of the newer prototypes’ designs. Lastly, the pilot study was using circles around the checkboxes to distinguishable from the other elements on the screen. This design choice was commented on in the pilot study to have been a hindrance in terms of user awareness as it could have potentially given the impression to the participants that they had to check the boxes. Checkboxes is a design solution proposed by Wang et al. (2013) in authorization dialogues for social applications. The pilot study prototype had an option to select all data at once which could have resulted in participants not being aware of their disclosures. Therefore, a new checkbox prototype was created using icons.




Even though one could argue this being the most important quality of research I will try to skip as much as possible. The reason for that is that it is just way too much to cover. I advice you to read the thesis if you are interested.

I conducted in total 100 usability tests for this study; 20 usability tests for each prototype (pilot, confirmation screen, swiping, drag-and-drop, and the new checkbox prototype). I observed the participants which gave me a direct look into the usage of the CREDENTIAL Wallet. I also tracked their timing of completing the authorization task. Consequently, a questionnaire was given to the participants to fill out: tracking user awareness scores of recall, precision, and uncertainty (see thesis for extensive explanation). Also, I used the system usability scale (SUS) as it is highly recommended – especially in the context of privacy systems. SUS is a ten item statement scale using a Likert scale, it generates a usability score between 0-100.


Usability and user awareness discoveries

Usability was measured by timing the participants when doing the tasks and also through using System Usability Scale (SUS) (Brooke 1996).

The timing varied in the different prototypes. The prototypes utilizing checkboxes had, naturally, a low average time. The average generated from the new checkbox prototype was 93 seconds. The confirmation screen added 18 seconds on top of the existing pilot prototype. The swiping prototype and drag-and-drop prototype had an average of 132 and 129 seconds respectively. The SUS score was close between four prototypes (pilot, confirmation screen, drag-and-drop, and checkbox prototype). The drag-and-drop prototype scored an SUS score of 81.00. The worse out of all the prototypes were the swiping prototype with an SUS score of 72.62. However, all designs are to be considered according to the recommendation set out by Ruoti et al. (2015). The swiping prototype only scoring 72.62 shows that details should be worked out in such an interface to make it even more understandable. Additionally, there was confusion at first sight for the participants using the swiping prototype which might have generated in the low SUS score. The participants thought, at first sight, that they had already selected the data types which indicate that the design was not clear enough.



During this study, usability tests were conducted to see if the different prototypes would generate different user awareness scores. Interestingly, the swiping and drag-and-drop prototypes achieved a recall score of 0.93 and 0.92 respectively. Perhaps the results were achieved through adapting to the mobile medium (Hoekman 2011) or because the avoidance of automated behaviour (Pettersson et al. 2005). However, the swiping prototype did not receive a high precision score whereas the drag-and-drop prototype had a precision score of 0.88. The highest precision score was the checkbox prototype with 0.89. This could be the case because of the limited screen space on smartphones as the checkbox design gave the participants a good overview of their data types. However, all the prototypes received a precision score between 0.85-0.89 meaning the differences in user awareness were much more prominent for the recall scores. Arguably, by looking at the values of recall and precision, the user awareness of data transactions was the highest in the drag-and-drop prototype. Adding the confirmation screen onto the pilot study prototype saw an increase in recall score as the participants had an easier time of realizing the data types they had disclosed when testing. Moreover, all prototypes showed a high uncertainty score with the checkbox prototype showing the most positive score.



Concluding remarks

This is only an out-take of the final version of the research. The thesis include a more extensive telling of CREDENTIAL Wallet and the findings of my conducted research. The thesis include future research and recommendations of similar types of applications.



Bauer, L., Bravo-Lillo, C., Fragkaki, E., & Melicher, W. (2013). A comparison of users’ perceptions of and willingness to use Google, Facebook, and Google+ single-sign-on functionality. In Proceedings of the 2013 ACM workshop on Digital identity management (pp. 25-36). ACM.

Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability evaluation in industry, 189(194), 4-7.

Cooper, A., Reimann, R., & Cronin, D. (2007). About face 3: the essentials of interaction design. John Wiley & Sons.

Dix, A. (2009). Human-computer interaction (pp. 1327-1331). Springer US.

Egelman, S. (2013). My profile is my password, verify me!: the privacy/convenience tradeoff of Facebook connect. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2369-2378). ACM.

Hoekman Jr, R. (2011). Designing the Obvious: A Common Sense Approach to Web & Mobile Application Design, second edition. Pearson Education.

Javed, Y., & Shehab, M. (2016). Investigating the Animation of Application Permission Dialogs: A Case Study of Facebook. In International Workshop on Data Privacy Management (pp. 146-162). Springer International Publishing

Material Design (n.d.). Cards. [Online] Available at: [2017-03-08].

Meyer, K. (2015). Flat Design: Its Origins, Its Problems, and Why Flat 2.0 Is Better for Users. [Online] Available at: [2017-02-09].

Murray, J. H. (2011). Inventing the medium: principles of interaction design as a cultural practice. MIT Press.

Norman, D. A., & Nielsen, J. (2010). Gestural interfaces: a step backward in usability. interactions, 17(5), 46-49.

Norman, D. A. (2013). The design of everyday things: Revised and expanded edition. Basic books.

Patrick, A., & Kenny, S. (2003). From privacy legislation to interface design: Implementing information privacy in human-computer interactions. In Privacy Enhancing Technologies (pp. 107-124). Springer Berlin/Heidelberg.

Pettersson, J. S., Fischer-Hübner, S., Danielsson, N., Nilsson, J., Bergmann, M., Clauss, S., & Krasemann, H. (2005). Making PRIME usable. In Proceedings of the 2005 symposium on Usable privacy and security (pp. 53-64). ACM.

Robinson, N., & Bonneau, J. (2014). Cognitive disconnect: understanding Facebook connect login permissions. In Proceedings of the second ACM conference on Online social networks (pp. 247-258). ACM.

Ronen, S., Riva, O., Johnson, M., & Thompson, D. (2013). Taking data exposure into account: how does it affect the choice of sign-in accounts?. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 3423-3426). ACM.

Ruoti, S., Roberts, B., & Seamons, K. (2015). Authentication melee: A usability analysis of seven web authentication systems. In Proceedings of the 24th International Conference on World Wide Web (pp. 916-926). ACM.

Wang, N., Grossklags, J., & Xu, H. (2013). An online experiment of privacy authorization dialogues for social applications. In Proceedings of the 2013 conference on Computer supported cooperative work (pp. 261-272). ACM.

Xu, H., Wang, N., & Grossklags, J. (2012). Privacy by redesign: Alleviating privacy concerns for third-party apps.