Disaster App Usability Evaluation

Description

This evaluation is a part of Independent Research Project for Ohashi Lab, Tokyo Institute of Technology, in the research topic "How to make Thai residents/visitors in Japan use disaster applications effectively (tentative)." This evaluation aimed to evaluate the usability of existing disaster applications in Japan by using Tan (2020)'s disaster application's usability guidelines.

App Selection

App selection process started with observing the official disaster guide from Cabinet Office, Director General for Disaster Management, and several travel websites suggesting Japan’s disaster apps that are compatible with iOS and Android. We applied three conditions when selecting apps by following Tan et al. (2020).

1. The apps can display in English or Thai languages.
2. We include apps that are or have sections that are purposefully designed to help users in the context of disaster preparedness in Japan.
3. We excluded tools that may be used in emergency situations but do not provide information or warning for disasters. For example, we excluded light and sound makers. We got five apps to evaluate usability. We included each app’s basic information and key features related to disaster preparedness in the following table. Requirements and storage used are based on iOS.

The selected five apps are

• Safety tips
• NHK WORLD-JAPAN
• Disaster Preparedness Tokyo
• NERV
• Yurekuru

Methodology

According to disaster apps’ usability guidelines from Tan et al. (2020), we applied 3 sets of 17 codes of guideline statements to evaluate the usability of each app. The audio cannot be tested in some apps, so we took 16 codes for evaluation. All apps were evaluated by iOS.

Set 1: Make critical information salient

• PRIO: Critical information should be prioritised
• EQUA: Avoid maintaining an identical level of priority when displaying different alerts
• VARY (TECH): Use varying techniques to make important information prominent (e.g. typographical emphasis, colour contrast)
• AUDI: Use audio output purposefully

Set 2: Consider cognitive load when designing the interface

• LEVE: Utilise existing interfaces that are familiar to users
• NEAT: Keep the design neat and simple
• INPT: Reduce the need for user input
• RTIO: Rationalise the use of text and images
• TEXT: Limit textual information
• VSUA: Use meaningful visualisation to enhance the content (e.g. image, logos, maps)

Set 3: Build trust, anticipating limited interaction

• RELV: Provide relevant content
• SIGN: Show information significant to time, location, and severity
• SRCE: Display the source of information
• PERS: Personalise or localise the content to the user
• ERRO: Minimise the impact of errors and reduce loading time
• VISI: Provide visible system status to evoke that the app is alive and updated
• PRVY: Show regard to users’ privacy (e.g. privacy statement)

Evaluation Results

Safety tips

Safety tips is a free disaster app developed under the supervision of Japan Tourism Agency (JNTO) giving information on disasters (such as earthquake early warnings, tsunami, and weather warnings) in Japan to foreign visitors. The app contains a multilingual display including English and Thai languages. The app got 8 out of 16 points on disaster apps’ usability guideline statements.

The app fails to make critical information salient. All earthquake and tsunami warnings at different levels are identical. The app does not provide access to a map showing which areas are critical. The user cannot find prioritized data in earthquake history.

Regarding cognitive load of the interface, the design of the app is familiar to the user, as it is near and simple without the need for user input. The interface includes icons in the menu to help the user to navigate. Although the map is not shown to display all events, the user can access the map to see the area of each event making it clearer to access disaster warnings. However, apart from icons on the menu, the information displayed is all texts leading to difficulty in reading earthquake history.

The app partially builds trust to the user. It provides relevant content and time, location, and severity of each disaster event. However, there is no specific source of information for each event. Also, there is no localized content that shows the specific events in the user’s location. No sign or cue is showing if the app is still active to show live information. The privacy policy is also not accessible within the app.

Overall, Safety tips can provide useful information about disasters to the user well. With icons and understandable interfaces, the app can reduce cognitive load making the user navigate to each page and access the information easily. However, the app can be improved in making critical information more salient and building trust.

NHK WORLD-JAPAN

NHK WORLD-JAPAN is a multilingual news app that provides disaster information and warning suggested by Cabinet Office, Director General for Disaster Management. The app includes English and Thai languages. This app got 10 out of 16 scores in usability evaluation.

The app partly achieves making critical information salient. The app prioritizes the latest event by opening the tab of the event to show its details by default. In each event, the map shows the color contrast in the severe zone of the earthquake, yet the earthquake history shows no color contrast or highlight of critical events. Therefore, it would be better to add typographical emphasis on the critical event.

Although the app is neat and familiar to the user, it has a lot of textual information with no or little visualization and icons making more cognitive load to the user. The top navigation bar and detail expand button to open the earthquake detail are neat and effective to lead the user to the details of the earthquake. However, the details exist only in texts and numbers with a map showing the severity of the surrounding zones. Adding some icons might be meaningful to improve the app’s usability.

There is room for improvement in building trust in this app. Sources and personalized content (specific information to the location of the user) could be added more. Apart from that, the app could provide relevant content with complete details and show that the app is up to date. There also is a privacy policy in the app.

NHK WORLD-JAPAN could give enough information and warnings to the user, but it still lacks visualization and prioritization. It would be better for the developer to add more icons and reduce the text as much as possible, and the critical information should also be highlighted more. Sources and personalized contents for each user are also good to have.

Disaster Preparedness Tokyo

Disaster Preparedness Tokyo is an official app by Tokyo Metropolitan Government to provide disaster warnings and disaster information both in daily life and in an emergency according to Disaster Preparedness Tokyo book. The app includes the warning system and information specifically in Tokyo and for Tokyo’s residents. In this evaluation, we focused only on the warning system. This app got a usability evaluation with a score of 15 out of 17.

The app partly succeeds to create salience for critical information systems, yet there is still room for improvement in alert differentiation and audio output. The app purposefully put evacuation information on the top of the page to prioritize the critical information, and it makes good use of typographical emphasis to highlight such information. However, a similar color is displayed although they are different alerts in warning history.

This app was successfully designed by keeping the cognitive load in mind. The neat and simple interface is familiar to the user. The app makes good use of images and icons to visualize each event. Some events may contain textual information, but they set the information in tables and cards making it easier to understand and distinguish the information.

Moreover, the app is able to build trust in the user. It shows all relevant content with sources and all needed information (location, time, and severity). There is personalization in that the user can choose the area they live in and focus only on the news of that particular area. The information is real-time updated, and the latest update is shown in the app. The user’s privacy policy is also provided within the app.

Disaster Preparedness Tokyo was designed keeping usability in mind. The app contains useful information on disasters with prioritization. The information is not so textual, and simple images and icons are used to visualize. The content is reliable to the user. There might be a small flaw in content differentiation in different alert and audio outputs, but overall, the app is a good app for disaster preparedness.

NERV Disaster Prevention

NERV Disaster Prevention app is a smartphone service that provides Japan’s disaster and weather information developed by Gehirn Inc. The app is free to use and suggested by a lot of travel websites and reviewers. It provides ad-free service with universal design based on user’s preference. This app got a usability evaluation with a score of 16 out of 17.

The app succeeds in prioritizing important information and makes good use of color and typographical emphasis. Only critical information is highlighted with color making user can distinguish data easily. Important data is shown in bigger text with colored icons. Box and card components help the user to see the emphasis. Moreover, the high-volume audio output is utilized when there is a critical alert purposefully, making the information more salient.

With simple interfaces and good use of icons, NERV could reduce the cognitive load of the interface. The app reduces text and uses simple icons with colors to give meaningful data to the user. The navigation menu is familiar to the user. The card and timeline list make the information more organized and not so textual.

The app could build user trust successfully except for the source of information. It provides all relevant data including time, location, and severity. The user can observe past events specifically in their location. The latest update is highlighted, and there is a user’s privacy policy in the app. One point to improve is the source of information. Although the source of information is provided clearly on the website, it is recommended to attach the source with all events to build user trust.

NERV Disaster Prevention app is well-designed with high usability. All important information is salient and well-highlighted. The interface is understandable and not textual, and images and icons could visualize information well. The app could build trust for the user through its content and personalization, yet it could improve more by adding sources of information.

Yurekuru

Yurekuru is a disaster app developed by RC Solution Co. The app was suggested by Japan’s travel websites and reviewers. Although Yurekuru app is free-to-use, it provides a lot of irrelevant ads. The app got a score of 8 out of 16.

The app used color to highlight severe events; however, there is no prioritization for critical information. The color tab on the top left of each event card shows the severity of earthquakes making the user distinguish easily. However, no specific event that the app prioritizes, and it may make the user fail to focus on any specific event.

The design of the app is simple and familiar. The bottom navigation is used to get to the history list. The history list is full of text without any images or icons to visualize. In each event, there is also a lot of text inside without any card or box component to help to understand the information, yet there is a map to access the disaster location easily.

The app fails to build trust, as there is a lot of irrelevant information such as ads. Location, time, and severity are shown in the earthquake history list, but there is no time on the earthquake detail page. There is no localization to the user’s location, and it takes time to perceive if the app is still active or not. Also, there is no privacy policy in the app.

Yurekuru should improve more in all aspects according to the disaster apps’ usability guidelines. They should remove irrelevant information and provide more complete information about all events. The developer should also consider making critical information salient and creating more trust for the user.

References

Tan, M. L., Prasanna, R., Stock, K., Doyle, E. E. H., Leonard, G., & Johnston, D. (2020a). Modified Usability Framework for Disaster Apps: A Qualitative Thematic Analysis of User Reviews. International Journal of Disaster Risk Science, 11(5), 615–629. https://doi.org/10.1007/s13753-020-00282-x

Tan, M. L., Prasanna, R., Stock, K., Doyle, E. E. H., Leonard, G., & Johnston, D. (2020b). Understanding end-users’ perspectives: Towards developing usability guidelines for disaster apps. *Progress in Disaster Science, 7, 100118. https://doi.org/10.1016/j.pdisas.2020.100118