REAL ID | My Site

Department of Homeland Security REAL ID Usability Research

My Role: Researcher & Facilitator

Timeframe: 6 weeks

Methods: Heuristic Evaluation, Usability Testing, SUS Scale, User Interviews

Tools: Zoom, Figma

Introduction:

The US Department of Homeland Security (DHS) features a microsite for the REAL ID Act. The site aims to educate users on what the REAL ID Act is, when the act will be enforced, if they have a REAL ID, and more general information about REAL IDs. DHS was receiving a lot of complaints that the site was difficult to use.

DHS tasked our team with conducting a summative usability study of the REAL ID site to figure out the issue. We utilized several methods to evaluate the usability of the site, including heuristic evaluations and remote testing via Zoom.

The Problem & Objective:

DHS was receiving a lot of complaints that the site was difficult to use. Ideally, the site should be easily accessible and organized so minimal effort from the user would be necessary to determine if they have a REAL ID or how they can get one. However, this may not be the case due to the wide variety of people who make up its users. The REAL ID microsite had a feedback score of 60%, the ideal feedback score is 75%.

The objective of this usability study was to evaluate the site and highlight the areas of the site that could be improved.

Methodology:

To conduct this study my team utilized several methods:

Heuristic evaluations from each team member
Remote usability testing using tasked-based scenarios
System Usability Scale post-test questionnaire
Post-test interview

Heuristic Evaluation:

Before conducting a usability test with participants, each team member conducted a heuristic evaluation of the website. We referred to Rolf Molich and Jakob Nielsen's devised set of heuristics while conducting the evaluations. The heuristic evaluation was an important part of the process. We were able to use our findings from our heuristic evaluation to create usability testing tasks later on in the project. We found the following findings from the heuristic evaluation:

Some headings are small and easy to miss
There were no navigational warnings when users left the site

Participants:

We sent out screeners to potential participants to determine if they were eligible for the study. DHS provided us with analytics that helped us choose the criteria and screener. Participants had to meet the following criteria to be eligible for the study:

The criteria were determined through several methods:

Checking the demographics of the site users via Google Analytics
Discussing the target audience of the site with DHS

We conducted usability tests with a total of 6 participants. I conducted a pilot run with the first participant. The pilot run went well and we were able to keep the data from it and conduct the rest of the testing sessions.

Tasks:

We asked our participants to complete the following task:

You are trying to ﬁnd information about the REAL ID Act. Find the oﬃcial DHS REAL ID website.
You are planning on traveling by plane soon and want to make sure you are REAL ID ready. Find out if your ID is REAL ID compliant.
You are traveling by plane with a friend who does not have a driverʼs license. You want to ﬁgure out what alternative documents will be accepted. Find at least two alternative documents that TSA will accept.
You are curious about the importance of having a REAL ID. Find out what may happen if you try to travel without a REAL ID. Find at least one scenario that could happen to travelers who do not have a REAL ID or an acceptable alternative.
You are a frequent traveler and want to know when the REAL ID Act will be enforced. Find the enforcement date.
Your driverʼs license is expiring soon, and you want to ﬁgure out how to renew it. Find at least one way you can renew your license.

I read each task aloud while the observers took notes on the participant's actions. The following table shows the task completion rate.

System Usability Scale Results:

We asked the participants to fill out a SUS consisting of 10 questions. The answer options were on a scale of 1 to 5 with 1 being strongly disagree and 5 being strongly agree. The following graph shows the SUS results.

I think that I would like to use this system frequently.
I found the system unnecessarily complex.
I thought the system was easy to use.
I think that I would need the support of a technical person to be able to use this system.
I found the various functions in this system were well integrated.
I thought there was too much inconsistency in this system.
I would imagine that most people would learn to use this system very quickly.
I found the system very cumbersome to use.
I felt very confident using the system.
I needed to learn a lot of things before I could get going with this system.

The SUS results were put on a range of 0 to 100 with 68 being a passing score. The participants in blue gave the website a passing score, while the red participants were below passing.

Post-Test Interview:

After testing and the SUS, we asked participants 6 questions:

How easy or difficult was it to figure out if you had a Real ID?
Was the questionnaire useful for you?
What is something you liked about the website, if any?
What is something you disliked about the website, if any?
Do you have any additional comments about the website?
Can you explain to me what a REAL ID is?

The answers from this interview helped us determine our findings and recommendations.

Some of the most common answers from the interview included:

“One thing I didn't like was that it took me to two different websites. It just opened up a new tab.”

- Participant 1

“The FAQ is not very user friendly at all.”

- Participant 2

“A REAL ID is another form of a driver’s license or passport if you are not able to get a regular one.”

- Partcipant 6

Findings:

After the completion of the heuristic evaluation, usability testing sessions, SUS questionnaire, and the post-test interview our team analyzed our data.

We grouped the data into categories and both positive and negative aspects were found.

Recommendations:

Based on these findings, our team was able to recommend the following changes.

Label the questionnaire and map to make each element more noticeably interactive.
Include a short and simple deﬁnition on the homepage of the REAL ID site.
Warn users when they will be leaving the DHS website.
Use drop-down tabs for each question in the FAQ section.

Questionnaire: Current

This is an image of the current questionnaire on the REAL ID microsite. The text letting the users know the questionnaire is interactive is very small and some participants did not notice it at first.

Questionnare: Recommendation

Our team recommends making the "Find our if you're REAL ID ready with our interactive tool!" larger, along with a blue font to make the questionnaire more noticeable to site users.

Headings: Current

Many participants scrolled past the map without realizing you could click on any state or territory to go to the desired DMV site.

Headings: Recommendation

Our team recommends making the heading for the map much larger, along with making the font blue so it stands out on the page. This change should make the map easier for participants to notice.

REAL ID Definition: Current

Currently, the site lacks a simplified definition of a REAL ID. None of the participants could give an accurate definition of a REAL ID.

REAL ID Definition: Recommendation

Our team recommends adding a simple definition of a REAL ID to the top of the site. Users will be more likely to read this short definition than read the paragraphs about the REAL ID Act.

Frequently Asked Questions: Current

Some participants complained that the FAQs were not user-friendly. Clicking a topic opens a large block of text that is difficult to scan.

Frequently Asked Questions: Recommendation

Our team recommends letting each question have its drop-down menu. Users will be able to easily scan and find the answers they are looking for.

Navigation Warnings: Recommendation

Some participants were not aware that when they clicked a link that it led them to a different government site, we recommend adding warnings to let site users know when they have left the site.

Conclusion

Our team was able to figure out which sections of the REAL ID site users were having difficulty with. With these recommended changes our team believes that users will have a better understanding of what the Real ID Act is, how to navigate the site, have an easier time finding information relevant to them, and have an overall better user experience on the site.

The director of the DHS Website Team was happy with our recommendations. They have implemented some of our recommendations and are in the process of adding the rest. A few months after adding changes to the site the REAL ID site had a feedback score of 76%.

Impact

This project and several other evaluations that I conducted with DHS helped DHS.gov receive multiple "A" grades on the 2022 Federal Plain Language Report Card. Several DHS.gov pages were selected as a model that other federal agencies should follow.

Reflection

This was one of the major projects I worked on for DHS.gov. I learned a lot about UX research. I also had the opportunity to implement a lot of my ideas on the website directly and also got experience with communicating and working with the development team.

I had the opportunity to continue working on this project as an intern at DHS. I also got to conduct similar studies on several DHS microsites.