UBC Faculty Research and Publications

Usability testing of ANSWER: a web-based methotrexate decision aid for patients with rheumatoid arthritis Li, Linda C; Adam, Paul M; Townsend, Anne F; Lacaille, Diane; Yousefi, Charlene; Stacey, Dawn; Gromala, Diane; Shaw, Chris D; Tugwell, Peter; Backman, Catherine L Dec 1, 2013

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-12911_2013_Article_767.pdf [ 469.24kB ]
JSON: 52383-1.0223444.json
JSON-LD: 52383-1.0223444-ld.json
RDF/XML (Pretty): 52383-1.0223444-rdf.xml
RDF/JSON: 52383-1.0223444-rdf.json
Turtle: 52383-1.0223444-turtle.txt
N-Triples: 52383-1.0223444-rdf-ntriples.txt
Original Record: 52383-1.0223444-source.json
Full Text

Full Text

RESEARCH ARTICLE Open AccessUsability testing of ANSWER: a web-basedmethotrexate decision aid for patients withrheumatoid arthritisLinda C Li1,2*, Paul M Adam3, Anne F Townsend2, Diane Lacaille2,4, Charlene Yousefi2, Dawn Stacey5,6,Diane Gromala7, Chris D Shaw7, Peter Tugwell8 and Catherine L Backman2,9AbstractBackground: Decision aids are evidence-based tools designed to inform people of the potential benefit and harmof treatment options, clarify their preferences and provide a shared decision-making structure for discussion at aclinic visit. For patients with rheumatoid arthritis (RA) who are considering methotrexate, we have developed aweb-based patient decision aid called the ANSWER (Animated, Self-serve, Web-based Research Tool). This studyaimed to: 1) assess the usability of the ANSWER prototype; 2) identify strengths and limitations of the ANSWER fromthe patient’s perspective.Methods: The ANSWER prototype consisted of: 1) six animated patient stories and narrated information on theevidence of methotrexate for RA; 2) interactive questionnaires to clarify patients’ treatment preferences. Eligibleparticipants for the usability test were patients with RA who had been prescribed methotrexate. They were asked toverbalize their thoughts (i.e., think aloud) while using the ANSWER, and to complete the System Usability Scale(SUS) to assess overall usability (range = 0-100; higher = more user friendly). Participants were audiotaped andobserved, and field notes were taken. The testing continued until no new modifiable issues were found. We useddescriptive statistics to summarize participant characteristics and the SUS scores. Content analysis was used toidentified usability issues and navigation problems.Results: 15 patients participated in the usability testing. The majority were aged 50 or over and were university/college graduates (n = 8, 53.4%). On average they took 56 minutes (SD = 34.8) to complete the tool. The mean SUSscore was 81.2 (SD = 13.5). Content analysis of audiotapes and field notes revealed four categories of modifiableusability issues: 1) information delivery (i.e., clarity of the information and presentation style); 2) navigation control(i.e., difficulties in recognizing and using the navigation control buttons); 3) layout (i.e., position of the videos, text,diagrams and navigation buttons); 4) aesthetic (i.e., the colour, look and feel of the online tool).Conclusions: Although the SUS score indicated high usability before and after major modification, findings fromthe think-aloud sessions illustrated areas that required further refinement. Our results highlight the importance offormative evaluation in usability testing.Keywords: Patient decision aid, Rheumatoid arthritis, Methotrexate, Usability test* Correspondence: lli@arthritisresearch.ca1Department of Physical Therapy, University of British Columbia, Vancouver,Canada2Arthritis Research Centre of Canada, Vancouver, CanadaFull list of author information is available at the end of the article© 2013 Li et al.; licensee BioMed Central Ltd. This is an open access article distributed under the terms of the CreativeCommons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, andreproduction in any medium, provided the original work is properly cited.Li et al. BMC Medical Informatics and Decision Making 2013, 13:131http://www.biomedcentral.com/1472-6947/13/131BackgroundRheumatoid arthritis (RA) affects about 1% of the popu-lation worldwide, with the peak onset between age 35and 50 [1,2]. There is ample evidence supporting earlyand persistent use of disease-modifying anti-rheumaticdrugs (DMARD) to prevent irreversible joint damage[3-5]. Among the available DMARD, methotrexate isgenerally considered the first-line treatment for RAbased on its benefits and potential side effects. However,a Canadian population-based research reported that only43% of the population with RA had used a DMARD overa five-year period [6].Patients’ decisions on medication use can be affectedby their concerns about side effects [7]. Several qualita-tive studies in chronic disease, including RA, have re-vealed patients’ ambivalence toward using medication[8,9]. On one hand, they described an aversion to drugsbecause of the anticipated side effects and, on the otherhand, they felt compelled to take medication due to afear of the potentially crippling effects of an uncon-trolled disease. The circumstance in which people de-cided to use or not use medications appeared to beinfluenced by the nature of the symptoms and the extentto which symptoms disrupt daily lives.In recent years, clinical practice has been expandingfrom traditional authoritative models, in which physi-cians make treatment decisions for patients, to includeshared decision-making. This involves an exchange ofinformation to prepare patients to make treatment deci-sions and engage in the process of decision-making withtheir healthcare providers [10,11]. One way to facilitateshared decision-making is through the use of patient de-cision aids [12]. These are evidence-based tools designedto help individuals to choose between two or more treat-ment options [13,14]. Decision aids help people topersonalize information about treatment effectiveness,outcomes and the inherent uncertainties of potentialbenefit versus potential harm. An important feature ofdecision aids is that they help individuals to clarify theirpersonal values towards benefits and harms, and to com-municate this information to health professionals. Pa-tients who have used decision aids tend to have moreknowledge about the treatment, more realistic expecta-tions and lower decisional conflict compared to thosewho received usual care [15]. Also, decision aid users aremore likely to participate in decision-making and toreach a treatment decision [15].To assist patients with RA to make decisions aboutusing methotrexate, we developed an online decision aidcalled ANSWER (Animated, Self-serve, Web-based Re-search Tool) [16]. The innovative aspect of ANSWER isits built-in patient stories that illustrate common situa-tions people experience when making decisions abouttheir treatment, as well as attributes required foreffective management of their healthcare. The primaryobjective of this study was to assess the user friendlinessof the ANSWER prototype. Our secondary objective wasto identify strengths and limitations of the tool from theuser’s perspective. This study focuses on the refinementof the ANSWER prototype so that it could be deployedfor use by the general public.MethodsDecision aid developmentDevelopment of the ANSWER decision aid was guidedby the International Patient Decision Aid Standards[17,18]. Our target users were individuals who had beenprescribed methotrexate for RA, but were feeling unsureabout starting it. As methotrexate was usually prescribedat the early stage of RA, we designed the ANSWER withthe needs of newly diagnosed patients in mind. This de-cision aid focused on two options: 1) to take methotrex-ate as prescribed; 2) to refuse methotrexate and talk tothe doctor about other treatment options. The design ofthe ANSWER was guided by Jibaja-Weiss’s EdutainmentDecision Aid Model [19]. Educative entertainment, oredutainment, is a process whereby educational messagesare imbedded within an entertaining medium, such asbroadcasting media, e.g., television [20,21] or performingarts (e.g., theatre) [22], and games [22-25]. Central tothe Edutainment Decision Aid Model is the focus onmaking the computer-human interface user-friendly[19,26]. We assembled a multidisciplinary team, involv-ing patients/consumers, digital media experts, cliniciansand health researchers, to develop the online tool. Therole of patients/consumers was particularly important asthey had firsthand experience in making treatment deci-sions. They informed the design of the ANSWER bysharing their experiences of using computers while hav-ing joint pain and fatigue. Further, they reviewed thecontent of the patient decision aid to ensure it is under-standable by people without medical background, al-though no readability program was applied.Figure 1 presents the navigation path of the ANSWER.Users were guided to start by completing the InformationModule, the Value Clarification Module, and then the stan-dardized health outcome measures. However, the tool alsoallowed users to access any component without following alinear path. The Information Module consisted of the latestevidence on methotrexate compared to placebo from aCochrane systematic review [27] and the current evidence-based recommendations from the 3E (Evidence, Expertise,Exchange) Initiative [28]. The latter was a multinationalcollaboration involving 751 rheumatologists from 17 coun-tries to develop recommendations for the use of methotrex-ate in RA using a Delphi process. The design of ANSWERwas guided by our previous qualitative study on the help-seeking experience of patients with early RA [29], and inputLi et al. BMC Medical Informatics and Decision Making 2013, 13:131 Page 2 of 12http://www.biomedcentral.com/1472-6947/13/131from the patient/consumer collaborators. The module ad-dressed six topics: 1) About RA; 2) About methotrexate; 3)Side effects of methotrexate; 4) Pregnancy; 5) Alcohol use;6) Other medication options and adjunctive treatments(e.g., exercise, joint protection techniques). Recognizingthat patients had different preferences in receivinginformation, the information was provided in text, voicenarration and animated vignettes.Each of the six vignettes was based on a unique, fic-tional character (Figure 2: Sample Storyboard). We usedthe animated graphic novel approach for the animatedcomponent, which is a relatively simple, inexpensive andFigure 1 ANSWER navigation pathway*. *The ANSWER is designed to guide patients to navigate each component in sequence. The dasharrows indicate that patients may also access any component without following a linear path.ACTION:Bob is seen using a workbench at his garage. He has various pieces of wood laid out in front of him in numerous shapes and sizes. He currently has adecent size piece in a clamp on the workbench and he's using a planer on it.CAMERA:Static, High angle, from behind. (Establishing shot)TIME: N/ANOTES:Garage door is OPEN. Daylight pouring in.ACTION:Bob is sitting in the doctor’s office.BOB(continued) ....., this... Methotrexate,CAMERA:High angle, static, wide (establishing shot)TIME: N/ANOTES: N/AFigure 2 Sample storyboard for ‘About Methotrexate – Bob’s Story’.Li et al. BMC Medical Informatics and Decision Making 2013, 13:131 Page 3 of 12http://www.biomedcentral.com/1472-6947/13/131visually appealing method for creating the animationsequences. This involved repeatedly photographing realactors in key poses based on the story script, processingthe images in Photoshop to create a comic-book-inspired look, and then sequencing the images to createa limited key-frame animation for the characters. Thisanimation method also allows us to make modificationto characters’ appearances, so that they appear to be‘race neutral’ for a multinational audience in Canada.We used a slow animated sequence of 3–5 frames/10 seconds, which allowed us the freedom to emphasizethe more important points in the story. Finally, actorvoice-over was added to complete the animatedvignettes.In the Value Clarification Module, two methods wereused to assist patients to consider the importance of theconsequences from each option. First, they were askedto rate on a 5-point scale the importance of: 1) improv-ing joint pain; 2) preventing joint damage; 3) improvingphysical function; 4) avoiding side effects; 5) becomingpregnant /starting a family; 6) drinking alcohol. This wasfollowed by the second method, in which they indicatedthe relative importance by allocating 100 points acrossthe same six items. Patients were also asked to list theirquestions and concerns about using methotrexate and toindicate their preferred choice out of the two options, orto declare that they remained uncertain.The ANSWER tool ends with two standardized healthstatus questionnaires: the Health Assessment Question-naire [30] and the RA Disease Activity Index [31,32].Scores of these measures and the individual’s responseto the value clarification questions were summarized ina 1-page printable report at the end of the online pro-gram. Patients could discuss this report with their physi-cians before reaching a final decision about usingmethotrexate. The ANSWER prototype was reviewed bythe research team, patients/consumers (OK, CK, CM)and a health education consultant (GE) to ensure thatthe content was understandable to people without aresearch or clinical background.Usability testingGuided by the methods outlined by Rubin and Chisnell[33], we used an iterative testing protocol, whereby we1) conducted onsite testing with participants to identifyusability issues in the ANSWER prototype, 2) stoppedtesting and made modifications when no new issueswere identified, and 3) resumed testing with the modi-fied version. A usability issue was defined as 1) when aparticipant was not able to advance to the next step dueto the decision aid design or a programming error, or 2)when a participant was distracted by a particular designor content of the online tool. Prior to the testing, werecognized that some usability issues would not bemodifiable. For example, because the animated storieswere in their final format, we were unable to change theanimation style or the storylines. We continued the test-ing until no modifiable usability issues were identified.Participants were recruited through study flyers postedat 1) rheumatologists’ offices and community health cen-tres in Vancouver, 2) Mary Pack Arthritis Program,Vancouver General Hospital and 3) classified advertise-ment websites such as Craigslist and Kijiji. Eligible indi-viduals were patients who had a diagnosis of RA andhad been prescribed methotrexate. After providing writ-ten informed consent, participants attended a two-hourtesting session at the Arthritis Research Centre ofCanada. The test was conducted in a small meetingroom in the presence of a trained research staff member.Participants were instructed to use the ANSWER as ifthey were looking for information about methotrexatefor RA. We used the concurrent think-aloud approach.The think-aloud protocol was developed in its currentform by Ericsson and Simon [34], and was introduced tothe field of human-computer interaction by Lewis [35].Participants were encouraged to verbalize thoughts andfeelings when navigating the decision aid. The researchstaff prompted the participant to elaborate on his/hercomments when appropriate or when they fell silent fora while. For example, participants were asked, “What areyou thinking?” or “Can you describe what are lookingat”, if they fell silent. In addition, the research staff inter-vened when participants indicated they did not knowhow to progress to the next stage while using the AN-SWER. All sessions were audio-recorded. To capture sit-uations which might be missed by the audio recording,the research staff took detailed field notes throughoutthe session.Participants were then asked to complete a question-naire including the System Usability Scale (SUS) [36]and socio-demographic and internet use characteristics.Developed by Brooke [36], the SUS consists of 10 state-ments that are scored on a 5-point scale of strength ofagreement. The total ranges from 0 to 100, with a higherscore indicating more user-friendly. Originally developedto measured system usability, the SUS has been adaptedfor testing a wide range of technologies, including hard-ware platforms and software programs [37].Data analysisWe used descriptive statistics to summarize participantcharacteristics and the SUS score after each testing cycle.No statistical comparisons were conducted between cy-cles, as hypothesis testing was not a goal of this study.Audio-recordings were transcribed verbatim. Contentanalysis was conducted to identify 1) modifiable usabilityissues and navigation problems, and 2) strength andlimitation of the ANSWER design. Our data analysis wasLi et al. BMC Medical Informatics and Decision Making 2013, 13:131 Page 4 of 12http://www.biomedcentral.com/1472-6947/13/131inductive, as we sought to understand participants’ ex-perience with the ANSWER rather than to prove a pre-conceived theory. We used a constant comparisonsapproach, whereby participants’ experiences in using theANSWER were coded. Codes that reveal similar naviga-tion problems were grouped into categories [38]. Thedata were constantly revisited after the initial coding,until it was clear no new categories emerged. The codingprocess was performed by one researcher (LCL) whoread each transcript and attributed a code to sentencesor paragraphs (open coding). Other team members werealso included in the coding process to assess causes ofusability problems from participants’ comments. Axialcoding was performed to develop connections amongthe categories of usability problems. LCL was also re-sponsible for discussing the modifications required withthe software programmer, and supervised the revisions.We stopped a testing cycle to make modifications whenno new problem was identified. The study protocol wasapproved by the University of British Columbia Behav-ioural Research Ethics Board (Application number: H09-00898).ResultsWe recruited 15 eligible participants between Augustand October 2010. Of those, 10 participated in Cycle 1and five tested the revised version in Cycle 2 (Table 1).We did not identify any new issues in Cycle 2. Over halfof the participants were aged 50 or older, with 85.7%being women and 53.3% being university or collegegraduates. The median disease duration was 5 years(interquartile range [IQR]: 0.83; 10.00), with participantsin Cycle 1 having a longer median disease duration(5.50 years [IQR: 0.65; 11:00] versus 2 years [IRQ: 0.92;15.50]). All participants have used methotrexate. Theyall used the internet for emails and 46.7% used it to playinternet games. Participants took an average of 56.80 mi-nutes (SD = 34.80) to complete the ANSWER. Table 2presents the total SUS score and the results of individualitems in Cycles 1 and 2. The SUS scores were similarbefore and after modification of the online tool (Cycle 1:81.25, SD = 14.92; Cycle 2: 81.00, SD = 11.81).Modifiable usability issues and changes madeFour categories of modifiable usability issues were identifiedduring Cycle 1 (Table 3, with examples of participants’comments). These include 1) Information Delivery, 2)Navigation Control, 3) Layout, and 4) Aesthetic. Figure 3presents the screenshots of the ANSWER homepage beforeand after modification.Information deliveryAll participants commented on the length of the ANSWERtool. The original version included details of benefits andrisks of methotrexate with each video lasting 6–8 minuteslong. During the testing, participants commented on therepetitiveness of the information and the video length. Inlight of these comments, we added short key messagesthroughout the online tool, reduced the video length, andincluded subtitles in the videos to highlight importantpoints. It should be noted that the videos were shortenedby condensing the storyline, not the evidence. The rheuma-tologist investigators in this team (Lacaille, Tugwell) hadensured that the change did not compromise the presenta-tion of evidence.Table 1 Participant characteristics and experience with internetAll (n = 15) Cycle 1: Before modification (n = 10) Cycle 2: After modification (n = 5)Age20–34 2 (13.3%) 1 (10.0%) 1 (20.0%)35–49 5 (33.3%) 4 (40.0%) 1 (20.0%)50–64 7 (46.7%) 4 (40.0%) 3 (60.0%)65 or older 1 (6.7%) 1 (10.0%) 0Women 13 (85.7%) 9 (90.0%) 4 (80.0%)University/college graduates 8 (53.3%) 6 (60.0%) 2 (40.0%)Disease duration in Years – Median (IQR) 5.00 (0.83; 10.00) 5.50 (0.65; 11.00) 2.00 (0.92; 15.50)Hours spent on Internet per day - Median (IQR) 2.00 (1.00; 3.00) 1.75 (1.00; 2.25) 2.50 (0.88; 3.75)Use of internet for:Email 15 (100.0%) 10 (100.0%) 5 (100.0%)Reading news 4 (26.7%) 3 (30.0%) 1 (20.0%)Entertainment 1 (6.7%) 1 (10.0%) 0Gaming 7 (46.7%) 5 (50.0%) 2 (40.0%)IQR Interquartile range.Li et al. BMC Medical Informatics and Decision Making 2013, 13:131 Page 5 of 12http://www.biomedcentral.com/1472-6947/13/131Navigation controlParticipants found it difficult to use the control buttonsto access the narrated content, adjust volume and con-trol the videos. In the original version, we created ourown navigation buttons for the ANSWER with the intentto achieve a unique look. This, however, became prob-lematic during the usability testing. While some partici-pants did not recognize these buttons, others did notknow how to operate them. One participant commentedthat average internet users might be more comfortablewith the YouTube navigation buttons and format (Bob,Table 3). Based on the feedback, we subsequently re-placed the navigation controls with the YouTube format.Further, the button size was enlarged to increase ease ofuse for patients with hand pain.LayoutIn the original version, each webpage under the tab ‘Ani-mated Stories’ started with the videos, followed by writtensummaries of the information. Participants found the for-mat unfriendly to navigate, especially for people who pre-ferred to watch the video and browse the text at the sametime. One participant commented that this layout requireda lot of scrolling up and down with a mouse, which wasparticularly difficult for people with RA as the hand jointswere often affected (Jamie, Table 3). In the revised version,we further condensed the key messages to reduce scrollingwith a mouse within a webpage. In addition, we addedhyperlinks throughout the tool to improve access to thevideos and written information.AestheticA major criticism of the original ANSWER tool was itsaesthetic. One participant commented that the colourwas ‘flat and uninteresting’ (Theresa, Table 3). Anotherparticipant felt that it needed more colour to make thesite ‘a little more fun’ and more inviting (Jamie). Basedon the feedback, we included pictures in the introduc-tory pages and throughout the Value Clarification Mod-ule. In addition, we added colourful screenshots fromthe animated stories in the Information Module.Limitations and strengths of the ANSWER patient decisionaidAlthough some components of the ANSWER were notmodifiable (e.g., storylines of the animated videos), weacknowledged participants’ comments regarding limita-tions of this online tool. Four additional themes relatedto limitations and strengths of the ANSWER emerged inour analysis. These included 1) authenticity, 2) informa-tion accuracy about living with arthritis, 3) modelingshared decision making, and 4) ease of use (Table 4).In general, participants from both cycles were able torelate to the characters in one or more patient stories,although some preferred the stories told by real actorsor patients rather than animated characters. Also, theyTable 2 ANSWER usability testing resultsAll (n = 15) Cycle 1: Beforemodification (n = 10)Cycle 2: Aftermodification (n = 5)Time to complete ANSWER in minutes (SD) 56.08 (34.80) 55.50 (37.98) 57.00 (33.28)Modified system usability scale items (SD)(1 = Strongly disagree; 5 = Strongly agree):1. I liked using ANSWER as a tool for making an informed decision about usingmethotrexate as a treatment option for my RA4.13 (1.06) 4.20 (1.23) 4.00 (0.71)2. I found ANSWER unnecessarily complex 1.20 (0.56) 1.10 (0.32) 1.40 (0.89)3. I thought ANSWER was easy to use 4.07 (1.34) 3.80 (1.55) 4.60 (0.55)4. I think I would need the support of a technical person to be able to useANSWER1.60 (1.06) 1.70 (1.25) 1.40 (0.55)5. I found the content and navigation in ANSWER was well integrated 3.73 (1.10) 3.60 (1.27) 4.00 (0.71)6. I thought there was too much inconsistency between the design andnavigation of ANSWER1.87 (1.19) 1.60 (0.84) 2.40 (1.67)7. I would imagine that most patients with RA would learn to use ANSWER veryquickly4.47 (1.06) 4.50 (1.27) 4.40 (0.55)8. I found ANSWER very cumbersome to use 1.73 (1.39) 1.50 (1.27) 2.20 (1.64)9. I would be very confident using ANSWER 4.13 (1.19) 4.00 (1.41) 4.40 (0.55)10. I would need to learn a lot of things about using computers before I couldget going with ANSWER1.67 (0.90) 1.70 (1.06) 1.60 (0.55)Total System Usability Scale score (SD) 81.17 (13.53) 81.25 (14.92) 81.00 (11.81)(Scores of the 10 items were transformed into a summary score ranging from 0to 100; higher = more user friendly)Li et al. BMC Medical Informatics and Decision Making 2013, 13:131 Page 6 of 12http://www.biomedcentral.com/1472-6947/13/131felt that the pros and cons about using methotrexatewere well integrated in the context of everyday life ofpeople with RA. For example, one participant remarkedpositively about the realistic depiction of fatigue in thestories (Jamie, Table 4). Participants also commentedthat the patient stories were helpful because the maincharacters demonstrated shared decision-making behav-iours, such as considering pros and cons of treatmentoptions and communicating questions and concernswith health professionals. Finally, although participantsfelt in general that the ANSWER was user-friendly, somecriticised the videos as less polished compared to otherexisting patient education programs that used real pa-tients or actors (Table 4).DiscussionIn this study, we employed rigorous methodology to as-sess the usability of a new online decision aid for pa-tients with rheumatoid arthritis. Our results showed thatthe ANSWER prototype was user-friendly even beforemodifications were made (overall SUS score beforemodification: 81.25, SD = 14.92; after modification:Table 3 Modifiable usability issues identified by participants in testing cycle 1 and changes madeCategory Examples of participant comment Changes made1. Information Delivery: clarity of theinformation and presentation style.• The narration is a bit long…a little bit repetitive.(Jamie – female, age group: 35–49)Added key messages for users who prefer asummary of the narrated content.• Six video clips. That’s quite a lot especially eightminutes long. (Theresa – female, age group:50–64)• Reduced the length of videos. The final versionranged from 4 minutes 26 seconds to 7 minutes55 seconds.• (On videos) I think it’s a bit long winded. Youcould have said the same thing with about2 minutes less so that’s like saying a bit boringfor someone to watch him doing the same thingtwice or three times. (Sherly – female, age group:50–64)• Added subtitles to highlight important pointsin the video.2. Navigation Control: Difficulties inrecognizing and using buttons to start/stopthe narrated content, adjust volume, andcontrol videos.• This could be a different colour maybe, thenarration (button), just because I didn’t see it rightaway, I went straight to the text to read.(Bob – male, age group: 35–49)• Used the YouTube format for all videos.• Enlarged the size of buttons.• (On accessing the videos) Well I’d be curiousso what I would do is I would probably click on,my first inclination is to click this because, youknow, you are programmed by YouTube to dothat. I saw the narration button later and that’swhy I was like, oh, okay, now what do I do?(Bob – male, age group: 35–49)• Added labels to navigation controls whenappropriate.3. Layout: Positions of the video, text,diagrams and navigation buttons.• (A comment on watching video and readinginformation at the same time) …I lose the videoso I am like back and forth, back and forth. Keepsme busy, keeps me entertained, but not all (thetime), you know, especially when you are dealingwith people with arthritis before medication, yourhands are not just scrolling down, trust me, it’svery, very hard. (Jamie – female, age group:35–49)• Further condensed the key points in order toreduce scrolling with a mouse while viewing awebpage.• Revised the webpage layout and addedhyperlinks for easy access to key summaries andvideo.4. Aesthetic: The colour and ‘look and feel’ ofthe program.• For aesthetics it might be nice to have acoloured box around each one of these(diagrams)…I don’t know if you can make themall the same or each one different coloursbecause it (the website) looks kind of bland…Or itdoesn’t look like a great beginning where noneof these really jump out at me… (Bob – male,age group: 35–49)• Added pictures in the introductory pages andthroughout the value elicitation module.Added a screenshot of the animated story at thetop of each page of the information module. Ahyperlink was set up to direct people to see thevideo in a bigger YouTube viewer.• As to the colour and layout, I think it needs, it’skind of flat and uninteresting… (Theresa –female, age group: 50–64)• Probably add a little more just colour. Make it alittle more fun so you can actually like you areeager to go into the site. (Jamie – female, agegroup: 35–49)Li et al. BMC Medical Informatics and Decision Making 2013, 13:131 Page 7 of 12http://www.biomedcentral.com/1472-6947/13/131Before modificationAfter modificationFigure 3 ANSWER homepage – before and after modification.Li et al. BMC Medical Informatics and Decision Making 2013, 13:131 Page 8 of 12http://www.biomedcentral.com/1472-6947/13/13181.00, SD = 11.81). Component scores of the two cyclesappeared to be similar, although the small sample sizehindered the opportunity for hypothesis testing. There isno consensus on what constitutes an acceptable SUSscore [36], however Bangor et al. [37] reviewed themeasurement properties of the SUS and suggested thatproducts with SUS scores between the high 70s and 80swere considered ‘good products’. Programs scoringbelow 70 required further improvement and those in thelow 70s were considered ‘passable’. Products scoring 90and above were deemed ‘superior’. Based on their rec-ommendation, the ANSWER has met the standard of auser friendly program. It should be noted that we de-signed the ANSWER for patients to use at their ownpace. Although participants were asked to complete theANSWER in one testing session, we expect that in real-ity some might complete the online tool in severalsessions.It was expected the usability of a new product couldbe improved by addressing issues identified during theusability testing and that this might translate into an im-proved SUS. What was interesting in this study was thatalthough the SUS score was high and met the standardas a good product in Round 1, the formative evaluationidentified a number of modifiable usability issues. Thissupports the use of the formative evaluation along withthe summative evaluation in usability testing. The smallchange in the SUS score between Round 1 and Round 2might be due to the non-modifiable issues, includingthose raised about the videos. However, given the smallsample size in each round, a direct comparison wouldnot be possible.Our study also demonstrated the value of formativeusability testing. Despite the favourable SUS scores inCycle 1, participants identified a number of usability is-sues. Our findings were similar to the usability issuesTable 4 Themes illustrating limitations and strengths of the ANSWER designCategory Participant comments1. Authenticity: Participants were able to relate to the patient stories, asthey cover different age groups and sexes. However, some preferred realactors as compared to the animated characters.• …it seems like real information, real people talking about the disease,pros and cons, you know, the fear to take it, uh, the fears or stopping itcould happen; are they going to be able to work? The fear of losing a job.So those are real situations. It just makes the site a little more human andrealistic. It’s not just scientific information. (Jamie – female, age group:35–49; C1)• (On ‘Rosa: About RA video’) I kind of related to it just because, well…,because my daughter’s, like I did have problems, like I did start flaring up.I was in remission and then I flared up after. So I was kind of relating toher …and then hearing her (Rosa) talk to her father really made me sadto actually go back to my parents’ place while my husband was workingand stay there for the week. It was tough, but yeah, I can relate to it.(Amy – female, age group: 20–34; C1)• I think the information is very plentiful, but I think what people, what thelayman person to look at this website is going to need to know morepersonable, real stories from people that are like not acting, not – youknow people that are actually taking the drug on a regular basis whatthey’re going through. (Rosemary – female, age group: 35–49; C1)• It might be good overall if these were real video clips (with actors).(Sheila – female, age group: 50–64; C2)2. Information accuracy about living with arthritis: Key feature of RAwas fairly portrayed.• People surrounding, you know, patients, um, with RA, they don’t knowlike…that tiredness you feel at all times the people around you they don’treally understand. So if someone in my family or within, you know, afamily watches this they might go like, oh it’s true, I mean she’s not likemaking it up, um, she is actually tired; it’s part of the information, so that’ssomething that haven’t seen in any of the websites to be honest.(Jamie – female, age group: 35–49; C1)3. Modeling shared decision-making: Some participants commentedon the ability of ANSWER to provide examples of active and engagedpatients.• I think we’re moving away from the old style where you just didwhatever your doctor told you and didn’t ask questions. And (ANSWER) ishelpful in expanding people’s thinking about it. As far as I can see ittouched on the key decision point. So it’s good for that. (Theresa –female, age group: 50–64; C1)4. Ease of use: Participants commented on the user friendliness, butthey were also hoping for a more sophisticated software product• It was user-friendly definitely. Yeah, I feel it wasn’t sophisticated enough.I mean methotrexate is a big name and it was a little gimmicky at somepoint (of the video presentation), maybe because of the graphics you know.(Jane – female, age group: 50–64; C2)C1 Usability Testing Cycle 1, C2 Usability Testing Cycle 2.RA Rheumatoid Arthritis.Li et al. BMC Medical Informatics and Decision Making 2013, 13:131 Page 9 of 12http://www.biomedcentral.com/1472-6947/13/131found in other patient-oriented online programs. Forexample, Stinson et al. [39] tested an electronic chronicpain diary for adolescents with arthritis and found theslider controls of pain visual analogue scales difficult tooperate. These slider controls were subsequently modi-fied to improve user experience. In another study evalu-ating an online self-management program for youth withjuvenile idiopathic arthritis, Stinson et al. [40] uncoveredperformance errors and design issues that were modifi-able to improve user satisfaction. Recently, in a full scaleusability evaluation of an online interactive game forpatients making treatment decisions for prostate cancer,Reichlin et al. [41] identified similar navigation andcontent-related issues that could impede user experi-ences. These studies indicated the importance of forma-tive usability testing to improve new online programsprior to field testing. Findings from the current usabilitytesting concur with this viewpoint.There are several limitations with this study. First, thetesting was conducted with participants with a long dis-ease duration (median = 5 years), hence the view of thosewith a recent diagnosis was under represented. Second,since only two out of 15 participants were men, ourfindings might not reflect the full range of user experi-ence of men. Third, most participants were educatedand computer-savvy; hence the results may not begeneralizable to people who are less educated orcomputer-savvy. Future studies including these popula-tions will be important as they may be in greater need oflearning about options, risks and benefits, and exploringtheir own preferences and engaging in shared decision-making. Fourth, we were unable to address all usabilityissues identified by participants since some componentswere already finalized at the time of the testing (e.g., theanimated videos). Our choice of animation style wasbased on a balance between aesthetic and budgetaryconstraints. Although some participants responded posi-tively to the animated graphic novel approach, othersconsidered it lacking sophistication. Finally, due to thesmall sample size, we were unable to further explore theinfluence of demographic characteristics (e.g., age, sex,education level), disease characteristics (e.g., disease dur-ation and severity) and individuals’ internet use (e.g.,time spent on internet per day) on the usability scores.This is important because some of the preferences (e.g.,animations) may be associated with specific patient char-acteristics, which if known, would assist in designingfuture decision aids targeted to particular populations.Despite the limitations, findings from the usabilitytesting have allowed us to refine the ANSWER proto-type. Recognizing that usability issues are major barriersto the adoption of health information technology [42],we have taken steps to address them over the courseof the ANSWER’s development. Yen and Bakkenrecommend three levels of usability evaluation [43]. Thefirst level aims to identify product components andfunctions needed by users to accomplish a task (i.e.,user-task interaction). Methodology includes direct ob-servation and needs assessment using qualitative orsurvey methodology. The second level assesses the user-task-program interaction using methods such as heuris-tic evaluation [44], cognitive walkthrough [45], and thethink aloud technique [46,47]. The third level examinesthe complex interaction among users, tasks, the programand the environment using a variety of experimental andobservational designs. All three levels are addressed inthe ANSWER development and were shown to be help-ful for different aspects of refining the tool.Strengths of the study include the emphasis on userexperiences. The ANSWER tool was informed by ourprevious qualitative research on RA patients’ help-seeking experience, especially their challenges in makingmedication decisions [29]. In addition, patient/consumercollaborators were involved at the outset to provide in-put on the program design. We subsequently evaluatedthe user-task-program interaction in the current usabil-ity testing and addressed all modifiable navigation issues.The next step will be to evaluate the ANSWER in aproof-of-concept field study with patients who are con-sidering methotrexate for treating RA. In addition tousing the tool online, individuals will be able print theone-page summary of their questions, concerns and pre-ferred option to bring to their rheumatologist appoint-ment. As such, they will have the full experience ofshared decision-making. Our goal will be to assess theextent to which the ANSWER reduces decisional con-flict and improves self-management knowledge and skillsin patients who are considering methotrexate for RA[16].ConclusionsWe have developed a user-friendly online decision aid toassist patients in making informed decision about usingmethotrexate for RA. Although the SUS score indicatedhigh usability before and after major modification, find-ings from the think-aloud sessions illustrated areas thatrequired further refinement. Our results highlight theimportance of formative evaluation in usability testing.Competing interestsThe authors declare that they have no competing interests.Authors’ contributionsThe study was conceived by LCL, PMA, AFT and CLB. All authors contributedto the development of the research protocol. LCL is the principal applicantand PMA is the decision-maker co-principal applicant. Writing of the manu-script was led by LCL and all authors approved the final version.AcknowledgementThe authors are grateful for the support of patient/consumer collaborators,including Otto Kamensek (Arthritis Research Centre Consumer AdvisoryLi et al. BMC Medical Informatics and Decision Making 2013, 13:131 Page 10 of 12http://www.biomedcentral.com/1472-6947/13/131Board), Cheryl Koehn (Arthritis Consumer Experts), Colleen Maloney(Canadian Arthritis Patient Alliance), health education consultant Gwen Ellertand information scientist Jessie McGowan. We also thank our digital mediacollaborators Jeannette Kopak and George Johnson (Centre for DigitalMedia) for organizing and supervising two Master of Digital Media studentteams to develop the ANSWER. The Design and Development Team: ConradChan, Fouad Hafiz, Felwa Abukhodair, Liam Kelly, Karin Schmidlin, Yamin Li,and Shao Yingyun. The Production Team: Shahrzad Aghasharifianesfahani,Erez Barzilay, Jason Ho, Milim Kim, Clark Kim, Natalia Mitrofanova, and AlSinoy. The ANSWER programming was led by Matt Jenkins. Original musicwas composed by Ben Euerby.This study was funded by a Canadian Institutes of Health Research (CIHR)operating grant (funding reference number: KAL 94482).Author details1Department of Physical Therapy, University of British Columbia, Vancouver,Canada. 2Arthritis Research Centre of Canada, Vancouver, Canada. 3Mary PackArthritis Program, Vancouver Coastal Health, Vancouver, Canada. 4Division ofRheumatology, Faculty of Medicine, University of British Columbia,Vancouver, Canada. 5School of Nursing, University of Ottawa, Ottawa,Canada. 6Ottawa Hospital Research Institute, Ottawa, Canada. 7School ofInteractive Arts and Technology, Simon Fraser University, Surrey, Canada.8Institute of Population Health, University of Ottawa, Ottawa, Canada.9Department of Occupational Science and Occupational Therapy, Universityof British Columbia, Vancouver, Canada.Received: 21 May 2013 Accepted: 27 November 2013Published: 1 December 2013References1. Engel A, Roberts J, Burch TA: Rheumatoid arthritis in adults. Vital HealthStat [1] 1966, 11:1–43.2. Spector TD: Rheumatoid arthritis. Rheum Dis Clin North Am 1990,16:513–537.3. Hochberg MC: Early aggressive DMARD therapy: the key to slowing diseaseprogression in rheumatoid arthritis. Scand J Rheumatol Suppl 1999, 112:3–7.4. Quinn MA, Conaghan PG, Emery P: The therapeutic approach of earlyintervention for rheumatoid arthritis: what is the evidence?Rheumatology 2001, 40:1211–1220.5. Nell VP, Machold KP, Eberl G, Stamm TA, Uffmann M, Smolen JS: Benefit ofvery early referral and very early therapy with disease-modifying anti-rheumatic drugs in patients with early rheumatoid arthritis.Rheumatology 2004, 43:906–914.6. Lacaille D, Anis AH, Guh DP, Esdaile JM: Gaps in care for rheumatoidarthritis: a population study. Arthritis Rheum 2005, 53:241–248.7. Lacaille D, Rogers P: Why are people with rheumatoid arthritis not usingDMARDs? Understanding gaps in care. Arthritis Rheum 2007, 56:S86.8. Townsend A, Hunt K, Wyke S: Managing multiple morbidity in mid-life: aqualitative study of attitudes to drug use. BMJ 2003, 327:837.9. Li LC, Townsend AF, Adam PM, Cox SM, Amarsi Z, Backman CL, et al:Crossing the threshold: Help-seeking for early symptoms in people withrheumatoid arthritis. Arthritis Rheum 2009, 60:S758.10. Kjeken I, Dagfinrud H, Mowinckel P, Uhlig T, Kvien TK, Finset A:Rheumatology care: involvement in medical decisions, receivedinformation, satisfaction with care, and unmet health care needs inpatients with rheumatoid arthritis and ankylosing spondylitis.Arthritis Rheum 2006, 55:394–401.11. Weston WW: Informed and shared decisoin-making: the crux of patientcentred care. Can Med Assoc J 2001, 165:438–440.12. Legare F, Ratte S, Stacey D, Kryworuchko J, Gravel K, Graham ID, et al:Intervenitons for improving the adoption of shared decision making byhealthcare professionals. Cochrane Collaboration 2010, 5. Art.No.:CD006732.DOI: 10.1002/14651858.CD006732.pub2.13. O’Connor AM, Graham ID, Visser A: Implementing shared decision makingin diverse health care systems: the role of patient decision aids.Patient Educ Couns 2005, 57:247–249.14. Legare F, Stacey D, Forest PG: Shared decision-making in Canada: update,challenges and where next! Zeitschrift für ärztliche Fortbildung und Qualitätim Gesundheitswesen - German Journal for Quality in Health Care 2007,101:213–221.15. O’Connor AM, Stacey D, Entwistle V, Llewellyn-Thomas H, Rovner D, Holmes-Rovner M, et al: Decision aids for people facing health treatment or screeningdecisions. Cochrane Database Syst Rev 2006, 2006:3.16. Li LC, Adam P, Townsend AF, Stacey D, Lacaille D, Cox S, et al: Improvinghealthcare consumer effectiveness: an animated, self-serve, Web-based re-search tool (ANSWER) for people with early rheumatoid arthritis. BMC MedInform Decis Mak 2009, 9:40.17. International Patient Decision Aid Standards (IPDAS) Checklist. Ottawa HealthResearch Institute; 2008. http://decisionaid.ohri.ca/methods.html#checklist.18. Elwyn G, O’Connor A, Stacey D, Volk R, Edwards A, Coulter A, et al:Developing a quality criteria framework for patient decision aids: onlineinternational Delphi consensus process. BMJ 2006, 333:417.19. Jibaja-Weiss ML, Volk RJ: Utilizing computerized entertainment educationin the development of decision aids for lower literate and naivecomputer users. J Health Commun 2007, 12:681–697.20. Glik D, Berkanovic E, Stone K, Ibarra L, Jones MC, Rosen B, et al: Healtheducation goes Hollywood: working with prime-time and daytimeentertainment television for immunization promotion. J HealthCommun 1998, 3:263–282.21. Marcus PM, Huang GC, Beck V, Miller MJ: The impact of a primetimecancer storyline: from individual knowledge and behavioralintentions to policy-level changes. J Cancer Educ 2010, 25:484–489.22. Blair C, Valadez JJ, Falkland J: The use of professional theatre forhealth promotion including HIV/AIDS. J Dev Comm 1999, 10:9–15.23. Thompson D, Baranowski T, Buday R: Conceptual model for the design ofa serious video game promoting self-management among youth withtype 1 diabetes. J Diabetes Sci Technol 2010, 4:744–749.24. Silverman BG, Mosley J, Johns M, Weaver R, Green M, Holmes J, et al:Computer games may be good for your health: shifting healthcarebehavior via interactive drama videogames. AMIA Annual SymposiumProceedings/AMIA Symposium :1075, 2003 2003, 2003:1075.25. Brown SJ, Lieberman DA, Germeny BA, Fan YC, Wilson DM, Pasta DJ: Educationalvideo game for juvenile diabetes: results of a controlled trial. Med Inform 1997,22:77–89.26. Jibaja-Weiss ML: Entertainment education for informed breast cancertreatment decisions in low-literate women: development and initialevaluation of a patient decision aid. J Cancer Educ 2006, 21:133–139.27. Suarez-Almazor ME, Belseck E, Shea BJ, Tugwell P, Wells G: Methotrexate fortreating rheumatoid arthritis. Cochrane Database Syst Rev 2008,3(2). Art. No.: CD000957. DOI: 10.1002/14651858.CD000957.28. Visser K, Katchamart W, Loza E, Martinez-Lopez JA, Salliot C, Trudeau J,et al: Multinational evidence-based recommendations for the use ofmethotrexate in rheumatic disorders with a focus on rheumatoidarthritis: integrating systematic literature research and expert opinionof a broad international panel of rheumatologists in the 3E Initiative.Ann Rheum Dis 2009, 68:1086–1693.29. Townsend AF, Backman CL, Adam P, Li LC: A qualitative interview study:Patient accounts of medication use in early rheumatoid arthritis fromsymptom onset to early post diagnosis. BMJ Open 2013, 3(2).30. Fries JF, Spitz P, Kraines RG, Holman HR: Measurement of patient outcomein arthritis. Arthritis Rheum 1980, 23:137–145.31. Stucki G, Liang MH, Stucki S, Bruhlmann P, Michel BA: A self-administeredrheumatoid arthritis disease activity index (RADAI) for epidemiologic research.Psychometric properties and correlation with parameters of disease activity.Arthritis Rheum 1995, 38:795–798.32. Fransen J, Langenegger T, Michel BA, Stucki G: Feasibility and validity of theRADAI, a self-administered rheumatoid arthritis disease activity index.Rheumatology (Oxford) 2000, 39:321–327.33. Rubin J, Chisnell D: Handbook of Usability Testing, Second Edition: How to Plan,Design, and Conduct Effective Tests. Indianapolis, IN: Wiley Pubishing, Inc.; 2008.34. Ericsson KA, Simon HA: Protocol Analysis: Verbal Reports as Data. Cambridge,MA: The MIT Press; 1984.35. Lewis CH: “Thinking Aloud” Method In Cognitive Interface Design (Technicalreport RC-9265). Yorktown Heights, NY: IBM; 1982.36. Brooke J: SUS: A “quick and dirty” usability scale. In Usability Evaluation inIndustry. Edited by Jordan PW, Thomas B, Weerdmeester BA, McClelland IL.London: Taylor & Francis; 1996:189–194.37. Bangor A, Kortum PT, Miller JT: An empirical evaluation of the systemusability scale. Inter J Human-Comput Interac 2008, 24:574–594.38. Strauss A, Corbin J: Basics of Qualitative Research: Grounded TheoryProcedures and Techniques. Newbury Park, CA: Sage; 1990.Li et al. BMC Medical Informatics and Decision Making 2013, 13:131 Page 11 of 12http://www.biomedcentral.com/1472-6947/13/13139. Stinson JN, Petroz GC, Tait G, Feldman BM, Streiner D, McGrath PJ, et al:e-Ouch: usability testing of an electronic chronic pain diary foradolescents with arthritis. Clin J Pain 2006, 22:295–305.40. Stinson J, McGrath P, Hodnett E, Feldman B, Duffy C, Huber A, et al:Usability testing of an online self-management program for adolescentswith juvenile idiopathic arthritis. J Med Internet Res 2010, 12:e30.41. Reichlin L, Mani N, McArthur K, Harris AM, Rajan N, Dacso CC: Assessing theacceptability and usability of an interactive serious game in aidingtreatment decisions for patients with localized prostate cancer. J MedInternet Res 2011, 13:e4.42. Nielsen J: What is usability. In User Experience Re-Mastered. Edited by WilsonC. Canada: Morgan Kaufmann; 2009:3–22.43. Yen PY, Bakken S: Review of health information technology usabilitystudy methodologies. J Am Med Inform Assoc 2012, 19:413–422.44. Nielsen J: Heuristic evaluation. In Usability Inspection Methods. Edited byNielsen J, Mack RL. New York: John Wiley & Sons; 1994:25–64.45. Mack RL, Montaniz F: Observing, predicting and analyzing usabilityproblems. In Usability Inspection Methods. Edited by Nielsen J, Mack RL. NewYork: John Wiley & Sons; 1994:293–336.46. Nielsen J: Usability Engineering. Boston, MA: Academic Press; 1993.47. Van den Haak M, de Jong M, Schellens PJ: Retrospective vs. concurrentthink-aloud protocols: testing the usability of an online library catalogue.Behav Inform Technol 2003, 22:339–351.doi:10.1186/1472-6947-13-131Cite this article as: Li et al.: Usability testing of ANSWER: a web-basedmethotrexate decision aid for patients with rheumatoid arthritis. BMCMedical Informatics and Decision Making 2013 13:131.Submit your next manuscript to BioMed Centraland take full advantage of: • Convenient online submission• Thorough peer review• No space constraints or color figure charges• Immediate publication on acceptance• Inclusion in PubMed, CAS, Scopus and Google Scholar• Research which is freely available for redistributionSubmit your manuscript at www.biomedcentral.com/submitLi et al. BMC Medical Informatics and Decision Making 2013, 13:131 Page 12 of 12http://www.biomedcentral.com/1472-6947/13/131


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items