Opt-in
user feedback
v0.4
Written10/16/00
Expires: 12/10/00
Author: Mike Kuniavsky (mikek@tired.com)
A system of software usage tracking and usability testing is proposed that attempts to leverage the large penetration of Internet connectivity to compensate for some deficiencies of existing user research methods.
The system recommends embedding usage tracking and usability question-asking mechanisms directly into software, which — with the user's permission — anonymously "phones home" to report on which features are used and to download questions and tasks which the user is asked at appropriate times as they use the product.
How does someone uses your software? Do they use it as designed? Which aspects of it do they use the most? Which do they find most valuable? Most frustrating?
Such questions are easier asked than answered. Understanding how people use your product is key to improve it and knowing why it's successful (or why it isn't), but gaining that understanding is not as easy as merely asking those questions. A product's audience consists of many people, with many interests, many desires and many abilities.
Existing methods of gaining insight into the user experience of a product are all either expensive, time-consuming or inaccurate (and sometimes all three at the same time).
It's difficult to collect usability information from people as they're use a product. Laboratory user testing, contextual inquiry and surveys — the most popular methods of collecting such information — all have their problems.
User testing happens in a laboratory, outside the environment that a product gets used. The people who participate do not necessarily use the product and the tasks they do are not necessarily representative of how the system is actually used. Although this allows elements of the interface to be isolated and studied, the designers' and researchers' understanding of it is can be distorted because the research happens outside the context of how the product is actually used.
Contextual inquiry involves observing how people use a product in their everyday environment. This is good for understanding task priorities and for seeing how the interface is really used, but the process is time-intensive and it is difficult to observe users using more than a small subset of the program's functionality.
Surveys do not measure usability at all, but the respondents' views. This is useful in determining how people view their use of the product, but it says nothing about how they actually use it.
Finally, all of these methods only measure the way that people use a product over a short period or with a small number of people. There is no straightforward way of knowing which people use which features, in what order or how that usage changes over time.
This proposal attempts to address these problems with a hybrid experience research method that's grounded in the software itself and which leverages the large number of persistent (or, at least, frequent) Internet connections among software users.
Software that collects data about how people use it can produces data that's as good or better than what an observer can produce. Many of the biases present in laboratory situations are eliminated because the software is just used, rather than its use being simulated. In addition, the software can be written such that it can ask questions of the user, relaying the answers directly to the developers. Together, these two methods create an unprecedented communication channel between the user and the developer, allowing the user's actions, perceptions and abilities to directly affect the development of the product they're using.
The information collected by this method augments and enriches the understanding created by more traditional methods, producing a large source of rich and rigorous information about the user experience with less work than existing methods. By distributing questions over time, it can also replace (to some extent) large, monolithic surveys and user tests.
A sketch of the proposed system
|
The key elements of the system are tight integration with the software product, opt-in participation by the users, feature usage tracking, remote usability testing and anonymous participation.
Deep integration with a software product is one of the most important elements to this proposal. Tightly coupling usage tracking and experience research with normal functionality it's possible to blend usage feedback seamlessly with a person's natural use of the software. External survey and monitoring systems create work for people, since they require a change of context and they interrupt the flow user's task. By merging feedback with how people work naturally, it's possible to get less forced response rate.
For example, if a question about the specifics of a printer configuration dialog box is asked on a survey, the context may be so far removed from the experience that the person taking the survey may not be interested in answering. Even if he or she answers it, the answers may not reflect their actual experience: they may not accurately remember either it or their own behavior and opinions. Asking the question as the dialog box is used may increase both the rate of participation and the accuracy of the responses.
Users should not be bothered by the system. The system should set expectations appropriately, informing the users of the value of their participation and their role in the development of the product, but it should neither collect information that they do not want collected, nor should it bother them when they do not want to be bothered.
People should be given the option to participate from the very beginning and they should have an easy way to reduce or eliminate their commitment at any time. The opt-in philosophy should permeate the working of the whole feedback system, rewarding and encouraging, but not forcing, participation.
Knowing which features are used, how often they're used and in what order they're used allows developers to know where to focus efforts and where problems may lie. Collecting this data automatically avoids the difficulty and subjectivity of direct observation or self-reporting while increasing accuracy and, hopefully, the degree of participation.
The system should record what parts of the software are used, for how long, in what order and in what combination, storing the results locally. At regular intervals, it contacts the application's "home" server and submits this data (or a summary of it) to a central database, where each user's data is kept together while maintaining the user's anonymity.
In addition, data collection tools used to analyze results are important and should be designed simultaneously with the development of the data collection software.
Asking questions as people are using a product for themselves (with their data, in their environment, when they want to use it) is a powerful way of gaining insight into their experience of it. It allows for the features to be seen as part of the larger task they're trying to accomplish, rather than as individual elements. It's possible to ask more detailed questions and perform more specific tasks than it would be in a regular user test, since users' attention is already on the task at hand.
In addition, periodic questions and tasks can lead to a greater cumulative amount of usability information than simple usability tests. One five-minute question asked once a week for a year yields more than three hours of detailed usability feedback information without requiring a lot of concentrated effort on the part of the user.
For certain applications, the tracking of usage patterns to a fine grain can be dangerously misused if it it's associated with a specific person. Thus, the system needs to protect the privacy of its user base while still gathering information that is useful to its developers.
This can be accomplished in a similar way to the way browsers protect the anonymity of their users. Although it's not perfect, this system appears to be sufficient for most people and options to increase or decrease anonymity are available through the browser configuration.
In this early draft, rather than concentrating on solutions, this document tries to specify the needs which any developed technologies will have to fulfill and to suggest a couple of solutions.
This is pretty raw. It's primarily a list of my thoughts on the technical implementation of this idea. It is not a rigorous specification, since I don't have the necessary technical knowledge to write such a thing.
Each piece of software will need to be able to "call home" and communicate with a central data store in a consistent, simple and robust manner. Following is a list of constraints on the communications layer of the protocol. It's based on the Web model, where there is a transport protocol that securely negotiates the transfer of content in a markup language between a client (the product running on the user's machine) and a central server or servers.
The protocol should assume guaranteed packet delivery in a reasonable amount of time. This can be assumed to be TCP/IP, since that's what most user application Net services talk and because accurate information delivery is more important than timely delivery.
The protocol will need to do three things:
Deliver usage information
Retrieve new user questions/tasks
Deliver the results of such tasks
It will need to do this securely in order to preserve the anonymity of users and to prevent server responses from being spoofed. This speaks to the need for bidirectional encryption and authentication, since that will keep responses private while enabling the tracking of how a single users' use changes over time.
Tracking an individual's use of a system without revealing that individual's identity is tricky. On one hand, it's important to uniquely identify a user, on the other hand, that identity should not be linkable to a specific individual. The crypto community probably has a better solution, but a system of unique cookies could work, provided that the cookie is encrypted when transferred to the server and the server keeps no logs that link IP addresses or personal information to specific cookies. The cookie would serve as an index to a database entry in the server that stored the user's behavior and participation preferences, but no more.
When retrieving questions and tasks from the server, the protocol should make sure that the questions are not spoofed, that the answers are delivered and that the right questions and tasks get to the right people at the right time. SSL encrypted HTTP, which is Open and for which code is available, is probably good enough for the needs of most people for the first two issues.
The last issue, question dissemination, may be more of a concern for the developers of the server than for the framers of the protocol. The contents of messages are opaque at this level of the protocol, but the system may allow for different levels of participation (as described below) and the preferences for what kinds of questions will be asked will be exchanged at the protocol level, in order to prevent unnecessary data transfer. For example, someone who just uses the word processor segment of Open Office may never want to see questions about the spreadsheet. Or they may be bothered by general questions and may only be interested in questions about specific features. There should be a way for the client program to tell the server that it's not interested in those questions.
[Then again, the amount of data related to the questions may be so small that the client can decide which questions it accepts. Or the server may keep track of preferences and only send the questions that are pertinent.]
This could be done in the form of a negotiation at the beginning of each session, where the client exchanges preference information with the server or the preferences could be stored with the server, so that the communications protocol only provides a means for storing preferences.
Questions and tasks produce results that need to be delivered to the "home" server. The delivery needs to be timely and secure. To ensure timeliness, questions and tasks should have expiration dates attached to them as part of the question definition. If they are not shown and answered by the expiration date, the information is deleted from the local machine. So that answers aren't spoofed and identities maintained, there needs to be a way to authenticate the responses.
An existing Open system that seems to do most of this is the HTTP's POST method encrypted in SSL coupled with a unique identity cookie, as used by most ecommerce sites.
The software will create a log file of features that are used. The format of this file should have several key features:
Easily parsed
Compact
Secure
Flexible
Since intense software use can generate thousands of commands per hour, it's important that the logging mechanism be space- and resource-conscious, while at the same time remaining easily parsed. An efficient file format is thus important. XML is human-readable and becoming a standard "base" file format for many kinds of files. A suitable DTD could be defined for the logging format. However, XML may be too space inefficient and a less verbose file format may need to be found. Alternately, for an application like Open Office that supports macro recording capabilities, the file format may be the same as the macro format (although, ideally, the file format would be Open and portable).
Experimentation and experience will show what the ideal file format is.
The kinds of information that it could log include:
Commands or indexes to commands ("Edit:AutoText..." or "2:10").
Actions (when something is cut: "Selection.cut()", when a new window is opened: "Application.Windows.Item(newwindow).Activate", etc.)
The time when commands are invoked.
How it is invoked (manually, automatically, from an external program, as part of a macro, etc.).
The module that is currently active.
Such information can reveal information which can be easily misused. Thus it's important to keep the log files secure. Granted, with Open Source products it's difficult to keep someone from altering the logging code to remove or pervert the security. Security provided in such a system would probably be either cumbersome (and would thus go against the goal of minimally impacting the user experience) or would only provide superficial protection. However, superficial, even partial, protection is better than nothing and should be attempted.
The file format should also be flexible. It should allow users to select level of logging with which they feel comfortable. Thus, in order to be able to unify the data logs as they arrive at the "home" server, the file format should have consistent but variable levels of granularity. Thus, one person may only choose to log that she used Open Writer, whereas another person may feel comfortable logging every single action short of typed keystrokes.
The format in which questions and tasks are transmitted to the client software should encapsulate several kinds of information:
The appropriate context for the question/task (Can it be asked at any time or should it only appear if a specific function or sequence is performed?)
The appropriate timing of the question (When should it be asked in the user's session? When will it expire?)
The information that should be reported (How long did it take to perform the task? How many actions did it take? How big is the document? Etc.).
The markup language should distinguish between a few different kinds question fauna so that the client can display them appropriately. For example, opinion questions could be displayed as dialog boxes that are overlayed on the regular interface, whereas multi-step tasks could be shown as collapsable sidebars. It should also be expandable so that other question types (and maybe whole other modes of inquiry, such as diaries) can be introduced at a later time.
The XML survey standard being hashed out by opensurvey.org may serve as the basis of this system.
There are two major interface groups from the perspective of the user, the Question/Task Interface and the Feedback Management Interface.
Question and task presentation should be integrated into the general interface of the product, not as a separate module. In order for people to feel that participation in the development process is a natural part of the way they use software, the question and task system should require no extra work from the user. It's the easiest way for them to communicate with the development team and improve the software they're using, so it's in their best interest, but it should also be easy, since it's a secondary goal when they're using the software.
The design should have the following qualities:
Clearly for feedback, rather than functionality.
Consistent.
Easily dismissed.
Contextual. This could involve a system that highlights parts of the interface with the intention of asking specific questions about those parts (much as some help systems work on a layer that's separate from the regular interface).
For more direct feedback, there should also be a direct "send the developers a note" and a bug/problem/request reporting system that's easily accessible from anywhere.
The first time the software is run the user is asked whether he or she wants to participate in the feedback program ("occasional surveys and usability studies to help the development team improve the product"). If they accept, then from that moment they will have to expend some amount of effort to manage their interaction with the system. The management interface should clearly explain the features of the service and provide controls which allow the user to make appropriate adjustments.
Controls should include the following:
An easy off/on switch.
An easy way to turn off/on major subsystems, allowing the user to turn off all logging or all usability testing.
Granularity controls that show how much information is being logging/reported and allow the user to adjust the quantity (either for the purposes of privacy or for space savings).
An observation window that allows the user to see exactly what is being logged for reporting.
Question/task controls that show what kind of questions and tasks are being downloaded and allow the user to select the kinds of questions he or she is interested in answering.
Security information and controls for managing encryption, cookies, log files, etc.
Demographic/Technographic information. The system will ideally have some demographic and technographic information about the user and the system they're using (so that specific audiences can be targeted for questions), but that information will need to be updated.
Network controls (proxy configuration, bandwidth collars, etc.).
There are three major interface elements from the developers' end: one to create questions/tasks, one to manage the responses and one to analyze results.
Since questions and tasks will not be seen by everyone equally (and shouldn't be), the interface for constructing questions should have two parts: one that helps the developers create appropriate and well-formed questions and a second one that allows them to target the placement of those questions in the interfaces of the user population.
Asking subgroups of the population questions will allow for a rich set of information to be collected about the needs, abilities and desires of specific groups of users (spreadsheet power users about complex function specification, newbies about the help system, etc.). In order to that, there need to be a robust interface for specifying who is asked what questions and when those questions should be displayed.
Apart from the database construction and management aspects of this, there will need to be an interface which allows the development team to manage various aspects of the question and task deployment process.
Appropriate analysis of the incoming information is critical. There will be a lot of data and it can be mined in a large number of ways. Creating the appropriate data analysis tools for the developers will be a sizable challenge onto itself. Since there will be two different systems (one, based on automated collection of feature usage; the other, survey/task-oriented), there will be two different interfaces. But, ideally, the two can interact to answer questions like this:
What are the most popular commands/features among expert users?
What are the sequences that commands/features are invoked?
What commands are most frequently followed by "Undo"?
What commands/actions are often repeated?
What dialog boxes are invoked, but canceled or that produce no change in the document or the state of the program?
Initially, the system should strive to merely tabulate and collate the information, but later, as the data volume grows larger and the development team learns to use the basic tools, the system will need to develop.
This is a first cut at an idea. The idea is to change the way that the producers and consumers of software interact with the software process. Widespread Internet communication has allowed for the breakdown of the walls between the development team and the user community, just as it has revolutionized software distribution. In the age of the golden master, it was necessary to maintain a distance between the user population and the developers because distribution cycles, and hence developer-user feedback loops, were on the order of years. With extensive penetration of connectivity, it's no longer necessary to look at the software development cycle monolithically. Likewise, it's no longer necessary to look at the user population as separate from the developer population. Just as developers are likely to be occasional users of their product, the users need to be thought of as occasional members of the development team. For that, they need good tools. This proposal outlines such a tool.
Copyright © 2000 Mike Kuniavsky. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is available at http://www.fsf.org/copyleft/fdl.html