Open Office User Survey Proposal
v0.1

Written: 10/19/00
Expires: 12/19/00
Author: Mike Kuniavsky (mikek@tired.com)

Executive Summary

To better target ongoing Open Office user experience research and feature development it's important to know who the existing StarOffice user base is and how they use StarOffice. This document describes a basic survey that can be fielded to StarOffice users in order to find out enough about them that subsequent experience research efforts can be focused.

The survey will be fielded to a random subgroup of as many existing StarOffice users as can be contacted, and will attempt to create profiles based on their personal characteristics (their demographic profile), their computer and network connection (their technological profile) and how they use StarOffice (their usage profile). The survey will be anonymous.

This is a first draft. Feedback is greatly appreciated. Please send feedback to mikek@tired.com or to the Open Office general discussion list, discuss@openoffice.org.

Background

In order to design a product that's right for an audience you have to know who that audience is, what they want and what they can do. Different groups of people have different needs, and knowing the needs of the group of users your product is targeted to increases the likelihood that what you design will be embraced by your audience.

In the case of Open Office, there is no established group of users. Presumably, the eventual target audience is the same audience as is being targets by the current commercial office suites. This is a very broad market: everyone who ever has need to use a word processor or a spreadsheet is now part of the same target audience. Even users of "lite" office suites such as Microsoft Works and Claris Works can be considered to be part of the market. Thus, the potential audience is incredibly broad, ranging from encyclopedia-writing work-groups to school children and from device driver-writing Übergeek teenagers to computerphobic executives.

Understanding what that entire audience wants is impossible. This survey is designed to represent the start of an ongoing research effort.

Since StarOffice, Open Office's parent program, has an existing audience, this survey will attempt to understand who that audience is. The StarOffice audience will not be identical to the Open Office audience — indeed it is likely to be quite different than the technically savvy audience Open Office will probably capture initially — but its members have experience with the product and are likely to be similar to the audience that Open Office will eventually attract. This makes them a reasonable subject for understanding. If future research reveals that Open Office's audience is different from StarOffice's, additional research can be done to understand who Open Office's audience really is.

Although this document uses the singular term "audience" it's important to recognize that the user population is not homogeneous. Although there will be certain trends in the population that can be extracted, this does not mean that everyone should be treated as a cookie cut from the average. The results of this survey will inform the constraints on the target audience, but they will not unequivocally define the audience and care needs to be taken treating the results as tendencies, rather than absolutes.

Unfortunately, there is no public information about StarOffice's users or their experience. This survey will attempt to answer the following questions:

• Who are StarOffice's current users? What are their distinguishing characteristics? Why are they using StarOffice?

• How do they use StarOffice? What do they like about it? What problems do they have with it?

This will create a base of knowledge from which additional research can be done. When we know something about who the users are we can focus our development efforts on improving their experience and how to attract other audiences. For example, if the current user base contains a lot of Spanish-speaking lawyers, then Spanish language support and legal document features can be bumped up in the development effort. Likewise, if we want to extend the user base to programmers, we will be aware of the fact that they're not a large part of the existing user base and we can do research as to why.

Thus, this survey is like flashing a camera flash in a dark room. For a split second, we will see some of what's in the room. We will understand some of it, we will misinterpret some of it and we will miss some crucial things. But without it, we'll have no idea about anything in the room. It will help guide us to the position where we can pop off the next flash and reveal some more. Eventually, we will have flashed enough to know pretty much what's in the room (and whether it's ready to eat us).

Recruiting

A survey is not a census. The idea is not to ask every single person in the world who meets a certain criteria, but to ask just enough people to have an idea of how the rest of who the rest are and what they believe.

Some concepts of sampling.

The group of people who fill out a survey is called a sample, but what is it a sample of? It's a randomly chosen subset of the sampling frame. The sampling frame consists of all of the people a sampling method can put us in contact with. In other words, out of the universe of all of StarOffice users, there's a subset we can contact and a subset we can't. The subset we can contact is our sampling frame; a random selection of people from the sampling frame is our sample.

A sampling frame

The big danger is picking a sampling method that does not contact the right group of people. When the sampling frame doesn't match the people we're interested in, the results can be inaccurate and misleading. For example, contacting the people who respond to a banner ad placed on Slashdot will likely over-represent power users and Linux owners relative to the actual population of StarOffice users while under-representing users of average technical skills and people who run Windows. Since (I'm assuming) a large portion of the existing StarOffice user base does not consist of Linux power users, any results from such research will be skewed toward the views of a minority.

Thus, in order to get a more accurate portrait of the StarOffice user population, it's important to invite people with as balanced a sampling method as possible.

Unfortunately, this is really hard. With a product that's been distributed through a number of different channels (StarOffice has been sold, downloaded, bundled with peripherals, etc.) on several continents, it's hard to keep track of every place it could have gone, much less how to contact its current users. Moreover, any information about the users that may have been collected in the distribution process has not been released by Sun. Thus, there's no information to start with, not even an estimate of how many active users the software has.

So we'll just have to muddle through and hope for the best.

Recruiting plan

Here's a first shot plan for how to get a balanced group:

1. Make a list of resources that large numbers of StarOffice users may use.

2. Create methods to access those resources.

3. Estimate the size of the StarOffice population each of those methods can reach.

4. Estimate the groups of users which we believe exist in quantity, but for which we can't figure out a method of access.

5. Field the survey.

6. Tracking which method each response came from and the number of invitations sent out versus the number of responses, we estimate how well the responses represent each subgroup, which we use as proportional weights when tabulating the results of the whole survey.

Resources

• Special Interest Groups (and their mailing lists), such as Werner Roth's

• StarOffice support sites (Sun StarOffice pages, ExpertCity, LinuxCare, staroffice.com, etc.)

• MIS managers who have installed StarOffice company-wide

• Sun's list of email addresses of StarOffice downloaders (not likely to be released).

Suggestions for additional resources will be greatly appreciated.

Sample size

Knowing little about the actual size of the StarOffice population, I will guesstimate that 1000 responses should be enough to be represent a "large enough" group of StarOffice users, even if it's not completely representative of the whole user base.

StarOffice user profile

Since the primary purpose of this survey is to be able to focus development and future research, the emphasis will be on gathering basic information about StarOffice users, rather than focusing on specific issues.

• Demographic. How is the respondent different from other people? Behavior, attitude and ability are concomitant with demographic information. In other words, understanding our users as human beings, rather than just as writers, illustrators or presentation makers, can make it easier to understand what they want to do with the software and how they will use it.

• Technological. What computer system do they use? Knowing the technological capabilities of our user base will allow us to gauge the potential acceptance of certain features and to better understand problems. We can't assume that everyone has a 1GHz Athlon, nor a 25 MHz 386. Knowing what the reality is can help set development parameters.

• Usage. How do they use StarOffice? StarOffice is a complex piece of software with lots of features. Knowing what parts of it actually get used and by how much can reveal a lot about the users' priorities and its effectiveness in satisfying them.

Questions

The following table is a list of questions to be asked on the survey. So that it can be completed in a reasonable amount of time by the respondents, the survey will be limited to 25 questions.

The table is divided into four columns. The Question column lists the text of the question as it will be asked on the survey. The Instructions column contains any additional instructions users will need to answer the question. The Answers column will contain a list of multiple-choice answers to the question or "Open-ended" if people can answer the question in a free text field. The Reasons field contains the reason why this question is being asked and how answers to it will be analyzed. The Reasons field also contains the expected comparison of the answers with other variables and any expected segmentation of the general population.

Demographic

Other potential questions:

• Size of company.

• Gender.

• Household income (compared with reasons for choosing StarOffice).

• Geographic location.

Technological

Other potential questions

• Hard disk size.

• CPU type.

• PDA integration.

Usage

Other questions:

• Reasons for choosing StarOffice.

Fielding

The survey fielding process will consist of several steps:

• The content of the survey will be finalized.

• It will be translated into several languages (as many as we can get translators for). Since the nuances of survey language are very important, this would ideally be done through a double translation procedure where it is first translated, then translated back by a different person (who has not seen either version), then the two English versions are compared and the translation is adjusted.

• Over a period of two to three weeks a group of people will be contacted by email and invited through banner ads to participate in the survey. The total number of people invited through each invitation method will be monitored. The invitation will consist of an explanation of the reasons for the survey, the benefits that participation, and a URL to a Web site that contains the survey (there may be different URLs for the different invitation methods).

• The survey results will be submitted automatically to a tabulation mechanism (possibly the phpESP platform, or a similar OSS tabulation mechanism).

Once tabulated, comparison tables are built and conclusions are drawn.

Analysis

First of all, the summary information provided by the survey (the distribution of operating systems, the distribution of common languages, the distribution of experience, etc.) will prove informative and useful by itself, since it will provide the development team with an idea of where to focus development.

However, it's possible to extract other information from the results of the survey. A number of tables will be created comparing various aspects of the survey in order to understand how one facet of the respondents' answers affects others. Although various comparison tables (crosstabs in survey speak) can be constructed, the survey will focus on several specific ones to answer specific questions.

Age vs. Education

As a face validity check on the data collection process, comparing age to education can produce an idea of how representative the collected information is relative to the population at large. It's not particularly useful for the continued research endeavor, but it can point out whether the collected data is wildly different than what's expected.

Industry vs. Features and Software

People in different industries may have different software needs and may concentrate on different aspects of the office application universe. If a relationship between certain industries and certain pieces of software or certain features can be established, then that can serve as the basis for further research as to whether those industries have specific needs which can be addressed by the software or whether they can serve as testbeds for the introduction of features before those features are introduced to the general population.

Industry vs. Frequently Used Features

Industry vs. Software Used

Industry vs. Experience

People in different industries depend with varying degrees on computers. This should create the situation where people in different industries have varying degrees of computer and StarOffice experience. In order to be able to tune features to the abilities and needs of certain people, it's useful to know whether such differences exist and whether they are significant (and, of course, if they exist, where they exist).

Industry vs. Computer Experience

Industry vs. StarOffice Experience

Experience vs. Operating System

In the interest of understanding the context that people approach a piece of software, it's useful to know what kind of operating system they primarily use. Knowing the relationship between what OS people use and the amount of experience they have further refines the understanding.

Reasons for using StarOffice vs. StarOffice features used

This is, of course, a preliminary list based on my gut-level understanding of what would be useful to the Open Office development community. There are many ways to analyze the data to focus on answering specific questions (provided that there are enough responses to be able to draw conclusions).

Copyright

Copyright © 2000 Mike Kuniavsky. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is available at http://www.fsf.org/copyleft/fdl.html