a. General questions

1. What are the advantages of WAMMI over a questionnaire I could write myself?

It is quite possible to develop a questionnaire whose reliability and validity are unknown, and which has no reference database behind it, in just a matter of days. There are many examples of such questionnaires.

The effort put into WAMMI should not be underestimated. And that's not all. The standardisation base behind WAMMI means you find out exactly how good or bad the results for your site are in a way that is simply not possible otherwise. And you find out how you stand compared to other websites.

Up

2. How exactly do I put a WAMMI questionnaire on my site?

Three things to do:

• Please contact us about setting up WAMMI.

• Together we will configure your WAMMI evaluation based on your specification. It will be placed on our server. We will provide you with a URL address to the questionnaire.

• When you want to start collecting data, you place a button or link on your website's home page. The visitor clicks on the link and the questionnaire loads up in the whole browser window. You should provide some text next to the button or link to encourage site visitors to answer the questionnaire.

After filling in the questionnaire users are thanked and presented with a link back to your home page.

Up

3. You don't seem to have WAMMI in my language, when will it be available?

WAMMI has an exacting system to ensure high quality translations. The languages in which WAMMI currently exists and those we plan to provide in the near future are listed on this website. Languages

Due to the nature of the translation process, we normally require at least six weeks, and may need your help in locating suitable people to help us in the translation process.

Up

b. Changes to questionnaire wording & design

1. Some of the statements sound strange for my site. May I change them?

The WAMMI statements have been developed using a very precise methodology known as psychometrics. And the databases have been created on the basis the statements provided.

Very often in psychometrics, apparently small changes in wording have very large effects on the way people respond to the statements. Some of the statements may sound strange to you, but they have been carefully constructed to address issues which have been found to be important to website users.

Furthermore, they have been tested with a wide variety of websites and users, so they are most probably perfectly intelligible to your ordinary end users. It is not advisable to change any item wordings in a psychometrically prepared questionnaire.

Alternatively, there is always the possibility of adding extra questions around the questionnaire.

Up

2. Some of the statements are not relevant for some of our users. Is it possible to have an "n/a" (not applicable) option or remove the question?

The statements in WAMMI have been carefully selected and edited so that they do in fact apply to a surprisingly large variety of sites and users.

Users must answer each of the statements and are asked to select the middle column if they feel the statement is not possible to answer. If an additional "n/a" option was used it would provide users with an easy "get out" option, making it tempting for them to use this response instead of fully thinking through their response.

Allowing an "n/a" option or removing a statement would therefore result in lowering the reliability of the questionnaire.

Up

3. May I add my own questions?

Yes, you may add your own questions, although these will of course not be analysed using our standardisation base. This will usually be considered as a customisation and may incur extra costs.

Up

4. May I change some of the options for the standard additional questions, as they don't suit me?

Yes, but this will usually be considered as a customisation and may involve extra costs.

Up

5. Can I have all the raw data that is collected in the database?

It is extremely rare that a client has need for raw data that is not provided in the WAMMI report. Normally clients find there is more than enough data to answer any questions they have in the various report pages.

As a result, we do not normally give out raw data files as these can contain data that may enable individual users to be traced. If your company really does require additional data for further analysis, then for a small charge we can provide a spreadsheet file containing all relevant data (anonomised if appropriate). You may also be asked to sign a special non-disclosure and non-tamper agreement with us.

Up

6. Is it possible to have my company's logo and typography on my WAMMI questionnaire?

Yes, but it is not always recommended. There are advantages of conducting an external, independent evaluation, which is why WAMMI comes with its distinctive style sheets. If you require special fonts and use of your company logo or other material in the questionnaire, then your WAMMI falls into the "customised" category and is priced according to your specific requirements.

Up

c. About the database

1. What sites are in the database?

The database contains a wide variety of different types of sites from different business sectors and from a variety of different countries. Aside from being confidential, the details of exactly which sites are used in our standardisation base change regularly as we maintain and update it.

Up

2. How many sites and responses are there in the database?

The WAMMI database is constantly being updated. At any time our standardisation base will reflect data collected from approximately 300 different sites selected to be a representative sample from the hundreds of sites recently analysed. Our entire database contains several hundred thousand sets of user-responses from hundreds of websites worldwide.

Up

3. How do the 20 statements relate to the different usability scales shown in the results?

Different statements relate to different usability scales, where each usability scale is composed of weighted contributions from the ratings received for its statements. The precise weighting values come from the current WAMMI standardisation base. The Global Usability Score (GUS) is an average of the five usability sub-scales.

Up

4. How is the data from my site compared to the data in your database?

First of all, the process of standardisation is a process of comparison. Therefore, the scores of each usability scale from your website are located along the distribution of scores in the database using percentiles. This is what enables us to say "you are better than 70% of the scores in our database on the usability scale X."

Secondly, in each statement we compare the average score from your website to the distribution of scores in the standardisation base. The further away your average score is from the database average, the more the discrepancy value for that statement.

Sometimes the discrepancy goes in a good way for your site. This could be good news as your site gets a more favourable response than the average response. Alternatively, it may mean that your site gets a less favourable response than the average response and its time to rethink this area.

Up

5. Is there a different database for each country?

No, the care we have taken over the translation of the WAMMI scores allows us to pool data from different countries. We do this in proportion to the number of uses we get from each country.

Up

6. How often is the WAMMI database updated?

We have a policy of updating it at least once a year but we are continually monitoring trends against what is in the database and issue an update more frequently if necessary.

Up

7. I do not want the data for my site going into your standardisation database. I assume that is okay?

WAMMI is normally available on the proviso that you give us the right to include your data into the statistical digest of our standardisation database in a way that cannot identify you, your site, your users, or your company.

However, if your company has explicit policies regarding the use anyone makes of your data - even in circumstances such as we have described here - then it will be possible to arrange this. As WAMMI works on the principle that shared data is there for the common good (so long as individual's privacy rights are well respected), such a request may incur costs.

Up

8. How does data collected from one country compare with others, are you sure that users rate the statements in exactly the same way in different languages?

Users respond differently for a number of reasons, and ultimately, it may not be possible to give a definite answer to this question.

The first major problem is words have different fields of meaning and associations in each language. We try to minimise this by the use of a strict back-translation protocol.

There is also the problem that users in different countries may have different beliefs and expectations about the web, technology, and self-disclosure. There is not much we can do about this. The Internet is a global community and we already know that such differences between countries are already sometimes as great as such differences within countries. As the Internet matures, such differences will fade.

Up

9. How can you have information and transactional sites in the same database? They are totally different!

WAMMI evaluates users' satisfaction with the sites they are visiting. Their satisfaction with a site is affected by a large number of factors. An important issue is users' expectations, they may well have different expectations for transactional as opposed to information sites.

Another extremely important factor is how successful they are. Even a basic information site may have a problem that means they can't find the information they want. Remember, there can be extremely satisfying transactional sites as well as extremely unsatisfying information sites.

However, we have developed specific databases for B2B sites and transactional sites. Although there are differences between these and the main overall database, the differences are on the whole very small.

Up

10. Can you compare my site with other sites of a similar type (e.g. competitors) in your database?

The privacy and confidentiality of all our clients is an important priority for us. We do not provide direct comparisons with other specific sites in our database. If you are interested in a comparison of your site with sites of a particular type, then we may be able to offer a tailored solution for you, but only if sufficient data is available from a number of sites of this type in order for the comparison to be completely anonymous.

Up

11. Can my organisation have its own database for its different websites?

Yes, we can offer a service whereby data is compared both with our overall database and your organisation-specific database. We would then maintain this database in the same way we maintain our overall database.

However, a substantial amount of data from different websites in your organisation is required before such a database can be created. There are of course additional costs involved in this service.

Up

12. Is there a "top 10" WAMMI site list (i.e. those with the best results)?

No, but we're working on the idea!

Up

13. Can I publish data from my WAMMI report?

Yes, of course! If you choose to do so, please let us know as we are always looking to add new publications on our References page.

Up

d. Using WAMMI & getting responses

1. How do you suggest presenting WAMMI on my website?

Perhaps the simplest and most common way is to place a clearly marked link or button on you home page. Next to it should be some text encouraging users to fill in the questionnaire. There are alternative methods, such as "popping up" a new window to ask the user. However, great care must be taken when doing this, and a user should never be presented with such pop-up windows more than once.

We always advise linking to WAMMI in the main browser window. The questionnaire should not be loaded with a left frame present - a thin top frame containing logo graphics, etc, may be used if necessary.

Remember that the setup and design of additional elements outside the WAMMI questionnaire is the website owner's responsibility, and care must be taken to make sure that these do not interfere with the questionnaire itself.

Up

2. Can I use WAMMI on an intranet?

WAMMI can be used on an Intranet website in exactly the same way as an Internet site, provided that users of the Intranet site have access to the Internet (as the WAMMI questionnaire itself will need to be accessed from our server). If this is not possible then we can support solutions whereby data is collected internally and then sent to us for analysis.

We have developed a specific reference database for Intranets. Although there are differences between this and the main international database, the differences are on the whole very small.

Up

3. Can I change my site during a WAMMI evaluation?

No, we recommend that you don't change the design until the evaluation is completed. Otherwise, you would not know which of the user-responses were to the earlier version of your site and which were to the later version.

Naturally, any content that is normally updated on a regular basis (e.g. news articles and other direct "feeds") should continue to be updated as normal.

Up

4. How long does WAMMI need to be on the site?

Unfortunately, the answer here is "it depends". Long enough to get a representative sample of user-responses. This depends on the number of people using your site and the frequency of visits to your site.

For large popular websites we recommend a minimum period of 2 weeks, in order to get a reasonable sample of results from both frequent and less frequent users. Smaller sites often require more time to get sufficient responses.

Up

5. How many responses do I need?

The key issue is not the number of responses, but that they be a representative sample of from your website users. A reasonable WAMMI profile can be obtained with only 30 sets of responses. More is of course desirable, however.

If you are using additional "fixed answer" questions with WAMMI, then the answers to these are cross-tabulated with the WAMMI profiles. More data will be required in order to make much sense of these cross-tabulations.

For example, if you choose a question that has 8 response options, you want to be sure to get a big enough sample so that at least some of the less usual response options are selected (to some extent this depends on the unusualness of the options).

Roughly estimated, we would recommend at least 20 users per response option (i.e.160 responses in this case). There's no statistical theory behind this, however, it's just another guesstimate based on our considerable past experience.

Up

6. How do you get users to answer the questionnaire?

So far, we have found with most sites that users are prepared to fill in the WAMMI questionnaire because it gives them an opportunity to express their feelings about the site.

Some site owners provide an additional small incentive by offering to donate a small amount of money to charity for each response they receive. However, we strongly recommend not providing users with incentives that would influence their answers to the questionnaire (or indeed encourage them to answer it several times).

Up

7 What happens if only a small percentage of users answer the questionnaire?

This depends on what you mean by "small". In a postal survey, for instance, 10% response rate is considered normal. If you are basing your statistics on server usage statistics then be sure you know what these statistics are showing and how they are calculated.

If you do find a very small WAMMI response rate as compared to your site usage, we would recommend that you set up a "user panel" consisting of 20 to 40 regular, committed users, and get an undertaking from them to visit your site in a certain period and fill out the WAMMI questionnaire for you.

You may also find this user panel useful for other activities when you are improving your site. The user panel may be small compared to your estimated hit rate, but you will have the satisfaction of knowing that this is quality data.

Up

8. How do we know that the "right" users are answering the questionnaire? What happens if the users answering the questionnaire are not entirely typical?

You never know exactly who is answering. It may be possible that the only people who respond are those who enjoy filling out questionnaires... Or those who love the site... Or those who hate it.

The presumption is that with a large sample, the various possibilities mentioned above (and others) will cancel each other out. In practice, we find that this is not a problem. In tests we have found that the results obtained from representative users who have answered WAMMI in controlled conditions compare very well with those received from visitors to the site who fill in wammi.

Should you site be subject to an attack then various techniques in the data collection and analysis process ensure that if people fill in the questionnaire many times (e.g. to generate a positive or negative result) then such data is filtered out. Naturally we cannot give 100 percent guarantees, but we are confident that the various measures we can take result in good quality data.

Up

9. Do you often find it is only the users that have something to say who actually answer the questionnaire? Or is it mainly those that feel motivated to answer questionnaires?

We've partly covered this in the previous answer, but this question comes at the problem from another angle. In one sense, this is an endemic problem in any research involving people. If you recruit people to help you, then you get only those who like to volunteer. If users are forced to participate (as sometimes happens in airport surveys), then some users will care more to give truthful answers than others.

In actual fact, even if the people who answer the questionnaire are those that like answering questionnaires, then this isn't necessarily a problem. The fact that they like answering questionnaires is unlikely to make them different from those people who hate filling out questionnaires in terms of their underlying attitudes to your site.

Up

e. Discounts & costs in special situations

1. Is there an educational or charity discount available for WAMMI?

Please contact Jurek Kirakowski if you are a bona fide educational user (i.e. student or teacher).

Up

2. I'm teaching an HCI course, can I use WAMMI for free as part of the course?

Contact Jurek Kirakowski. You can begin by introducing your students to WAMMI by giving them the URL to this site.

Up

3. Can my organisation license WAMMI so that we can run evaluations and analyse results ourselves?

Due to the complex nature of data collection and analysis using our reference databases, we do not offer a WAMMI license at this time. However, we can offer special arrangements for any client who wishes to use WAMMI regularly and often.

Please contact Nigel Claridge for more information about prices and discounts available.

Up

4. If we use WAMMI several times, is there a discount?

Yes, we offer discounts for multiple WAMMI evaluations of the same site, or for the same WAMMI evaluation across a number of sites.

Please contact Nigel Claridge for more information about prices and discounts available.

Up

5. What happens if my site has multiple languages and I want WAMMI to cover every one?

If you wish to use WAMMI in multiple languages there are two options:

1. Multiple copies of the questionnaire in different languages contributing to the same analysis. The content/design of the site must be the same in all languages. You receive one analysis report and pay an additional 20% for each additional language.

2. Carry out separate WAMMI evaluations for each language. This gives you the benefit of a detailed analysis for each language and means you can compare between the different language sites. This is especially interesting if the design varies slightly between languages or if you wish to assess national variations.

In this case you must pay for multiple evaluations, but a discount will apply depending on the number of languages you require.

Up