What does the number of criteria depend on? If you are operating on a tight budget, little time to make the selection or not enough people, it is not possible to analyse a large number of criteria. In such cases it is best to choose a small number of criteria. If your resources are greater and need a detailed analysis of a certain program, choose more criteria.

Small number of criteria

When the resources are insufficient, when an institution operates on a tight budget or it is limited by time or the number of available people, it is not possible to conduct an analysis based on a great number of criteria. That is why it is extremely important to choose exactly the criteria which you want to analyse. They should not be too general, since in that case they will not provide any more information than what you can find in promotional materials of the software manufacturer.

In order for the software that will be chosen to best suit our needs, it is necessary to plan the questions carefully. Bad choice of criteria could result in the selection of software which will satisfy criteria which are not so important, but none of the reasons why we are buying it.

Medium number of criteria

This is the most common evaluation type. It requires resources and it can still rely on experts. It is not too different from evaluation with a small number of criteria, but there are some changes. With good selection of criteria it is possible to make a very good specialized analysis of software. Such an analysis should satisfy most users for who it was made. Because of their small number it is not easy to define a sufficient number of criteria which would offer a review of software that would satisfy a large number of the analysis' users.

Even if some of the selected criteria are unnecessary, the analysis should still be of the same quality in regard to the features that are of real interest, although some of the answers we will get are not necessary.

Large number of criteria

This type of analysis is conducted when there are enough resources and a need for detailed analysis of a specific program. This analysis should result in a very objective image of the program, since even if some of the criteria are not important or do not answer a particular question truly, some other criterion will. The likelihood that an important detail would be missed is far lower, but even in this case good preparation remains very important.

With large number of criteria the biggest problem is how to show the results. Several A4 sheets filled with zeros and ones would more likely remind of "Matrix" than of the results of a serious research. This is one of the bigger problems related to a large number of criteria. The criteria should certainly be well grouped. In most cases it is not even enough to just group them. These groups will have subgroups, which will have subgroups of their own.

Another big problem is defining a larger number of criteria. Although it seems very simple to think of many questions, the problem appears when they have to be uploaded into a computer. After just a few questions, you run out of ideas. In order to prevent that from happening it is best to first define the basic criteria groups that you are most interested in. After that subgroups are defined, and so on.

One example for grouping a large number of criteria intended for our market can be found in Appendix. It omits the criteria themselves, but gives main groups of criteria and their subgroups. The criteria which are being analyzed are in the subcriteria group.