Abstract – The aim of this paper is to make a step towards usability evaluation of adaptive systems. Hence, there is no universal method for evaluating usability of a system, we used evaluation methods divided into three groups according to their mainly activities: inspection, inquiry and testing. In order to provide usability engineering approach for evaluation of adaptive systems we pointed out characteristics of usability evaluation methods regarding adaptivity according to the following criteria: stages, impact, intrusiveness, time, layers, rules and type of data. We gave an overview of usability evaluation methods according to above criteria so that further work on discovering proper approach towards usability evaluation can be performed.
INTRODUCTION
In various application domains, user-adaptive software systems have already proved to be more effective and usable than non-adaptive systems. As pointed out in [9], adaptive systems with clear user benefits are user-adaptive (personalized) tutoring systems which significantly improve the overall learning progress.
Adaptivity of Web-based educational systems assumes collecting information about the student working with the system and creating the appropriate student model, which can then be further used to adapt the presentation of the learning material, navigation through it, and sequencing and annotation to the student. From the learners’ perspective it supports pro-active learning that at the same time adapts itself to the potentials and needs of the individual learner.
Recently, a lot of work has been done under the theme of adaptivity of web-based educational systems to psychological learner’s characteristics and is still undergoing. It is a joint effort of educational psychologist, knowledge engineers, educational system developers and developers of educational system authoring tools. As pointed out in [10] the main problems in a Web-based learning environment is to determine which attributes should be used (are worth modeling) and how (what can be done differently for students with different styles).
For assuring usability of a system it is necessary to identify proper requirements and goals which cover the critical aspects of usability. An elementary need for a systematical
approach, involving thinking, modeling and analyzing, exist from the beginning of system’s life-cycle. There is also a need to develop and take advantage of the usability principles or classifications of usability aspects. These principles can be assured with the evaluation methodologies which enables us a methodological accessibility for the evaluation. However, evaluation can be difficult to perform, time consuming and also expensive in case we do not take the right approach.
USABILITY EVALUATION METHODOLOGIES
Because a universal method for usability evaluation of systems does not exist, it is reasonable to use evaluation methods together. Table 1 enlists some of the evaluation methods for non-adaptive systems [1] [2]. We divided them into three groups according to their mainly activities: inspection, inquiry and testing.
Inspection methods do not contain any users (except of the pluralistic walkthrough where a user is a member of a group). Evaluators need medium to high expertise. The most popular inspection method is the heuristic evaluation, which is very quickly to conduct and not much equipment is needed. The other three inspection methods (cognitive, pluralistic walkthrough and action analysis) require more time and higher expertise in comparison to heuristic evaluation. The difference between cognitive and pluralistic walkthrough make the users. Action analysis divides tasks into individual actions, which evaluator performs and measures time needed for an action.
Inquiry methods involve focus groups, questionnaires, interviews and field studies. They differ in a manner of interaction. Interviews possess an individual inquiry, while in focus groups more users discuss about topics. Focus groups have also a moderator, who is directing the conversation flow based on questions. The main purpose of questionnaires lies in statistical results of a broad demographic population. Because there is no direct interaction with participants, the realization time of questionnaires is low. The simplest inquiry method is observing users in the field (for example, a working place) where we note down information we see.
User testing is the last group of methods and is a fundamental activity by evaluating usability. It enables a direct access to information about system’s usability and problems that users run into during the testing. A frequently used method is the thinking aloud method, where a user verbally express his thoughts and behavior. It is a good approximation of system in practice, because we see the cause why a user has done his move. Since the thinking aloud method is not a natural way of user testing, while actions are described in loud, a better choice in this aspect is the constructive interaction, where two users test the system simultaneously. Consequently, we are dealing with higher costs.
Table 1. Comparison of usability evaluation methods
ADAPTIVITY ASPECTS
There is no doubt that the evaluation of a system should be taken into account during the entire development life cycle of applications. However, can we use abovementioned methods for evaluating adaptive systems under a usability engineering approach?
Because of the various adaptivity factors the evaluation of adaptive systems is difficult to perform. Therefore, a layered approach has been introduced. There are defined two layers at least: content and interface [6]. Totterdell and Boyle define the layered approach as: “Two types of assessment were made of the user model: an assessment of the accuracy of the model’s inferences about user
difficulties; and an assessment of the effectiveness of the changes made at the interface” [5]. In other words, evaluation of adaptive systems should not be evaluated as whole but as components – layers.
According to [7], the evaluation methodologies can be seen
as generative models, since they contribute to the improvement of adaptable systems during the evaluation phases by combining requirement specification and evaluation.
An important aspect of human-computer interaction plays the participants. They seek different kind of information. Likewise, they are the target population of the applications and represent the main source of information. Because of that their role is significant. A user-centered approach for evaluating adaptive systems has been proposed by [4].
Adaptive systems differ regarding the activities for achieving changes: initiative, proposal, decision and execution [3]. Therefore, a group of these stages have been considered in the first criteria. Full adaptivity occurs if all adaptation stages are feasible by the system. Another mode of adaptation is a user-controlled self-adaptation, where the user decides upon action to be taken. By the term system we do not refer only to the system itself, but also to the system
The intrusiveness of adaptivity is the third criterion. By intrusive we refer to the frequently given suggestions of a system. Since there is no direct user interaction with the system, all inquiry methods are not intrusive (with the exception of field studies).
The time aspect of adaptation is divided into three phases: before the first use, during the system usage and between two accesses into the system. It shows the adaptivity appearance.
Presentation, navigation and content are the main adaptivity criteria. In general, adaptive presentation the adaptable aspect of the content difficulty (a simpler content for a less advanced user and additional, deeper information for more advanced user). The goal of adaptive navigation is to assist student by orientating and navigating him, to easier achieve goals [10]. The content shows the information which is relevant for an individual user.
With the use of reusability adaptation rules, creation of new adaptation rules from scratch is being avoided. Our criterion shows simple or complex reusability adaptation rules.
The last criterion represents qualitative and quantitative data which can be gathered during the evaluation of a system. It can be helpful to decide which method will produce necessary data. Qualitative results can be gathered from an
interpretative evaluation [8], focusing on individual users which give a rich set of information and a deep knowledge. Qualitative approach comes into account by measuring processes and calculating dependence of variables for making generalizations on the obtained results.
CONCLUSION
We presented an approach towards usability evaluation of adaptive systems. Since, there is a variety of adaptivity factors; the evaluation of adaptive systems is difficult to perform. Therefore, a layered approach has been introduced. We have considered the evaluation during the entire development life-cycle and we suggest that evaluation takes place before the implementation started. Discussion has been performed about characteristics of usability evaluation methods regarding adaptivity according to the following criteria: stages, impact, intrusiveness, time, layers, reusability rules and types of data. The next step would be further development of evaluation methods concerning specific adaptivity criteria.
REFERENCES
[1] Holzinger, A. Usability engineering methods for software developers, Communications of the ACM, januar 2005, pages 71-74 [2] Nielsen J. Usability Engineering, Morgan Kaufmann,September, 1994
[3] M. Schneider-Hufschmidt, T. Kühme, and U. Malinowski, eds. Adaptive User Interfaces: Principles and Practise. Amsterdam: North Holland Elsevier, 1993 [4] Gena C. »User-Centered Approach for Adaptive Systems« [5] Totterdell, P and Boyle, E, 1990, The evaluation of adaptive systems, Adaptive User Interfaces. London: Academic Press, pp. 161–194. [6] Brusilovsky P. The Benefits of Layered Evaluation of Adaptive Applications and Services [7] Dix, A, Finlay, J, Abowd, G and Beale, R, 1998, Human Computer Interaction, 2nd edn. Englewood Cliffs, NJ: Prentice-Hall. [8] J. Preece, Y. Rogers, and H. Sharp, Interaction Design: Beyond Human-Computer Interaction, Wiley & Sons, 2002 [9] Fink, J., Kobsa, A. A Review and Analysis of Commercial User Modeling Servers for Personalization on the World Wide Web. User Modeling and User-Adapted Interaction, UMUAI, Kluwer Academic Publishers, Vol. 10, No. 2-3, 209-- 249. (2000)