Murray W. Goldberg, "CALOS: First Results From an
Experiment in Computer-Aided Learning". Accepted for publication in the
Proceedings of the ACM's 28th SIGCSE Technical Symposium on Computer Science
CALOS: First Results From an Experiment in Computer-Aided Learning
Murray W. Goldberg
Instructor, Senior Faculty Advisor
Department of Computer Science, University of British Columbia,
This paper describes the outcome of an experiment in Computer-Aided
Learning (CAL) using the World-Wide Web (WWW). The purpose of the
experiment is to evaluate WWW-based course delivery both in terms of
academic performance and in terms of student acceptance. A complete
WWW-based offering of an existing third-year university course in
Computer Science was produced. Students enrolled in the course were
divided into three groups:
Academic results were evaluated and qualitative student impressions were
solicited. In general the WWW-only method of course delivery was an
equal success compared to the lecture-only delivery, both in terms of
academic results and in terms of student acceptance. On average,
students in the combined lecture and WWW-based delivery performed better
academically than students in the other groups.
- those taking the course using only the WWW-based resource (no lectures),
- those taking the course by attending lectures (no access to the
WWW-based resource), and
- those with access to both lectures and the WWW-based resource.
- WWW Education Computer-Aided Learning
This paper describes the outcome of an experiment in Computer-Aided
Learning (CAL) using the World-Wide Web (WWW). This paper will be of
interest to educators for its evaluation of the educational
effectiveness and student acceptance of the WWW as a teaching resource.
More generally it will be of interest to anyone designing WWW-based
courses or training material.
The WWW provides educators with an outstanding opportunity to deliver
courses, or to supplement existing courses, using a networked computer.
There are clear advantages to the use of the WWW for CAL:
These opportunities for educators have lead to the development of a
large number of WWW-based courses and other training resources , ,
, , , , .
In the department of Computer Science at the University of British
Columbia we, too, have taken advantage of the educational opportunities
presented by the WWW by creating a WWW-based version of a third-year course
dealing with the fundamentals of Operating System design (CALOS: Computer-Aided
Learning for Operating Systems) . This
WWW-based course is a rich resource including: interactive exercises,
interactive simulations and demonstrations, on-line notes, student -
instructor communication mechanisms, student - student communication
mechanisms, progress tracking, navigational aids, student evaluations,
student annotations, a glossary and a bibliography. We make use of text,
images, animations and voice. Implementation makes heavy use of CGI
programs and dynamic GIF and HTML creation. Click here
for an image of the startup page for the course.
- courses can be developed once, by experts, and delivered at any
scale. This creates the opportunity for improved course quality.
It is also possible that a greater number of students can be served at
a reduced cost. Finally, this ensures uniform delivery of courses;
the quality will not vary with that of the instructor delivering
- the pace, path and depth of the learning can be adapted by the
learner to suit his or her background, level of aptitude and
- interactive Web-based exercises offer more opportunity for repeated
experimentation and practice than is available with a chalk-board or
pen and paper.
- the networked nature of the Web allows traditionally isolated
distance-education students to be brought into a community of
communicating peers through the use of bulletin-boards, chat areas and
- courses can be delivered at any location. All that is required is access
to a computer with a network connection or modem. This is an advantage
for all students but is a special advantage for physically
challenged students or those students for whom travel to a common
campus is otherwise difficult.
- instant student self-evaluation is available through the use of
multiple-choice questions or intelligent exercise evaluators.
- the Web is a familiar resource and Web browsers are
quickly mastered even by students new to computers.
It was our intention from the beginning not to simply create the course
and use it, but also to measure the effectiveness of the WWW and its level of
student acceptance as a teaching resource. It was also our intention to
experiment with the use of the WWW resource in varying contexts both in
combination with lectures and as a stand-alone course. We have done our
first round of experimentation in this respect and this paper presents
our first results.
The remainder of this paper is organized as follows: section 2 presents
an overview of the composition and implementation of our course. Section 3
describes the structure of the experiment in course delivery.
Section 4 presents both the quantitative and qualitative
results of our experiment. Finally, sections 5 and 6 present
conclusions and future directions.
2. COURSE OVERVIEW
This section gives an overview of the WWW-based course implementation and contents.
A more detailed description of the course can be found in .
2.1 Implementation Summary
The WWW version of CPSC 315 is highly interactive. In order to achieve
its interactive nature, HTML and images for the pages are generated
dynamically. The pages are composed in response to the course notes,
student information, system navigation information and student input to
exercises. The course contents are written in
an augmented version of HTML which is passed through PERL scripts. These
scripts generate HTML and GIF images for display. Operating system
simulations and interactive exercises are written in C and PERL. These
generate HTML and GIF images dynamically.
This mechanism allows for significant flexibility and interactivity, yet
requires no special software or tools on the client (browser) side of
the connection. The advantage is that our tools have to run only on our
server; no complex porting or installation of software to client
machines is necessary.
2.2 Course Contents
2.2.1 Page Layout
Every course page has a button-bar at the top and bottom allowing students
access to course features (navigation, glossary, exercises,
bibliography, chat, bulletin-board, etc.). These button bars are
generated dynamically according to
For example, if the student's preferences file indicates the use of a
black-and-white display, then the button bars are generated from
black-and-white buttons (color buttons do not show up well on black and
white displays). Likewise, if there is a set of multiple-choice
questions embedded in the source file, these questions are stripped
before being sent to the browser, and a question icon is generated for
the button bar on that page. Clicking the question icon produces a form
displaying the multiple-choice questions and prompting for their answers.
Click here for a sample page of notes.
- the information in the external navigation file (discussed below),
- the student's log and preferences files, and
- the HTML and supplementary (non-HTML) information contained in the
Our navigation goals are four-fold:
In order to accomplish the first two goals, we have provided both a
hierarchical view of the course, and a linear view. Using the
hierarchical view a student can view the course at any level of
granularity and skip directly to sections of interest. Using the linear
view a student can proceed down the usual path of course material, from
one page to the next. No navigation information is recorded in the
course content pages. Instead, there is a single file containing the
sequence of pages on the usual path. This file is referenced by the PERL
scripts in order to generate forward and backward arrows on the course
pages. This makes reordering and insertion or deletion of pages trivial.
- make the usual path through the course material obvious and direct
- facilitate unusual paths through the course material to accommodate students
already familiar with certain aspects of the course material
- encourage excursions off the usual course path by allowing the student
to instantly return to the point at which the path was left. A student
should be able to do this using a single mouse click no matter how far
off the main path they veer.
- when starting a new session a student should be placed in the same
navigation context that they were in when they terminated their previous
visit to the course.
In order to accomplish the latter two goals, we need to know the
identity of the student making each request. Access controls, which are
used to restrict course access, allow us to know the student's identity.
Once the identity of the student is known, it is a simple matter to log
that student's accesses to the course pages. Having a recorded history of
each student's references allows us to place that student either back on the
usual path (after an excursion off that path) or back at the same
location from which they ended their previous session. Such a log
also allows us to make inferences about the student and various course
sections based on the number of accesses.
Communication among students, and among students and the instructor is
accomplished using a chat service and bulletin-board.
The chat service allows real-time communication among course participants.
It presents a virtual set of rooms that can
be entered. The names of the occupants of each room are displayed for all to see. The door
to each room can be locked to facilitate private conversations. Each room
can be labeled by its occupants in order to advertise the nature of the
conversation taking place.
The bulletin board allows the posting of messages in a public forum.
It can be accessed in either of two ways: either from the central
bulletin-board page, or from any of the course-notes pages. When accessed
from the central bulletin-board page, messages are displayed for the
student to peruse. Knowing the identity of the reader allows us to refrain from
presenting messages that have already been read by that student (unless he or
she so desires). The usual range of bulletin-board features exist including
threading and responding to messages with an embedded quote of a previous
When the bulletin-board is accessed from a page of course-notes, only
messages pertaining to that page are presented. A message pertains to a
page if it was posted from that page, or if it is a response to a message
originally posted from that page. When viewed from a course-notes
page, all messages, including ones previously read by the student, are
presented. This essentially creates a global annotation mechanism where
discussions about that page are recorded for others to see. The expectation is
that questions, answers and other discussions about each page become part of
the permanent record for that page. They supplement the learning experience,
actually forming part of the content of the course. Shortcomings in the
course-pages will usually be addressed in the related bulletin-board
articles. This adds a dynamic, self-correcting nature to the course notes.
2.2.4 Interactive Exercises and Simulations
Interactive, graphical computer simulations allow students to explore
operating system concepts. They are both student-driven and computer
driven. Student-driven interactive examples simulate operating system
responses to student-generated input conditions. Computer-driven
simulations exercise the student's knowledge by presenting a set of
conditions and asking the student to indicate outcomes. The student's
response is checked for correctness by the simulation. The computer can
generate an almost infinite set of starting conditions; no two exercises
are the same.
The simulations include six disk scheduling algorithms, three memory
allocation algorithms, five page replacement algorithms, four process
scheduling algorithms (including exponential averaging), process
creation constructs, and others. Click here
for an image of a disk-scheduling simulation.
These interactive exercises and simulations form an integral part of the
course, adding an experience that cannot be matched in the classroom. In
a typical class only one or two such examples can be presented due to
the time required to set up and evaluate the problem. The WWW-based
interactive examples relieve the student of having to set up the problem.
As a result it is often the case that five or
ten such exercises can be performed in the time required to do one
corresponding pen-and-paper exercise.
Click here for an image of an interactive
2.2.5 Student Evaluation
Multiple-choice questions are provided that enable the student to
perform self-evaluation at any time. Questions related to the current
pages are available by clicking an icon on the button bar. For each
question answered, a result is displayed (correct or incorrect) and a
textual or graphic explanation of why the answer was correct or
incorrect is given. Click here for an image
of a multiple-choice question page.
There is also a mechanism through which timed quizzes can be administered.
A form with questions and text-entry boxes for the student's responses are
presented. A timer is used to record the interval between when the student
is presented with the quiz and when the response is given.
2.2.6 Other Course Features
Other features include the following:
- a complete, search-able course glossary containing text, images, and
references to course notes
- each notes-page has an icon which, when clicked, presents the list of page
and chapter references to the course texts for that page of notes
- an annotation mechanism that allows students to make private annotations of
- each page of notes includes an icon which, when clicked, presents a page
containing learning goals for that page. This provides the students with
topic emphasis that would otherwise be provided in lectures.
3. THE EXPERIMENT
This section presents an overview of the structure of the experiment
that evaluates the use of the WWW-based course material both alone and
in combination with lectures.
3.1 Course Background
CPSC 315 is a popular third-year Computer Science course dealing with
the fundamentals of Operating System design. Traditionally it has been
offered in a regular lecture format (three hours per week) two to three
times per year to roughly 80 students per offering. Student evaluation consists
of four programming assignments, one midterm exam, and one final exam.
The author has been
teaching this course for the last five years, and has consistently
enjoyed top or near-top student ratings. The popularity of the course
makes it particularly suitable for a comparison with a WWW-based
offering. Here we can be sure that it is not the case that students
over-rate the WWW-based offering because they consider anything better
than the current, lecture-based, offering.
3.2 Experiment Setup
In the fall of 1995 students of CPSC 315 were asked to volunteer to be part of an
experiment comparing various offerings of the course. The understanding
was that a volunteer could opt-out of the experiment at any time and
return to the traditional lecture-only course format. Sixty-four of the
sixty-eight students agreed to volunteer.
Those that volunteered were split into three groups. The placement
was random except that an attempt was made to balance the groups so that
all levels of aptitude were represented in each group. Aptitude level was
measured from the student's performance in the main prerequisite course: CPSC
216; Data Structures.
The three groups were defined as follows:
All three groups of students wrote the same final exam and did the same
assignments. Scheduling problems prevented the writing of the same
midterm; the WWW-only group wrote a different midterm than the remainder
of the students. The two midterms, however, tested exactly the same
material, and an attempt was made to try and present questions of equal
difficulty on both exams.
- Group 1 was the WWW-only group. This group of students was not allowed
to attend lectures. They took the course entirely on-line. These
students were expected to keep pace with the regular offering, though
initially no attempt was made to be sure they did so. This group met
with the instructor once per week. This meeting was not a lecture,
but was used as a time to discuss the student's perceptions and
problems with the course.
- Group 2 attended lectures and had full access to the WWW-based material.
- Group 3 did not have access to the WWW-based material but received the
course in the traditional format: lectures only. This group was the control
Two forms of results were expected. First, quantitative results could be
measured by comparing the academic performance of each group. Second,
qualitative results were sought through the administration of a
questionnaire. The questionnaire was created by the UBC Center for
Applied Studies in Evaluation. To ensure honest answers, students were
assured that the questionnaire would be treated as anonymous, and that
it would not be viewed until after course grades were submitted. The
questionnaire was given out during the final exam and returned over the
following week. Both sets of results are presented in the section that
4. EXPERIMENT RESULTS
This section deals with the results and experience gained from conducting
the experiment as outlined in the previous section. The reader should be
cautioned that given the small class size (under seventy students) no
statistically significant outcome can be reported here. Regardless, we
feel that the anecdotal results are still of interest.
4.1 Experience Gained During the Offering of the Course
The WWW-based version of CPSC 315 had never before been used to the
extent it was during the experiment. As such, the students provided a
great deal of valuable insight in terms of changes that would improve
the WWW-based course. For example, the structure of the bulletin-board was
redefined as a result of student input. Also, a goals section was
added to each page of notes in order to indicate the relative importance
of the course topics. This was something that the WWW-only students felt
would be present in a lecture-based course, but was not initially
present in the WWW-based course. As a third example, many students
requested that we make printed copies of the WWW-based notes available
for photocopying (which is easier than each student having to print each
page). These students wanted to be able to read the notes at their leisure
(for example, in bed).
Another interesting result is that the students in the WWW-only group
were divided in terms of their degree of self-motivation. One subset of
WWW-only students greatly appreciated the ability to self-pace this
course in order to accommodate the changing demands of their other
courses. The remainder felt that without external incentive, they were
unable to maintain the motivation required to keep pace with the
lecture-based offering. The problem did not seem to be one of boredom
with the material or the delivery method (most students seemed to enjoy
the novel experience and the ability to work on the course anywhere, and
at anytime). Instead, the problem was that each of these students was
faced with at least four other courses being taken concurrently. Each of
these other courses was lecture-based and was therefore not self-paced.
As a result, when the overall load on the student was high (as it is
much of the term) it was the self-paced course (CPSC 315) that tended to
Competition with other lecture-based courses became the most serious
obstacle to the WWW-only student's acceptance of the course delivery
method. Of the four students that opted out of the WWW-only group (and
thus out of the experiment), each cited this competition with
lecture-based courses as the reason. During our weekly meetings, the
author and the students decided that the way to combat this problem was
to institute bi-weekly quizzes. This would require that the students
keep pace in order to be prepared for the quizzes. Those students that
were highly self-motivated declared themselves and were exempt from the
quizzes. Once the quizzes were instituted, the problem greatly
diminished and no further students opted out of the experiment. We found
it interesting that many students lacked the motivation to keep up on
their own and were happier when the course pace was forced upon them.
This may not be surprising, though, given that the bulk of most students'
academic experience is not self-paced. One might speculate that if the
competition for student's time was from other WWW-based courses (and not
from lecture-based courses) that this would not have been a problem.
4.2 Academic Results
It is not useful to simply compare the average academic performance of
the three groups of students. While an attempt was made to have all
levels of academic aptitude represented in each group, the groups were
small enough that individual differences in background and ability could
significantly skew the results. Also, simple averages do not
differentiate the case where, for example, each member of the group
scores a little better than expected, from the case where a couple
members of the group score much better and the remainder of the group is
Instead we adopted the following approach. We decided to count the number
of students in each group that performed:
In order to determine expected performance, the students' grades in
the main prerequisite course, CPSC 216, were used. This course is usually
a good determiner of the grade in CPSC 315.
- better than expected
- as expected
- worse than expected
In order to determine whether a student performed as expected, we
declared a value (call it MARGIN) to be used as follows. If the grade in
CPSC 315 was within MARGIN (either way) of the grade in CPSC 216, the
student was said to have performed as expected. If the grade was less than
the 216 grade - MARGIN, then the student was said to have performed worse
than expected. Finally, if the grade in 315 was higher than the 216 grade
+ MARGIN, the student was said to have performed better than expected.
Determining the value of MARGIN was somewhat arbitrary. Even so the
choice did have an effect. If we chose the value too small, a small
difference in performance could move the student from better than
expected to worse than expected (or the other way). If we
chose the value too large, then most of the class would appear to have
performed as expected. We experimented with many values for MARGIN. As
expected, changing the value of MARGIN moved students either into or out
of the performed as expected group, but did not move them
between the better than expected and worse than
expected groups. Thus, all values for margin gave the same
qualitative outcome. For the purposes of the following results, a MARGIN
value of 5% was chosen. The results are as follows (each number
represents the percentage of the students in that group that performed
better than, worse than, or as expected):
While these results can only be interpreted as anecdotal, they do correspond
with general intuition. Those students that had access to lectures and the
WWW-based material performed better than either of the other groups. Students
that had access to only the WWW-based material or the lectures performed
roughly the same. It is encouraging that it seems possible for a WWW-based
offering to be as effective as a traditional lecture-based course.
4.3 Student Acceptance
Here we look at results from the questionnaire completed by all students
that participated in the experiment.
What follows is a summary of the responses to many of the questions on the
If you would like to view the raw data from
which this summary is generated, please refer directly to the
question of interest:
- All groups of students were generally satisfied with their course
offering. The WWW & lecture group was the most satisfied, followed by
the WWW-only group and then the lecture-only group.
- All groups felt that their method of course delivery was effective. The
WWW-only group and the WWW & lecture group felt equally strongly that
their course delivery method was effective. The lecture-only group felt
less strongly that their delivery method was effective than did the
other two groups.
- When asked which components of the WWW-based course were most useful, students
overwhelmingly cited the interactive exercises, bulletin-board, goal statements
- In all groups there was a large range in the number of hours students
spent working on the course (including time spent attending lectures).
However, the WWW-only group had the largest proportion of students
that reported spending under five hours per week. Also, as expected, the
WWW-only group spent more time on-line than did the WWW & lecture group.
- WWW-based students guessed that they had learned roughly the same (or perhaps
a bit less) than they would have in the lecture-based course. On average,
they felt that they had to commit a bit less time than they would have to
a lecture-based course. Also, most agreed that taking WWW-based courses requires
more personal responsibility that a lecture-based course.
- WWW-based students felt that access to the instructor was important, and
on average felt that having this access once per week was sufficient.
Preferred methods of meeting with the instructor were either one-on-one
or in a classroom setting with other students. The use of a
bulletin-board was also a reasonably popular choice for instructor
contact. When asked how
often lectures should be held (if at all) in conjunction with a
WWW-based course, all agreed that lectures should indeed be held, and
most said once or twice per week was preferred.
- When asked whether there was anything missing from their education because
they took the course on-line (and could not attend lectures), most
students said no. Those that said yes cited in-class discussions and
- On average, students taking the WWW-based course preferred that mode of
delivery over lecture-based courses. Roughly one third far preferred
WWW-based courses, and another 40% somewhat preferred WWW-based
courses. When asked whether they would take another WWW-based course,
almost 90% said yes.
- Given the choice between a WWW-based course and a traditional
distance-education course (with paper-based notes and telephone contact
with the instructor), almost all of the WWW-based students said they
would prefer the WWW-based course.
- When asked what they liked best about the WWW-based course and taking
WWW-based courses in general, the following points were mentioned (roughly
in order of decreasing frequency):
- The ability to self pace in response to the availability of time
and the difficulty of the topic
- Not having to attend classes
- It was easy to refer back to previous topics
- The interactive exercises and computer simulations
- The whole course is available at once
- There was no arbitrary 1-hour division in learning sessions (some
topics required more time, some less)
- The ability to learn anytime and anywhere
- The novelty of the experience
- Finally, when asked what they liked least about the WWW-based course
and taking WWW-based courses in general, the following points were mentioned
(roughly in order of decreasing frequency):
- The extra motivation and responsibility required
- The inability to get immediate answers to questions
- They didn't meet and interact with other students
- They disliked being in the first offering as they helped debug the
- How satisfied were you with the method by which you took CPSC 315?
- How effective do you feel the method used to deliver the course to you was?
- On average, how much time did you spend working
on the course each week (including lectures)?
- On average, how often did you sign on to the on-line course?
- Compared to a lecture-based course,
I (in the WWW-based course) have learned (more, less):
- Compared to a lecture-based course, I
(in the WWW-based course) have spent (more time, less time):
- Compared to a lecture-based course,
the WWW-based course requires (more or less personal responsibility):
- When taking an on-line course,
how important is it for you to have contact with the instructor?
- When taking an on-line course,
how much access to the instructor do you feel you need?
- How often should lectures be held in conjunction with an on-line course?
- Did you feel there was anything missing from your education
because you did not attend lectures?
- Given the choice between lecture-only courses and WWW-based courses
such as CPSC 315, I would prefer:
- Would you take another course on-line?
- Given the choice between a traditional distance-education
course (paper-based notes, textbook & telephone contact with TA) and WWW-based
courses such as CPSC 315, I would prefer:
In arriving at conclusions, the reader must remember that this was a
limited experiment in terms of both the number of students involved
(under 70) and in terms of the number of WWW-based courses these
students were exposed to (one). It is possible that a larger student
base might produce a different set of results. It is also true that the
effectiveness of WWW-based offerings cannot be compared to lecture-based
offerings until students have more experience with WWW-based offerings.
Even so, the number of students that participated in this experiment,
and the fact that these were reasonably mature university students
(hopefully capable of careful consideration and comparison) make the
results interesting at least in an anecdotal sense. This experience
opens the door just a little farther to an understanding of the
tradeoffs of lecture-based, WWW-based, and combined course offerings.
Some of the most interesting results of this experiment are summarized below:
It is difficult to speculate on whether the same results would have been
observed had the course audience not been Computer Science students. It
could be argued that Computer Science students are predisposed to doing
well in a WWW-based course due to their familiarity with, and interest
in computers. It could equally be argued that students outside Computer
Science would view the WWW-based course as more novel. This might help
with the most significant problem; that of self-motivation. It would be
interesting to study this further.
- The group of students in the WWW-only offering achieved roughly the
same level of academic performance as did the group of students in the
lecture-only offering. As one might expect, students with access to
both sources of information achieved a higher level of performance.
- The level of student acceptance of the WWW-based offering was high. Most students
preferred the WWW-based offering to lecture-based offerings. Almost
90% of the WWW-only students said they would be willing to take
another WWW-based course.
- When taken concurrently with other lecture-based courses, the most
serious impediment to success in a WWW-based offering seems to be self
motivation. Yet other students seemed to thrive on the self-paced nature
of the course.
6. FUTURE WORK
There are several fronts on which we plan to continue this project.
First, now that we have some student opinion as to the best combination
of WWW-access and lectures, we plan to follow this up by experimenting
with a WWW-based course offering supplemented by one or two lectures per
week (as opposed to the traditional three lectures per week). Perhaps
this would provide the missing human contact and motivation, yet still
yield most of the advantages of the WWW-based offering.
Next, WWW-based distance offerings are scheduled for the Spring and
Summer of 1996. It will be interesting to review the opinions of those
students that are accustomed to taking traditional distance-education
courses. It will also be interesting to observe the interaction on the
bulletin-board and chat areas between the local and distance students.
It is our hope that the distance students will experience a far greater
connection with the distributed community of students than is possible in
Finally, new technologies (for example, Java) being incorporated into
popular WWW browsers enable us to create interactive exercises that are
more attractive and responsive than our current cgi-based exercises. We
plan to take advantage of this opportunity.
 E. Bilota, M. Fiorito, D. Iovane and P. Pantano, "An
Educational Environment Using WWW." Computer Networks
and ISDN Systems 27 (1995) 905-909.
 J. Campbell, S. Hurley, S. Jones, N. Stephens, "Constructing
Educational Courseware Using NCSA Mosaic and
the World-Wide Web," Computer Networks and ISDN Systems
27 (1995) 887-896.
 The World Lecture Hall,
 Introduction to Object Oriented Programming Using
 B. Ibrahim, S. Franklin, "Advanced Educational Uses of
the World-Wide Web," Computer Networks and ISDN Systems 27 (1995)
 D. Nicol, C. Smeaton, A. Falconer Slater, "Footsteps:
Trail-blazing the Web," Computer Networks and ISDN Systems 27
 D. Dimitroyannis, "Virtual Classroom: A Case Study,"
 Murray W. Goldberg, "CALOS: An Experiment with Computer-Aided
Learning for Operating Systems," To appear in Proceedings of the 1996 SIGCSE
Technical Symposium, February, 1996.
About The Author
Murray W. Goldberg,
Department of Computer Science, University of British Columbia, Canada.