SCOPE logo

Evaluation Documents
SCOPE home page

SPACER SPACER SPACER

SCOPE Project

Stage 5 User Satisfaction Survey Report
June 1997


1. Introduction

This report summarises the evaluation results from Stage 5.


2. Stage 5 Courses 1996/97

Autumn 1996:

Sociology(a) pack
Sociology(b) pack
Economics pack
Business Applications Software pack

Spring 1997:

Sociology(c) online
Sociology(d) pack
Engineering pack

Two of the above course packs were cancelled. Sociology(c) was intended to be a pilot for online delivery. However, the company producing part of the software experienced delays and the software was not finished in time. The Sociology(d) course pack was cancelled because the original course organiser was seconded to another course and the content of Sociology(d) was subsequently changed to the extent that the previous course pack was not relevant. In addition, lecturers felt that the packs could not affordably contain enough material to avoid spoon-feeding students.
Sociology(a) and (b) were the same course, Sociology(a) was thaught during the day, Sociology(b) was taught as an evening class.


3. Methodology

Evaluation was done by a combination of interviews, focus groups and questionnaires.

Questionnaires were distributed to students, and the results analysed using either PinPoint 3 for Windows, or Excel, depending on which institution was doing the evaluation. The questionnaires were all based on one design, and then modified according to local needs. Most results were quantitative, with a few qualitative questions. The questionnaires were usually distributed by lecturers during classes and handed back at the end.

Focus groups were also used for student evaluation, ideally as a follow up to the questionnaires, to provide more qualitative information on areas which were highlighted by the questionnaires. These were run, for the most part, by an external facilitator which avoided bias and took advantage of their expertise in keeping conversations going and steering the conversation onto the relevant topics. Some groups were self selecting, consisting of students who turned up to an advertised group, and others were requisitioned: the engineering lecturer allowed a focus group to be run during a tutorial period.

Structured interviews were held with the academic staff and booksellers, either face to face or by telephone. In some cases, the interview was recorded, in others, notes were taken during the interview.

Due to the absence of a Liaison Officer during February to April, some evaluation work was not completed. There was no student evaluation of the Business Applictions Software, and student evaluation of the Engineering pack was incomplete - not all students received questionnaires, and due to imminent exams the focus groups had to be run before the questionnaires had been analysed.


4. Overall Results

4.1 Preparation

The amount of work required from lecturers to produce the pack depended on the format of the pack. The Engineering pack consisted of handouts from different lecturers, and the time taken to collect all of these and provide clean copies was described by one as exorbitant, but the same lecturer felt that the amount of time and effort it saved later in the year made the investment worth while. On the other hand, the Business Applications Software pack was derived from previously collected handouts, so required minimal work. The Sociology(a) and Sociology(b) packs involved a lot of work due to the fact that so many items were carefully considered and chosen - nothing similar had been done before.

4.2 Delivery

All packs were delivered on time.

4.3 Sales

Economics pack 68 printed, sold out.
Business Applications Software 450 printed, 423 sold.
Engineering 52 printed, 48 sold.
Sociology(a) and (b) 306 printed, 134 sold

All packs were sold by John Smith and Sons, usually from Campus bookshops. Most of the sales occurred during the first two months of the semester (i.e. September and October), with a small amount of sales later on in the year. The remaindered packs are expected to be sold next year.

4.4 Cost

The costs of most of the packs was felt to be appropriate, with the exception of the Social Sciences pack which were not perceived as being good value for money. Some students objected to paying for materials which they used to get free, as happened in the case of the Engineering packs. Lecturers felt that the packs were competitively priced, particularly the Business Applications Software pack.

4.5 Binding

There were complaints about the use of perfect binding on the Sociology packs - some students reported that they fell apart on heavy usage. Since then packs have been spiral bound, which has been well received by both lecturers and students. Spiral binding allows the packs to lie fully open without placing any stress on the spine, so students find it easier to read and make notes in their packs.

4.6 Composition

There was a broad range of pack contents. The Sociology packs contained 38 published journal articles / book chapters spread over three alternative packs; the Engineering pack consisted of lecturers' material and two chapters of previously published, but out of print material; the Business Applications Software and Economics packs contained only material written by lecturers.

4.7 Relevance

On the whole, students seem to have reacted better to the packs containing lecturer's materials than those which contain only published. This may also be due to the a number of factors.

The material written by lecturers tended to be used and referred to throughout the course, and all the material contained in it was directly relevant to what the students needed to know. They also contained most, or all, of the basic information which students on the courses needed to know. For this reason, students tended to indicate that these packs were good value for money.

The packs containing a selection of previously published materials were intended to be an introduction from which students could get the general ideas and then move on to other sources. The packs were never intended to provide students with all they needed to know, therefore they were seen as being less relevant, and the students were less satisfied with their value for money.

In addition, the ethos behind the Sociology course was that students should do as much reading as possible, the lecturers did not want to tell students what to read, they were asked to form their own opinions and ideas. In contrast, in the Economics and Engineering courses, the lecturers were happy to recommend one book. This is probably to do with the fact that these latter two are far more fact-based.

Where courses are taught in modules and assessed on an ongoing basis students prefer to have packs focusing on specific subject areas, rather than an overview of all areas.

4.8 Use of packs

Use of the pack depended on what it was intended for. The Sociology packs were not read as broadly as the lecturers had hoped - only 18% of respondents to the questionnaire said that they had used more than 50% of the material. There was some evidence that they were used for revision, and as an introduction to the subject.

The Business Applications Software and Engineering packs were worked through during the course, and the Engineering pack was also heavily used for revision.

There is no data available on how students used the Economics text - presumably the same as any other textbook.

4.9 Effect of alternative provision

Library usage:

Some Engineering students stated that they used the library less because the material they needed was provided by their course pack. Comparing the number of date stamps on copies of the relevant books shows some drop in issues over the past year, but not a conclusive reduction.

The Sociology students who bought packs showed some evidence of using the library books less than those who did not use the packs, but there is little quantitative evidence for this.

The Economics textbook in the library showed a 46% drop in issues since the SCOPE pack was produced, both in the short loan collection and the main collection.

Book Buying Habits:

In the case of Engineering, Business Applications Software and Economics, the lecturers stated that there was no one book which covered the topics they wanted to teach, in the correct format. From this point of view it can be said that they did not replace any substitute books. In the case of Economics and Business Applications Software, the packs replaced course handouts, so no revenue was lost to publishers. However, if the packs had not been available for the Engineering course, students stated that they would have had to buy a book, albeit a less suitable one. The book recommended for the materials part of Engineering only sold one copy in 1996-97, despite it being a new edition. This is compared with 30 copies in 1995-96, and 27 previous to that. The attitude of the bookseller was that it was a shame but that at least they were getting some revenue from the SCOPE pack. They had also been warned by the lecturer that he would not be recommending the text to buy. The book which would have been recommended for the manufacturing part of the course is out of print, therefore the inclusion of this material earned the rights holder some money.

Departmental Spending:

In the case of those packs which replaced handouts, departments were saved the time and money spent on photocopying. In addition, the fact that the notes are bound together and students have to pay for them means that students treat them with more respect and they are harder to lose than loose-leaf handouts. Therefore lecturers no longer have to make replacement copies of lost notes.

4.10 Electronic delivery

There were two foreseeable problems which arose out of electronic delivery. The first was lack of student skills in using the system, and the second was concerned with the infrastructure of the institution and whether students would actually be able to access computers. This latter concern included not only a lack of computers, but a lack of access to them (e.g. passwords taking several months to be issued).

On the positive side, those lecturers who felt that course packs were spoon-feeding were very much more in favour of electronic delivery, where the student had to go through several steps in selecting the material they wished to use, and had many alternative materials from which to choose.


5. Conclusions and Recommendations

5.1 Preparation:

Most lecturers feel that while the packs take a lot of work to produce, the end product is worth the initial investment of time. The amount of time required can be expected to fall as more publishers sign contracts, and is also affected by the amount of previously prepared work (e.g. course handouts) that is included. Once the packs have been produced, most have been used again with relatively little, or no alterations, making the investment of time even more worth-while.

5.2 Sales:

Apart from the Sociology packs, all packs sold very well. No single reason has been identified for the failure of the Sociology packs to repeat their initial sales, and more study of this will be required when the remaining packs are sold in Autumn '97.

The Economics text and the Engineering coursebook sold out completely, and only 4.6% (21 copies) of the Business Applications Software pack remain in stock.

5.3 Cost:

One of the main problems of the packs is that many lecturers feel uncomfortable about telling students that they have to buy materials. This attitude depends on the individual lecturer - some see no problem with telling students that they must buy books. there is little point in producing a SCOPE pack unless the lecturers are prepared to strongly recommend that it be bought.

5.4 Binding:

The use of spiral binding has had very positive feedback with no complaints. It makes the packs easy to use in a small space, and allows them to be opened fully. The perfect binding was not successful, since opening the packs fully placed too much strain on the spine, and some packs fell apart.

5.5 Use:

There is a fundamental difference between what students and lecturers expect from the packs. Students want packs to contain all the material they need so they don't need to look at other sources. For the most part, lecturers want the students to read more widely and use the packs as an introduction. Therefore it is easy for both parties to be dissatisfied by the packs and what they achieve. One solution may be to have a brief introduction at the start describing the aims of the pack - i.e. it is not meant to be the entire reading for the course, only the introductory, or essential material. This might help to prevent student misconceptions of the purpose of the pack, although they will probably continue to demand more content for the same price.

5.6 Composition/Relevance:

With the exception of the Sociology pack, the feedback from students was positive. Most students thought the packs were good value for money and found them to be useful. There was some call for more detail in both the breadth and depth of information provided (e.g. Engineering and Economics), which ties in with the finding that students would prefer to have all their material in one place and not need to do any additional reading. Again, there is a correlation between those amount of published material and the students satisfaction with the usefulness of the pack. This is probably also related to the amount of reading students are asked to do: Sociology students are asked to do far more reading than the other students who used packs, and therefore any pack produced for them cannot affordably cover the same percentage of material as does the Engineering one, for example. For this reason it is probably that courses with large reading lists are better suited to online access.

5.7 Effect on Alternative Provision:

There is quantitative evidence of reduced library usage caused by the provision of the Economics text. There is less evidence of effect on library provision caused by the Sociology pack. There was a slight drop in issues, but not marked enough to be able to say that it was related to the SCOPE pack. This is unsurprising considering the small percentage of the Sociology reading list which was actually contained in the pack.

In the case of Engineering, book sales were reduced from 30 in the previous year, to one. This is because the course was taught following the pack that was produced. The attitude of the bookshop was philosophical: they had been warned by the lecturer that the book would not be recommended for purchase, and they were at least getting a cut from the sales. It is worth pointing out that this pack was designed by the lecturer to replace the need for students to buy books which while adequate, did not cover the needs of the course as well as might have been wished. The SCOPE packs containing collections of previously published material do not replace sales of the original texts, or at least, very few of them, since very few students would be able to afford to buy copies of all the original sources.

5.8 Electronic provision

There is little that SCOPE can do about solving problems of institutional infrastructure, apart from making sure that the equipment is available for students to use, and that the staff know how to use it themselves. The user interface is designed to be as simple and self-explanatory as possible, and guides to using the system are being designed.

As has already been stated , it has been concluded that some courses are intrinsically more suited to electronic provision that others, due to their structure and the teaching methods employed.

When online access was first raised as a possibility staff and students were very positive and enthusiastic about the idea. As time has gone on and people have had practical experience of using online resources, enthusiasm has been tempered by the reality of the practical issues - one CMI has decided against using online services because their infrastructure will not support it. However, many CMIs still want SCOPE to provide electronic access for students - five out of the seven CMIs being supplied in Autumn '97 will be using online materials.


6. Summary

Evaluation Documents
SCOPE home page