|
|
SPACER SPACER SPACER |
SCOPE ProjectStage 6a User Satisfaction Survey Report
|
1. Introduction
This report summarises the evaluation results from the first half of Stage 6, Autumn 1997.
2. Stage 6 Courses Autumn 1997
| Biology | Online |
| Business Applications Software | SCOPE pack |
| Communication and Information Science | Online |
| Economics | SCOPE pack |
| Engineering(a) | Online |
| Engineering(b) | SCOPE pack |
| Engineering(c) | SCOPE pack |
| Philosophy | SCOPE pack |
| Sociology(a) | SCOPE pack |
| Sociology(b) | SCOPE pack and online |
| Sociology(c) | SCOPE pack and online |
| Sociology(d) | Online |
Of these courses, four did not use the SCOPE resources provided. This was due to failure to get the relevant hardware/software running in time for Engineering(a) and Communication and Information Science courses (see 4.2 Delivery for details); building work which disrupted project involvement for the Biology course; and insufficient promotion of the Engineering(c) pack by the lecturer.
3. Methodology
Evaluation was done by a combination of interviews, focus groups and questionnaires.
Two generic questionnaires were designed at SCOPE for evaluating packs and for online delivery. These were then modified by Library Representatives according to local needs. Most results were quantitative, with a few qualitative questions. The questionnaires were usually distributed by lecturers during classes and handed back at the end, except one Sociology(d) course where questionnaires were sent out via Royal mail with prepaid return envelopes (providing a response rate of 50%), and another Sociology(c) course where questionnaires were sent via Royal Mail with course evaluation (this had a very poor response). All questionnaire analysis was done by SCOPE using PinPoint 3 for Windows (except where one institution analysed their own results using Excel), this allowed comparability of results.
Focus groups were also used at three institutions, in addition to the questionnaires. These provided qualitative information on areas which were identified by the Library Representative. One institution used an external facilitator, whilst the others used lecturers from the course. Some groups were self selecting, consisting of students who turned up to an advertised group, and others were requisitioned by running focus groups during timetabled tutorials.
Most academic staff wrote reports about their experience of using SCOPE. These reports were supplemented with telephone calls and emails to expand or clarify some points.
On the advice of institutional representatives, Economics students were not asked to evaluate the pack, and some previously produced Sociology(b and c) packs were not evaluated in depth. In the case of Economics, the pack was well received the previous year and the only comments were requests for changes in layout which were carried out for this year. The Sociology(b anc c) packs were exactly the same as the previous year, therefore it was felt that no new data would be produced by evaluating the packs.
The methodology, where implemented, generally worked well, although some modifications to the questionnaires should have taken place, for example the prices of the packs sometimes fell at the boundary of two categories when student's were asked what price they would be willing to pay.
4. Overall Results
4.1 Preparation
Most lists are submitted with a good level of bibliographic detail. The most commonly lacking aspects are details of illustrations, page ranges of associated notes, and the start and end points for extracts which are mid-page and not part of a clearly defined range.
All lists underwent some revision before the final materials were produced:
All lecturers questioned felt that the time spent preparing SCOPE materials was worthwhile.
There have been some problems in getting hard copy of materials to the technical unit for scanning. This arises during progressive mounting of online materials, because library representatives are understandably unwilling to remove recommended texts from the library once teaching has started.
Delivery of the packs was relatively painless, but to a very tight schedule which did cause some concern as to whether packs would be available at the start of the semester, particularly where one bookseller was involved for the first time. The Sociology(a) pack was from the departmental office, which did not find selling the packs to be particularly disruptive, but would not be able to do this on a large scale (i.e. for other modules).
Online delivery threw up more problems than expected. Online Sociology(b,c and d) materials were available in an unstaffed computer laboratory (also used for teaching), and later in the semester through a library computer. The machine in the library was only available during the last two weeks of teaching, but was heavily used when it became available. Material for Sociology(b and c) was available for the start of the semester; Sociology(d) had material mounted on the server throughout its duration, prior to it being required. However, there were complaints from the students that material was not up in time to be useful. Students also complained about the limited number of terminals from which SCOPE could be accessed, and the fact that they could not access SCOPE from terminals outside the university (i.e. from home and work). The main source of complaints was printing problems, such as failure to print the whole document and the fact that they could not attempt printing again without getting staff to reset the database. In addition, there were instances of paper being used by other documents queued on the networked printer. Only Sociology(d) had extensive demonstration and support from the lecturer, where SCOPE was covered in IT tutorials. There was no library induction on the subject, or demonstration to Sociology(b and c), leading to students requesting more support/demonstration of the system from staff, particularly at the start of the semester.
Engineering(a) and Communication and Information Science courses had the follwing problems. Installing the software and use of image-based PDF files caused local problems which were not solved before the Engineering(a) course had finished. The institution running Communication and Information Science had severe technical problems caused by lack of on-site support, late delivery of software and general installation. By the time the equipment was up and running, students had already started their assessment for the course and did not make use of the SCOPE material. Students were made aware of the service by the lecturer, but no demonstrations took place.
4.3 Sales
All packs were sold through John Smiths campus bookshops, unless otherwise stated.
| Philosophy pack: | 40 packs, sold out, reprinted 6, sold out (via James Thin). |
| Sociology(a) pack: | 50 printed, sold out, 48 copies reprinted, 2 remaindered (via the Department). |
| Business Applications Software pack: | Initial 420 copies sold out, 67 reprinted, 24 remaining. |
| Engineering(b and c) pack: | Initial 20 copies printed, 60 reprinted, 48 remaining. |
| Economics pack: | 40 packs supplied, 15 remaining. |
| Sociology(b and c) pack: | 165 copies supplied, 36 remaining. Some second hand sales were apparent. |
| Sociology(c): | 3 printouts were made during the first semester. |
| Sociology(b): | 4 printouts were made during the first semester. |
| Sociology(d): | 166 printouts were made during the first semester. |
Most sales occurred during the first two months of the semester (i.e. September and October), with a small amount of sales later on in the year. It was suggested by one bookseller that having had experience of SCOPE packs, students would be less likely to buy them in subsequent years if they had not been completely satisfied. This could be a partial explanation for the decreasing sales of the Sociology(b and c). The remaindered packs will be saleable to future courses.
Sociology(d) printouts were made throughout the course as material became available, especially near the end of semester when exams were due and the library printer came online. However, it is unclear whether this trend marks an actual increase in demand or simply the fact that the library computer and printer were set up at this time allowing existing demand to be fulfilled. It should be noted that evaluation results indicate that students are not always asking for the database to be reset when printouts fail, so the number of printouts registered is higher than the number of items received by students. However, the figures do give an indication of demand.
4.4 Cost
The cost of the packs ranged from £3.50 to £15, depending on size and copyright content. Satisfaction with value for money appears to be directly linked with the integration of the materials in the pack with the course. Philosophy rated highest with 81% of students satisfied with value for money, and all items being required reading for tutorials; Sociology(a) was next with a 66% satisfaction and 60% of the material being essential reading for tutorials. Business Applications Software and Engineering(b and c) had a lower satisfaction rating at 53% - probably explained by the lack of direct reference to the Engineering(b) course in the Manufacturing and Materials packs, and a perception that lecturer-written materials in the Business Applications Software pack were overpriced: '..fair enough making a profit but [it] does not have to be so vast.' Only 26% of students indicated satisfaction with value for money of Sociology(b and c) packs, this is probably due to the fact that none of the material contained was essential to the course, and it was not directly integrated into teaching.
Format also affected perceptions of value - the Philosophy pack included image files which looked like photocopies, so students did not attach very high value to them. Another student using the Sociology(a) pack pointed out that '[Packs] can never be cheap enough'. Backing up lecturers' views that students will always complain about the price, no matter how cheap it is.
All lecturers approved the prices of the packs. Restrictions included keeping the cost to less than a photocopied equivalent for the Philosophy pack, and that packs should not cost more than £15 for Sociology(a). These objectives were fulfilled by using a public domain translation of Plato for the Philosophy pack, and by prioritising the reading list for Sociology(a): the lecturer dropped the least important item from the list to keep the cost to £15.
Online printing at for Sociology(b,c and d) did not incur any costs to the students, apart from providing their own paper. This is an institutional policy. Some anecdotal evidence that students would find it more convenient to pay for use rather than supply paper, one also stated that even the cost of buying paper was too much (a slightly surprising statement since the cost of paper is less than photocopying). The institution providing Communication and Information Science were going to pass all costs on to students, who would pay for the printouts when they collected them from the library desk. At the institution runningEngineering(a), the library was going to cover the cost of copyright.
The cost to libraries who requested, but did not use the service will be minimal, except where a flat fee was charged in advance by the rights holder. Library costs where material was used will have been increased by students failing to get the database reset after unsuccessful printouts.
4.5 Binding/Covers
Generally speaking, users were not very satisfied with the covers of the packs. They were felt to be too flimsy and did not stand up to the amount of use that the packs got. It may be that the acetate covers are more of a hindrance than a help, since once they are bent, they do not fold back as easily as the card does. However, there is no feedback to either support or refute this.
The packs produced ranged from entirely institutional/author copyright through to entirely commercial copyright, with one pack containing a significant amount of public domain material.
Philosophy: 10 items: 8 book extracts, 2 journal articles; rights held by three publishers, seven public domain items. Initial list included details of alternative sources for extracts: the final choice was determined by the speed at which publishers responded and their price. The public domain material was substituted for material by a publisher who would not give permission.
Sociology(a): 9 items: all book extracts; rights held by: one author, one editor, one agent, four publishers. Initial list of twelve items requested, six items dropped due to four uncooperative publishers. Additional six extracts supplied as; list prioritised to eleven extracts, one dropped due to high pictorial content, one dropped to keep price to acceptable level.
Business Applications Software: material comprised of lecturer-written materials; rights held by authors.
Engineering (b and c): material consisted of lecturer-written material and extracts from one out of print book; rights held by University and publisher.
Economics: material consisted of lecturer-written material; rights held by author.
Sociology(b and c) online: 11 items: 8 book chapters, 3 journal articles; rights held by eight publishers. Initial list of thirteen extracts requested, two dropped due to two uncooperative publishers, no substitutions made.
Sociology(d): 24 items: 18 book extracts, 3 journal articles, 3 official publication extracts. Rights held by 14 publishers, one author, one exectutor, three public domain items. Initial list of 12 items requested, 18 extra requested as additional reading, six items dropped due to three uncooperative publishers, no substitution made.
Engineering(a): 1 item: out of print Open University volume. Rights held by publisher. Inital request for extracts from two books, one dropped due to uncooperative publisher.
Biology: 8 items: 1 journal extract, 7 newspaper extracts. Rights held by four publishers. Initial list of 21 items requested, thirteen items dropped due to three uncooperative publishers, no substitution made.
Communication and Information Science: 7 items: six book extracts, one journal article. Rights held by 6 publishers. Initial list of ten items requested, three items dropped due to two uncooperative publishers, no substitution made.
Format of the materials (pack or online) was usually dictated by the availability of computing equipment. However, some first year students indicated that they found that being expected to use the Resource Bank during their first semester was overwhelming (in terms of the amount of new skills they were already learning) and preferred packs. The expected profile of students is therefore a factor for consideration when deciding on the format of materials. (This will also affect the necessary training for students.)
4.7 Relevance
As with the cost, perceptions of the relevance of the pack are directly linked to the integration of the materials and the course. The majority of students on all courses were satisfied with the usefulness of the pack. However, the fact that students consider the material to be useful is no guarantee that they will actually use it.
4.8 Use of material
Most of the material which SCOPE produced for this semester was required reading for its course. Philosophy, Sociology(a) and Sociology(d) required students to read specific extracts in advance for discussion in tutorials. Engineering(b and c) Economics used the material to support lectures. Sociology(b and c) was the only course for which use of the extracts was not essential for the course.
All material was designed to be of use for preparing for assessments, be it background on-going assessment or revision for end of semester exams.
4.9 Effect on alternative provision
4.9.1 Library usage:
Only the Philosophy pack had any significant effect on library usage. Comparison done with the main library collection showed a decrease in loans of the journal articles included in the pack. However, there was an increase in loans of Plato dialogues, which may reflect the dissatisfaction with the translation used in the pack, and also the decrease in sales of the recommended book (see 4.9.2 below). Other library provision at the institution included a short loan collection and a small departmental library, but the effect on these collections was not assessed.
The Sociology(a) course was new so no previous figures were available. The lecturer stated that library provision was important complement to the pack and provided additional reading.
The situation remained static for Engineering(b and c): too few books were issued to allow comparison.
There was no equivalent book for the Business Software Applications course.
No library usage data was provided for Sociology(b and c), but it is probably safe to assume that last year's patterns were repeated - i.e. no conclusive drop in issues for social science courses.
Sociology(d) was a new course so there are no previous figures to compare issues against.
4.9.2 Book Buying Habits:
At the institution running Philosophy, sales of a recommended textbook halved, despite an increase in students numbers. It was expected by the bookshop that 50% of students would buy books, but in fact only 15% made purchases. However, this was not reflected in students' responses to their questionnaires: 45% indicated that they had bought new books. It is likely that the SCOPE played some part in the reduction of sales, and was exacerbated by students making more use of the library (see 4.9.1 above), and buying books from sources other than the campus bookshop.
25% of Sociology(a) students indicated that they had bought new books. The course was new and there is no campus bookshop at the institution, so comparative figures are not available.
Sociology(b and c) students were given suggestions of books they could buy if they wanted, but were not told that they had to buy any. The students' questionnaires indicate that those who bought SCOPE readers were statistically more likely to buy other books as well. 30% of students had bought new books.
4.9.3 Departmental Spending:
Sociology(a) packs were sold through the department office; although this did not cause the staff any problems, would have cost the department in terms of administrative staff time.
Provision of Engineering(b) packs and Business Software Applications packs decreased department spending. This was in terms of less staff time and departmental resources spent on photocopying.
4.9.4 Teaching practice:
Responses from staff have been generally positive. SCOPE materials give students guaranteed access to key reading materials, which is important in allowing staff to plan their teaching sessions, and students to prepare for them. Staff can focus teaching and discussions more precisely, and teaching can progress faster because less time is required to introduce new subjects.
One Engineering(b) student complained that lecturers who use the engineering course pack, speak to the pack during lectures, rather than to the students.
4.9.5 Student reading
Results are split between those who have used the packs as a starting point for studying, and those who have tried to use SCOPE materials as their sole reading. All packs (except for Business Software Applications and Economics packs) were designed specifically to provide introductory/background reading, which would be relevant but insufficient as the sole source of information for assessments. The packs also avoided making the student dependent on the views of one particular text and author perspective. Students were informed of this by the lecturers, and most appear to have used the packs in this way.
4.10 Electronic delivery
Perceptions by non-users
Most students welcomed the idea of providing materials online - they felt it would be cheaper and would save paper, but were put off by the problems of accessing machines. There was also evidence of computer literacy problems and technophobia, but access to machines was the main concern. Those who used packs liked them for their tangibility and convenience: once purchased, all the key reading was available.
Perceptions of users:
Not being able to access the resource bank from computers off campus caused complaints, as did having to print off entire articles. Anecdotal evidence suggests that some students, having located the information they required using the resource bank, used the photocopier to copy the pages they needed rather than printing off the whole article. Students did use the facility to search the text on screen, and some took notes from the screen, especially when printing problems occurred.
Printing problems meant that many students gave up attempting to use the system and relied on traditional library provision and the photocopier. Other students were put off by hearing about problems and did not even attempt to use the system. Many of those who used the online resources for Sociology(d) stated that they would have preferred a pack. This may be partially caused by inadequate library provision resulting in a lack of material for the course if the resource bank failed, and progressive (rather than advance) mounting of the material.
5. Conclusions and Recommendations
5.1 Preparation:
Library representatives are usually too busy with their own work to promote SCOPE to all interested parties. Therefore this has limited the uptake of SCOPE services. If possible, the Liaison Officer should work more directly with the lecturers involved.
Lists continue to be passed to the SCOPE office with incomplete details, and results in delays in clearance. This is very frustrating for the SCOPE team since the required information can often requires sight of the hard copy, and can sometimes only be provided by the lecturer.
Most lists required some prioritisation of requested materials. The substitutions which took place were caused by publishers refusing to grant permission and the need to speed up the clearance process.
Lecturers seem happy with the amount of effort required to produce SCOPE materials, especially when it directly benefits their teaching methods.
Library Representatives dislike providing hard copy materials to the technical unit once the course is running, because this prevents student access to materials. The only solutions are to use photocopies (requiring clean copies and high accuracy scanning/OCR software at the technical unit) or to have materials cleared and scanned before teaching starts.
5.2 Delivery
All packs were delivered on time, despite the tightness of the schedules arising from late delivery of reading lists. There was some concern from lecturers and booksellers about whether they would arrive on time for the start of teaching, and it would be preferable to try to have packs ready sooner, although this is obviously dependent on getting reading lists earlier and copyright clearance going to schedule. This problem is less serious for online materials than for packs, since online materials can be mounted as permissions are granted, rather than having to wait for all items to clear before printing.
Online delivery was thought to be a good idea in principle, but was hampered by initial technical problems, which meant that printing was unreliable. There were also complaints about being unable to print selected pages. Printing problems were the most common cause of complaint. PDF files composed of images are not suitable for delivery over the internet to networked printers, due to their large size. From the experience of Sociology(b and c), it is recommended that students receive some sort of SCOPE training. The ideal minimum requirement is a demonstration using the powerpoint presentation designed for this purpose.
5.3 Sales:
Print runs, with the exception of Sociology(b and c), Economics, and Engineering(b), were appropriate to the numbers of students and had a very high percentage uptake. The remaining packs can probably be sold next year.
Use of the resource bank was sporadic, partly due to technical problems, and also due to late mounting of materials. These technical problems are expected to be less significant in future.
5.4 Cost:
There are no new conclusions to draw from this year's results. There was a wider spread of prices this year, but the cheapest and the most expensive packs both had a high uptake. It appears that price is, within reason, far less important than the lecturer's recommendation and the integration of the material into the course.
5.5 Binding/Covers:
There continue to be complaints about the covers of the packs, but it is unknown whether students would be willing to pay the extra money to use stronger cardboard.
5.6 Composition/Relevance:
All new materials produced consisted of commercially published extracts, a small minority of which was public domain. Satisfaction with the usefulness of most material was very high, although this did not guarantee usage. Incorporation of material into the course (i.e. it is directly relevant) has more effect on usage, and increases perceptions of its value.
5.7 Use:
Material generally enabled lecturers and students to focus teaching and learning on specific, key readings, and also to prepare students for assessment. In addition, the material helped introduce students to new subjects and increased their confidence because they knew that they had access to essential materials.
5.8 Effect on Alternative Provision:
It is impossible to draw any firm conclusions about the effect that SCOPE materials have on other provision. The results from Sociology(b) course indicate that students who buy SCOPE packs also buy other books, however, where the Philosophy packs were provided, the sales of the recommended book from the campus bookshop halved.
It may be that where essential reading is integrated into the course, sales of other books are affected, but where SCOPE provides general reading, there is little effect on commercial sales.
5.9 Electronic provision
SCOPE's first experience of providing online information has been a success, despite the early difficulties which arose with printing. Students and staff support the idea in principle, and are enthusiastic about the possibilities once initial printing problems are solved.
6. Summary