3. Interim evaluation results

This section reports on the findings which are emerging from SCOPE evaluation activities and comments in particular on any general outcomes, effects and impacts. Difficulties or delays experienced in carrying out evaluation activities are also noted and explained

3.1. Difficulties and delays

3.1.1 General

The SCOPE team is confident now that our approach to development of the resource bank was correct; a wide but shallow resource bank would probably have been of little use to lecturers and students at SCOPE CMIs. We now have a sufficient number of agreements with publishers and other copyright holders to offer a useful bank of resources in one subject area and have produced resources tailored for classes in other subject areas that may be offered to classes at other institutions. On the other hand we have widened the range of subjects into related areas to seize opportunities of working with enthusiastic academics in more institutions. We have also set out to broaden our range of activities in order to increase the likely long-term viability of the project; to date we have published institutions' own materials.

3.1.2 Liaison

Responsibility for conducting and/or co-ordinating evaluation activities falls primarily on the Liaison Officer post which was vacant for a large proportion of Year 2. Assistance was employed to conduct evaluation activities at Stirling University. SCOPE is dependent on academics and library representatives of member institutions for ensuring that evaluation activities are conducted at appropriate times, and in a few cases this did not happen.

3.1.3 Copyright clearance

By the end of Year 2 of the Project, the Copyright Officer has increased to 55 the number of copyright holders that have agreed terms of use with SCOPE. However, although the Project has made gradual progress which accelerated during Year 2, there are negative developments and findings to report. Mobilization in this area is evident in that many more publishers now have developed or are developing policies on licensing their copyright materials for electronic use in academic libraries. Unfortunately, the policy developed by some publishers is to deny permission for such use. A few publishers that agreed to license material to SCOPE during Year 1, refused further permissions requests following a policy decision or change of management in Year 2. The rate charged for copyright materials may also be effected by mobilization among publishers. SCOPE offers to pay for copyright at a rate of 2.5 pence per page and this rate is accepted by a majority of rights holders agreeing terms with the Project. However, a powerful minority continue to charge at a higher rate and some UK publishers are now adopting the rate of 5 pence per page proposed by the Publishers Association.

The SCOPE team has always expected that copyright clearance would be streamlined when agreements were in place. We are aware now that permission to use specific items may be denied even by publishers with whom we have agreements and amicable relations. We have been advised recently by the rights director at one publishing company that she will agree contractually to license materials to us but that every request will be decided with reference to the forecast effect on sales of the source text at the institution for which an extract is selected. The price of materials licensed by that publisher also will vary with the predicted effect on sales of originals, for example, the price will be considerably higher for an extract from a book that sold 100 or 200 copies to students in the year previous to the request compared to one that sold only 3 copies to the library.

3.1.4 Content and appropriate delivery method

The SCOPE team is now more aware that certain media are more appropriate for delivering resources in specific subject areas and that certain types of materials cannot viably be delivered on-demand in academic libraries because of the rights status of their content. That course packs appear to be appropriate for students in certain subject areas, for example engineering, and less appropriate in others, e.g. sociology is covered below. One factor that militates against use of certain types of scientific materials used commonly by undergraduates, for example review papers, is that the number of third-party copyright holders renders copyright clearance prohibitively expensive. SCOPE was asked to produce a course pack for a psychology class at St Andrews University during Year 2 and discovered that much of the material included figures which were copyright not of the author or publisher of the work but of a third party. One extract of 50 pages contained materials controlled by 33 different rights holders; if fully costed, the task of clearing copyright would have increased the price of the course pack well beyond what students are prepared to pay.

3.1.5 Demand for SCOPE resources

Regarding uptake of resources, it appears that students assessment of value is based largely on lecturer endorsement in their first year. Anecdotal evidence suggests that in later years, peer recommendation is more persuasive. It is difficult to influence the latter. However, lecturers who have invested time and effort in developing SCOPE resources may be expected to have some sense of 'ownership' over them and thus, may be expected to recommend them to classes. First year students who find SCOPE resources useful, beyond their function simply as introductory materials that confer confidence while a student develops further information skills, may be expected to use them in future years and to recommend them to their peers. Assessment of value by students who have used SCOPE resources would appear to be based on the perceived relevance of the material to course content (or to that part of the course content that they focus on in writing essays or planning for workshops) and value for money when compared with other available resources.

3.2 SCOPE's outcomes and effects in relation to the eLib programme's overall evaluation preoccupations as specified in the Guidelines for eLib Project Evaluation

3.2.1 Course packs vs. online delivery

3.2.1.1 PERFORMANCE.

Evaluation findings confirm the suspicions of the SCOPE team that online delivery of SCOPE resources will be welcomed at certain institutions and by lecturers in specific subject areas while course-packs will continue to be more valuable at institutions with limited infrastructure for network access and on courses with less emphasis on independent learning and information skills development. For example, sociology lecturers have designed packs to give a broad overview of the course with a view to encouraging wider reading among students. Students, however, tend not to read widely but focus their studies on essay topics and workshops, they resent paying for material in course packs that they do not use and have asked for packs that focus on essay topics. Lecturers refuse to supply students with themed packs as they consider this to be 'spoon-feeding' and to undermine basic skills that are required to perform well in sociology. Online access may reconcile the requirements of lecturers and students. Students will have the facility to print only those resources that they intend to use, as they require them throughout the year; there will be no need for them to pay for materials that they have not selected themselves.

3.2.1.2 CULTURAL CHANGE/MOBILIZATION.

Sociology lecturers do not consider online access to be spoon-feeding. On the contrary, one lecturer considered incorporating use of SCOPE electronic resources into the course to help students become more familiar with electronic media. This approach would help students to address general reticence about computer use as well as those problems that SCOPE resources are designed to address.

3.2.1.3 VALUE ADDED

In contrast, lecturers in engineering consider course packs to be valuable because they give students convenient access to all of the basic material that they require. Online access would reintroduce the problem that class hand outs tend to get lost and thus, are less likely to be read by students. Students of engineering reiterate that a key positive feature of course packs is the fact that all materials are compiled and bound together and thus, are less likely to be lost.

There is a fundamental difference between what students and lecturers expect from the packs. Student want packs to contain all of and only the material that they are required to read so that they need not seek other sources. For the most part, lecturers want students to read more widely and to use the packs as an introduction. Therefore, it is easy for both parties to be dissatisfied by the packs and what they achieve. SCOPE will aim to address student misconceptions during Year 3 by including in course packs a brief introduction which describes the aim of the pack and explains that it does not represent all of the reading material for the course but contains only essential materials.

3.2.2 Barriers to adoption of online delivery mechanism

3.2.2.1 SUSTAINABILITY.

The SCOPE team are aware that some institutions are better equipped to deliver material online than others and that the latter are likely to use course packs rather than risk congestion at PCs in the library. However, lecturer concerns about online access are not confined to availability. There is some concern about the practicalities of offering the online option to newly arrived first-year students who require to be registered, given passwords and trained in the space of a few days. At one 'wired' institution, lecturers are concerned that first-year students do not register to obtain user IDs and consequently do not use networked resources. The reasons that students fail to obtain user IDs are evident and are related to policy of the Information Services department. Clearly plans for introducing on-demand publishing should not be made in isolation but must be integrated into teaching and learning and into the information strategy of the institution.

3.2.3 Costs and benefits of online delivery

3.2.3.1 SUSTAINABILITY

A variety of issues has been raised by librarians and lecturers regarding the costs and benefits of online access. Some libraries are concerned about reduced profits from photocopying facilities and academic departments supplying printing facilities free of charge are concerned about paper costs.

3.2.4 Libraries concerns about recovering costs for student use of materials delivered on-demand

3.2.4.1 COST EFFECTIVENESS

Although it appears that some chief librarians are prepared to pass copyright fees to students, librarians 'on the ground' are more reluctant to do so. There appear to be mixed views on the desirability of passing on to students royalty charges for texts delivered online (see above).

3.2.4.2 SUSTAINABILITY

While libraries would prefer not to pass costs on to academic departments at this developmental stage, considering SCOPE to be too important to jeopardize support before the benefits are clear, if fees are high, academic departments may be asked to pay or to contribute to the cost. Payment for SCOPE resources will be negotiated within each of the consortium member institutions but SCOPE will document relevant dialogue and influences on decision making in the final report of the Project. Few institutions have systems or hardware in place which would allow cost effective recovery of charges from students.

3.2.5 Sources are price sensitive

3.2.5.1 COST EFFECTIVENESS/ DEMAND

One thing that is clear to the SCOPE team is that resources are very price sensitive. SCOPE has produced course packs for a variety of classes over the first 2 years of the project but, to date, only one of those courses has been offered SCOPE resources in two consecutive years allowing us to monitor the effect of variation. In the first year, course packs for that class consisted of approximately 300 pages of photocopied book extracts and journal papers. The packs were subsidized by over £15 per copy and were sold for £17.50. They were very successful; a total of 320 packs were sold to approximately 320 students. In the following year, the material in the packs was digitized and occupied a smaller text area. Packs consisted of approximately 200 pages of original material but the reformatted page extent was approximately 120 pages. Packs were sold for £15 and would have covered production and copyright costs had all copies been sold. Unfortunately, only 40% of a print-run of 300 was sold. Students cited price as the most important factor determining their decision to purchase or not to purchase a course pack.

Course packs tend to be particularly valuable to mature part-time students living off campus because these students find library provision inconvenient and often do not use the short-loan collection because they do not visit the library frequently enough to return books within the restricted loan period. There is discontentment among part-time students that full-time students are able to access texts free of charge in the library while part-time students consider reading materials to be accessible conveniently only if they buy course packs. Students are acutely aware of disparity of access to required reading materials, for example, at one CMI it has been suggested that photocopy costs represent a 'tax on part-time students'. On courses supplied with packs consisting of material 'tailor-written' for the course, lecturers acknowledge that students who could not afford to buy a pack were at a disadvantage. One lecturer actually believed that nobody would be denied access because the pack was priced at a rate that was affordable to all students. His attitude may be naïve. Our limited evaluation suggests that some students spend nothing on reading materials. Perhaps some of these students cannot afford to buy reading materials and thus, are effectively denied access when core reading materials are sold rather than provided free of charge in the library.

3.2.5.2 MOBILIZATION/SUSTAINABILITY

Even those students who found packs to be invaluable resented the fact that reading materials for the course cost money while those for other courses were supplied free of charge (but in a less convenient form). SCOPE is aware that student feedback is important to academics. In attempting to mobilize academics in adopting SCOPE resources, we would hope to harness this influence. For example, if a respected academic were to adopt SCOPE resources and students found them to be useful then the academic may act as an exemplar who effectively encourages uptake by other academics of alternative means of providing reading materials. Meanwhile, the SCOPE team would call for clarification of institutional policies with regard to charging for reading materials. If student complaints are to be avoided, students must be made aware in advance that they will be expected to purchase some reading materials.

3.2.5.3 VALUE ADDED/CULTURAL CHANGE

Questions may be raised also about the validity of charging students for basic teaching materials such as those included in lecturer-authored course packs; perhaps institutions where this is practised should advise students in advance that they will be required to buy such materials. SCOPE must also differentiate between materials that may be sold legitimately and those produced as part of normal preparations for a course. Future SCOPE policy is likely not to allow royalty fees when supplying course packs for distribution at an academic's own institution. Royalties would be charged and would be paid if her/his material is selected for use more widely, and thus is distributed at other institutions.

3.2.6 Issues raised by 'publishing' lecturer-authored materials

3.2.6.1 SUSTAINABILITY/ USEFULNESS

Publication of lecturers' own material has raised two issues that are important but cannot be addressed by SCOPE within the lifetime of the Project. First, typographical accuracy is important and is cause for complaint if quality is poor. SCOPE were aware of this problem on accepting materials directly from lecturers but, owing to cost, made a policy decision during Year 1 not to edit heavily any materials supplied by lecturers; the lecturer herself/himself should be responsible for typographical accuracy. Nevertheless, the SCOPE Liaison Officer must impress on any lecturer writing for SCOPE that accuracy is very important and should encourage the academic to take responsibility for proof-reading.

Second, lecturers writing course materials for SCOPE have licensed the material for distribution throughout the consortium. To date, packs consisting of lecturers' materials have been distributed only at the institution for which they were written. SCOPE would like to distribute these materials more widely so that a greater contribution towards overheads may be recovered from SCOPE activities. Library representatives have been made aware that these resources are available but the SCOPE team cannot afford to spend more time on promotional activities for 'additional' materials.

One issue relating to lecturer-authored material is rights ownership. To date only one of the lecturer-authored course packs has been agreed contractually with the institution at which the author is employed rather than with the author her/himself. Although the Copyright, Designs and Patents Act 1988 makes clear that material written in the normal course of employment is copyright of the employer, practice varies and institutional policy may not always be clear. Clarification on copyright ownership of teaching materials is required if use of this type of material is to increase as predicted.

3.2.7 Effect on student study habits

3.2.7.1 VALUE ADDED/ PERFORMANCE/ CONTRIBUTION TO OVERALL PROJECT GOALS

Lecturers on two courses supplied with SCOPE course packs to date believe that student study habits have improved as a direct result of possessing a pack containing basic reading materials for the course. A sociology lecturer reported that students in earlier years had encountered difficulty in obtaining reading materials for tutorials and workshops. As a result many of the students attending these sessions had read nothing relevant and could contribute little to discussion. The class supplied with SCOPE course packs were more prepared and although some had difficulty in obtaining reading materials from the library, many had read the relevant material from the pack.

3.2.7.2 USEFULNESS/ PERFORMANCE/ CONTRIBUTION TO OVERALL PROJECT GOALS

A lecturer on an engineering course had included in course packs materials that previously had been distributed in lectures as 'hand-outs'. He reported that having paid for course packs, students are more likely to respect the content and less likely to lose it as it is bound together in a substantial product.

3.2.8 Student satisfaction

3.2.8.1 USEFULNESS/ PERFORMANCE/ DEMAND

Feedback from students in all but one of the classes supplied with course packs was positive; the exception being a sociology class. Most students found the packs useful and considered them to be good value for money. Some students called for more detail in packs which appears to reiterate their call for all required reading materials from one source. Student satisfaction appeared to vary with the proportion of lecturer-authored material, presumably because this type of material is tailored directly to student requirements. Sociology packs were less successful for the reasons given above; the packs contained less than 30% of the total reading list and covered the whole course rather than focusing on specific topics. Courses such as sociology that draw on a wide range of sources and for which students must select from an extensive list of readings are probably better suited to online access than to course-pack provision.

3.2.9 Effect on library

3.2.9.1 CONTRIBUTION TO OVERALL PROJECT GOALS

The effect of course-pack provision on library use is difficult to determine. Library issues for books from short loan which were selected for an economics class supplied with SCOPE course packs fell by 46%; issues from the main collection also dropped. This reinforces our findings in Year 1. In contrast, library issues to engineering students supplied with SCOPE course packs barely fell but the base level was very low anyway suggesting that students rely on core reading materials distributed at lectures or contained in texts that they purchase.

3.2.10 Effect on publisher income/publisher relations

3.2.10.1 SUSTAINABILITY

The SCOPE team is aware that the impact of ODP on publishers' profits is related to good relations with publishers and thus, to the success of the Project. Publishers are concerned that SCOPE resources will substitute for sales of original hard copies. Generally, SCOPE resources do not substitute for sales of texts. Students that buy SCOPE resources tend also to buy texts and to spend money on photocopies. Students that do not buy SCOPE resources tend not to spend money on reading materials of any type. It would appear that students are divided into 'haves' and 'have nots' and that the latter group rely principally on the library for all of their reading materials.

While copyright fees are transaction-based, publisher revenue generated by the Project is related to payment per copy made. A second-hand market could reduce the income that publishers may expect from SCOPE. Our limited experience suggests that there is no significant second-hand market for course packs in classes supplied with SCOPE resources to date. Some students annotate their course packs thus reducing their resale value and some consider the resale value to be worth less than the inconvenience of reselling the pack.

3.2.10.2 COST EFFECTIVENESS/ SUSTAINABILITY

Those publishers that agree terms with SCOPE may expect their materials to be adopted more widely in SCOPE CMIs. Evidence of this would be that lecturers are prepared to substitute affordable items for those that are too expensive or that SCOPE cannot secure permission to use. It is clear that academic staff are prepared to substitute recommended texts in certain circumstances. SCOPE communicates with academics throughout the copyright clearance process and requests substitutes when royalty charges are very expensive or when permission cannot be secured in time to supply the material at the beginning of semester. Occasionally, a lecturer will judge material to be too valuable to omit but generally, expensive items are replaced or are omitted without replacement. One lecturer replaced items in the pack owing to expense regardless of the fact that the cost would not have affected the price of packs as SCOPE intended to absorb the expense. She wishes to move towards viability of course packs now rather than to select prohibitively expensive items that she will be required to replace in later years if and when costs are passed to students.

3.2.10.3 MOBILIZATION

When the SCOPE Project began we expected that by Year 3, copyright clearance procedures would be streamlined and largely administrative. We aimed to negotiate and agree terms of use with publishers based on our model contract. We believed that having agreed on the terms of a contract with a publisher, we could expect to add to the SCOPE resource bank any materials published by that company simply by sending an amended appendix to the contract; this would be an inexpensive administrative task. Thus, the workload of the Copyright Officer should have consisted primarily of negotiation with companies that had not previously agreed a contract with SCOPE. Over the last year, the number of publishers agreeing terms with SCOPE has continued to rise. However, we are aware now, that our initial perception was false. Until on-demand publishing is more mature, and publishers have established and operated policies on licensing electronic rights, we cannot take for granted that having agreed terms with us, a publisher will grant further permissions. Many of the publishers that agreed terms with SCOPE and signed our contract during the first half of the Project did so before developing policies on licensing electronic rights. Subsequently, a few of those publishers have developed and now operate a policy of denying such rights; they have denied requests to include further copyright materials in the SCOPE resource bank. These developments indicate mobilization among publishers but mitigate the growth of SCOPE activities.

The number of publishers developing policies on licensing copyright materials in electronic form has increased significantly over the last year. Some of those publishers that signed the SCOPE contract during the first half of the Project now operate a policy of denying requests to use material for on-demand publishing.

3.3 Feedback on emerging models of HE library provision particularly as regards the provision of teaching resources

3.3.1. The relationship between content and appropriate delivery method

Emerging models for library provision of teaching resources vary across subject areas and across institutions. The most successful course packs produced by SCOPE to date have contained a significant proportion of lecturer-authored materials; students were satisfied that all of the material included in packs and for which they had paid, was relevant. Production of lecturer-authored materials for delivery on-demand requires the lecturer to prepare materials well in advance of the course start date. This is relatively inconvenient but, with hindsight, lecturers consider it to have been a worthwhile investment as time spent preparing materials during semester is reduced and, in specific subject areas, the perceived benefits of teaching from a pack can be considerable.

Students in subject areas such as sociology find pre-prepared course packs to be less useful than those studying certain other subjects. In future, students that read from a wide range of sources are likely to benefit more from delivery online or from the facility to self-select the contents of a customized course pack. Clearly, librarians selecting the most suitable means of delivering SCOPE resources at their institution must do so with reference to available resources; students at an institution choosing to deliver materials online must have access to a sufficient number of PCs.

3.3.2. Necessary conditions for ODP

In common with other initiatives promoting use of technology to deliver teaching and learning materials (TLTP and LTDI projects for instance), the SCOPE team is aware that problems with copyright clearance have effectively stymied developments in this area. When developing teaching resources, academics express interest in incorporating commercial copyright and lecturer-authored materials. An efficient mechanism for clearing copyright at an affordable rate would facilitate academic development of resources that utilize fully available technology and may relieve pressure on libraries.

SCOPE was a major contributor to an eLib-commissioned study on the impact of on-demand publishing on higher education in the UK (see JISC 1997, in press). It is slowly becoming clear within HEIs that ODP will not necessarily present a more efficient model for delivering teaching materials in the short term. Any library or project seeking to deliver a collection of high-demand resources in electronic form is likely to find that the cost of clearing copyright (that is the process of negotiating rights rather than simply the royalty rate commanded) and the cost of digitizing from hard copy are prohibitively expensive for a small-scale operation. The copyright clearance process must be streamlined and it is widely believed that a central body operating a one-stop-shop for copyright clearance is required to make the process efficient. Digitization also is likely to be efficient only to an organization exploiting economies of scale.

Nevertheless, it is very likely that in the longer term, ODP will provide a more effective means of delivering teaching and learning materials in a variety of forms. There has never been so much variety among students with regard to lifestyle and approach to study and trends towards wider access and lifelong learning are set to continue (see National Committee of Inquiry into Higher Education 1997) . Lecturers and librarians are now aware of the flexibility offered by on-demand publishing from an electronic resource bank for distance learners, part-time students and traditional students. Although many lecturers and librarians seeking information about ODP from SCOPE team members have yet to consider all of the issues raised by ODP and the obstacles to be overcome before a working system may be implemented, there is definitely growing interest in ODP. As the means of delivering academic information change, so too does the balance of responsibility for producing and providing resources. All stakeholders must be flexible and must take advantage of opportunities as they arise. Projects like SCOPE will be most useful in informing planning and mobilizing change within HEIs.

[Contents]