SCOPE Evaluation Plan

Introduction

This document details an evaluation plan for year 1 of the SCOPE project, drawn up in the light of the Tavistock guidelines and April workshop. An outline of evaluation activities for years 2 and 3 is also given.

Contents

1.

SCOPE's aims

2.

Purpose and priorities of evaluation

3.

Stakeholders and their interests, with planned data gathering methods

4.

Year 1 Evaluation

i)

methods

ii)

resources

iii)

reporting

iv)

schedule of data gathering and reporting responsibilities

5.

Years 2 and 3 evaluation outline

1. SCOPE's Aims

SCOPE is engaged in innovation: it is attempting to introduce relatively new and cost-effective methods of student text provision. To be successful SCOPE must:

i)

persuade academic staff of the advantages of customized, on-demand course packs and on-line delivery;

ii)

persuade students that it is in their interests to purchase packs and (later) to use the resource bank, probably paying for individual articles;

iii)

persuade publishers of the necessity of reaching new agreements and pricing models and streamlining permissions procedures;

iv)

demonstrate (potential) savings in libraries in areas such as short loan/reserve collection administration;

v)

persuade booksellers that greater revenue will result from high volume sales with lower than average mark-ups.

2. Purpose and Priorities of Evaluation

The first purpose of evaluation is therefore to monitor how effective SCOPE's efforts are in these areas. The other priority areas are product design and production: course packs and on-line delivery. (Note that, at this stage, we do not expect to cease course pack production when on-line delivery is introduced. It is probable that one product will be more suitable in some institutions or with some types of course than the other. This requires further investigation.)

3. Stakeholders and Their Interests

The major stakeholders have been identified as academic staff and student representatives, publishers, booksellers, librarians and eLib itself. A list of their main information needs are identified below, ( a revised version of one mounted on lis-scope and lis-scurl for general comment). Note however that most stakeholders will be interested in the wider picture.

AQ

questionnaire or structured interview for academic staff for whom pack is produced;

SQ

questionnaire for students for whom pack is produced;

CCD

copyright clearance document;

LQ

questionnaire or structured interview with library staff of institution using course packs;

i) Both Academic Staff and Student Representatives need to know:

• that student access to relevant reading is improved, ie. it is available
  • when required [production reports, SQ]
  • in a convenient format [SQ]
  • at an affordable price [sales figs, SQ]
• that SCOPE resources afford increased educational benefit eg. better
  • participation in tutorials, and wider or more in depth reading [AQ, focus groups]

[For comparison and where practicable, data should be gathered on levels of student satisfaction pre-SCOPE: eg. library user satisfaction surveys, course questionnaires, book sales figures.]

ii) Academic Staff also need to know:

• that participation in SCOPE does not take up an undue amount of time [AQ] (set-up time may be investment)
• the extent to which packs/individual articles are used [sales figures, SQ/on-line access log (later)]
• the extent of overlap/sharing between institutions [copyright database]
• how SCOPE has affected teaching practice [AQ]
• to what extent material is used where copyright belongs to institutions or individual academics (ie. is not commercially published) [AQ, CCD]

iii) Publishers need to know:

• the effect on book sales to students and libraries [sales figures, SQ]
• the purpose of SCOPE texts (eg. supplementary or replacing core commercial texts) [AQ]
• the nature and extent of the potential market [pack sales, SQ, AQ, focus groups]
• how costs are broken down [production figures]
• the effect on the volume of photocopying done by students and institutions [SQ, library figures for photocopied articles in collections and for photocopies made by and for users]

iv) Booksellers need to know:

• the effect on sales [sales figures (through booksellers and other distribution points)]
• the extent and nature of the potential market [SQ, focus groups, AQ, pack sales]
• what is their role when SCOPE enters the electronic delivery phase [pilots, production report]
• a breakdown of costs [production costs]

v) Librarians need to know:

• if students and academic staff feel that access to relevant reading is improved [SQ, AQ, focus groups]
• what savings they might make in terms of:
  • purchases of multiple texts [LQ]
  • maintenance of reserve and photocopy collections [LQ]
  • reduction in theft and vandalism [LQ, surveys of original material used for digitisation]
• the extent to which they still have to make provision for those who cannot/ will not purchase SCOPE resources [LQ]
• that they have adequate facilities for access, printing of on-line articles [SQ, focus groups, LQ]
• that students find the system user-friendly [SQ, pilots]
• that mechanisms for cost recovery and payment are straightforward and take little effort [pilots, production reports, LQ]
• whether on-line text offers added value (searching facilities etc.) [pilots, AQ, SQ]
• what resources they have to put into making SCOPE work (e.g. liaison with academic staff, evaluation etc.), and generally how they have contributed to the project [LQ]
• whether procedures/channels of communication are effective [promotions survey]
• what would be required of them in the future [LQ]
• whether any consortium members have been unable to participate and why [co- ordinator's report]

vi) Elib needs to know:

• the extent to which model contracts have been agreed [CCD]
• which models work best (and more generally publisher and bookseller reactions) [CCD]
• the extent to which general statements can be made re student and academic and library behaviour (and especially evidence of cultural change) [user satisfaction survey: analysis of results]
• which behaviour patterns appear to relate to specific institutions (inter- institutional comparisons) [user satisfaction survey: analysis of results]
• the cost-effectiveness of system/software developments [production reports]
• the transferability of any developments [site pilots, production reports]
• the on-going feasibility of SCOPE [cost – benefit analysis]
• that project funds have been used responsibly [financial reports]

4. Year 1 Evaluation

The following plan has been drawn up in the light of the Tavistock guidelines and workshop. Note, the SCOPE team has been engaged in previously planned evaluation activities which fall short of those suggested here in several respects. As all the first year's activities will be repeated in year 2 there will be opportunities to carry out the entire plan then. It will also be necessary to carry out additional evaluation work in preparation for future developments.

i) Methods of evaluation

The following methods of evaluation will be used: questionnaires, structured interviews, focus groups, PRINCE quality assurance as given in the Project Initiation Document. Data from additional sources (eg. book sales and interlibrary loan figures) will be assessed for their significance to the project.

Questionnaires:

a)

The student questionnaire will be brief, easy to analyse and will seek to obtain basic quantitative data from those who bought or could have bought course packs.

b)

The questionnaires for academics and library staff will be much more open, and will provide a series of headings on which staff are asked to comment and provide data if available. These questionnaires will be backed up with interviews or 'phone discussions. The purpose is to make an in-depth investigation of the importance of packs for particular courses, recommendations for future developments and detail of the impact on libraries.

Focus groups:

Are used to explore in-depth the characteristics of the market for SCOPE products, and to shape future developments.

PRINCE quality assurance methods:

Will be used as part of regular project management, e.g. for evaluating production quality of course packs.

The copyright negotiations process provides an example:

• a draft agreement was approved by the Copyright Advisory Panel (CAP)
• this agreement was sent to publishers
• publishers' comments were fed back to the CAP
• the draft agreement was revised, and an alternative version also was made available.

The quality of model agreements will be assessed by the Copyright Advisory Panel with reference to the standards devised at the initial Copyright Advisory Panel meeting. Legal assessment will be made by the University of Stirling Copyright Officer.

ii) Resources

The project co-ordinator will supply draft questionnaires which will be adapted by the library representative to suit local conditions. The chief resource requirement is staff time to administer and analyse questionnaires, to conduct structured interviews and write reports. The library representatives have undertaken to do this as part of their contribution to the consortium. Where focus groups are deemed useful in individual institutions, each will be encouraged to organize and resource these; however if there is no local expertise, SCOPE would be prepared to pay the costs of a facilitator.

The chief direct cost in year 1 to date has been the fees of a facilitator for focus groups and interviews at Stirling, £345. In year 2, £2000 will be earmarked for evaluation activities: up to £1400 for focus groups, including possibly an inter- institutional one for academic staff. Up to a further £600 would be available to pay for additional help to carry out survey work identified by the library representatives' working group on evaluation. Institutions will however be encouraged to use their own resources as far as possible.

iii) Reporting

The project team consists of the Copyright Officer (Leah Halliday), Production Manager (George Pitcher) and Library Co-ordinator (Carolyn Rowlinson). The team shares all information, usually at fortnightly meetings.

The team makes regular reports to a variety of other bodies which amongst them represent most of the stakeholders. Most of these stakeholders have representation on the project board or on an advisory group, or another body with which SCOPE has close ties:

Project Board:

- 4 librarians including Project Director & Co-ordinator (both from the lead site), a librarian from the partnership and one representing the other libraries of the consortium

- an eLib representative

Copyright Advisory Panel:

- e-Lib Copyright Advisor (expert in field who is also an academic)

- two academics with close contact with publishing industry

- bookseller

- an e-Lib nominee who has close ties with the commercial publishing industry (under discussion)

Technical Advisory Panel:

- University Computer Services Manager

- librarian with cataloguing/IT experience

- bookseller

- two academics with print production experience

The Project Board meets at least quarterly (usually more often). The Advisory Panels meet irregularly and much contact is by e-mail. The academic staff are expected to play dual roles: i) as experts on the copyright or technical sides; and ii) as teachers of students.

In addition there is a group of Library Representatives responsible for co- ordination of SCOPE activities in their institutions. This meets quarterly.

Finally the project is under the aegis of SCURL (Scottish Confederation of University and Research Libraries). SCOPE reports regularly to the chief librarians who meet approximately bi-monthly.

SCOPE News the project newsletter and Web pages will be used for brief articles on all aspects of the project and distributed to consortium member institutions, publishers and booksellers. A synthesis of all evaluation activities will form a major part of the annual report which will also have wide distribution. In addition, we shall continue to seek opportunities and accept invitations to give presentations at a wide range of professional meetings and to write articles for appropriate periodicals.

iv) Schedule

Notes:

a)

milestones are identified by emphasis

b)

project year ends May 14; business plan due with eLib 31 July

c)

other packs may be produced outside of the September/February peaks. This would be to fit in with the needs of particular institutions. Further, we aim to even out production over the year as far as possible

d)

in addition to data listed in the schedule below, financial reports are received monthly from Napier and Stirling Finance Offices and sales figures are frequently checked at booksellers or other distribution points

e)

consideration will also be given to relevant data from other projects

Review of whether any useful data can be gathered in a participating institution library before delivery of a SCOPE product (CR)

Pre-SCOPE book sale figures (LH)

Production report September/October (GP)

Production (including copyright) costs September/October (GP, LH)

Copyright clearance document October (LH)

Student questionnaire October/November (CR)

Focus groups - if appropriate October/November (CR)

Library questionnaire December/January (CR)

Final pack and book sales figures December/January (LH)

Review of whether any useful data can be gathered in a participating institution library before delivery of a SCOPE product (CR)

Pre-SCOPE book sales figures (LH)

Production report February/March (GP)

Production (including copyright) costs February/March (GP, LH)

Copyright clearance document March (LH)

Student questionnaire March/April (CR)

Focus groups - if appropriate March/April (CR)

Library questionnaire and reports May/June (CR)

Final pack and book sales figures May/June (LH)

May:

- set of model agreements in place

- technical procedures for document delivery, storage, retrieval and dissemination in place

5. Years 2 & 3 Evaluation Outline

We shall continue to evaluate along the lines of Year 1, concentrating on monitoring course pack production and its impact on users and libraries where new institutions, new subjects, and/or different levels (ie. years) of student are involved.

Two further areas are critical:

- on-line delivery

- cost–benefit analysis and assessment of overall demand.

i) On-line delivery: design and implementation.

Issues include:

- payment policy and mechanisms

- user friendliness

- accessibility

- comparison with market for printed course packs

- copyright security.

Full implementation is critically dependent on agreements with a wide range of publishers; however a pilot will be possible using a restricted number of texts for which permissions for electronic delivery have been obtained. The pilot should allow all technical aspects to be tested, but may not contribute to a view of demand, for which we may have to rely on other data.

Activities are likely to include:

- GP to visit all sites and build up picture of IT facilities likely to be available by Year 3 and to discuss the preferences of individual institutions. Conclusions to be discussed in paper to SCOLAR/SCURL.

- User interface to be tested in a pilot towards end Year 2 and reactions sought from library representatives, students and academic staff. The pilot should be tested in several sites.

- A variety of payment scenarios including site licencing will be developed with the help of the Copyright Advisory Panel and consultation will follow with publishers, library representatives and SCURL.

- Using data obtained from student questionnaires and from academic staff, models will be developed to compare the costs and revenues associated with printed packs and on-line delivery of single articles. Data relating to demand for interlibrary loans in relevant subject areas, and particularly from students, should also be used.

ii) Cost – benefit analysis and assessment of overall demand

iii) Also to be monitored:

- success and usefulness of local printing option (in co-operation with at least one other institution and possibly a bookshop). (GP)

- build up of copyright material belonging to academics and/or institutions. Try to determine whether this and use of non-copyright material is a deliberate strategy for avoidance of copyright negotiations with commercial publishers. (Through interviews with contributing academics). (CR)

- documentation of overlap or sharing of material between institutions. (CR/LH)

- documentation of any particular problems associated with taking material directly in electronic form. (GP)

- cost of true on-demand publishing compared with existing print strategy. (GP)

Further details of evaluation activities in years 2 and 3 will be included in PRINCE stage plans.

Go to the SCOPE home page