SHEFC

An enhancement led approach to quality assurance

Consultation paper HEC 02/2002

Response by the National Postgraduate Committee

On behalf of the National Postgraduate Committee I am pleased to respond to SHEFCs consultation paper HEC 02/2002 on its new approach to quality assurance in higher education. I would like to thank Peter McColl of Edinburgh University Students Association for his assistance in drafting this submission. I hope you find our comments helpful.

James Groves
General Secretary
National Postgraduate Committee

3/5/02

Executive summary

SHEFC should make additional funding available to institutions to ensure: effective support structures are developed for student representatives; efficient systems are developed for evaluating student feedback; external examiners receive more robust support; and information for public dissemination can be produced without over-burdening staff.

The institutional audit process should focus on: the accuracy, effectiveness and transparency of external examination and student feedback procedures; how well institutions respond to and utilise the information obtained during these procedures; the extent to which each institution is abiding by the QAA Code of Practice; and the extent to which each course complies with the relevant subject benchmark thresholds and (where appropriate) professional accreditation requirements. The audit process should generate commentary and recommendations only, not graded outcomes.

All audit teams should include a student member. Students association should be asked to nominate individuals who would be willing to serve on audit teams; the QAA would then, for each audit, select a student from this pool of nominees.

The publicly available information from institutions should include summaries of external examiners reports and summary statements of periodic programme and departmental reviews.

Institutions should think about how to make effective use of student representatives. Students associations have an essential part to play in developing and training representatives; institutions should make sure associations receive sufficient block grant income for this purpose.

The LTSN will be a vital partner in developing quality enhancement engagements.

Preamble

The National Postgraduate Committee (NPC) is the representative body for postgraduate students in the UK. We organise meetings and conferences, publish best practice guidelines and seek to influence public policy on all aspects of postgraduate education. Our membership consists of affiliated students unions from across the UK; we have one full-time officer, the General Secretary, and fourteen voluntary officers. We work closely with the National Union of Students and the lecturers unions.

The NPC has a Scottish Subcommittee, NPC Scotland, which focuses specifically on matters affecting postgraduates in Scotland. It has its own officers, all of which are current postgraduates in Scottish higher education institutions.

Responses to consultation questions

Issue 1 Internal quality assurance procedures

1A Do you agree with the above characteristics of internal reviews? Are there any other general features of internal review which should form part of the Councils expectations for the sector?

We agree. We have some additional comments on the use of student feedback:

(i) To ensure effective information, it is necessary for institutions to have effective systems of student representation within their internal structures, not just feedback. Student representatives are more likely to supply detailed qualitative information; furthermore they will be able to express the views of those students who do not, for whichever reason, wish to directly offer their feedback to the institution.

(ii) As far as possible, feedback should be obtained from students on each course offered by the institution. In particular its important that the distinct views of postgraduate students within each subject area are obtained, collated and assessed.

(iii) Student feedback should be obtained from students after their course has completed as well as during their course. Students are more likely to speak freely and without intimidation once their course is complete; they also have the benefit of hindsight and a sense of perspective. Ways to obtain this feedback include enhanced first destination surveys and selective use of exit interviews.

1B While recognising that many members of internal review teams are highly experienced, would it be useful to develop appropriate training opportunities for team members? This may be particularly relevant for team convenors and for members from outside the Scottish HE community. If so, who is best placed to deliver such training?

We certainly believe team convenors should receive appropriate training. The QAA would seem best placed to deliver this.

1C How can institutions best keep the Council informed about the progress and outcome of internal reviews? Rather than introduce new reporting processes, it may be appropriate for institutions to provide this information as part of annual planning documents relating to the institutions strategy for continuous improvement of learning and teaching. Alternatively, we could ask the QAA to provide this information based on review documentation which it receives from institutions.

Part of each institutions annual operating statement to SHEFC should include a commitment to produce, on a regular basis, transparent summaries of the outcomes of that institutions internal reviews. In return, SHEFC should make additional funding available to institutions, to ensure such summaries can be produced without over-burdening staff.

We believe it would minimise bureaucracy if these summaries were submitted both to SHEFC (to keep SHEFC informed of the progress and outcome of internal reviews) and the QAA (to enable the QAA to accurately compile evidence for institutional audit), rather than all summaries being sent to the QAA in the first instance, then sent in revised form by the QAA to SHEFC.

1D Do you agree that further value would be added by asking QAA to analyse the reports of internal review on an ongoing basis? If so, what form should this analysis take?

There is certainly merit in enabling the QAA to assess emerging trends and themes within and across subject areas, as this will inform the QAAs ongoing approach to institutional audits. We also feel the QAA could use the reports to develop and publicly promote systems of institutional quality enhancement.

Issue 2 The institutional audit process

2A Do you agree with the proposed scope and remit of institutional audit? Do you agree that the audit process should generate commentary and recommendations rather than graded outcomes?

In our response to HEFCE 01/45, July 2001, we regretted the demise of subject review, believing it had been much maligned. However, we now recognise that comprehensive subject review is a thing of the past. In its place, we feel that revised systems of quality assurance should centre on two vital areas: external examiners reports and students feedback. These ensure the views of, respectively, the higher education community and the consumers of higher education are taken fully into account when assessing each course. We expressed this wish in our response to HEFCE 01/66, November 2001.

The institutional audit process should focus on:

(i) the accuracy, effectiveness and transparency of external examination and student feedback procedures, and how well institutions respond to and utilise the information obtained during these procedures;

(ii) the extent to which each institution is abiding by the QAA Code of Practice;

(iii) the extent to which each course complies with the relevant subject benchmark thresholds and (where appropriate) professional accreditation requirements.

We strongly agree that the audit process should generate commentary and recommendations only, not graded outcomes. The tendency to reduce QAA assessments to a numerical score greatly weakened the reputation of subject review, and generated widespread resentment amongst staff.

2B How can we best ensure that the team can take account of employer perspectives? Do you agree that teams should include a representative of industrial and professional bodies or learned societies?

We find it difficult to see how one could select a representative from an industrial body, professional body or learned society who would reflect any more than a tiny proportion of most institutions intended vocational outcomes. The only exceptions are, perhaps, the specialist art, music and drama colleges.

We would rather SHEFC dropped this unattainable goal and, instead, proposed that audit teams should include someone who could simply offer his/her opinions on the extent to which the institution complied with its stated aims in developing students transferable skills and promoting regional and industry-funded initiatives. Industrial bodies, professional bodies and learned societies should be invited to nominate people for possible inclusion within audit teams on this basis.

2C How might the Council best address the international perspective in audits? What form of evidence base might be helpful? Do you agree with the suggestion that audit teams might include someone with international experience?

We agree that quality assurance is becoming increasingly trans-national. One thinks of the Bologna declaration, and developments within European higher education since then on implementing common qualifications frameworks; one also thinks of worldwide initiatives such as the Observatory on Borderless Higher Education. Any new system of Scottish quality assurance should certainly take these dimensions into account.

We believe each audit team should either include, or be able to consult, someone with experience of international developments in higher education, including moves towards adopting common standards for qualifications and quality assurance. This person would not necessarily need to be based outwith the UK; experience is what counts, not present location.

2D How can we best ensure that the audit team can take account of the student perspective? Do you agree that teams should include a student or someone nominated by a student body?

We strongly endorse the proposal that all audit teams should include a student member. The very fact that each team includes a student will, we hope, make institutions think particularly carefully about their systems of student feedback, their mechanisms for student representation, their academic and non-academic student support structures, and the extent to which the students association is included within the institutions decision-making processes.

We believe the most effective way of implementing this proposal is to ask each students association to nominate a small number of individuals who would be willing to serve on an audit team as a student member. The QAA would then, for each audit, select a student from this pool of nominees to serve.

2E How can the Council and the QAA best ensure that audit, and the composition of the audit team, take account of the specific mission and context of each institution, while at the same time providing a consistent approach to the conduct and reporting of audits across all institutions?

Each institution will have its own distinctive mission and offer its own distinctive courses. It will, however, be expected to comply with certain national guidelines, notably subject benchmarking standards and the QAA Code of Practice.

Audit teams will therefore have two classes of documentation against which to judge each institution: the material produced by the institution itself; and the national guidelines issued by the QAA and other bodies. They should also assess institutional procedures against basic principles of openness, transparency and natural justice. The Nolan principles on standards in public life are, for example, wholly relevant to the conduct of higher education institutions.

Issue 3 Public information on quality

3A Do you agree with the principle that the format and scope of public information about educational and quality issues should be principally determined by the needs of students and other stakeholders?

We agree.

3B How can we best balance the needs for standardisation of information with the recognition of institutional diversity?

We commend the final report of the HEFCE task group chaired by Prof Sir Ron Cooke (HEFCE 02/15, March 2002), which, we feel, balances these factors admirably.

It is obviously vital that all publicly available information is easily accessible to the public! Moreover, the public should be able to readily compare reports on courses from different institutions which: are in similar subject areas; or are delivered in the same geographical region; or are at the same level of attainment; or are delivered in similar teaching formats (e.g. distance learning). This will only be achieved if a single portal for accessing the information (maybe housed by HERO) is established, and if a system of reporting is implemented which allows such comparisons to be easily made.

We do not believe this will lead to an excess of numerical ratings or, even worse, league tables, provided enough thought goes into the format for reporting data and the software for comparing data. Developing these areas will, necessarily, require significant funds from SHEFC; we hope these will be forthcoming.

3C How can we best contextualise information on entry qualifications, progression and outcomes?

We believe all such information should be positive, in the sense of recording students achievements and not failures. This does not mean institutional failure should go unnoticed; if significant numbers of students graduate having achieved (on their terms) little, the institution should be held accountable.

3D Do you agree with the proposed distinctions between public and private information?

No. We strongly disagree with your assertion that: we will not ask institutions to publish the reports of internal reviews, or external examiners reports, or summaries of these. How can students and other stakeholders have any confidence in the quality of a course if there is no publicly available information on the courses external assessments? Without selective dissemination of the contents of external examiners reports and major internal reviews, the information will lack any credibility.

We agree with the final report of the HEFCE task group (HEFCE 02/15, March 2002) that the publicly available information should include summaries of external examiners reports on each programme and summary statements of the results of, and the actions taken in response to, periodic programme and departmental reviews.

If English institutions adopt the HEFCE task groups recommendations, while Scottish institutions adopt the suggestions contained within your consultation document, this would adversely affect potential students ability to choose between English and Scottish institutions; it could also make Scottish institutions appear secretive and conspiratorial.

Issue 4 Student involvement in quality processes

4A Do you have any comments on improving the design and use of student feedback surveys?

We refer you to our response to question 1A. We have one additional comment on the design of such surveys. Tick box surveys, though very straightforward to complete, often convey little useful information without more detailed additional components. It is the qualitative reasons why a student feels satisfaction or dissatisfaction that is important, not the quantitative fact that he/she feels it.

4B Do you have any comments about our proposal to develop a long-term longitudinal study of student and graduate cohorts?

We would query Scottish institutions assertion that extending the scope of the current Fist Destination Survey would not be effective. In our view this survey is an under-exploited opportunity to obtain valuable feedback from past students. It would be interesting to know the institutions reasons for asserting this.

In general, however, we strongly encourage the establishment of a longer-term longitudinal study of student and graduate cohorts. We are particularly interested in any data that would analyse and evaluate the value and effect on career progression of postgraduate qualifications, both taught and research.

4C Do you agree with the principle that student representation on course committees should be standard practice?

We believe effective student representation, backed up by induction and training programmes, should be standard practice.

There is little use served if every committee has a space for a student representative, but there are insufficient volunteers willing to fill all the places; similarly it is to no-ones benefit if a student representative ends up being placed on more committees than he/she has the time for. A student representative can only be truly effective if he/she is presented with committee papers well in advance, has been properly briefed on their contents and implications, has sufficient time to consider the papers and, where necessary, discuss them before the meeting, and is treated in a respectful manner by his/her fellow committee members during the meeting.

4D How can institutions involve students more effectively in the management of course delivery? Do you agree with the principle of a national development service to offer support to student representatives?

Institutions should think about how to make effective use of those who are willing to volunteer their free time to represent their fellow students. They should be welcomed and valued. Students associations have an essential part to play in ensuring effective student representation; institutions should make sure associations receive sufficient block grant income for this purpose.

We remain to be convinced that a national development service would be the best way of supporting student representatives and students associations; there is a risk that the training and support on offer would be too generic to be effective. Wed need to see more detailed proposals before taking a firm view, however. In our view, allocating funds to each students association (either directly or via the institution) for the purpose of training and supporting student representatives may represent a better use of SHEFCs resources.

Issue 5 Quality enhancement engagements

5A Are the proposed curriculum groupings appropriate? Are there alternative ways to categorise provision into 4-6 sensible groupings?

They seem reasonable, although doubtless there are other combinations that would also be perfectly acceptable. We urge dialogue with the LTSN subject centres to determine the final groupings.

5B We believe that these broad curriculum groupings can be used to identify issues of generic interest across (say) science and engineering; but there may also be scope and justification for some specific activities of particular interest to (say) biologists or computer scientists. Do you have any comments on the balance between generic and discipline-specific activities within each curriculum group?

We have no comment here.

5C It will be important to select cross-curricular themes which are broad (in order to encompass a wide range of developmental activities) but also specific enough to ensure coherence in the overall running of the theme. Do you have any comments on a methodology for the selection of generic themes?

We have no comment here.

5D Do you agree with the proposal to set up a planning committee for each theme? Do you have any other comments on the organisation of each theme?

We agree with the proposed planning committees, and endorse your suggested composition and timetable for these.

5E How can the Council best ensure that all practitioners have opportunities to participate in these processes?

We endorse your proposal to use the money saved by the ending of universal subject review to allow all relevant practitioners access to quality enhancement activities.

5F We will wish to engage the help of external agencies, particularly QAA and LTSN, in supporting and delivering these engagements. Roles will include supporting the planning committees, organising events, interacting with institutions, disseminating good practice, and maintaining a strategic overview of the engagements as a whole. What roles can best be played by each of these organisations in the enhancement process?

We believe the LTSN will be a vital partner. It subject centres are widely respected within the academic community; they are developing, expanding and starting to show signs of real maturity. The LTSN generic centre has an important role to play in disseminating general best practice.

References

HEFCE 01/45, July 2001: Quality assurance in higher education Proposals for consultation
http://www.hefce.ac.uk/Pubs/hefce/2001/01_45.htm

HEFCE 01/66, November 2001: Information on quality and standards of teaching and learning Proposals for consultation
http://www.hefce.ac.uk/Pubs/hefce/2001/01_66.htm

HEFCE 02/15, March 2002: Information on quality and standards in higher education Final report of the Task Group (Prof Sir Ron Cooke, chair)
http://www.hefce.ac.uk/Pubs/hefce/2002/02_15.htm