NPC/03/09/A: Response to the Joint Funding Bodies Research and Assessment Review

This section...

Also in Consultations 2003...

by Tim Brown

Please find below our responses to the recommendations as requested. We have presented our answers in this form so that we can communicate our response to our members via electronic means.

Preamble

The National Postgraduate Committee (NPC) is a charity with the aim to advance, in the public interest, postgraduate education in the UK. We organise meetings and conferences, publish best practice guidelines and seek to influence public policy on all aspects of postgraduate education. Our membership consists of affiliated student representative bodies from across the UK; we have one full-time officer, the General Secretary, and fourteen voluntary officers. We work closely with the National Union of Students and the lecturers unions as well as other bodies relevant to postgraduate education.

Recommendation 1 (see paragraphs 113-116 of the review)

Any system of research assessment designed to identify the best research must be based upon the judgement of experts who may, if they choose, employ performance indicators to inform their judgement.

Do you agree or disagree with recommendation 1? Place a cross beside the appropriate answer: Strongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree? Please give comments.

We strongly agree

Based on previous responses we argue that any research should have a peer review to validate its contribution to the international research community.

Recommendation 2 (see paragraphs 117-126 of the review)

a. There should be a six-year cycle.
b. There should be a light-touch mid-point monitoring. This would be designed only to highlight significant changes in the volume of activity in each unit.
c. The next assessment process should take place in 2007-8.

Do you agree or disagree with recommendation 2? Please answer the same as for recommendation 1 for each of the three above points.

We agree with point (a), strongly agree with point (b) and agree with point (c).

Under the new forms of assessment we agree that increasing from 5 years to 6 years would help accommodate the extra aspects of the assessment particularly with regard to the assessment of research competences, which we see as important to postgraduate researchers.

Recommendation 3 (see paragraphs 127-133 of the review)

a. There should be an institution-level assessment of research competences, undertaken approximately two years before the main assessment.
b. The competences to be assessed should be institutional research strategy, development of researchers, equal opportunities, and dissemination beyond the peer group.
c. An institution failing its assessment against any one of the competencies would be allowed to enter the next research assessment but would not receive funding on the basis of its performance in that assessment until it had demonstrated a satisfactory performance.

Do you agree or disagree with recommendation 3? Please answer the same as for recommendation 1 for each of the three above points.

We agree with point (a), strongly agree with point (b) and neither agree nor disagree with point (c)

We certainly welcome the introduction of research competences within institutions and would encourage these to be managed at institutional level so there is consistency in the individual departments. With regards to development of researchers and equal opportunities especially we strongly feel that there is still a considerable level of reform to take place in a number of academic institutions. Based on this we would question the fairness of holding back funding for institutions when this could be at the expense of losing research. We would, however, point out that institutions should be demonstrating evidence that reforms are being made with a strict time plan.

Recommendation 4 (see paragraphs 134-155 of the review)

a. There should, in principle, be a multi-track assessment enabling the intensiveness of the assessment activity (and potentially the degree of risk) to be proportionate to the likely benefit.
b. The least research intensive institutions should be considered separately from the remainder of the HE sector.
c. The form of the assessment of the least research intensive institutions would be a matter for the relevant funding council.
d. The less competitive work in the remainder of institutions should be assessed by proxy measures against a threshold standard.
e. The most competitive work should be assessed using an expert review assessment similar to the old Research Assessment Exercise.

Do you agree or disagree with recommendation 4? Please answer the same as for recommendation 1 for each of the five above points.

We agree with all the above points apart from point (d) where we disagree

Although we identify this new research model will help to create a better assessment method for the different kinds of research unit, we still raise some concern over emerging research. We would consider proxy measures in this instance to be used heavily and would wish to see appropriate measures introduced that identify the value of the emerging research.

Recommendation 5 (see paragraphs 156-171 of the review)

a. The output of the Research Quality Assessment should be a quality profile indicating the quantum of one star, two star and three star research in each submission. It will not be the role of the assessment to reduce this profile to summary metrics or grades.
b. As a matter of principle, star ratings would not be given to named individuals, nor would the profile be published if the submission were sufficiently small that individual performance could be inferred from it.
c. Panels would be given guidelines on expected proportions of three star, two star and one star ratings. These proportions should normally be the same for each unit of assessment. If a panel awarded grades which were more or less generous than anticipated in the guidelines, these grades would have to be confirmed through moderation.

Do you agree or disagree with recommendation 5? Please answer the same as for recommendation 1 for each of the three above points.

We agree with points (a) and (b) and neither agree nor disagree with point (c)

While we welcome the point that this method will now evaluate the density of research active staff we do find point (c) possibly indicates that research panels will be only motivated to generate enough research active staff to secure funding. We would encourage an assessment being taken against the necessary level of research active while also maintaining the necessary teaching activity.

Recommendation 6 (see paragraphs 172-197 of the review)

a. There should be between 20 and 25 units of assessment panels supported by around 60 sub-panels. Panels and sub-panels should be supported by colleges of assessors with experience of working in designated multidisciplinary thematic areas.
b. Each panel should have a chair and a moderator. The role of the moderator would be to ensure consistency of practice across the sub-panels within the unit of assessment.
c. Each panel should include a number of non-UK based researchers with experience of the UK research system.
d. The moderators of adjacent panels should meet in five or six super-panels whose role would be to ensure consistency of practice between panels. These super-panels should be chaired by senior moderators who would be individuals with extensive experience in research.

Do you agree or disagree with recommendation 6? Please answer the same as for question 1 for each of the four above points.

We neither agree or disagree for all points

We are not in a position to identify how effective this method of assessment will be within academic structures. However, we do identify that there is an introduction of peer assessment on an international level. However, it does concern us that the assessment of sub-panels will largely be from people outside the area. It would therefore be desirable for sub-panels to present evidence of recognition of their work within their research community.

Recommendation 7 (see paragraphs 198-204 of the review)

a. The rule that each researcher may only submit up to four items of research output should be abolished. Research Quality Assessment panels should have the freedom to define their own limits on the number and/or size of research outputs associated with each researcher or group.
b. Research Quality Assessment panels should ensure that their criteria statements enable them to guarantee that practice-based and applicable research are assessed according to criteria which reflect the characteristics of excellence in those types of research in those disciplines.

Do you agree or disagree with recommendation 7? Please answer the same as for question 1 for each of the two above points.

We neither agree or disagree with both points.

Again, we would consider ourselves not in a position to be able to comment on these areas.

Recommendation 8 (see paragraphs 205-213 of the review)

a. The funding councils should work alongside the subject communities and the research councils to develop discipline-specific performance indicators.
b. Performance against these indicators should be calculated a year prior to the exercise, and institutions advised of their performance relative to other institutions.
c. The weight placed upon these indicators as well as their nature should be allowed to vary between panels.

Do you agree or disagree with recommendation 8? Please answer the same as for question 1 for each of the three above points.

We strongly agree with all the above points.

Recommendation 9 (see paragraphs 214-234 of the review)

a. Where an institution submits to Research Quality Assessment in a sub-unit of assessment all staff in that sub-unit should become ineligible for the Research Capacity Assessment, even if they are not included in the Research Quality Assessment submission.
b. The funding councils should establish and promote a facility for work to be submitted as the output of a group rather than an individual where appropriate.
c. The funding councils should consider what measures could be taken to make joint submission more straightforward for institutions.
d. Where an institution submits a sub-unit of assessment for Research Quality Assessment, no fewer than 80% of the qualified staff contracted to undertake research within the sub-unit of assessment must be included in the submission.
e. All staff eligible to apply for grants from the research councils should be eligible for submission to Research Quality Assessment.

Do you agree or disagree with recommendation 9? Please answer the same as for question 1 for each of the five above points.

We strongly agree with point (a) and agree with all other points.

We would further emphasise that a sub unit should be assessed either by the RQA or the RCA, mixing the two would cause extensive problems.

Recommendation 10 (see paragraphs 235-238 of the review)

Each panel should consider a research strategy statement outlining the institutions plans for research at unit level.

Do you agree or disagree with recommendation 10? Please answer the same as for question 1 for the above point.

We agree with this point.

Question 11 Burden for institutions

The review proposals have been designed to make the burden of assessment proportionate with the possibility of financial reward. Do you agree that this has been achieved? Please answer the same as for question 1 for the above point.

We neither agree nor disagree

Question 12 Value of research assessment

What value do you place on the research assessment if the financial reward is likely to be small? Would it be high, medium or low? Please add any comments

We say high.

While there are cost implications, we still identify the need to recognise truly world class research. Although the main aim of the RAE is to define a means to obtain funding, it is inevitable that the assessment results will partly be used internationally as a means to identify quality research. Prospective postgraduates will also consider the research quality results to be of significant importance to them.

Question 13 Equality of opportunity for all groups of staff

The funding councils wish to promote equality of opportunity for all staff regardless of age, sexual orientation, political belief, disability, gender, race or religion and seek to ensure that its research assessment policies are compatible with this objective. How successful do you consider that the proposals of the research assessment review are in this respect? Are they very successful, Successful, Neither successful nor unsuccessful, Unsuccessful, Very unsuccessful? Please add any comments.

We would say neither successful nor unsuccessful.

We have identified that human resources and postgraduate recruitment in many higher education institutions are largely of medieval nature compared to commerce and industry. It would be of added benefit for the research and assessment review to consider standards of good practise from which institutions will be expected to reform. This would then create a more rigorous assessment.

Question 14 Overall approach of the review

Notwithstanding your views on any specific recommendations, and given the responses to the earlier Invitation to contribute, do you agree or disagree with the broad approach taken by the review to the question of research assessment? Please answer the same as for question 1 for the above point.

We strongly agree.

Question 15 Further comments

Our main reason for responding to this consultation is the importance of the research and assessment in terms of the value it will be to the development of researchers as well as the accessibility of quality research to postgraduates. We identify that despite lower levels of funding that will be available long term, this method of assessment has reformed significantly to ease the burden on less research intensive units and emerging research.

We would strongly promote the plans to assess equality of opportunity and the development of individual researchers. As well as this consultation we will respond to the Improving Standards in Research Degree Programmes, which we feel has been led largely under a different management. It is of vital importance to postgraduates that these two consultations bring results that complement each other. Research quality and standards in research degree programmes are two separate entities and it is of vital importance that an individual postgraduate has access to both to bring out true development in their research.