CHMS Response to theRoberts Review of Research Assessment

Annex A

Review of researchassessment: response form

Please complete andreturn as Word attachment to an e-mail, to responses@ra-review.ac.uk. The deadline for responses is 30 September2003.

Response by (name of person or organisation):Council of Heads of Medical Schools..

Corporate response (representing the views of the group ororganisation): Yes

Private response (representing the views of one or moreindividuals): No

Contact in case of queries:

Name: Dr Katie Petty-Saphon, Executive Secretary

Tel: 020 7419 5494

e-mail:katie.petty-saphon@chms.ac.uk

Recommendation 1 (seeparagraphs 113-116 of the review)

Any system of research assessment designed to identify thebest research must be based upon the judgement of experts who may, if theychoose, employ performance indicators to inform their judgement.

Do you agree or disagree with recommendation 1? Place across beside the appropriate answer:

Strongly agree

Agree X

Neither agree nor disagree

Disagree

Strongly disagree

Comments on recommendation 1:

We recognise the general desire for retention of peer-review, but it would have been interesting to see analysis of a variety of metrics and comparison with RAE scores. For instance, is there any data on previous RAE scores and correlation with external peer-reviewed funding, citation indices, etc.? The rationale for substantial use of metrics is efficiency and reduction of potential for games-playing. In the absence of any such analysis, it is difficult to make a judgement on the role of performance indicators.

Recommendation 2 (seeparagraphs 117-126 of the review)

a. There should be a six-year cycle.

b. There should be a light-touch mid-point monitoring. Thiswould be designed only to highlight significant changes in the volume ofactivity in each unit.

c. The next assessment process should take place in 2007-8.

Do you agree or disagree with recommendation 2? Pleaseindicate your views using the grid below:

	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
Point a		X
Point b			X
Point c			X

Comments on recommendation 2:

A six-year cycle is acceptable. The Report does not say what the volume indicators of the mid-point monitoring might be and it is therefore difficult to express an opinion on this proposal. They presumably will be data currently returned by universities and therefore not require any additional work. If not, implementing a mid-cycle exercise will impose major additional work. The Report suggests that funding should not be decreased unless very significant fall-off is detected. There is no mention of increasing funding if increased activity is detected. This should be an option, given the longer time between assessments.

b) New Medical Schools need to be treated differently. As their research capacity develops mid-point monitoring with provisional allocations would be valued.

c) 2007 8 is acceptable only if rules and procedures are clearly defined well in advance. If not, provision should be made further to delay the exercise.

Recommendation 3 (seeparagraphs 127-133 of the review)

a. There should be an institution-level assessment of researchcompetences, undertaken approximately two years before the main assessment.

b. The competences to be assessed should be institutionalresearch strategy, development of researchers, equal opportunities, anddissemination beyond the peer group.

c. An institution failing its assessment against any one of thecompetencies would be allowed to enter the next research assessment but wouldnot receive funding on the basis of its performance in that assessment until ithad demonstrated a satisfactory performance.

Do you agree or disagree with recommendation 3? Pleaseindicate your views using the grid below:

	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
Point a				X
Point b		X
Point c					X

Comments on recommendation 3:

There are major concerns about this proposal. Submission of this data two years before the RAE i.e. in year four - will require preparation in year three the year of mid-point monitoring. Unless this is very light touch, this begins to look like continuous assessment, rather than an increased duration of cycle as proposed in Recommendation 2. Additionally, what criteria will be used? How will equity be guaranteed? How much funding will be withheld on failure of a single or several research competencies? When/how does an institution re-submit? How is it to be judged? Why is funding to be withheld surely if the RQA gives a score, then funding should reflect this?

There is a danger here of mixing up the RAE (a retrospective assessment of research activity) with a laudable desire to encourage good practice in research management. There are other methods of achieving the latter without involving the RAE. This is simply adding to the burden and complexity. Alternatively, if it is felt to be an essential part of a RAE, then it should be incorporated into the RAE proper and not separated out.

Recommendation 4 (seeparagraphs 134-155 of the review)

1. There should, in principle, be a multi-track assessment enabling the intensiveness of the assessment activity (and potentially the degree of risk) to be proportionate to the likely benefit.
2. The least research intensive institutions should be considered separately from the remainder of the HE sector.
3. The form of the assessment of the least research intensive institutions would be a matter for the relevant funding council.
4. The less competitive work in the remainder of institutions should be assessed by proxy measures against a threshold standard.
5. The most competitive work should be assessed using an expert review assessment similar to the old Research Assessment Exercise.

Do you agree or disagree with recommendation 4? Pleaseindicate your views using the grid below:

	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
Point a	X
Point b	X
Point c			X
Point d		X
Point e		X

Comments on recommendation 4:

This is, in principle, sensible. This should allow new medical schools to submit to the appropriate assessment process.

Recommendation 5 (seeparagraphs 156-171 of the review)

1. The output of the Research Quality Assessment should be a quality profile indicating the quantum of one star, two star and three star research in each submission. It will not be the role of the assessment to reduce this profile to summary metrics or grades.
2. As a matter of principle, star ratings would not be given to named individuals, nor would the profile be published if the submission were sufficiently small that individual performance could be inferred from it.
3. Panels would be given guidelines on expected proportions of three star, two star and one star ratings. These proportions should normally be the same for each unit of assessment. If a panel awarded grades which were more or less generous than anticipated in the guidelines, these grades would have to be confirmed through moderation.[1]

Do you agree or disagree with recommendation 5? Pleaseindicate your views using the grid below:

	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
Point a	X
Point b	X
Point c					X

Comments on recommendation 5:

The concept of a quality profile is welcomed, as is the avoidance of publishing individuals scores. A concern is that the definition of work to be assessed is not described. While it is to be left to each panel to decide its criteria (see below Recommendation 7), there is danger that if greatly varying definitions of work are used, it will be impossible to provide comparability between panels.

While comparability is absolutely essential, the idea of achieving this by pre-determining the proportion of 3*, 2* etc, work in all subjects is interesting there will clearly be widely varying amounts of international/sub-national research in different areas. There is ambiguity in the Report about this proposal. Does this mean the proportion of each star will be the same in all subjects, or simply that the methodology for deciding the proportions will be the same? Even allowing that each subject will have different proportions, the idea that the proportion of 3*, 2* etc. is pre-ordained seems to vitiate the whole point of the assessment process. Although panels will be allowed to vary from the guidelines, if they can show good reason (moderation), it is likely that this will be difficult, or at the very least time-consuming.

While one endorses the desire to reduce variation between panels, this device has the potential to reduce the whole exercise to a confirmation of the Funding Councils pre-conceptions. Panels should assess the work presented as objectively as possible, free from pre-determined proportions. The use of moderators and the genuine involvement of international experts between panels (see below) is essential and the best mechanism for reducing inter-panel variation. The difficulty of achieving this is recognised and so steps must be taken to ensure that it actually occurs.

It is disappointing, if predictable, that the Funding Councils have not sought consultation on Sir Gareths recommendations on the value of each star rating.

Providing institutes with information on the value of each star rating would be of value, and welcome. By providing opportunity for prior discussion, this should reduce the anomaly of the recent exercise where there are differing levels of funding attached to the same grading between different UofAs. However, a concern must be that this will tempt each panel (especially in weaker areas) to maximise the funding returns for its own UofA. But, trying to limit this by pre-determining the proportion of star grades in each UofA is open to the objections described above.

Given the centrality of Recommendation 5, it would be helpful to have some worked examples. There is a degree of ambiguity in aspects of this Recommendation, especially the funding outcome.

Recommendation 6 (seeparagraphs 172-197 of the review)

1. There should be between 20 and 25 units of assessment panels supported by around 60 sub-panels. Panels and sub-panels should be supported by colleges of assessors with experience of working in designated multidisciplinary thematic areas.
2. Each panel should have a chair and a moderator. The role of the moderator would be to ensure consistency of practice across the sub-panels within the unit of assessment.
3. Each panel should include a number of non-UK based researchers with experience of the UK research system.
4. The moderators of adjacent panels should meet in five or six super-panels whose role would be to ensure consistency of practice between panels. These super-panels should be chaired by senior moderators who would be individuals with extensive experience in research.

Do you agree or disagree with recommendation 6? Pleaseindicate your views using the grid below:

	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
Point a		X
Point b		X
Point c	X
Point d				X

Comments on recommendation 6:

In principle, these mechanisms of providing consistency may work better than Recommendation 5. No information is given on numbers of panel members and a concern is that proposed overlap and cross-reference will require large numbers and may produce confusion and conflict. Care should be taken to ensure that the existing large number of units for medicine are not further expanded. This is especially relevant in view of Recommendation 7.

CHMS would prefer a simpler system with a main Panel and sub-panels

Recommendation 7 (seeparagraphs 198-204 of the review)

1. The rule that each researcher may only submit up to four items of research output should be abolished. Research Quality Assessment panels should have the freedom to define their own limits on the number and/or size of research outputs associated with each researcher or group.
2. Research Quality Assessment panels should ensure that their criteria statements enable them to guarantee that practice-based and applicable research are assessed according to criteria which reflect the characteristics of excellence in those types of research in those disciplines.

Do you agree or disagree with recommendation 7? Pleaseindicate your views using the grid below:

	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
Point a		X
Point b		X

Comments on recommendation 7:

While this is in general sensible, too large variation in criteria will act against the attempts to minimise inter-panel variation (see Recommendation 5).

Recommendation 8 (seeparagraphs 205-213 of the review)

1. The funding councils should work alongside the subject communities and the research councils to develop discipline-specific performance indicators.
2. Performance against these indicators should be calculated a year prior to the exercise, and institutions advised of their performance relative to other institutions.
3. The weight placed upon these indicators as well as their nature should be allowed to vary between panels.

Do you agree or disagree with recommendation 8? Pleaseindicate your views using the grid below:

	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
Point a				X
Point b				X
Point c			X

Comments on recommendation 8:

While these recommendations appear uncontroversial, there is concern that this may result in pre-judging of the results of the RAE. It is difficult to believe that panels will not be influenced by the bandings.

Additionally, this exercise is to take place one year before the RAE. Thus there will be a volume indicator exercise at year three, a competency assessment at year four, a banding exercise in year five and an RAE in year six. As mentioned in Recommendation 3, this looks like continuous assessment and rather than reducing burdens, this process may well increase both burden and stress.

Furthermore, given that these bandings will guide the choice of institutions about which route of assessment they choose, the criteria and methodology of the performance indicators will require full transparency and consensus.

Recommendation 9 (seeparagraphs 214-234 of the review)

1. Where an institution submits to Research Quality Assessment in a sub-unit of assessment all staff in that sub-unit should become ineligible for the Research Capacity Assessment, even if they are not included in the Research Quality Assessment submission.
2. The funding councils should establish and promote a facility for work to be submitted as the output of a group rather than an individual where appropriate.
3. The funding councils should consider what measures could be taken to make joint submission more straightforward for institutions.
4. Where an institution submits a sub-unit of assessment for Research Quality Assessment, no fewer than 80% of the qualified staff contracted to undertake research within the sub-unit of assessment must be included in the submission.
5. All staff eligible to apply for grants from the research councils should be eligible for submission to Research Quality Assessment.

Do you agree or disagree with recommendation 9? Pleaseindicate your views using the grid below:

	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
Point a		X
Point b			X
Point c		X
Point d				X
Point e		X

Comments on recommendation 9:

Recognising work of a group composed of individuals may give rise to considerable complexity, although it will allow inclusion of individuals who are otherwise not returnable. It should be noted that the ability to transfer staff between the RQA and the RCA processes might in itself increase the possibility of games playing.

It is not clear why only a minimum of 80% of eligible staff must be submitted. Given the Report recognises that games-playing needs to be reduced and that the RAE should provide an accurate picture of a department, CHMS would prefer 100% of staff to be submitted. This would provide an accurate picture, more accurately than one which excludes 20% of the staff. Deciding whom to exclude simply perpetuates the games-playing.

Furthermore, since low-ranking individuals do not contribute stars (or funding), excluding them seems pointless. The total number of stars is the same. If funding is unaffected, then the only justification for setting a minimum limit is if the use of a grade-point average is adopted. This will encourage institutions to reduce the number of individuals in their submissions, thereby increasing their grade-point average and the diminution of individuals, which the Report seeks to avoid.

Recommendation 10(see paragraphs 235-238 of the review)

Each panel should consider a research strategy statementoutlining the institutions plans for research at unit level.

Do you agree or disagree with recommendation 10? Place across beside the appropriate answer:

Strongly agree

Agree X

Neither agree nor disagree

Disagree

Strongly disagree

Comments on recommendation 10:

This is sensible but opens the way to re-duplication since research strategy is being assessed as a competence. It would be preferable for this solely to be part of the RAE. This should also inform the work of the sub-panel.

Question 11 Burdenfor institutions

The review proposals have been designed to make the burdenof assessment proportionate with the possibility of financial reward. Do youagree that this has been achieved? Place a cross by the appropriate answer:

Strongly agree

Agree

Neither agree nor disagree

Disagree X

Strongly disagree

Comments on question 11 burden for institutions:

The burden for HEFCE and panels may be reduced, but the research-intensive institution will have to do at least as much as before. Indeed, since there are to be several more types of assessment (competencies, mid-point monitoring, etc.) arguably the burden for these institutions is increased. Whether this is proportionate to financial reward depends on the financial reward. The unexpected under-funding of the last RAE raises concerns that the vagaries of funding mechanisms will not make the rewards of the new system of RAE any more proportionate to the effort. This particularly applies to medical schools which were disproportionately disadvantaged by the 2001 RAE funding decisions.

Question 12 Value ofresearch assessment

What value do you place on the research assessment if thefinancial reward is likely to be small? Place a cross by the appropriateanswer:

High

Medium

Low X

There is merit in a relatively objective RAE for the UK as it forces institutions to think and plan strategically about the research activity. It also can help in reputation and recruitment and retention of staff and students. However, the converse is true if RAE scores are low. Arguably the RAE has raised research standards in the UK, but this has probably only happened because of the large financial rewards. Institutions found it worth making the effort. If the financial reward is small, the strong likelihood is that much less effort will be expended.

Question 13 Equalityof opportunity for all groups of staff

The funding councils wish to promote equality of opportunityfor all staff regardless of age, sexual orientation, political belief,disability, gender, race or religion and seek to ensure that its research assessmentpolicies are compatible with this objective. How successful do you considerthat the proposals of the research assessment review are in this respect? Placea cross by the appropriate answer:

Very successful

Successful

Neither successful nor unsuccessful X

Unsuccessful

Very unsuccessful

This may risk discrimination against younger researchers and women who have taken time out.

Question 14 Overallapproach of the review

Notwithstanding your views on any specific recommendations,and given the responses to the earlier Invitation to contribute, do you agreeor disagree with the broad approach taken by the review to the question ofresearch assessment? Place a cross by the appropriate answer:

Strongly agree

Agree X

Neither agree nor disagree

Disagree

Strongly disagree

The review has proposed methods which are likely to address the key issues of comparability and continuous distribution of assessment. There are concerns that the burden for research-intensive institutes will rise and whether the proposed system of appointments to panels will provide greater transparency is debatable. The key issue of linkage between funding and RAE is recognised, but other than saying there should be prior clarity, is not addressed. By proposing a minimum of 80% staff return, the opportunity for game-playing is not completely negated.

Question 15 Furthercomments

Question 15 any further comments:

No comment.

---

[1] This consultation questionreflects an edited version of recommendation 5. The recommendation in thereview report also states that thefunding councils should provide institutions with details of the relative value,in funding terms, of one star, two star, and three star research, and ofresearch fundable through the Research Capacity Assessment in advance of theassessment. These ratios might vary between disciplines. In the event that thereview recommendations are accepted, each funding council will develop its ownpolicies for reflecting the assessment results in funding, taking properaccount of Sir Gareths recommendation.