Gatekeeping or filtering?: Investigating the connection between peer review and research quality

Arguably, the most important opportunity to acquire the standards and norms of the discipline and develop researchers' judgement is the peer review process - but this depends on the quality of the reviews. 'Good' feedback - which we take to mean feedback that has the capacity to improve subsequent practice - has been identified as being timely, specific and relevant. Yet often reviews lack these basic qualities. In this paper we report an investigation of the peer review process at the 2010 Australasian Association of Engineering Education (AAEE) conference. Authors at the conference were given the chance to rate their reviews and we subsequently analysed both the nature of the reviews and authors' responses. Findings suggest that the opportunity to use the peer review process to induct people into the field and improve practice is being missed. As in other disciplines there is also ample evidence that the review process does little or nothing to ensure the standard and relevance of conference presentations. It is therefore legitimate to ask whether there may not be better processes to attain these ends and we conclude with some discussion of how the review process may be made more helpful for everyone involved.