Guidance note: External referencing (including benchmarking)
Documents
Providers should note that Guidance Notes are intended to provide guidance only. They are not definitive or binding documents. Nor are they prescriptive. The definitive instruments for regulatory purposes remain the TEQSA Act and the Higher Education Standards Framework as amended from time to time.
What does external referencing encompass?
In the context of the Higher Education Standards Framework (Threshold Standards) 2015 (HES Framework), external referencing means a process through which a higher education provider compares an aspect of its operations with an external comparator(s) e.g. comparing the design of a course of study and/or student achievement of learning outcomes with that of a course from another provider.
The purposes of external referencing are varied, but typically include:
- providing evidence of the quality and standing of a provider’s operations
- offering an external evidence base as context for the development of internal improvements, especially to student outcomes
- establishing or fostering collaborative improvement efforts across providers.
Monitoring, review and improvement processes can and should encompass review against comparators, both internal to the provider and external.
A number of approaches and techniques can be used for external referencing, such as benchmarking, peer review and moderation. Benchmarking is perhaps the most elaborate form of external referencing and typically consists of focused improvement through relationships with a benchmarking partner or partners, but can also include comparing course design against publicly-available information and market intelligence. Further detailed guidance on typical benchmarking practices is given in the Appendices A and B below.
The main focus of external referencing in the HES Framework is on comparisons of courses or units of study and of student achievement to inform improvements. This does not preclude a provider from undertaking much broader comparative activities across any aspect of its operations, including through benchmarking.
The purposes and forms of external referencing relied on by a provider should be reflected in the provider’s policy framework or otherwise documented (e.g. in quality assurance processes).
Relevant Standards in the HES Framework
The Standards primarily concerned with external referencing are those relating to:
- monitoring, review and improvement as part of institutional quality assurance at Standards 5.3.1, 5.3.4 and 5.3.7 and
- the specification of learning outcomes at Standards 1.4.1, 1.4.3 and 1.4.4.
There is also a requirement for external referencing for providers that are seeking self-accrediting authority for their courses of study (Part B2, Section 2c). The Standards for learning outcomes (Section 1.4) and course design (Section 3.1) require the course and its learning outcomes to be consistent with the level of qualification involved, thus requiring referencing against the level classifications of the Australian Qualifications Framework (AQF). External referencing can also be linked to the provider’s representation of itself and its offerings (Section 7.1). Data on student progress that is needed to inform both internal and external referencing is required to be collected (Standard 1.3.5) and each provider’s repository of information about its operations and courses (Standard 7.3.3) is likely to be an enabler of external referencing in the sector.
Intent of the Standards
The intent of the Standards is to ensure that providers inform their own operations, particularly teaching and learning, through external comparisons. This provides an insight into the standing and quality of a provider and its offerings within the sector in one or more respects. It also provides a rational evidence base to guide and focus improvement initiatives. The Standards mostly contemplate national and international comparators (e.g. courses and data), although this does not preclude useful comparisons within a provider (see above). External referencing is expected to inform improvements (see Standard 5.3.7), and to provide a backdrop to the monitoring of student success and the quality and validity of assessments (Standard 5.3.4).
While the intent of the Standards is to encourage external comparisons and build a provider’s perspective of other offerings and other ways of operating, the Standards are not intended to be an instrument of ‘standardisation’. While providers are expected to be aware of other offerings and approaches, they are also expected to tailor their offerings to:
- their circumstances
- their mission
- their scale
- the particular needs of their students
- their chosen market positioning.
Risks to quality
A provider that does not undertake external referencing, aside from not meeting the requirements of the HES Framework as required by the Tertiary Education Quality and Standards Agency Act 2011 (TEQSA Act), is at risk of being increasingly insular and inward-looking with a diminished evidence base to inform decision making.
The provider may also be vulnerable in relation to backing any claims about its offerings in the market and/or student success. Failure to undertake external referencing may limit awareness of recent advances or leave a provider ‘reinventing the wheel’ when an external reference may have been equally suitable, perhaps at lower cost. A lack of international perspective also jeopardises any claims that may be made about the quality and standing of Australian higher education.
Failure to undertake some form of peer review of assessment that is fit for the specific purpose may result in local grading and marking outcomes tracking away from standards in the wider academic community.
What TEQSA will look for
This part of the guidance note covers the full extent of the Standards, and corresponding evidence that TEQSA may require, in relation to external referencing.
For new applicants seeking initial registration and course accreditation, TEQSA will require evidence to be provided in relation to all relevant Standards.
For existing providers, the scope of Standards to be assessed and the evidence required may vary. This is consistent with the regulatory principles in the TEQSA Act, under which TEQSA has discretion to vary the scope of its assessments and the related evidence required. In exercising this discretion, TEQSA will be guided by the provider’s regulatory history, its risk profile and its track record in delivering high quality higher education.
TEQSA’s case managers will discuss with providers the scope of assessments and evidence required well ahead of the due date for submitting an application.
The evidence required for particular types of application is available from the Application Guides on the TEQSA website.
Providers are required to comply with the Standards at all times, not just at the time of application, and TEQSA may seek evidence of compliance at other times if a risk of non-compliance is identified.
TEQSA will expect providers to be able to demonstrate that the external referencing required by the HES Framework has been undertaken and that the objects and processes employed for referencing are credible, tenable and recent. A provider could clarify the positioning of its offerings through undertaking comparisons against both peers and other providers that have achieved higher levels of good practice. Where both national and international comparators are required but claims are made that one or the other is not available, TEQSA will need to be convinced that useful comparators do not exist.
External referencing should encompass not only course design and methods of assessment, but also student achievement of learning outcomes, including through cohort analysis of student performance data (such as analysis of variations in retention, progression and completion rates by entrance pathway, international students compared with domestic students, different fields of education etc.). Peer review of assessment is a valuable means of validating that grades awarded reflect the level of student attainment, including through calibration of different markers’ grading (refer to Standard 5.3.4).
Where the HES Framework requires external referencing to inform improvement planning and processes, TEQSA will need to be satisfied by examples that this is the case and will continue to be so. The more the provider’s operations are informed by well-established external referencing that flows through to improvement processes, the greater TEQSA’s confidence in the provider is likely to be. This may lead to expedited assessment of applications of renewal of course accreditation.
Providers that are seeking self-accrediting authority will need to satisfy TEQSA that the external referencing required by Part B2, Section 2c is well established in the provider’s review and improvement processes, and likely to be transferrable to new courses or levels of study.
Resources and references
FlipCurric, Online guide for assessment practitioners.
Shapland, N. (2011), Assessment Matters!, Griffith Institute for Higher Education.
Innovative Research Universities, Academic Calibration Project
Longden, B., & Yorke, M. (2009), Institutional research, Perspectives: Policy and Practice in Higher Education, 13(3), 66-70.
McAllister, S., Lincoln, M., Ferguson, A., Davidson, B., Hill, A., Davenport, R., et al. (2011). The benchmarking COMPASS® database: A confidential interactive web based strategy to benchmark learning outcomes, paper presented at the Australian Universities Quality Forum: Demonstrating Quality, Melbourne.
Meade, P. H. (1998), A guide to benchmarking, Dunedin, New Zealand: University of Otago.
Office for Learning and Teaching Resource Library[1].
Office for Learning and Teaching, Peer Review of Assessment Networks.
Peer Review Portal. The Peer Review Portal is an online support mechanism which individuals, education providers, industry, networks and professional associations in meeting national and international standards in external peer review. The Portal is an optional support mechanism that higher education providers can use for external referencing activity and reporting.
TEQSA (2016) Guidance Note on Course Design, Learning Outcomes and Assessment.
TEQSA welcomes the diversity of educational delivery across the sector and acknowledges that its Guidance Notes may not encompass all of the circumstances seen in the sector. TEQSA also recognises that the requirements of the HESF can be met in different ways according to the circumstances of the provider. Provided the requirements of the HESF are met, TEQSA will not prescribe how they are met. If in doubt, please consult your TEQSA case manager.
|
Version # |
Date |
Key changes |
|---|---|---|
|
1.0 |
April 2015 |
|
|
2.0 |
19 August 2016 |
Updated for the HESF 2015 and made available as beta version for consultation. Replaces previous guidance note on ‘Benchmarking’. |
|
2.1 |
23 August 2016 |
Incorporated additional feedback. |
| 2.2 | 11 October 2017 | Addition to ‘What will TEQSA look for?” text box. |
| 2.3 | 19 July 2018 | Inclusion of information on Peer Review Portal. |
| 2.4 | 28 March 2019 | Updated as a result of consultation feedback. |
| 2.5 | 16 April 2019 | Resources and references section updated to include link to IRU Academic Calibration Project. |
Appendix A
Benchmarking
Purpose
This appendix provides additional guidance drawn from typical practices; it is not intended to be prescriptive or a template.
What is benchmarking?
Benchmarking can be defined as a structured, collaborative learning process for comparing practices, processes or performance outcomes. Its purpose is to identify comparative strengths and weaknesses, as a basis for developing improvements in academic quality or performance. Benchmarking can also be defined as a quality process used to evaluate performance by comparing institutional practices with identified good practices across the sector.
It is important for higher education providers at the outset to identify the purpose of benchmarking. The purpose of benchmarking is not to standardise all courses and all assessment outcomes, but to reveal variations, and establish whether those variations arise from the individual nature of the courses or the student cohorts, or from variations in quality or academic standards.
Partnering
Benchmarking is typically conducted via a mutually beneficial relationship with one or more institutional partners on terms agreed by the parties involved. Such relationships are typically envisaged to be enduring rather than confined to a single comparative exercise. Depending on the nature and goals of the relationship, the costs, benefits and improvement efforts involved in benchmarking may be shared.
Characteristics of benchmarking in higher education
The broad types of benchmarking include:
- Organisational benchmarking in which comparisons are made at the organisational level (institution, faculty/department, school, course and unit levels).
- Course benchmarking of course design and student performance.
- Process benchmarking involving comparisons of particular processes and practices e.g. of cycle times, efficiency.
- Outcomes benchmarking relating to the comparison of outcomes data, especially student outcomes such as attrition and completion rates.
- Best-practice benchmarking in which the provider selects a comparator thought to be at the forefront in the area to be benchmarked.
Institutional benchmarks
Institutional benchmarks might include minimum acceptable entry criteria, progression and completion rates, grade distributions, criteria for academic appointments and many others, several of which may be embedded in academic policy frameworks.
Higher education providers may use one or more different types of benchmarking at the same time. The HES Framework does not prescribe that a provider should undertake all or any of these types of formal benchmarking, although some are the subject of external referencing requirements.
‘Apples and oranges’
It is sometimes argued that it is only meaningful to compare like with like, i.e. ‘apples with apples’. On the other hand, little may be learned from such a comparison. With some provisos, more is likely to come from a comparison of differences e.g. a more advanced and a less advanced process. In some cases, e.g. for process benchmarking, it may even be helpful to benchmark outside the sector, e.g. an admission process may be informed by comparisons with a hotel or hospital.
Benchmarking exercises
Benchmarking exercises will characteristically investigate the interaction of processes and outcomes.
Before embarking on a benchmarking exercise, providers should consider:
- who are the prospective/suitable partners?
- what will be measured?
- how will results be compared between the partners?
Choices or trade-offs will need to be made between the breadth (number of partners) and depth of assessment, as it will be more feasible to investigate more deeply with a smaller number of well-matched partners. Some providers will prefer to benchmark with partners at the level they aspire to rather than with their peers.
In some parts of the sector it may be difficult to establish partnerships because of competitive tensions. Peak bodies (e.g. Council of Private Higher Education, Australian Council for Private Education and Training) are playing an increasing role in assisting providers in these circumstances.
For providers considering benchmarking that is more investigative in nature and which goes beyond the sharing of data, there are six commonly adopted key phases, which are explained further in Appendix B.
As outlined above, benchmarking is more than the comparison of data. It is a process of continuously monitoring and evaluating the quality and effectiveness of a provider’s operations.
The HES Framework does not prescribe any particular process, but below are some indicative elements that could contribute to meeting the expectations for external referencing:
- identify areas for improvement and areas of good practice
- evidence could include benchmarking reports
- analyse the reasons for any variation or commonality
- evidence could include benchmarking reports and follow-up interviews
- formulate improvement strategies
- evidence could include action plans or elements of other plans
- report on the results and analysis of benchmarking, which should be internally considered by the appropriate governance body or person
- evidence of consideration could include minutes of meetings, emails, or file notes
- implement the agreed action plans, and
- review the outcomes of the implemented actions, both against the expected outcome and against subsequent benchmarking results
- evidence could include progress reports back to managers and governance bodies.
Institutional context
For benchmarking to be successfully implemented in higher education providers, it has to become the ‘way things get done’. Institutional processes need to support benchmarking exercises, including policies and procedures, and adequate resourcing needs to be considered, including administrative and management support. For benchmarking to be a key strategy for institutional organisational change, it needs to be endorsed and supported by senior executive leadership. For benchmarking exercises to be truly successful, there has to be a bottom-up empowerment where there is a sharing of expertise. Collaboration and openness are key ingredients for effective benchmarking exercises. Benchmarking then becomes a shared conversation and a form of peer development, as well as a mechanism to drive institutional change and quality improvement.
Appendix B
Six phases of benchmarking
Purpose
This appendix provides additional guidance drawn from typical practices; it is not intended to be prescriptive or a template.
Phase 1: Concept and scope
This phase is about deciding what type of benchmarking to use, how to undertake it and with whom. Providers need to have a shared commitment and understanding of benchmarking goals.
Phase 2: Plan & design
This phase is critical in identifying and agreeing on the benchmarking project’s objectives, scope (as well as what is not in scope), schedule, allocation of tasks, timelines and deliverables and development of a project plan, project team, reference group, and communication plan.
Phase 3: Self-review
This phase includes:
- planning self-review activities
- identifying key stakeholders to answer particular sections
- writing up institutional context statements for peer review workshop
- collecting evidence from as many sources and stakeholders as possible
- triangulating the evidence using focus groups, surveys and interviews to gather data for analysis and consideration.
Phase 4: Peer review
A common and useful element of the review process is to conduct a peer review workshop (face-to-face and/or online) with the benchmarking partners comparing processes and data. The key aims of the workshop are:
- to identify areas of good practice
- to identify areas for improvement
- to identify areas for sharing and collaboration.
The peer review workshop is also an external validation and calibration exercise to assist providers to understand the quality of their processes.
Phase 5: Communicate and implement improvements
This phase is about communicating the findings from the peer review workshop and implementing improvements. Discussions need to consider who is responsible for carrying out these improvements and whether these improvements carry significant budget implications.
Reports need to be submitted to the appropriate academic governance body or manager.
Phase 6: Evaluate and review
This phase is about evaluating the effectiveness of the benchmarking exercise. Providers should ask themselves:
- What were the outcomes?
- How useful were they?
- Were they implemented?
[1] This library contains a collection of higher education learning and teaching materials flowing from projects funded by the Commonwealth of Australia, including those from the Australian Learning and Teaching Council.