How TEQSA Rates Risk Indicators
TEQSA’s Risk Assessment Framework (RAF) outlines TEQSA’s approach to undertaking risk assessments of higher education providers, including the set of risk indicators used. This document provides more detail on how TEQSA draws on different types of information to derive ratings for risk indicators.
A TEQSA risk assessment is based on accumulated information and understanding about a provider. This includes the provider’s history, own risk management, and outcomes of regulatory events in particular, as well as sectoral information which can provide insights into common issues and trends. As such, information may be quantitative and qualitative. The following is a summary of information sources that may be drawn on by TEQSA:
- Data accessed from the Department of Education’s national collection ‘HEIMS’
- Data collected through TEQSA’s Provider Information Request (PIR)
- Survey data provided by Graduate Careers Australia
- Provider materials e.g. strategic plan, annual reports, risk management plans
- Regulatory history and interactions between TEQSA and a provider e.g. re-registrations and course accreditations, requests for information
- Providers’ responses to previous risk assessments
- A key aspect in TEQSA’s approach to rating an indicator is to consider the specific circumstances of a provider, which will provide the context for understanding any potential risks that the indicator is designed to capture. Examples of provider context that may be relevant to TEQSA’s rating of an indicator are provided in an information sheet available at TEQSA's Application guides and support section.
- TEQSA also considers the calculated quantitative value of the indicator, based on its published technical definition, and with reference to ‘risk thresholds’. Risk thresholds provide guidance to TEQSA’s risk analysts on areas in which potential risks may be present. Considered together with the individual circumstances and context of the provider, TEQSA derives a risk rating (represented by traffic lights).
- The assessment of individual indicators is then considered holistically to inform an overall risk evaluation. It is possible that certain types of risk can only be identified after combining different observations across indicators. TEQSA also takes into account a provider’s regulatory history and standing in arriving at an overall evaluation of risk.
- Where potential concerns are identified, the findings are shared with the provider, with explanatory notes, so that further context or risk control information can be considered. This may lead to amendments to the risk assessment.
TEQSA has adopted a systematic approach to developing its risk thresholds, which includes consideration of the following dimensions:
Documentation such as past regulatory and quality assurance reports, and providers’ risk management and strategic plans, can provide views on common issues such as attrition and student-staff-ratio. These views are useful to TEQSA in ascertaining the depth and breadth of the issues.
Statistical analysis provides insights on the status quo and trend in the sector, and can shed light on the discriminating power of a proposed threshold. Statistical analysis provides further context as to whether the thresholds are representative of genuine levels of risk.
Methodologies explored include mean, medium, standard deviation, moving average and linear regression, for example.
The experience of applying the thresholds, in terms of percentages of providers rated red/orange before and after contextual considerations, can help to ascertain the efficacy of thresholds. This includes, for example, frequent adjustments across the sector for a particular indicator.
Some indicators may lend themselves to a more absolute setting of thresholds, based on a priori logic, while other indicators may more appropriately focus on activity that varies from other providers. In exercising this judgement, TEQSA does not prescribe a ‘standard operating model’. However, if an observation suggests that a provider’s approach may be significantly different from other similar providers, or the approach carries inherent risks, there is ground for further exploration.
Before a risk assessment cycle is commenced, available sectoral information is used to calibrate the risk thresholds, adopting the above approach.
TEQSA has not released its risk thresholds but shares information about its approach and provides advice directly to the provider about the basis of a risk assessment. This position has been informed through consultation with the sector, with the majority of the feedback against releasing the risk thresholds. Reasons for this position include:
- A risk assessment does not, on its own, result in definitive conclusions in relation to compliance with the Threshold Standards but identifies potentialities
- Ill-informed commentaries surrounding the risk thresholds could be very damaging to the reputational and commercial standing of providers and the sector
- Qualitative information, including contextual and control information, is an important input to TEQSA’s rating of a risk indicator, which goes beyond quantitative risk thresholds
- There is a high risk of third parties misusing the risk thresholds in trying to replicate risk assessments. For example, to rank providers.
The confidential nature of the risk assessment between the provider and TEQSA is also seen as an important component in fostering an open dialogue between the two parties in which potential risks can be openly discussed, supporting effective regulatory processes.