Risk Assessment Framework Consultation Paper
|EFTSL||Equivalent Full Time Student Load|
|ESOS Act||Education Services for Overseas Students Act 2000|
|ESPSE||Education Services (Post-Secondary Education) Award 2010|
|FTE||Full Time Equivalent|
|PIR||Provider Information Request|
|QILT||Quality Indicators for Learning and Teaching|
|RAF||Risk Assessment Framework|
|SES||Student Experience Survey|
|SSR||Student to Staff Ratio|
|TAFE||Technical and Further Education|
|TEQSA||Tertiary Education Quality and Standards Agency|
|TEQSA Act||Tertiary Education Quality and Standards Agency Act 2011|
The Tertiary Education Quality and Standards Agency (TEQSA) risk assessments are a core component of TEQSA’s proportionate, risk-reflective approach to assuring higher education standards and safeguarding student interests. The Risk Assessment Framework (RAF) outlines the main components of the risk assessment cycle and provides detailed information on the risk indicators adopted (TEQSA 2019).
Since TEQSA’s establishment in 2011, the agency has completed six cycles of provider annual risk assessments to date. As the current RAF has been in place since 2014, it is timely to call for a review and consider ways in which the RAF could be improved upon to ensure that it continues to effectively and comprehensively reflect the nature and operation of higher education (HE) providers.
This consultation paper has been devised to be open-ended in order for TEQSA to be able to inclusively incorporate sector views in the next iteration of the Risk Assessment Framework. Through stakeholder feedback, the objectives of this consultation paper are to:
- identify specific changes that can be made to refine the current risk framework
- enhance the validity and usefulness of the risk framework in identifying emergent or existing risks.
This consultation paper discusses the current risk approach and puts forth a number of broad considerations for sector input. The questions raised in this consultation paper aim to guide sector feedback, and providers are encouraged to use this consultation process to submit their responses and ideas to TEQSA.
TEQSA is committed to refining the RAF over time, with experience of applying the RAF in its assessment processes, written feedback from providers, and through consultation with the sector.
TEQSA has established a consultation process to ensure that any proposed changes are carefully explored and considered by both TEQSA and the sector. This paper is part of TEQSA’s broader sector consultation outlined below.
TEQSA has engaged with a number of critical friends across the sector in discussion to help shape the format, content, and focus of the risk assessment workshops (discussed below). These critical friends represent various parts of the sector, such as independent providers, faith-based providers, pathways, TAFEs, and universities. As a result of this consultation process, TEQSA was able to determine the ways in which the risk assessment workshops will be carried out.
This consultation paper is published to provide the higher education sector an opportunity to assist the agency to consider whether its current approach to risk assessment should be amended. TEQSA encourages all stakeholders to make written submissions throughout the consultation period either by responding to the questions posed in this paper or by presenting other relevant considerations for TEQSA to factor into its future planning.
TEQSA has planned a number of Risk Assessment Consultation Workshops beginning in July 2019 and concluding in September 2019. All currently registered higher education providers in Australia will be contacted and invited to attend. These workshops aim to:
- better inform the sector of TEQSA’s current risk assessment approach and process
- allow for open-ended discussions between providers and dialogue between TEQSA and providers on ways in which TEQSA can further enhance its monitoring and assessment of risks.
The consultation process will conclude with the closing of the consultation paper submission period, following which, TEQSA will consolidate all written submissions and ideas proposed by providers during the workshops. TEQSA will endeavour to carefully consider the necessity, viability, and importance of all issues raised through this consultation process before devising a plan for feedback implementation. Given the open-ended nature of this consultation, TEQSA will release further updates on the implementation timeframe in due course.
The risk assessments assist TEQSA to give effect to its principles of reflecting risk, proportionality and necessity in regulating the sector. More specifically, they indicate TEQSA’s assigned risk levels for each provider and help guide TEQSA's focus in undertaking its regulatory activities.
TEQSA’s first risk assessment cycle started in 2013 with 46 risk indicators across three broad areas. Currently, there are 11 risk indicators in the current RAF which are applied in combination to provide an overall evaluation against Risk to Students and Risk to Financial Position. Each risk indicator relates to the Higher Education Standards (Threshold Standards) 2015 under the TEQSA Act and/or the National Code of Practice for Providers of Education and Training to Overseas Students 2018 under the ESOS Act (as outlined in Appendix 1 of the RAF).
TEQSA uses the risk assessments to support its regulatory assessments by identifying potential areas of providers’ practices that may require further analysis and monitoring. Through the risk assessments, TEQSA aims to achieve a number of objectives:
- inform a differentiated approach to evidence and reporting requirements in assessment processes based on risk assessment outcomes, which in turn reduces regulatory burden on providers
- strengthen the protection of students' interests and the sector's reputation by monitoring key aspects of providers' operations during registration periods
- support TEQSA staff and providers to engage in early discussions about emergent issues prior to any scheduled assessment process
- support quality improvement activities through the sharing of information with providers about potential risks and good practices in the sector.
The purpose of the RAF is not to identify all institutional risks or to replace or replicate a provider's internal risk management. The RAF focuses on key risks across the sector that can be readily monitored on a regular basis.
The current RAF has been in place for five years (since 2014) and has proven to be fit-for-purpose, where TEQSA has found correlation between risk ratings and regulatory assessment outcomes (TEQSA 2018). However, considering the ever-changing and diverse nature of the higher education sector, TEQSA remains open to the ways in which the RAF can be further refined in line with the principles of reflecting risk, proportionality and necessity.
It is important to note that the RAF takes into account both quantitative and qualitative factors. While the data for each risk indicator provides a starting point for TEQSA, this is weighed alongside other contextual information which includes: case manager intelligence; more recent regulatory outcomes; providers supplying more contemporary data through the response process; providers’ risk management strategies; monitoring of risks within the provider’s academic board and corresponding actions; and other relevant information. These factors culminate into a final assigned risk rating for each risk indicator and the overall ratings for Risk to Students and Risk to Financial Position1.
In considering the key dimensions and corresponding indicators below, TEQSA seeks feedback on the existing indicators and for changes that may be introduced in the next version of the RAF.
As guidance for feedback, the following are the proposed criteria for providers to consider when putting forth ideas or recommendations for the risk framework:
- agnostic: Indicators should not deliberately discriminate against a provider based on factors such as delivery model, course offering, geographic location, business model, etc
- avoid calculation bias: The indicator needs to be able to be calculated using data which is available on a consistent basis for all providers (or as many as possible)
- minimise burden: The indicators need to be able to be calculated using existing data collections
- commonly accepted: The indicators should be commonly accepted measures of performance within the sector and avoid the use of exotic or untested indicators
- understandable: The indicators and calculation methodology should be easily understood by those in the sector
- inform decision making: The indicators should help to inform (but not be the sole source) Commission decision making, provider case management scoping and provider risk profile.
This considers the current Risk Assessment Framework (RAF) and outlines the underlying rationale for how current risk indicators are applied to examine key dimensions of the operations of higher education providers that contribute to a view of potential risks to academic standards. These areas include: student profile and performance, student outcomes, academic staffing profile, and financial viability and sustainability2.
When crafting the risk assessments, TEQSA applies a standardised set of risk thresholds (low, moderate, high) for each risk indicator. These risk thresholds are confidential, and are approved each year by the Commission in line with TEQSA’s risk appetite. In determining a provider’s risk for each indicator, TEQSA adheres to the established thresholds to form a baseline risk finding before incorporating relevant contextual information that may inform the final risk rating.
Given the confidential nature of the risk thresholds, TEQSA is aware that there have been calls from some parts of the sector for TEQSA to publish this information. If the risk thresholds were to be published in line with enhancing transparency, it would then follow for TEQSA to also publish provider-level data for each risk indicator.
Publishing the risk thresholds and data may facilitate goal-setting in guiding providers’ operations, thereby contributing towards TEQSA’s efforts to increase transparency with the sector.
It is expected that the publication of risk thresholds and data will also allow for easier benchmarking and more effective comparative analysis across providers. It will also offer insights into the higher education sector from TEQSA’s perspective, and in turn provide further clarity on TEQSA’s approach to monitoring risk.
However, it is important to note that the risk assessments are not linear processes that directly translate quantitative measurements into specific ratings, but often involve extended discussions that take into account a range of qualitative considerations and contextual determinants. Publishing the risk thresholds draws attention towards the quantitative component of the risk assessments and may lead to a misinterpretation of ‘risk’. In addition, the thresholds are by no means fixed or static across the risk cycles as the threshold for each indicator may slightly differ depending on TEQSA’s risk appetite.
Presently, the confidential nature of the risk thresholds and provider-level data fosters an environment for open and constructive dialogue between TEQSA and the provider to discuss any risks identified in the risk assessments. However, publishing the risk thresholds and provider data may jeopardise confidentiality and in turn distort the TEQSA-provider dynamic during the risk assessment process.
Moreover, there are concerns around possible gaming of the system. There is a risk that making the thresholds public for each indicator may inadvertently lead to providers directing their attention towards meeting the risk thresholds rather than focussing on continuously improving their delivery of higher education using best practice.
These factors raised are important to take into consideration when assessing whether it would it be appropriate for TEQSA to publish the risk thresholds and provider-level data. If they were published, the potential adverse outcomes would need to be sufficiently mitigated against.
|Discussion Question: Transparency and Confidentiality|
Student performance is examined as a proxy for the extent to which a higher education provider is adequately supporting students through their courses of study and identifying emerging and existing risks to academic performance. Three risk indicators are directly associated to student performance—(first-year) student attrition, progress rates, and completions.
TEQSA also considers any change in student profile by monitoring changes in student load. TEQSA considers the increase in student load as a measure for volatility, as significant growth in student load has the potential to impact on the quality of student experience unless planned for and managed. Unpredicted or rapid changes to the size of a provider are likely to have adverse effects on the quality of education provided, particularly in relation to resulting strains on the level of support and resources available to students.
For student load, TEQSA considers associated factors including, but not limited to, provider size, the provider's strategic planning objectives, student support arrangements and capacity to accommodate and manage changes in student population, as well as trends in student load.
TEQSA also considers a variety of factors that may be relevant towards student performance which include data trends, graduate/ student satisfaction measures, and relevant programs to increase student retention and to assist students to progress and complete their respective courses
Taken together, changes in load and the three student performance indicators function as indications of a provider’s course quality, delivery of higher education, academic standards, and the effectiveness of policies, strategies, and processes in place. More importantly, poor student performance indicates potential issues that warrant a closer look by TEQSA.
|Discussion Question: Student Performance|
The graduate satisfaction and graduate destination indicators are measures of student outcomes and their perceptions of the overall educational experience.
Low graduate satisfaction levels across an institution signal potential issues in relation to the quality of the course, the level of staff and support available to students, the quality of teaching, and adequacy of learning resources. Low levels of graduate destinations (i.e. low full-time employment or further study rates) indicate that students may not be well equipped with the necessary graduate attributes to successfully advance into their chosen profession or further study.
When assessing a provider’s graduate outcomes, TEQSA also takes into consideration relevant contextual factors such as fields of education, provider mission, location, survey sample size and response rates before assigning a final risk rating for these indicators.
Graduate satisfaction is a lag indicator that captures the views of students who have completed their studies. Therefore, TEQSA is considering the relevance of incorporating student experience survey (SES) data for coursework students from Quality Indicators for Learning and Teaching (QILT) as an additional indicator. A student satisfaction indicator based on SES data (or equivalent) will function as a lead indicator that reflects the experiences of currently enrolled coursework students, thereby providing a more comprehensive insight into different aspects of a provider’s overall student and graduate experience.
For graduate destinations, the Graduate Outcomes Survey (QILT GOS) and the Provider Information Request (PIR) Graduate Survey collection are used to determine the percentage of graduates in full-time employment or full-time further study. However, the Australian higher education sector is highly heterogeneous, encompassing pathway providers where students articulate into their partner universities, and providers operating in specific industry contexts where graduates are more likely to engage in casual or part-time work. As such, TEQSA is interested in feedback on alternative ways to measure graduate destinations.
|Discussion Questions: Student Outcomes|
The staffing profiles of providers indicate the levels of staff resourcing and academic support available to students. TEQSA’s Assessment Insights report described over 80% of all adverse decisions made by TEQSA relating to re-registration assessments between 2013 and 2017 referenced the standards covering corporate and academic governance. Strong academic governance and management is highly correlated with the strength of academic leadership and the management and resourcing of academic staff. Three indicators are used to inform a provider’s staffing profile:
- senior academic leaders: the ratio of the number of senior academic leaders to broad fields of education
- student-to-staff ratio: the ratio of onshore coursework student load (equivalent full-time student load, EFTSL) to total onshore teaching staff (full-time equivalent, FTE)
- academic staff on casual work contracts: the percentage of academic FTE employed on a basis other than full-time or fractional full-time to total academic FTE.
Having an adequate level of senior academic staff who are well-engaged in a scholarly community and intellectual climate is core to the teaching and learning functions of providers, which in turn safeguards the quality of higher education delivery. In contrast, the absence or low ratio of senior academic leaders per broad field of education may compromise the provider’s academic capability as senior academic leaders typically make significant contributions to key academic policies for the organisation.
A high student-to-academic-staff ratio (SSR) places potential constraints on the level of support available to students and the average teaching workload. It is important to note that SSR is not proposed here as a proxy for class size. Finally, a high proportion of casual academic staff may decrease the likelihood for these staff to provide a continuity of support for students and to be active contributing members in a community of scholarship.
For providers that are subject to the Higher Education Industry Award, senior academic leaders are defined as academic staff at levels D and above. For providers that are subject to the Education Services (Post-Secondary Education) Award (ESPSE Award), TEQSA defines senior academic leaders as academic staff who are formally employed at Level C but undertake academic leadership roles beyond that of a typical Level C in areas such as curriculum and assessment, pedagogy, staff management, and professional development, research, and/ or scholarship.
In assessing the staffing indicators, consideration may be given to context such as the size and scope of a provider's operations, delivery model and mode, field of education, close institutional relationship with another higher education provider, in addition to strategies in place to mitigate any risks posed by the provider’s staffing profile.
Other than the current indicators, TEQSA is open to alternative measures that may be used to inform the availability and quality of teaching and learning provisions.
|Discussion Questions: Academic Staffing Profile|
The Financial viability indicator assesses potential risk to a provider's current and immediate-to-short-term financial strength and capacity. This is measured by the provider's profitability, liquidity, gearing, debt servicing coverage, and operating cash flow ratio. The Financial sustainability indicator informs a view of a provider’s longer term financial strength and capacity, as well as whether the provider exhibits structural characteristics that support operating endurance. This indicator is measured by revenue changes, assets, employee benefits, enrolments, and revenue diversification. When rating a provider’s financial viability and sustainability, consideration is also given to the provider’s corporate structure and ownership model.
There may be limitations to the current approach to financial analysis that impede a better informed view of a provider’s financial standing. For instance, while providers that are part of a larger group may not have made any investments in physical resourcing, it is possible that their investments are sufficient as asset replacement is made by their parent or related parties. However, there is a lack of systematic way to provide documentation to support contexts such as these.
Identifying additional information is key to improve the accuracy and comprehensiveness of TEQSA’s risk assessment of the financial standing of Australia’s higher education providers. Given the differences between provider types and the varying operating models in resourcing and financial arrangements for providers across the sector, TEQSA is open to considering ways in which the current financial measures may be further enhanced, as well as other indicators that can be applied to better capture providers’ financial viability and sustainability.
TEQSA is also seeking proposals for mechanisms that could be used to facilitate standardised reporting of any proposed additional information or measure without significantly increasing reporting burden on providers.
|Discussion Question: Financial Viability and Sustainability|
- Should TEQSA publish the risk thresholds and provider-level risk indicator data or maintain confidentiality of this information? What are the benefits and disadvantages, and what are some possible ways to mitigate the potential adverse outcomes?
- How can the current student profile and performance indicators be improved to ensure they remain fit-for-purpose? What other measures should TEQSA employ as risk indicators for student profile and performance?
- Would student satisfaction be a more accurate and appropriate reflection of the overall student experience compared to the current Graduate Satisfaction indicator? Alternatively, should both student and graduate satisfaction data be incorporated in the RAF?
- How can TEQSA’s definition of graduate destinations be revised to ensure that it is fit-for-purpose?
- For providers that are subject to the ESPSE Award, is TEQSA’s definition of senior academic leaders sufficiently robust?
- How can the current staffing indicators be improved to ensure that they are fit-for-purpose? Are there other measures that TEQSA can use to monitor risks posed by providers’ academic staffing profiles?
- How can changes be made to the current financial indicators for further enhancements? Are there any other financial measures that TEQSA should consider in its financial analysis without significantly increasing the reporting burden on providers?
This paper is being made available on TEQSA’s website and has been sent directly to peak bodies for providers regulated by TEQSA under the TEQSA Act.
Responses to the questions posed in this paper and other relevant feedback can be provided through the following email address: consultation [at] teqsa.gov.au.
Submissions close at 5.00pm on 20 September 2019.
Interested parties can also email consultation [at] teqsa.gov.au with queries about this paper or to seek clarification of the consultation questions.
Please note that TEQSA intends to publish a summary of submissions received. If you do not wish your submission to be published in full or in part, please indicate this in your response. TEQSA may alter the format or content of submissions before they are published, or decline to publish particular submissions, having regard to the requirements for Australian Government websites.
- Examples of risk controls that TEQSA considers in relation to the risk assessments.
- For definitions and details of risk elements and calculation for each risk indicator, refer to appendices 1 and 2 of the Risk Assessment Framework.