TEQSA Regulator Performance Framework Report 2016–2017

With the exception of the Commonwealth Coat of Arms, TEQSA’s logo, any material protected by a trade mark and where otherwise noted, all material presented in this document is provided under a Creative Commons Attribution 3.0 Australia licence.

The details of the relevant licence conditions are available on the Creative Commons website (accessible using the links provided) as is the full legal code for the CC BY 3.0 AU licence.

The document must be attributed as the TEQSA Regulator Performance Framework Report 2016–2017.

Contacts

More information about the Tertiary Education Quality and Standards Agency, including electronic versions of this report, is available online.

Comments and enquiries about this report may be directed to:

Communications
Tertiary Education Quality and Standards Agency
GPO Box 1672
MELBOURNE VIC 3001

T: 1300 739 585
E: comms [at] teqsa.gov.au

Acknowledgements

This report reflects the efforts of many people. Special thanks go to TEQSA staff involved in contributing and coordinating material.

Letter to the Minister

Senator the Hon Simon Birmingham
Minister for Education and Training
Parliament House
CANBERRA ACT 2600

22 November 2017

Dear Minister,

As the accountable authority of the Tertiary Education Quality and Standards Agency (TEQSA), we have pleasure in presenting to you TEQSA’s Regulator Performance Framework Report for the year ended 30 June 2017.

TEQSA’s report has been prepared in accordance with the Regulator Performance Framework (RPF) released by the Commonwealth Government in October 2014. The RPF requires the accountable authority to give a report on the RPF to the responsible Minister for noting.

This report describes the progress made over the course of 2016–17 to meet the performance measures in the RPF, as self-assessed by TEQSA. The report has been reviewed and externally validated by the Higher Education Standards Panel.

In addition, we, as the accountable authority of TEQSA, have certified this report as required by the RPF. In our opinion, the TEQSA Regulator Performance Framework 2016–2017 accurately reflects the performance of TEQSA, and complies with the RPF.

Yours sincerely,

Professor Nicholas Saunders, AO
Chief Commissioner

Professor Cliff Walsh
Commissioner

The Higher Education Standards Panel

Professor Nicholas Saunders
Chief Commissioner
Tertiary Education Quality and Standards Agency
Level 14/530 Collins St
Melbourne VIC 3001

Dear Nick,

The Higher Education Standards Panel is pleased to provide formal external validation of the Tertiary Education Quality and Standards Agency’s (TEQSA) self-assessment of its 2016-2017 performance  as the regulator of higher education in Australia against the Government’s Regulator Performance Framework.

The Higher Education Standards Panel has assessed TEQSA’s Regulator Performance Framework Report 2016-2017 and at its meeting of 10 November 2017 agreed that the report has been prepared in accordance with the Australian Government’s 2014 Regulator Performance Framework.

The Higher Education Standards Panel is therefore happy to externally validate the report prior to it being formally certified by the accountable authority and submitted to the Minister for Education and Training for noting.

Yours sincerely

The Higher Education Standards Panel
Professor Peter Shergold AC, Chair

On behalf of:
Professor Greg Craven AO, Dr Krystal Evans, The Hon Phil Honeywood, Emeritus Professor Alan Robson AO, CitWA, Ms Karen Thomas (Members)

10 November 2017
50 Marcus Clarke Street, Canberra ACT 2600

1. Introduction

1.1 Background

The Commonwealth Government released its Regulator Performance Framework (RPF) in October 2014. The RPF was developed in consultation with a range of stakeholders, and consists of six outcomes-based key performance indicators covering:

• reducing regulatory burden
• communications
• risk-based and proportionate approaches
• efficient and coordinated monitoring
• transparency
• continuous improvement.

Commonwealth regulators that administer, monitor or enforce regulation are required to implement the RPF.

Under the RPF, TEQSA is required to self-assess its performance, and then obtain external validation of that self-assessment. The Higher Education Standards Panel has provided this external validation and the TEQSA accountable authority has certified the report. The report has also been provided to the Minister for Education and Training for noting.

TEQSA’s RPF consists of the following six key performance indicators (KPIs):

1. Regulation by TEQSA does not unnecessarily impede the efficient operation of higher education providers.
2. TEQSA’s communication with higher education providers is clear, targeted and effective.
3. Regulatory actions undertaken by TEQSA are proportionate to the risks being managed.
4. TEQSA’s compliance and monitoring approaches are streamlined and coordinated.
5. TEQSA’s dealings with higher education providers are open, transparent and consistent.
6. TEQSA’s regulatory framework continues to be improved in consultation with stakeholders.

These evidence metrics were published on the TEQSA website. For more information, refer to TEQSA’s Regulator Performance Framework 2015-16 (Version 1.0).

1.2 Basis of self-assessment

The RPF report is based on operational qualitative and quantitative data and the results of a sector-wide survey completed in July 2017. The sector-wide survey was largely based on TEQSA’s Regulator Performance Framework 2015-16 (Version 1.0).

Some of the operational data and survey results also appear in the TEQSA Annual Report 2016–2017.

1.2.1 Operational data

Operational data considered for the RPF includes metrics relating to:

• risk ratings and outcomes of decisions
• decision-making timeframes
• feedback from direct engagement with the sector (for example from provider briefings)
• statistics on website views and document downloads for reports and support materials published by TEQSA.

1.2.2 About the survey

The 2017 survey consisted of:

• one provider-specific version for principal contacts of all higher education providers (those registered as well as those who had submitted initial registration applications), including a section for Vice-Chancellors and Chief Executive Officers (VC/CEO) to offer personal comments
• a brief survey for the operational head of selected peak/professional/student bodies.

The survey was sent to 192 higher education providers (those registered as well as those who had submitted initial registration applications) and 39 relevant peak, professional and student bodies (PPSBs). A total of 143 principal contacts and 28 PPSBs completed the survey. This represents a response rate of 74% for both categories of respondents.

The objective of the survey was to obtain feedback to increase TEQSA’s accountability, better understand its impact on higher education providers, and improve its performance. The use of the survey results in the publicly reported TEQSA RPF is part of TEQSA’s approach to increase its transparency and accountability. Consistent with this approach, a summary of the results of the 2016–17 stakeholder survey will be published in conjunction with the publication of the TEQSA RPF Report 2016–2017, on the TEQSA website.

1.2.3 Reporting period

This report is for the 2016–17 financial year.

1.2.4 Enquiries

For enquiries relating to this report, contact comms [at] teqsa.gov.au.

1.3 Executive summary

For TEQSA’s second survey of stakeholders there were some excellent results, but also some results that have declined since 2016. Various sub-groups within the higher education provider population had quite different views on aspects of TEQSA’s performance. There were some providers (usually large, low risk and self-accrediting) who were extremely satisfied with TEQSA’s performance while smaller, for profit providers were less positive about TEQSA’s performance.

The survey clearly indicated where providers thought TEQSA was doing well (quality and relevance of guidance materials and regulatory information) and where TEQSA could improve (streamlining, speed of response, consultation and case management). The survey gives TEQSA clear guidance on where to focus its service improvement initiatives.

The survey’s five highest rated individual questions by provider principal contacts were as follows:

1. Guidance and Support Materials: The quality of information (94 per cent excellent or good)
2. Guidance and Support Materials: Relevance of information (93 per cent excellent or good)
3. TEQSA Conference: Opportunity to interact with other delegates (92 per cent excellent or good)
4. Guidance and Support Materials: Usefulness of information (91 per cent excellent or good)
5. TEQSA Conference: The quality of speakers (90 per cent excellent or good)

In November 2016, TEQSA held its first conference titled Sharing Excellence: Assuring Quality. The conference was attended by more than 420 delegates and, with a further waiting list of over 200, demand exceeded the capacity of the conference venue. Planning for the second TEQSA conference, to be held in November 2017, commenced in 2016-17 and utilised feedback from the first conference as the basis for an expanded program of engagement with the sector.

TEQSA released 24 revised and new guidance notes to support providers. These included guidance notes on: academic leadership; workplace integrated learning; research and research training; TEQSA and the Australian Qualifications Framework; financial standing; technology enhanced learning; external referencing; diversity and equity; academic integrity; course design, learning outcomes and assessment; and wellbeing and safety.

The survey’s five lowest rated individual questions by provider principal contacts were as follows:

1. Case Management: Knowledge of your organisation’s specific needs / issues / environment (52 per cent excellent or good)
2. KPI 3: The consultative approach taken to confirm the annual risk assessment results with your organisation (53 per cent excellent or good)
3. KPI 4: Timely coordination of TEQSA staff visits to your organisation (53 per cent excellent or good)
4. KPI 2: Timeliness of information provided by TEQSA after TEQSA makes a regulatory decision (54 per cent excellent or good)
5. Case Management: Consideration of your organisation’s specific needs / issues / environment for tailoring the application process (55 per cent excellent or good)

In 2016-17 median processing times for assessments increased as a result of heavy caseloads, increased complexity, and a large increase and carryover of prospective providers applying for registration. Constraints on staffing levels were also a contributing factor. TEQSA will continue to look for further streamlining opportunities to ensure it addresses this issue over the next 12 months.      

Importantly, overall TEQSA was well-regarded by providers and peak bodies as a regulator assuring the quality of Australia’s higher education. The rating of TEQSA’s overall performance for 2017 (80% rated excellent or good) remained very similar to that reported in the 2016 survey (82%).

TEQSA will now develop initiatives in follow up of this performance report for approval by the TEQSA Accountable Authority. These initiatives will inform the TEQSA Corporate Plan for 2018-22.

2. Survey results at a glance

The stakeholder survey results were analysed to produce the top two scores (the proportion of respondents selecting the two most positive ratings points – ‘excellent’ or ‘good’). ‘Don’t know’/‘not applicable’/‘no answer’ responses have been excluded from all top two score calculations.

Consistent with results observed in the 2016 stakeholder survey, TEQSA is well-regarded by its stakeholders with 80 per cent of provider principal contacts rating TEQSA’s performance as ‘good’ or ‘excellent’, and 88 per cent of peak/professional/student bodies rating TEQSA’s performance as ‘good’ or ‘excellent’.

There was a strong response rate (74 per cent) to the survey by provider principal contacts, and the distribution of responses was proportionate to the size of the categories of providers. The survey included registered higher education providers as well as those applying to be registered as higher education providers (Table 1).

Table 1. Provider population by category and responses

CATEGORY	PRINCIPAL CONTACTS POPULATION		RESPONSE SAMPLE
	Freq	%	Freq	%
Australian University	40	20.8	33	23.1
Australian University of Specialisation	1	0.5	1	0.7
Overseas University	2	1.0	2	1.4
Higher Education Provider (HEP)	122	63.5	85	59.4
Prospective Higher Education Provider (HEP)	27	14.1	22	15.4
Total	192	100.0	131	100.0

 

Table 2 illustrates the results for each KPI by provider principal contacts in 2016 and 2017. KPI 4 - TEQSA’s compliance and monitoring approaches are streamlined and coordinated recorded the largest decline between 2016 and 2017 moving from 73.1 per cent to 54.9 per cent. KPI 2 - TEQSA’s communication with higher education providers is clear, targeted and effective was the highest rated KPI. Interestingly providers rated TEQSA’s overall performance higher than any individual KPI.

Table 2. Summary of  provider principal contact top two scores 2016 and 2017

	% OF SCORES AS GOOD OR EXCELLENT
SUMMARY OF TOP 2 SCORES BY KPI	PRINCIPAL CONTACT (2017)		PRINCIPAL CONTACT (2016)
KPI 1 - Regulation by TEQSA does not unnecessarily impede the efficient operation of higher education providers	66.0		74.2
KPI 2 - TEQSA’s communication with higher education providers is clear, targeted and effective	71.7		80.8
KPI 3 - Regulatory actions undertaken by TEQSA are proportionate to the risks being managed	57.0		60.8
KPI 4 - TEQSA’s compliance and monitoring approaches are streamlined and coordinated	54.9		73.1
KPI 5 - TEQSA’s dealings with higher education providers are open, transparent and consistent	67.6		72.5
KPI 6 - TEQSA’s regulatory framework continues to be improved in consultation with stakeholders	60.8		72.5
Overall	79.7		82.3

 

Table 3 presents the results for Peak Professional and Student Bodies (PPSB). With the exception of KPI 6, the results for PPSB respondents are largely consistent or better than those observed in 2016.

Table 3. Summary of Peak Professional Student Bodies (PPSB) top two scores 2016 and 2017

	% OF SCORES AS GOOD OR EXCELLENT
SUMMARY OF TOP 2 SCORES BY KPI	PPSB (2017)		PPSB (2016)
KPI 1 - Regulation by TEQSA does not unnecessarily impede the efficient operation of higher education providers	91.7		93.3
KPI 2 - TEQSA’s communication with higher education providers is clear, targeted and effective	84.6		73.3
KPI 3 - Regulatory actions undertaken by TEQSA are proportionate to the risks being managed	89.5		84.6
KPI 4 - TEQSA’s compliance and monitoring approaches are streamlined and coordinated	85.0		71.4
KPI 5 - TEQSA’s dealings with higher education providers are open, transparent and consistent	88.9		87.5
KPI 6 - TEQSA’s regulatory framework continues to be improved in consultation with stakeholders	66.7		100.0
Overall	87.5		100.0

 

Consistent with the result observed in 2016, a majority of respondents from both the provider principal contact group and the Peak Professional and Student Bodies group view TEQSA’s performance in each KPI as ‘good’ or ‘excellent’.

3. Performance by KPI

3.1 KPI 1 - Regulation by TEQSA does not unnecessarily impede the efficient operation of higher education providers

TEQSA consults with higher education providers, peak industry bodies and government on streamlining regulatory processes and reducing the administrative burden on providers to comply with TEQSA’s requirements. In 2015–16, TEQSA reduced the evidence requirements for low risk providers by extending the risk differentiated Core+ model from re-registration to course accreditation and re-accreditation. In 2016-17 case managers also continued to work closely with providers to tailor the required information/evidence based on a variety of risk and other factors.

WHAT DID OUR STAKEHOLDERS SAY?

For KPI 1, 66 per cent of principal contacts (and 78% from low risk providers) and 92 per cent of peak/professional/student bodies rated TEQSA’s performance as ‘good’ or ‘excellent’.

3.1.1 Performance indicators for KPI 1

For KPI 1, TEQSA has assessed its performance against the following four indicators using both internal metrics and feedback from the stakeholder survey.

3.1.1.1 Evidence of regular, constructive consultation with the sector

TEQSA consults regularly with the higher education sector, in meetings and via written correspondence. The agency meets regularly with peak body representatives including Universities Australia, the Australian Council for Private Education and Training, the Council of Private Higher Education, TAFE Directors Australia, English Australia, the National ELT Accreditation Scheme and the International Education Association of Australia.

TEQSA entered into MoUs with a large number of peak domestic bodies in 2016-17. These are listed in Table 4.

Table 4. Memorandums of Understanding signed with peak domestic bodies in 2016–17

ORGANISATION	MoU ESTABLISHED
Australian Medical Council Limited	July 2016
International Centre of Excellence in Tourism and Hospitality Education (THE-ICE)	September 2016
Queensland College of Teachers	September 2016
Australasia Veterinary Board Council Inc. (AVBC)	October 2016
Australian Institute of Quantity Surveyors	October 2016
Teacher Registration Board South Australia	November 2016
The Speech Pathology Association of Australia Limited	December 2016
National ELT Accreditation Scheme Limited	January 2017
Victorian Institute of Teaching	January 2017
Australian Council for Private Education and Training	February 2017
Professions Australia	February 2017
NEAS	February 2017
Council for Private Higher Education	February 2017
Australian Nursing and Midwifery Accreditation Council	February 2017
Australian Osteopathic Accreditation Council	February 2017
Australian Children’s Education and Care Quality Authority	March 2017
Australian Pharmacy Council	April 2017

 

3.1.1.2 A series of reductions in administrative burden already achieved, with further progress planned

In 2015–16, TEQSA reduced the evidence requirements for providers assessed as low risk by extending the Core+ model from re-registration to course accreditation and re-accreditation. This approach was piloted in late 2014–15, and fully deployed in September 2015.

Following extension of this risk-differentiated Core+ model to course accreditation and re‑accreditation in September 2015, all renewal applications assessed in 2016-17 were undertaken in accordance with the Core+ methodology.

In 2016-17 TEQSA explored options to streamline assessments, ranging from trialling abridged case team assessment reports for course accreditation applications from low risk providers to trialling a streamlined approach for assessment of initial registration/course accreditation applications. However, there were limited low risk assessments for course reaccreditations received in 2016-17 to trial the options.

3.1.1.3 Engagement with international agencies to contribute to development of transnational policy

TEQSA plays an important role in protecting, enhancing and promoting the quality and integrity of Australia’s higher education sector in the international sphere. TEQSA is a member of the following international networks:

• International Network for Quality Assurance Agencies in Higher Education
• Asia-Pacific Quality Network
• Council for Higher Education Accreditation International Quality Group
• Quality Beyond Boundaries (QBB) group, a network of international quality assurance agencies from the major sending and receiving countries of cross-border education. The group addresses common challenges by creating a platform to collaborate, share information and best practices and work together to improve quality assurance systems for cross-border higher education.

In 2016–17, TEQSA continued to strengthen links with international quality assurance agencies through these networks, with ongoing discussion on Australian providers, quality assurance and regulation.

During 2016, TEQSA developed an International Engagement Strategy to document how it will work with international stakeholders, government departments and international quality assurance agencies. After discussion with Commissioners and stakeholders, TEQSA publicly released the TEQSA International Engagement Strategy 2016-2020 in February 2017.

The Department of Education and Training approached TEQSA in mid-2016 to manage a project focused on the quality assurance of online higher education for countries in the APEC region. The project deliverables include the development of a Discussion Paper and Toolkit, a workshop on the Toolkit with representatives from a range of APEC economies followed by a number of validations in Indonesia, Vietnam and Mexico, and the completion of a final discussion paper, toolkit and report. These project deliverables have now been completed.

TEQSA signed five MoUs with international bodies during the year, as listed in Table 5.

Table 5. Memorandums of Understanding signed with international bodies in 2016–17

ORGANISATION	MoU ESTABLISHED
Ministry of Education and Training Vietnam	March 2017
Malaysian Qualifications Agency	March 2017
Chinese Service Centre for Scholarly Exchange (CSCSE)	September 2016
Higher Education Evaluation Centre of the Ministry of Education (HEEC), Beijing	September 2016
Council for Private Education, Singapore	July 2016

 

In 2016-17, TEQSA undertook various international activities, including:

• participation in the Asia Pacific Quality Network (APQN) Conference and Annual Meeting in Russia in May 2017
• participation in the Groningen Declaration Network Annual Meeting in Melbourne in April 2017
• hosting of delegations from the Korean Ministry of Education in February 2017 and the Federal Democratic Republic of Ethiopia in April 2017
• attendance at the ASEAN University Network Quality Assurance (AUN-QA) International Conference in Kuala Lumpur and visited the Malaysian Quality Agency in March 2017 to renew the Memorandum of Cooperation with TEQSA
• participation in the International Network for Quality Assurance Agencies in Higher Education (INQAAHE) Conference and Board Meeting in Bahrain in March 2017
• hosting of three staff members from the National Institution for Academic Degrees and Quality Enhancement of Higher Education, Japan, as part of TEQSA’s international quality assurance agency staff exchange program in February 2017
• attendance at the Council of Higher Education (CHEA) and CHEA International Quality Group (CIQG) meetings and other meetings with the Lumina Foundation, the United States Senate Committee on Health, Education, Labor and Pensions (HELP Committee), the American Council on Education (ACE), and the Federal Department of Education in Washington in January 2017
• participation in a global summit in September 2016 on Higher Education in Bengaluru India
• attendance at an Australian-India Qualification and Quality Assurance Workshop in Delhi in September 2016
• visit to the Chinese Education Association for International Exchange (CEAIE) and the China Academic Degrees and Graduate Education Development Centre (CDGDC) in September 2016.

TEQSA hosted the annual meeting of the Quality Beyond Boundaries Group (QBBG) in Melbourne in October 2016. QBBG members focused on addressing common challenges, sharing information and best practices and working together to improve quality assurance systems for cross border higher education.

3.1.1.4 The use of a case management model for regulatory processes allowing tailoring of processes according to circumstances of individual providers

The majority of respondents (90.2 per cent) indicated that TEQSA’s case management approach was very important to their organisation, as shown in Figure 1.

Figure 1. Principal contacts’ rating of the importance of TEQSA’s case management approach

Principal contacts’ rating of the importance of TEQSA’s case management approach	%
Very important	90.2
Somewhat important	7.7
Not important at all	1.4
Don’t know/no answer	0.7

 

Principal contacts made a range of observations across many of the survey questions about the case management approach and the quality of TEQSA case managers. Common themes are summarised here, rather than across the KPIs. Many respondents were very positive, commenting on the professionalism and competence of case managers by name.

A number of other providers (in both principal contact and VC/CEO responses) raised concerns over various aspects of the case management approach. These concerns are consistent with issues identified in the 2016 stakeholder survey, including:

• workload for individual case managers
• lack of consistency of information provided
• turnover of case managers dealing with their organisation
• concerns about capability and understanding of the provider's business.

Figure 2 presents a word cloud summarising the comments from provider principal contacts in relation to how the case management approach can be improved. The single biggest issue identified in the commentary was that TEQSA should endeavour to limit the turnover of case managers and provide more continuity and stability in this area.

Figure 2. Word cloud indicating most common word used by principal contacts in relation to how the case management approach can be improved

Providers that had interacted with a TEQSA case manager in the last 12 months were asked about different aspects of TEQSA’s case management approach, as shown in Table 6. The highest scoring item was responsiveness, with 62.9 per cent rating this as ‘good’ or ‘excellent’. Note that this rating varied greatly when assessed by TEQSA market grouping, with 56 per cent of private for profit providers rating TEQSA as ‘good’ or ‘excellent’ in this regard, while 71 per cent of faith based and 79 per cent of university based principal contacts rated TEQSA’s performance as ‘good’ or ‘excellent’ in this regard.

The item with the largest deviation between TEQSA market grouping was ‘consideration of your needs’ ranging from 83 per cent for university-based principal contacts and 82 per cent for faith-based principal down to 29 per cent for principal contacts from private for profit providers.

Table 6. Percentage of principal contacts rating TEQSA’s case management approach as ‘good’ or ‘excellent’

PC: CASE MANAGEMENT APPROACH	TOP 2 SCORE (%)*
Responsiveness n=140	62.9
Consideration of your needs n=130	55.4
Knowledge of your organisation n=136	52.2

*Includes only respondents who interacted with case managers in the last 12 months.

Top two scores and n counts exclude ‘don’t know’/‘not applicable’/‘no answer’ responses.

3.1.2 Stakeholder survey results

Figure 3 shows the frequency distribution of answers (proportion of respondents choosing a particular answer) for KPI 1 items.

Close to three quarters (73 per cent) of principal contacts rated TEQSA’s performance as either ‘good’ or ‘excellent’ for the item ‘opportunity to give feedback’. Over half (59 per cent) of principal contacts rated TEQSA’s performance as either ‘good’ or ‘excellent’ for the item ‘streamlining to reduce burden’.

Figure 3. Breakdown of responses from principal contacts for KPI 1

 

	Streamlining to reduce burden	Opportunity to give feedback
Excellent	14.0%	23.8%
Good	39.9%	45.5%
Fair	21.7%	20.3%
Poor	8.4%	3.5%
Very poor	7.0%	2.1%
Don’t know/no answer	7.0%	3.5%
Not applicable	2.1%	1.4%

 

 

3.1.3 Common survey themes

In relation to this KPI, principal contacts have raised clear concerns about response times. This concern is consistent with findings reported in the 2016 stakeholder survey.

As reported in the TEQSA Annual Report for 2016-17, the percentage of decisions made within six months for course accreditations, re-accreditations and re-registrations decreased significantly from the previous year. During 2016-17, although the median processing time for applications for accreditation of new courses from registered providers remained almost consistent with 2015-16 at 160 days, the median processing time for applications for re-accreditation of courses increased by 100 per cent from 182 days in 2015-16 to 363 days and for re-registration of providers there was an increase of 54 per cent from 245 days in 2015-16 to 377 days. The median processing time for applications for registration from prospective entrants to the sector also increased by 42 per cent from 245 days to 349 days.

A total of 64 comments (excluding comments like ‘don’t know’ or ‘not had enough experience to answer’) were offered. The most common themes across these comments were:

• case management problems, including turnover leading to lack of continuity and lack of knowledge, not knowing who case managers were and inconsistencies between managers
• limited or no consultation around risk assessments
• positive comments about case management, communication and providers’ relationships with TEQSA
• slow, protracted processes, including release of 2016 risk assessments, some of which led to other issues like re-work.

Of the peak/professional and student bodies, 11 respondents provided comments in relation to KPI 1. Six of these included positive comments about TEQSA’s engagement with the sector, which was seen as consultative, collaborative and supportive. Areas where TEQSA could improve included the timeframes in processing/assessing re-accreditation applications.

3.2 KPI 2 – TEQSA’s communication with higher education providers is clear, targeted and effective

TEQSA actively and openly engages with the higher education sector and as part of these activities TEQSA developed a framework in 2016-17 to guide engagement activities. The engagement framework aligns with the TEQSA Corporate Plan 2016-20 and addresses feedback from the 2015-16 Stakeholder Survey.

TEQSA Commissioners and senior staff are frequently invited to present at higher education conferences and events. In 2016–17, the agency presented at or attended more than 30 events to explain and discuss the agency’s approach to quality assurance and regulation. The agency issued a monthly e-News to the sector in 2016-17, including a special edition focused on the transition to the Higher Education Standards Framework (Threshold Standards) 2015 (HES Framework 2015). As of 30 June 2017, the e-News had a subscriber list of 2300, including students, peak bodies, professional staff, academics and CEOs/vice chancellors.

In November 2016, TEQSA held its first conference titled Sharing Excellence: Assuring Quality. The conference was attended by more than 420 delegates, and, with a further waiting list of over 200, demand exceeded the capacity of the conference venue. Feedback from conference attendees revealed that the sector was keen for more interaction of this type with TEQSA. Planning for the second TEQSA conference, to be held in November 2017, commenced in 2016-17 and used feedback from the first conference as the basis for an expanded program of engagement with the sector.

WHAT DID OUR STAKEHOLDERS SAY?

For KPI 2, 72 per cent of principal contacts (and 85 per cent from low risk providers) and 85 per cent of peak/professional/student bodies rated TEQSA’s performance as ‘good’ or ‘excellent’.

3.2.1 Performance indicators for KPI 2

For KPI 2, TEQSA has assessed its performance against five indicators using both internal metrics and feedback from the stakeholder survey.

3.2.1.1 TEQSA’S decisions are provided in a timely manner, clearly articulating the reasons for decisions

Over 70 per cent of all principal contacts rated TEQSA performance as ‘good’ or ‘excellent’ for providing a reasonable opportunity to address matters relevant to a regulatory decision.

Decisions to register a new provider are included on the National Register of Higher Education Providers (National Register).  Figure 4 shows the external usage made of the National Register website (based on Google Analytics statistics of web traffic). Monthly views of the register ranged from five thousand to eight thousand views each month.

Figure 4. National register views in 2016-17 compared to 2015-16

WHAT DID OUR STAKEHOLDERS SAY?

More than 70 per cent of principal contacts rated the completeness and clarity of information provided about TEQSA’s regulatory decision as ‘good’ or ‘excellent’.

55 per cent of principal contacts rated the timeliness of information provided by TEQSA after a regulatory decision is made as ‘good’ or ‘excellent’.

89 per cent of principal contacts rated TEQSA’s performance as ‘good’ or ‘excellent’ in relation to the quality of information provided on the National Register showing the results of regulatory decisions.

3.2.1.2 TEQSA gives all higher education providers a reasonable opportunity to address matters relevant to a decision by TEQSA before making a decision that affects the provider

Where adverse decisions are made, TEQSA always sends providers the reasons for decision. Providers are advised of proposed adverse decisions and no decision is made until the provider has had a reasonable opportunity to comment.

In some instances, after notification from TEQSA, providers have been able to rectify issues that form the basis of a potential adverse decision. Where issues have been rectified a positive decision may result.

WHAT DID OUR STAKEHOLDERS SAY?

72 per cent of principal contacts rated TEQSA as ‘good’ or ‘excellent’ in terms of being provided with a reasonable opportunity to address matters relevant to a regulatory decision, prior to a final decision being made.

3.2.1.3 Specific consultation with the sector occurs before proposed changes are made to TEQSA’s practices

Consultation with the Australian higher education sector is a crucial component in maintaining TEQSA’s relations with stakeholders and improving quality assurance processes. For more information, refer to the Information Sheet: TEQSA’s Approach to Consultation available on the TEQSA website. A summary of the review process and consultation outcomes is also available.

Consultation undertaken in 2016−17 includes:

• proposed changes to the publication of TEQSA’s decisions. The consultation focused on proposed changes to the frequency and way TEQSA published decisions. TEQSA proposed that a simplified set of principles be adopted, informed by approaches of other Australian Government agencies.
• consultation on TEQSA's External Reporting Program. TEQSA holds a range of information from its data collection, risk analysis and assessment functions which may inform the public about the operations of the higher education sector. This consultation sought feedback on the suitability of a range of areas of information for publication that would be of greatest value to students, providers and the general public.

The TEQSA website is a key tool for communicating important information to the higher education sector and those with an interest in Australian higher education.

WHAT DID OUR STAKEHOLDERS SAY?

59 per cent of principal contacts (and more than 85 per cent of those from low risk providers) rated the opportunity to give feedback on proposed changes to TEQSA’s practices as ‘good’ or ‘excellent’.

3.2.1.4 Comprehensive current guidance material for regulatory policies and processes is publicly available and updated regularly

The transition to the HES Framework 2015 (effective 1 January 2017) was successfully achieved with:

• the development of a comprehensive framework of guidance materials
• a series of briefings for providers
• the development of new application guides based on stakeholder feedback, as well as new online forms and templates.

TEQSA will continue to monitor the impact of the implementation of the HES Framework 2015 on providers during 2017-18.

Figure 5 shows the number of page views of materials related to the HES Framework 2015 transition, based on Google Analytics web traffic statistics. The data reveals a high degree of interest in the information across the year with the contextual overview being the most popular document.

Figure 5. Views of support material for HES Framework 2015 (July 2016 – June 2017)

TEQSA’s guidance and support materials were rated highly, with all top two scores rating between 81 per cent and 94 per cent. The quality of the information and relevance of the information were the highest rating items (94 per cent and 93 per cent).

Table 7. Percentage of principal contacts rating TEQSA’s performance as ‘good’ or ‘excellent’ for guidance and support material

PC: GUIDANCE AND MATERIALS	TOP 2 SCORE (%)*
The quality of the information	94.1
Relevance of information	92.9
Usefulness of information	90.6
Ease of access to that information	82.4
Amount of information	81.2

 

3.2.1.5 All general information that is required by providers is current and publicly available

As well as the Standards Transition guidance notes and application guides mentioned previously, TEQSA published a video on YouTube of the briefing sessions on the Standards Transition, held in April and May 2016.

The TEQSA website contains comprehensive information for providers covering the breadth of TEQSA’s functions, its interaction with providers, good practice and guidance notes and TEQSA’s risk-based regulatory approach.

Information available to providers on the TEQSA website is clearly laid out and organised into sections including:

• TEQSA’s regulatory approach
• the HES Framework
• how to apply under the new standards
• provider obligations
• registration and renewal
• course accreditation and renewal
• TEQSA fees
• Guidance Notes
• TEQSA’s Risk Assessment Framework

WHAT DID OUR STAKEHOLDERS SAY?

More than 76 per cent of providers rated the clarity of online forms and application guides as ‘good’ or ‘excellent’.

3.2.2 Stakeholder survey results

Four of the seven items under KPI 2 rated above 70 per cent. Quality of regulatory information scored highest, with 89 per cent of principal contacts rating this item as either ‘good’ or ‘excellent’. Relevance of information was the second highest scoring item with a top 2 score of 87 per cent.

The items relating to clarity of information (69 per cent), communicating streamlining initiatives (59 per cent) and timeliness of information after making a decision (55 per cent) all rated significantly lower this year.In the chart below, note the relatively high proportion of ‘don’t know’/‘not applicable’/‘no answer’ for some items relating to TEQSA’s communication. This suggests that not all respondents had the specific interaction with TEQSA to provide a rating response for these items. This may not be negative, but simply reflect lack of opportunity or experience.

Figure 6. Breakdown of responses from principal contacts for KPI 2

 

	Timeliness of information after making a decision	Communicating streamlining initiatives	Clarity of information	Completeness of information	Reasonable opportunity to address matters	Relevance of information	Quality of regulatory information
Excellent	8.4%	12.6%	9.8%	10.5%	12.6%	22.4%	22.4%
Good	30.1%	41.3%	39.9%	41.3%	38.5%	62.9%	64.3%
Fair	20.3%	24.5%	11.2%	12.6%	9.8%	8.4%	7.7%
Poor	4.9%	9.1%	7.7%	5.6%	4.2%	2.1%	2.1%
Very poor	7.0%	3.5%	3.5%	2.8%	5.6%	2.1%	1.4%
Don’t know/no answer	4.2%	9.1%	4.9%	4.2%	4.2%	2.1%	2.1%
Not applicable	25.2%	0.0%	23.1%	23.1%	25.2%	0.0%	0.0%

 

3.2.3 Common survey themes

A total of 64 comments were received from provider principal contacts. The most common themes across these comments were consistent with those reported under KPI 1, namely:

• the turnover of case managers leading to rework or the loss of corporate knowledge
• lack of consultation in the risk assessment process
• long processing times.

Of the peak/professional and student bodies asked to provide comments about TEQSA’s communication, fourteen respondents offered feedback. All comments offered were positive, and included feedback about the reports provided by TEQSA, the TEQSA conference, and TEQSA’s clear and targeted advice. Two respondents offered suggestions for improvements in relation to communicating with their constituents and wanting more tailored communication.

3.3 KPI 3 – Regulatory actions undertaken by TEQSA are proportionate to the risks being managed

TEQSA has developed and implemented an innovative standards-based risk-modulated approach to regulation. For low risk providers, most re-accreditations and re-registrations continue to be granted for the maximum seven-year period. TEQSA continues to encourage low risk providers to seek self-accrediting authority (SAA).

WHAT DID OUR STAKEHOLDERS SAY?

For KPI 3, 57 per cent of principal contacts (and 76 per cent from low risk providers) and 90 per cent of peak/professional/student bodies rated TEQSA’s performance as ‘good’ or ‘excellent’.

3.3.1 Performance indicators for KPI 3

For KPI 3, TEQSA has assessed its performance against three indicators using both internal metrics and feedback from the stakeholder survey.

3.3.1.1 A comprehensive capacity for multifactorial risk analysis of all provider types

The Cycle 4 Annual Risk Assessment was completed in 2016-17 using Department of Education and Training verified and validated data. The Cycle 5 Annual Risk Assessment process commenced in May 2017 and will be completed in 2017-18. The delivery of the assessment was delayed due to the introduction of new analytical software and competing priorities arising from reduced staffing levels. Implementation of additional mechanisms for achieving more streamlined, efficient risk assessments processes will also continue in 2017-18.

A project to establish each registered provider’s profile of international education activity was undertaken and the data verified with providers in 2016-17. The project brought together data from the Higher Education Information Management System (HEIMS), the PIR, the Provider Registration and International Student Management System (PRISMS), Universities Australia, and other open source material to provide a snapshot of a provider’s operations in relation to international students studying in Australia and students studying accredited Australian higher education awards offshore. This project was identified as critical to TEQSA’s ability to understand international operations in the sector.

The identified purposes of the project included:

• the ability to understand a provider’s international activity
• support for TEQSA's sector-wide scanning capability
• the ability to understand sector-wide trends or issues relevant to international activity.

The project produced an international activity profile for each registered provider which was subsequently sent to the provider and used by TEQSA risk analysts and case teams to further inform the case management of providers.

3.3.1.2 Integration of risk analysis and regulatory decision making, by use of comprehensive detailed current datasets gathered and maintained to inform risk analyses and regulatory interventions

TEQSA uses datasets from previous risk analysis and regulatory history, and data collected from external sources such as the department’s Higher Education Information Management System (HEIMS). Data from HEIMS has been supplemented with information collected through the Provider Information Request in the last financial year.

A review of the relationship between the risk profile of a provider and the outcome of regulatory decisions was undertaken for 2015-16 and 2016-17 (Tables 8 and 9). The analysis reveals a strong correlation between the outcomes of renewal of registration and course accreditation assessments, and the risk rating of providers.

Table 8. Risk ratings and outcomes for Renewal of Registration

	RISK RATING	ADVERSE OUTCOME*	POSITIVE OUTCOME
JULY 2015 – JUNE 2016	MODERATE TO HIGH	70%	30%
	LOW	14%	86%
JULY 2016 – JUNE 2017	MODERATE TO HIGH	81%	19%
	LOW	0%	100%

* Condition, < 7 years or rejection of application

 

Table 9. Risk ratings and outcomes for Course Accreditation and Renewal of Accreditation

	RISK RATING	ADVERSE OUTCOME*	POSITIVE OUTCOME
JULY 2015 – JUNE 2016	MODERATE TO HIGH	28%	72%
	LOW	0%	100%
JULY 2016 – JUNE 2017	MODERATE TO HIGH	66%	34%
	LOW	0%	100%

* Condition, < 7 years or rejection of application

 

Providers with a risk category of moderate to high were those which received at least one high or moderate risk rating for Risk to Students or Risk to Financial Position in either of the years shown.

The data reveal a strong correlation between adverse findings and provider risk rating. This is most strongly shown for renewal of registration applications in both 2015–16 and 2016–17, where:

• the majority of low risk providers received positive outcomes (86 per cent for 2015–16 and 100 per cent for 2016–17)
• between 70 per cent and 81 per cent of moderate or high risk providers received a form of adverse outcome (i.e. conditions, re-registration for less than 7 years or rejection).

3.3.1.3 Progressive development of the scope and application of the differentiated model (known as Core+) to further reduce burdens on demonstrated low risk providers

The increased level of enquiries and applications from prospective providers in 2016-17 necessitated more resource-intensive assessment processes due to the lack relevant historical data and regulatory history of most applicants. Initial registration applications are assessed against all standards in the HES Framework due to the absence of regulatory history with TEQSA and therefore cannot be assessed under Core+. The impact of the continuing demand from prospective providers and the loss of experienced staff meant that throughput for the year did not increase, with 374 assessments completed in 2016-17 compared with 381 completed in 2015-16. In addition, 208 assessments were carried over into 2017‑18.

Every renewal application assessed in 2016-17 was undertaken in accordance with the Core+ methodology. TEQSA also further explored options to streamline assessments in 2016-17, including trialling abridged case team reports for low risk course accreditation and streamlined approaches for initial registration/course accreditation applications.

3.3.2 Stakeholder survey results

Compared with all other KPIs, top two scores were considerably lower for KPI 3 with scores of 61 per cent and 53 per cent for the two items. The item consultative approach was the lowest scoring KPI item for principal contacts.

Referring to the table and chart below, note the relatively high proportion of ‘don’t know’/‘not applicable’/‘no answer’ responses for both items, suggesting that not all Principal contacts had the specific interaction to provide a rating.

Figure 7. Breakdown of responses from principal contacts for KPI 3

 

	Consultative approach	Actions proportionate to risks
Excellent	14.0%	8.4%
Good	27.3%	30.8%
Fair	17.5%	14.0%
Poor	13.3%	8.4%
Very poor	5.6%	2.8%
Don’t know/no answer	10.5%	7.7%
Not applicable	11.9%	28.0%

 

3.3.3 Common survey themes

A total of 39 comments were offered by provider principal contacts. Themes were consistent with the 2016 survey including:

• poor communication/interaction including lack of status updates
• lengthy processing times
• the perception that regulatory burden has increased, case manager turnover and a lack of continuity contributing to this
• positive comments about interactions/relationships.

Of the peak/professional and student bodies, eleven respondents offered feedback about TEQSA’s risk approach. The majority of comments were positive, with respondents stating TEQSA’s actions were proportionate. Three respondents indicated they didn’t have enough information to answer.

3.4 KPI 4 – TEQSA’s compliance and monitoring approaches are streamlined and coordinated

TEQSA is engaged with other Commonwealth agencies and other regulators (e.g. the Australian Skills Quality Authority [ASQA], international agencies and professional bodies) to enhance, streamline and share data, and to minimise regulatory impact.

In 2016-17 TEQSA developed a significant number of cooperative arrangements with domestic and international agencies and peak bodies designed to share intelligence and data, and to develop coordinated responses to emerging areas of shared risk.

WHAT DID OUR STAKEHOLDERS SAY?

For KPI 4, 55 per cent of principal contacts (and 76 per cent from low risk providers) and 85 per cent of peak/professional/student bodies rated TEQSA’s performance as ‘good’ or ‘excellent’.

3.4.1 Performance indicators for KPI 4

For KPI 4, TEQSA has assessed its performance against four indicators using both internal metrics and feedback from the stakeholder survey.

3.4.1.1 Collaboration with the department to streamline and automate data collection on providers, and enhance access and sharing

TEQSA worked with the department to successfully verify and validate all student and staff data collected under the 2016 PIR. This milestone represents the successful validation of the student and staff data for the first year of the PIR transition for over 130 providers. As a result of the transition, student and staff data previously reported to TEQSA is submitted through the department’s Higher Education Information Management System (HEIMS).

The PIR is a request for information under section 28 of the Tertiary Education Quality and Standards Agency Act 2011 (TEQSA Act). Data is collected to ensure a core dataset is available across all providers in the sector. PIR data informs tailored regulatory processes that take into account provider history, standing and risk. This means that regulatory burden on providers can be significantly reduced. The use of PIR data also strengthens protection for students’ interests and the sector’s reputation by enabling TEQSA to monitor significant changes and identify any emerging issues

3.4.1.2 Collaboration with professional bodies to enhance data sharing and thus reduce regulatory burden on providers that are regulated by both TEQSA and a professional body

As described in Table 4 under KPI 1, TEQSA entered into a number of additional MOUs with peak domestic bodies in 2016–17 to share information for the benefit of providers.

TEQSA consults regularly with the higher education sector, in meetings and via written correspondence. The agency meets regularly with peak body representatives including Universities Australia, the Australian Council for Private Education and Training, the Council of Private Higher Education, TAFE Directors Australia, English Australia, the National ELT Accreditation Scheme and the International Education Association of Australia.

3.4.1.3 Specific interactions with international regulatory agencies as warranted for assessments of cross border education from Australian providers

TEQSA plays an important role in protecting, enhancing and promoting the quality and integrity of Australia’s higher education sector in the international sphere. The agency is a member of the following international networks:

• International Network for Quality Assurance Agencies in Higher Education (INQAAHE)
• Asia-Pacific Quality Network (APQN)
• Council for Higher Education Accreditation (CHEA) International Quality Group
• Quality Beyond Boundaries Group (QBBG).

In 2016-17, the agency continued to strengthen links with international quality assurance agencies through these networks, with ongoing discussion about Australian providers, quality assurance and regulation. TEQSA signed a number of MoUs with international bodies and domestic groups and agencies, as listed earlier in the report in Table 5.

In February 2017, TEQSA released its International Engagement Strategy 2016-20. The Strategy builds on TEQSA’s statutory objective to protect the strong reputation of the quality and integrity of Australia’s higher education sector in cross border higher education.

3.4.1.4 Demonstrated transparency of assessment and monitoring arrangements

TEQSA uses a case management model to manage its relationship with providers, assigning staff as liaison for all quality assurance and regulatory processes. Regulatory staff work in ‘provider teams’, responsible for the quality assurance and regulatory assessment of a group of providers. The provider team arrangements enable providers to have contact with more than one staff member, allowing greater responsiveness and consistency of advice. Allocation of providers to teams is reviewed periodically.

Provider teams meet with representatives from providers and conduct provider visits as part of their assessment activities. Provider teams are also allocated to liaise with entities that have submitted an application seeking to register as a higher education provider.

Following the reduction in average staffing level (ASL) from 60 to 48 in the 2016-17 Budget, considerable effort was made to recruit contract staff to undertake regulatory work. However, TEQSA experienced a high turnover of case managers, particularly for those case managers engaged as contractors where the turnover rate was 136 per cent.

Compounding this turnover was the shifting ratio of APS staff to contractors in the assessment groups, from 3.8 APS staff for each contractor to 1.5 APS per contractor. This loss of corporate memory and experience contributed to the significant increase in the time taken to complete assessment of applications compared with 2015‑16.

To address the high turnover in case managers, considerable work was undertaken in 2016-17 to update and enhance internal systems to support case managers. It is worth noting that the stakeholder survey results identified that 90 per cent of respondents considered the TEQSA case management approach ‘very important’ to their organisation. However, despite positive comments in the performance of individual case managers, a significant number of providers commented negatively on the high turnover of case managers.

3.4.2 Stakeholder survey results

The ratings for both items within KPI 4 dropped significantly from 2016 to 2017. The item reuse of material was rated as ‘good’ or ‘excellent’ by 56 per cent and timely coordination of visits by 53 per cent (compared to 71 per cent and 75 per cent in 2016).

Not all respondents were able to rate the items, with a considerable proportion of ‘don’t know’/‘not applicable’/‘no answer’ responses, particularly for timely coordination of visits (48 per cent). It is likely these respondents had not experienced a TEQSA visit in the last 12 months and were therefore not able to provide a rating.

Figure 8. Breakdown of responses from principal contacts for KPI 3

 

	Timely coordination of visits	Reuse of material
Excellent	11.9%	13.3%
Good	16.1%	26.6%
Fair	11.2%	21.0%
Poor	5.6%	7.0%
Very poor	7.7%	2.8%
Don’t know/no answer	3.5%	14.0%
Not applicable	44.1%	15.4%

 

3.4.3 Common survey themes

Principal contacts were largely silent on streamlining initiatives in their comments. Where streamlining or increased regulatory burden was mentioned, it was in the context of a perceived lack of continuity in case managers and concerns about processing times and TEQSA’s processes.

Eight comments were offered by peak/professional/student bodies about TEQSA’s approach to compliance and monitoring. Respondents were largely positive. One indicated that they had noticed improvements, but further improvements could be made. There was some feedback about the length of time taken for TEQSA decisions and case manager turnover. Two respondents indicated they didn’t have enough information to answer.

3.5 KPI 5 – TEQSA’s dealings with higher education providers are open, transparent and consistent

TEQSA is open and transparent in its dealings with higher education providers; it regularly publishes reports on trends and observations of sector performance and continues to improve its materials based on feedback from the sector, conducting consultations where appropriate and regularly providing guidance and good practice guidelines on its website for the benefit of the sector.

WHAT DID OUR STAKEHOLDERS SAY?

For KPI 5, 67 per cent of principal contacts (and 80 per cent from low risk providers) and 89 per cent of peak/professional/student bodies rated TEQSA’s performance as ‘good’ or ‘excellent’.

3.5.1 Performance indicators for KPI 5

For KPI 5, TEQSA has assessed its performance against four indicators using both internal metrics and feedback from the stakeholder survey.

3.5.1.1 A clear publicly available risk framework and articulated risk analysis process

The TEQSA Risk Assessment Framework sets out the risk assessment process and information about key components of risk assessments:

• key risk areas
• risk indicators
• risk thresholds.

TEQSA first released a Regulatory Risk Framework in early 2012. A review was conducted in late 2013 drawing on the experience of the first cycle of risk assessments and feedback from the sector. The framework document was last reviewed in February 2016 with the most current version being Version 2.1.

The Risk Assessment Framework is available from the regulatory approach section of the TEQSA website.

Figure 9 shows the number of page views of the Risk Assessment Framework in 2015-2016 and 2016-17, the data reveals interest is consistent across the 2 years.

Figure 9. Risk Assessment Framework views in 2016-17 compared to 2015-16

3.5.1.2 Transparency in the results of the regulatory decision making process

As described under KPI 2, TEQSA publishes the National Register on a monthly basis, updating decisions and sharing decision-related information such as conditions and shorter approval periods.

WHAT DID OUR STAKEHOLDERS SAY?

79 per cent of principal contacts (and 71 per cent of high risk providers) rated quality of information on the national register as ‘good’ or ‘excellent’.

63 per cent of principal contacts (and 61 per cent of high risk providers) rated availability of information as ‘good’ or ‘excellent’.

3.5.1.3 Public sharing of aggregate observations of performance and risks derived from regulatory experience with the sector, by publication of analytical reports

TEQSA is committed to providing information and analysis about the Australian higher education sector. In the 2016-17 financial year, TEQSA published three key documents:

1. Statistics Report on TEQSA Registered Higher Education Providers: this is the fourth statistics report. It includes analysis of 2015 sector data
2. Key financial metrics on Australia’s higher education sector: this is the second key financial metrics report on Australia’s Higher Education Sector
3. Characteristics of Australian higher education providers and their relation to first-year student attrition: this is the first time an analysis of attrition has been published covering the entire Australian Higher Education Sector.

The reports are available from the Publications section of the TEQSA website.

3.5.1.4 Consultative approach taken for new application guides and guidance notes for the transition to the HES Framework 2015

Following the successful engagement with the sector during 2015-16 to brief providers on the transition to the HES Framework 2015 that came into effect on 1 January 2017, TEQSA focused on finalising the application materials and guidance for the sector. In 2016-17, TEQSA worked on several activities related to improving the quality assurance framework, including:

• releasing a contextual overview and 19 guidance notes for consultation to support providers transition to the HES Framework 2015
• conducting a briefing on the HES Framework 2015 in September 2016, for 43 peak bodies with responsibility for registration or accreditation for professional practice
• trialling new online forms and templates for the HES Framework 2015 with selected providers before being made available to all providers
• updating applications guides for registration, re-registration, course accreditation, re‑accreditation and registration in university categories based on stakeholder feedback.

All materials required by providers to lodge applications under the HES Framework 2015 were made available. As of 30 June 2017, 55 assessments had been submitted under the HES Framework 2015.

WHAT DID OUR STAKEHOLDERS SAY?

88.2 per cent of provider principal contacts rated TEQSA’s guidance and support material as ‘good’ or ‘excellent’ (including 87.8 per cent of high risk providers).

3.5.2 Stakeholder survey results

TEQSA’s performance on KPI 5 was rated fairly positively, with two items rated ‘good’ or ‘excellent’ by more than 68 per cent of respondents. Positive ratings for Consistency of decisions decreased significantly this year; this was also the lowest scoring item for this KPI (61 per cent).

Figure 10. Breakdown of responses from principal contacts for KPI 5

 

	Consistency of decisions	Availability of information	Consistency of information	Quality of information on National Register
Excellent	11.9%	14.7%	18.2%	13.3%
Good	32.2%	46.9%	47.6%	58.0%
Fair	16.8%	30.8%	17.5%	15.4%
Poor	7.7%	4.2%	8.4%	2.8%
Very poor	4.2%	0.7%	5.6%	0.7%
Don’t know/no answer	4.9%	2.8%	2.8%	9.8%
Not applicable	22.4%	0.0%	0.0%	0.0%

 

3.5.3 Common survey themes

Principal contacts largely identified case manager turnover as a concern and requested more stability and continuity in TEQSA’s case management. Where case managers change, providers feel there have been problems with handover leading to re-work. Some providers felt there was a lack of consultation in the provider risk assessment process and wanted more input into the process.

Eleven respondents commented on TEQSA’s approachability. Nine respondents provided a positive response; four of these indicated they had noticed improvements over the last 12 months. One respondent indicated that dealings with TEQSA could be more regular or based on clear understanding or responsibilities.

3.6 KPI 6 – TEQSA’s regulatory framework continues to be improved in consultation with stakeholders

TEQSA communicates its approach to its core functions to a wide range of stakeholders. In 2016–17, the agency focused its efforts on:

• outlining the agency’s approach to its work and building confidence in its capacity
• increasing provider understanding of their quality assurance obligations and compliance with them, to support a culture of self-assurance in the sector
• increasing provider understanding to support the transition to the HES Framework 2015
• supporting the development of policies and processes to reduce the regulatory burden for providers
• promoting international recognition of TEQSA’s expertise to assure the quality of Australian international education and reduce provider compliance burden.

WHAT DID OUR STAKEHOLDERS SAY?

For KPI 6, 61 per cent of principal contacts (and 75 per cent from low risk providers) and 73 per cent of peak/professional/student bodies rated TEQSA’s performance as ‘good’ or ‘excellent’.

3.6.1 Performance indicators for KPI 6

For KPI 6, TEQSA has assessed its performance against the following two indicators using both internal metrics and feedback from the stakeholder survey.

3.6.1.1 Well-established, productive consultative mechanisms with stakeholders, and use of a variety of media and channels to convey information to stakeholders

As described in KPI 4, TEQSA consults regularly with the higher education sector, in meetings and via written correspondence. In 2016-17, TEQSA actively engaged with students, including with peak student bodies, to discuss approaches to student engagement and how TEQSA can work with these bodies in the future. The agency also entered into discussions with Study Melbourne and the Overseas Student Ombudsman regarding information sharing of student complaints about higher education providers.

In November 2016, TEQSA held its first conference titled Sharing Excellence: Assuring Quality. The conference was attended by more than 420 delegates, and was one of the highest-rated items in the 2017 stakeholder survey. Planning for the second TEQSA conference, to be held in November 2017, commenced in 2016-17 and used feedback from the first conference as the basis for an expanded program of engagement with the sector.

WHAT DID OUR STAKEHOLDERS SAY?

The TEQSA conference was very well-received with 90 per cent of principal contacts rating the conference as ‘good’ or ‘excellent’.

Activities also included reviewing the information for students on the TEQSA website, with the prominence of the student voice and student access to the website as key principles for a refreshed website which will be launched in late 2017. Student representatives were also consulted in relation to the TEQSA Conference for 2017.

TEQSA utilises a variety of media to engage with its stakeholders, including through its:

• website
• social media channels Twitter and LinkedIn
• meetings with peak bodies and providers, and attendance at industry conferences
• forums
• annual conference.

3.6.1.2 Regular engagement with the minister and the department of education and training

TEQSA collaborates with the department and other Australian Government agencies to access and share knowledge, expertise and resources. The TEQSA CEO, staff and Commissioners meet regularly with relevant areas of the department to discuss common issues and identify opportunities for cooperation.

During 2016-17, following changes to the loan arrangements for vocational education and training students, the agency met regularly with areas of the department managing the transition to the replacement loan scheme. Information regarding dual sector providers from the department was used to monitor the impact of changes in access to the scheme on the financial viability of the higher education operation of providers.

The Commissioners and CEO met with the Higher Education Standards Panel during 2016–17 to discuss:

• the review of the impact of the TEQSA Act
• the consultation on transparency in admissions
• links with the QILT project
• regulatory options for upholding the integrity of academic assignments
• international activities in the sector
• the regulatory impact of professional accreditation
• the transition to the HES Framework 2015
• reporting parameters for the Regulatory Performance Framework
• research into attrition rates
• updates on the impact of TEQSA’s reform agenda.

The Panel also provided external validation of TEQSA’s self-assessment of its 2015-16 performance against the RPF.

3.6.2 Stakeholder survey results

Variety of media and making process improvements were the best scoring items for KPI 6 with a top 2 score of 62 per cent. Positive ratings for Engagement dropped significantly this year to 59 per cent from 79 per cent in 2016.

Figure 11. Breakdown of responses from principal contacts for KPI 6

 

	Engagement	Making process improvements	Variety of media
Excellent	18.9%	10.5%	14.0%
Good	37.1%	46.2%	42.7%
Fair	23.8%	23.8%	30.1%
Poor	7.0%	4.2%	2.8%
Very poor	8.4%	7.0%	2.1%
Don’t know/no answer	4.2%	6.3%	8.4%
Not applicable	0.7%	2.1%	0.0%

3.6.3 Common survey themes

Several principal contacts from private for-profit providers continue to express the opinion that TEQSA’s regulatory processes and risk framework are tailored towards universities and disadvantage their particular working models.

Eleven peak/professional/student bodies responded on TEQSA’s continuous improvement with an overall theme of TEQSA being committed to this KPI. One respondent commented positively on TEQSA’s workshops, and another on TEQSA’s stakeholder briefings.

4. Learnings for TEQSA

TEQSA appreciates the willingness of the sector to provide open and frank feedback, as attested by the high level of participation in the surveys, with responses from 74 per cent for both principal contacts and peak/professional and student bodies.

It is encouraging that overall TEQSA is well-regarded as the regulator assessing the quality of Australia’s higher education, with 80 per cent of principal contacts and 88 per cent of peak/professional and student bodies rating TEQSA’s performance as ‘good’ or ‘excellent’.

The survey results show where TEQSA is performing at a high level, while also highlighting areas for improvement, based on both the areas where performance was rated less highly as well as the constructive responses to the four key questions about what TEQSA should continue to do, change, stop and do more.

4.1 Highest and lowest rated performance areas

Table 10 shows the items that achieved a top two score of 80 per cent and over (when rounded). Guidance and support materials were well-rated, as were the items relating to the TEQSA Conference. CRICOS applications also scored highly, as did KPI 2 items which related to TEQSA’s communication with providers.

Table 10. 2017 highest scoring individual items

PC: TOP SCORING ITEMS	2017 (n)	2017 TOP 2 SCORE (%)
Guidance & Support materials: The quality of the information	85	94.1
Guidance & Support materials: Relevance of information	85	92.9
Conference: Opportunity to interact with other delegates	84	91.7
Guidance & Support materials: Usefulness of information	85	90.6
Conference: The quality of speakers	83	90.4
CRICOS Application process: Clarity of the online form	69	89.9
Conference: The program	83	89.2
KPI 2: Quality of regulatory information	140	88.6
KPI 2: Relevance of information	140	87.1
Conference: Relevance of material presented	83	86.7
CRICOS Application process: Clarity of the application guide	71	84.5
CRICOS Application process: Helpfulness of portal information	71	84.5
Guidance & Support materials: Ease of access to that information	85	82.4
Guidance & Support materials: Amount of information	85	81.2
Overall performance	138	79.7

*N/A means the result is not available as these items were not included in the 2016 survey.

The lower scoring items (below 60 per cent top two score) in the PC survey are displayed in Table 11. These issues were all reflected in respondent comments, particularly around:

• timeliness
• stability in case management
• need for greater consultation or understanding
• requests for some or more visits.

The key areas where the providers would like improvement are TEQSA’s:

• case management (knowledge of organisations)
• streamlining
• consultation.

A number of providers commented on streamlining initiatives going backwards and some wanted greater coordination between TEQSA and ASQA. A view still exists within the non-university sector that TEQSA is university-centric and does not understand the uniqueness of many small, private providers; and that TEQSA is even hostile toward them.

Table 11. 2017 lowest scoring individual items compared to 2016

PC: LOWER SCORING ITEMS	2017 n	2017 TOP 2 SCORE (%)	2016 TOP 2 SCORE (%)
KPI 1: Streamlining to reduce burden	130	59.2	71.4
KPI 2: Communicating streamlining initiatives	130	59.2	78.9
KPI 6: Engagement	136	58.8	78.9
KPI 4: Reuse of material	101	56.4	70.8
Case mgt approach: Consideration of your needs	130	55.4	70.9
KPI 2: Timeliness of information after making a decision	101	54.5	76.5
KPI 4: Timely coordination of visits	75	53.3	75.0
KPI 3: Consultative approach to risk assessments	111	53.2	56.6
Case mgt approach: Knowledge of your organisation	136	52.2	71.1

 

4.2 Providers differentiated by risk

Providers that TEQSA rated as having a low risk to students in 2017 also rated TEQSA’s performance the highest on all items in the below table. There was a considerable number of differences for these groups of providers, including for TEQSA’s overall performance rating.

Table 12. Comparison of issues where low and high risk providers differ significantly

SIGNIFICANTLY DIFFERENT ITEMS TOPIC / ITEM	2017 RISK TO STUDENTS TOP 2 SCORES (%)
	High / mod max n=63	Low max n=47
KPI 2: Timeliness of information after making a decision	48.1	77.8
KPI 2: Clarity of information	60.4	86.2
KPI 2: Quality of regulatory information	85.5	97.8
KPI 3: Consultative approach	39.7	66.7
KPI 3: Actions proportionate to risks	42.6	86.2
KPI 4: Reuse of material	47.2	77.4
KPI 5: Consistency of information	56.5	82.6
KPI 5: Consistency of decisions	46.2	80.0
KPI 6: Engagement	46.7	71.1
KPI 6: Making process improvements	55.7	83.3
Application process: Helpfulness of information about how to prepare an application	57.5	91.7
CRICOS Application process: Any follow up assistance that was required	63.0	93.8
Case mgt approach: Responsiveness	55.7	80.9
Case mgt approach: Knowledge of your organisation	44.1	68.9
Case mgt approach: Consideration of your needs	39.7	85.7
Overall performance	73.3	95.6

Total n varies by item because not all respondents were eligible to answer all questions. The maximum possible n is displayed in the table header.

The survey clearly indicates that low risk providers are very positive about TEQSA’s performance whilst high or moderate risk providers are less positive about TEQSA’s performance.

4.3 Providers differentiated by market grouping

Table 13 presents providers classified by University, Faith Based, For Profit, Other and Prospective Higher Education Providers and highlights where a particular market grouping significantly diverges from the average score for a particular item. Universities scored significantly higher, and For Profit providers significantly lower, than most other market groups on all the items shown in Table 13.

Table 13. Differences by market grouping

MARKET GROUPINGS - TOP 2 SCORES (%)
Item	Faith based	For profit	Uni	Other	Prospective HEP
	n=14	n=35	n=33	n=40	n = 21
KPI 2: Timeliness of information after making a decision	72.7	33.3	82.4	60.1	33.3
KPI 2: Clarity of information	61.5	43.3	94.4	87.0	54.5
KPI 3: Actions proportionate to risks	54.5	24	94.7	77.2	42.9
KPI 4: Reuse of material	54.5	34.4	90.5	60.7	44.4
KPI 5: Consistency of information	64.3	44.1	81.3	76.6	70.0
KPI 5: Consistency of decisions	53.8	38.7	90.9	67.2	50.0
KPI 6: Engagement	57.1	36.4	77.4	66.4	55.0
KPI 6: Making process improvements	57.1	50	89.3	64.4	38.9
CRICOS Application process: Any follow up assistance that was required	100	52.6	91.3	84.8	100
Case mgt approach: Knowledge of your organisation	53.8	32.4	71.9	59.3	40.0
Case mgt approach: Consideration of your needs	81.8	29.4	83.3	54.3	44.4

Significantly higher results highlighted in bold, significantly lower results highlighted in italics.

4.4 Common survey themes

In addition to requesting feedback on performance using the rating scale, TEQSA also asked participants to respond to four key questions. Table 14 outlines the common themes that emerged from stakeholder responses.

Table 14. Common survey themes

Questions	Common Themes
What does TEQSA DO WELL that it should continue doing?	A total of 156 principal contacts offered ideas about what TEQSA does well. There were two clear/common themes: TEQSA’s case management & case management approach TEQSA’s information offering such as guidance notes and other material offered through its website.   Other themes less commonly mentioned were: Positive communication and consultation TEQSA’s considerate and professional approach TEQSA’s Risk based approach Forums and the TEQSA conference.
What one thing should TEQSA IMPROVE or change that would make the most difference to its effectiveness as a regulator?	Principal contacts offered 121 suggestions with the most common suggestions being: Improving response times More streamlining, including with professional accreditation bodies Be more of a partner / work with us Obtain more resources Hold more forums including webcasts More provider visits, contact, proactivity and consistency.
What one thing should TEQSA NOT DO that it is currently doing?	A total of 43 suggestions were offered. The most common themes among these were: Using inappropriate/ inexperienced experts (including those with a lack of private sector experience) Slow responses.
What should TEQSA be involved in or MORE INVOLVED in than it is currently?	A total of 77 principal contacts offered suggestions about activities that TEQSA should be more involved in. The most common themes were: Be more of a partner and/or understand our situation more Engage with the sector more Offer good/best practice examples and case studies Conduct more thematic quality assurance activities.   An emerging issue across several open-ended comment opportunities was TEQSA’s role in relation to professional accreditation bodies, including duplication of activities and monitoring the quality/standards of these bodies.

 

Results of this self-analysis and stakeholder survey will now be used by TEQSA to revise its Corporate Plan and set priorities for 2018.