(Field-test version)
Ross Woods, 2020
Based on the SRTO, an auditor can audit a site only by looking at paperwork and evaluating it for compliance. However, that is a poor indicator of program quality.
A better approach also includes:
industryif the program is in-house training. This gives a picture of program effectiveness and whether implementation matches the paperwork.
This particular audit approach is designed specifically for ACAS and its series of forms. Processes might look quite different In a more centralised RTO or an all-on-one-site institution.
ACAS central office appoints auditors. (ACAS member colleges may not do so.)
To be able to act as auditor, you will need to do the following:
You will also need to develop several particular attitudes:
fitness for contextis often required, and
one size fits allis often noncompliant.
As the auditor, you still need to wear the policeman role to ascertain compliance, but you can also act in a helping role so that people will feel that you have suppported and helped them. In fact, part of an auditor's role is to promote improvement. Be warned that people will probably ask you to do it for them.
Think of your role as a teacher. You can explain, ask questions, and give examples, but the final work must be the students' own work.
As auditor, the purpose of the audit is to reach accurate, defensible conclusions based on the evidence. Consider the two extremes, both equally unsatisfactory. The fault-finding expedition means that you feel you've failed if you haven't found something wrong. At the the other extreme is the encouragement tour, where the auditor is willing to forgive anything so people will feel good about the audit.
Encourage auditees to aspire to excellence, within the limits of practicality. They should not feel that it is enough to do the minimum amount of work to just fall over the line.
Principal auditee: The person representing the program being audited, whether it is a qualification, a site, or a department. The person is usually the head of the program, but may be someone else authorized to be principal auditee.
Training and Assessment Strategy (TAS): A long form that covers program planning and related compliance requirements.
Approval to deliver: The staff form that documents which units that each staff member has the skills and qualifications to instruct.
Evidence: Information, materials, or products that show whether or not a student has met the competencies of the unit. For example, an instructor could observe a student, make observation notes, and record an assessment result.
The guide contains mapping to the SRTO. The mapping has various meanings:
Some parts of the SRTO do not apply to ACAS at all, such as teaching TAE qualifications. Other parts are done completely by ACAS centre and do not apply to sites.
The first step is to contact the head of the program and set an audit date. Brief them on what will be involved and give them a link to What's an audit?.
What will be done to make sure everybody comes and who will do it? What will you do if some people can't come? At what point will you cancel the whole meeting because not enough people can come?
Suggestion: Prepare an attendance roll with spaces for name, basic contact details and signatures. Get people to fill it in as they come in. You can also use it as a Professional Development record and issue attendance certificates to attenders.
Plan your meetings beforehand. For example, you might want to re-arrange the sequence of items so that it is easier for people to follow and understand.
Which parts of the whole audit do all teaching staff need to attend? Which parts don’t they? This will affect the order of reviewing items. The recommended approach at this stage is that all of them come for everything.
However, you might decide on another approach, for example:
It also depends on the staff. Even if some items are not relevant to most staff, they might need to learn the compliance requirements or how the audit process works. Most staff are either unaware of them or have only learned one particular compliance system. If the site is subject to a franchise agreement, you might speak with the principal auditee alone to sign the latest version of the agreement. Then again, you might want staff to understand the terms of the agreement.
You might find that, at some sites, they only want to send administrators. This is not a good option. At some points, their input is absolutely necessary. More than that, the purpose of this approach is partly to educate more widely on compliance obligations.
The validation meeting might require personnel other than regular instructors. The SRTO says that validation is to be done by one or more persons who are not directly involved in the particular instance of delivery and assessment of the training product being validated, and who collectively have ... vocational competencies and current industry skills relevant to the assessment being validated; ... current knowledge and skills in vocational teaching and learning; and ... the training and assessment credential ....
Your ideal solution is to do it as part of the whole workshop if you can.
Industry consultation can be done at different levels:
The meeting will cover the report on industry consultation. If you are doing the consultation for that site, you have several options:
In a new program or a program with many new staff, giving them a set of forms as pre-reading might confuse them rather than help them.
In an ongoing program, however, staff should already be familar with ACAS systems, so you might ask people to download some forms and read them beforehand, (especially the TAS and Approval to deliver). They might even draft some of their responses.
In an ongoing program, the audited site will already have most or all the required documents. The task of the audit will then mostly be checking and updating them. Inform people what documents to bring:
I suggest sitting people around tables, probably with coffee and tea available in the room. One table might be enough for a small group.
Use a data projector to put the main points and key questions on a screen. That means you'll have to prepare your screen content.
It’s common to appoint a scribe and a discussion leader for each table, but watch the dynamics. Some scribes write everyone else’s thoughts down, while others dictate all answers. Groups might need time to reflect and edit responses so they are not just a list of brainstorm ideas.
Collate contents on the day and copy them into the relevant forms. Don’t let people go home with a promise to email them later because some people will forget. They could email items at the time or put all notes on a thumb drive for you.
You will need to explain the purpose of the item (Why are we doing this?), state what you want them to do, answer or discuss their questions, and then get them to produce answers..
For each item during the audit below, explain the SRTO requirements and rationale, and answer any questions. For each item, you will probably need to clarify who is the intended audience. Some of them are for instructors/assessors, some are for students, and some are for administrators. And a few things are only for ASQA auditors to read, and serve no other known purpose.
It is helpful to discuss possible examples, but, if you provide the examples or your examples are too few, beware that people might simply copy them.
Decide how much you will orient people to the text of the Standards. Many trainers and assessors are are unaware of compliance requirements and reasons behind them, so it could be a good opportunity to teach them.
Are you sure that's what it means?,
Why do you do it the way you do?and
Why don't you do it this way ...?Beware that you might get so many questions that your schedule will go overtime and you will have difficulty keeping up.
This schedule is intended to put most of the audit into a one-day workshop, and might be too rushed for many sites. Most sessions are only 45 minutes, and there is no time for interviews on implementation. It might be most difficult for schools as their schedules are often quite rigid.
9.00 – 9.30 a.m. | Session 1. | Introductions, goals for the workshop, any housekeeping announcements, entry meeting, signed form for the latest version of the auspice agreement, grand tour. . |
9.30 – 10.15 a.m. | Session 2. | Training and Assessment Strategy (TAS). |
10.15 – 10.30 a.m. | Break. | ☕. |
10.30 – 11.15 a.m. | Session 3. | Information for the public, Course information for students. |
11.15 a.m. – 12.00 noon. | Session 4. | Staff expertise. |
12.00 noon – 12.30 p.m. | Lunch. | 🍴. |
12.30 – 1.15 p.m. | Session 5. | Delivery materials, Equipment list, Admission checklists. |
1.15 – 2.00 p.m. | Session 6. | Assessment materials, RPL assessment tools. |
2.00 – 2.45 p.m. | Session 7. | Assessment validations, sample of assessment records, retention of assessment evidence. |
2.45 – 3.00 p.m. | Break. | ☕. |
3.30 – 4.15 p.m. | Session 8. | National recognition, Program review and improvements. |
4.15 – 5.00 p.m. | Session 9. | Exit meeting & conclusion. |
Meet the key person (the principal auditee) and any other key people.
Do the audit entry meeting.
Do the grand tour. Get an overall picture of what the organization does, how they do it, and who their clients are. This often includes a look around the site. This overall picture will probably answer many of your questions before you get to ask them.
View latest version of agreement | Form
This must be updated annually for each qualification, although some courses may be allowed to go over multiple years without change. The form includes many different compliance requirements. The task isn't finished until it is done well, that is, you are convinced that the program is well planned and will run effectively.
If they already have a Training and Assessment Strategy (TAS) document, check that it is the current form and update anything so that it is correct and up to the minute.
If it's a new program and they don't already have a TAS, work with them to write a new one. As there are so many items, I suggest that you delegate items to separate tables, so different questions get answered simultaneously.
If the ACAS website information is correct, they need no other publicity materials to be compliant.
Much of this information is usually advertising literature and the site's own website. Some other forms of information for prospective students are not ads, such as school handbooks and newsletter articles. Even small ads and micro-ads are included, because the standards do not set a minumum size ad for compliance. Some sites do not issue ads at all and refer only to the ACAS website information. If the ACAS website information is accurate, their other information may be incomplete, as long as it is accurate and not misleading.
Course description
This is a written description for students informing them how the whole course will be taught and assessed:
• It is the version of the training and assesment strategy
given to students that informs them about the program.
• It may be the same as the advertising literature or prospectus.
• The ACAS website gives the minimum amount of information, unless electives offered vary between groups, or unless locations and schedules vary according to employer.
• It is good practice to incude the purpose of the program and a job description of the role for which students will be trained.
• Some programs issue only separate unit descriptions for each unit. This only works where each unit is a standalone. Students still need information about their whole program.
Unit descriptions
• Look at the unit descriptions. Do they inform students about expectations (e.g. what's in the unit, what's the schedule, how they'll be assessed, etc.?
• Some programs combine all unit descriptions into a handbook for the whole course. This is very effective if the units are integrated into a rather seamless whole.
(The ACAS website gives the minimum amount of information to be compliant but it is good practice to give detailed information.)
Each instructor needs to provide a current Approval to deliver form, and copies of all relevant credentials.
• Use this procedure to categorize staff and see what credentials are required. (Link opens new window.)
• Note: The Approval to deliver form now includes the staff form.
Check qualifications and any other credentials of all teaching and assessing staff. Get new paper copies and check that they are complete. Verify your copies: E.g. Original sighted, 31 Feb, 2020, [signed] J. Bloggs.
If they already have "Approval to deliver" forms, check them and update anything so that they are up to the minute on the current form. If they don't, check them (or oversee them filling them in).
The point is to show that the site has sufficient delivery materials for all units offered. Materials will vary greatly according to how the program is delivered, e.g. online, practicum, classroom, outsourced. Delivery materials must cover all units taught and all specific unit requirements.
It's not possible to include all training materials in the audit report, but auditors should describe them and go through a sample of at least two units and check that materials cover all requirements. The evidence may vary, for example:
• copies of local materials,
• links to online materials,
• list of contents of commercial materials,
• photograph of the library.
The point is to show that the site has all equipment prescribed as necessary for all units offered.
• If units specify equipment (and most do), make a checklist of all required equipment and do an inspection. Sign the checklist showing what equipment is present and what is missing. To pass the audit, all required items must be present.
Some units and some whole qualifications do not specify any required equipment, and this will also need to be recorded.
Check assessment materials for all units of every qual to be offered.
See the footnote for a working definition of assessment tools.
A site must be able to offer RPL for all programs it has in scope for all units listed in the Training and Assessment Strategy form:
• If the above tools are unsuitable for RPL, the site needs a separate set of RPL tools.
• ACAS recommends that, where possible, assessment tools should be written to be equally suitable for both taught and RPL students.
• The ACAS website has generic RPL instructions but they might not be specific enough for the course being audited.
Check that validations have been done for every unit of every qual. Ideally these are already done, but not always. There are two kinds, which are often done at different stages of the program:
• Validate assessment tools every year.
• Validate assessment judgements every year, unless no assessments were done in a course.
What to require:
These checklists record compliance with admission procedures.
Overall, what has student feedback been like?
A list of glitches, risks, and program weaknesses and a plan to make improvements for each one. (The purpose is to show that the program is continually improving cf. SRTO 2.2 b.)
If the program is already running:
Ask if they have examples of accepting credit from other RTOs or AQF institutions. They might not have, but if they have, it is evidence that they practice national recognition.
Ask how they store assessment evidence.
Hold the exit meeting.
At the end, ask for ideas and improvements:
In consultation with the principal auditee, you might need to tidy up the documentation. For example:
Audits must be documented and the final full report comprises several elements: the auditor's report, the auditor's evaluation of processes, and a collection of evidence. These then need to be collated into one document of the whole audit.
Keep good notes as you go and write your final report as soon as possible after the audit is finished.
Inside the report, give the date of each meeting and the names of everybody who attended. If there were more than one meeting, list them as separate meetings and mention what each meeting was about. (If you used an attendance roll, simply photocopy it for the report. This will make it easier to write your report.)
Go through all the audit items, and for each one:
State what was done in the exit meeting.
Sign and date the report, and give your name as the writer of the report.
On a separate sheet, state any difficulties encountered in the audit process.
Give your suggestions for improving the audit process, ACAS forms and processes, and this ebook.
Forward a copy of the review to the ACAS office.
The best answer is probably both. Considerations include:
The goal is to present the audit in a way that readers can easily understand and confirm your conclusions, and can easily find any document in it. To achieve this goal, present the whole audit inside one set of covers, complete with any evidence that you collected. Being well-organised is even more essential in larger programs that produce a fairly large collection of staff documents or represent multiple qualifications.
Arrange the full report like a book:
Send a copy of the report to the ACAS office.
SRTO Standard 2, esp. 2.2
As part of ACAS quality management, audits include a risk analysis key aspects of a program. This means “Is the program viable and healthy, with little to go wrong?” ACAS needs to know whether programs are strong and healthy or at risk. This approach uses specific sets of criteria, with ratings from 0 – 5. Scores of 5 represent an excellent program in that aspect. Some scores of 0 represent a “non-compliant” audit outcome, while others represent a new program that is unproven but compliant. New, unproven programs will normally be 0 for at least some criteria, even if they have some very strong points.
This risk assessment is intended only for new institutions wanting to apply to ACAS, although some criteria might also apply to established colleges. It is generally unlikely that ACAS would now accept a new member college unless it is almost all low risk.
There are easy and difficult approaches to starting a new college. Low risks represent programs that are relatively easier to implement. Some higher risks might not be any better at the first stage of a new program; they generally represent programs that are very difficult, or perhaps impossible, to implement in an RTO. Items written in dark red like this indicate an automatic fail in an audit of compliance and quality standards.
When they commence, new programs will normally be scored 0 for some items, even if they have some strong points. Beware also the possible effects of the hype cycle when evaluating new programs.
Give the program a star rating of zero to five for each criterion, based on the descriptions of weakest and strongest ratings, explain your reasons, and mention any obvious opportunities for improvement. Beware also the possible effects of the hype cycle when evaluating new programs. (See below.)
Programs are expected to improve. If a program does not improve or is high risk, the auditor will refer it to the board for further evaluation.
Items written in dark red like this indicate an automatic fail in an audit of compliance and quality standards.
Highest risk ☆ ☆ ☆ ☆ ☆ |
Lowest risk ★ ★ ★ ★ ★ |
Rating | ||
---|---|---|---|---|
Qualifications | ||||
1. | Wants to offer multiple qualifications initially. | Wants to offer only one qualification initially. | ||
Trainers & assessors | ||||
2. | Needs to recruit new staff to teach. | Uses current staff as supervisors. | ||
3. | Proposed trainers-assessors will need to learn new skills. | Programs match the core skills & expertise of proposed trainers-assessors. | ||
4. | All proposed trainers & assessors still need to get all formal qualifications. | All trainers & assessors already have all formal qualifications and experience. | ||
5. | Has only one main trainer-assessor. | Has multiple trainer-assessors. | ||
Structures | ||||
6. | Establish a new school structure and ignores current organizational structure. | Uses current organizational structures. | ||
7. | Not clear about the role students would be trained to meet. | Clear statement of the role students would be trained to meet and already has people in that role. | ||
Recruitment | ||||
8. | Needs to advertise the program to recruit students. | Already has prospective applicants for the program. | ||
9. | Ignores current constituency and intends to pioneer a new one from scratch. | Intends to recruit students from current constituency. | ||
10. | Plans to recruit full-time class-room students who will get Austudy. | Plans to have only part-time students. | ||
Delivery | ||||
11. | Plans to teach regular classes in a classroom atmosphere. | Plans to train on the job. | ||
12. | Needs to buy new equipment for classrooms. | Use existing equipment. | ||
13. | Teach everything in the classroom. | Only conduct classes and training weekends as you need them. | ||
14. | Needs to write or gather all teaching and assessment materials. | Already has all teaching and assessment materials. |
Take-up of new programs might follow the so-called "hype cycle," a pattern that warps perceptions. At some points, perceptions are overly optimistic, while they are overly pessimistic at others. The stages are below:
|
|
In your report, give the program a star rating of zero to five for each criterion, based on the descriptions of weakest and strongest ratings, explain your reasons, and mention any obvious opportunities for improvement.)
Programs are expected to improve. If a program does not improve or is high risk, the auditor will refer it to the board for further evaluation.
Items written in dark red like this indicate an automatic fail in an audit of compliance and quality standards.
Weakest ☆ ☆ ☆ ☆ ☆ | Strongest ★ ★ ★ ★ ★ | Rating | |
---|---|---|---|
1. | Has no clear purpose, or has no clear planning that ensures it will achieve its purpose. If it has a plan, it might be more like a wish list. | Has a clear purpose and has clear, concrete planning that ensures it will achieve its purpose. The program is already running well. | |
2. | New or has a very small number of students. It hopes to recruit students from the general public because it does not have a well-defined constituency from which it can draw new students. | Has either growing student enrolments, or as many students as it can handle well. It has a well-defined constituency, which is a reliable source of new students for the foreseeable future. | |
3. | Instructors are not teaching effectively and/or students are not learning effectively. | Instructors are teaching effectively, and students are learning everything they are supposed to. (This is not just a review of training and assessment documents.) | |
4. | Has no professional development program for instructors. | Professional development for instructors is effective in improving the quality of instruction. | |
5. | Materials are new and untested, and might be incomplete. | Training and assessment materials work well and need only minor adjustments from time to time. | |
6. | Has difficulty demonstrating compliance. | Has no difficulty demonstrating compliance. | |
7. | Often does not comply with ACAS policies and procedures. | Consistently complies with ACAS policies and procedures. | |
8. | Income currently does not cover costs. For example, it tries to offer courses at very low fees, or even for free, and has no income from other sources. It might have to cross-subsidise courses from other sources in order to survive. | Income covers costs with a safety margin. For example, it charges reasonable fees or has stable, assured income from other sources such as employers. It has few bad or slow debts receivable. | |
9. | Either has no clear budget or its estimates and projections are unrealistic. | Has a clear budget with realistic estimates and projections. | |
10. | Would be in trouble if key people were to drop out. For example, it depends heavily on volunteers. | Would still be healthy if key people were to drop out. Has enough staff and does not depend heavily on volunteers. | |
11. | Record-keeping is poor. Required documents were never created or are now missing. | Keeps good records and can easily demonstrate compliance. | |
12. | Satisfied to meet minimum requirements. | Trying to achieve excellence and best practice. | |
13. | Unaware of its areas of weakness and/or future improvements. | Understands its areas of weakness and/or future improvements. |
While you should do a full risk assessment of government schools, they tend to be low risk due to the following factors:
assessment tools?
The Standards for Registered Training Organisations 2015 mentions Assessement tools
but does not contain a definition. Consequently, it provides no criteria for what must be included and what may be excluded.
For the purpose of ACAS audits, assessment tools are defined as follows:
An official ASQA definition (not in the Standards) of assessment tools is as follows:
It would be helpful to also incude the following:
Content 20