Two different kinds of evidence?

Ross Woods Rev. Jul 2010, 2018

 

Part 1: "At least two different kinds of evidence?"

Russell Docking, a Curtin lecturer in VET Studies and a senior auditor, once proposed the following assessment practices:

1. Two kinds of evidence. The minimum requirement was that assessment of each unit must have two kinds of evidence, and must assess competence in all outcomes. It may include assessment of knowledge. The idea of two independent items of evidence for each outcome, not just each unit, is good practice and dovetails with many other aspects of assessment.
2. An extended period and a range of contexts. Evidence must represent the student's abilities over an extended period and in a range of contexts, and assessments must comply with endorsed or accredited guidelines.
3. Evidence items should corroborate each other. The two items of evidence should differ in some way so that they corroborate each other. The skills might be performed at different times or in different contexts or with different equipment, etc.
   For example, you might observe the student for an assessment. The student's supervisor might then provide a reference based on the outcomes and mentioning consistency of performance over a period of time. Having two kinds of evidence also shows that the student can transfer learning from one context to another.
4. Use different modes of assessment. It is better to use two different modes of assessment. Every mode has limitations or weaknesses and a single mode cannot cover all aspects equally well. Even directly-observed abilities need corroboration through oral questioning and workplace supervisors.
   It's also fairer. A single assessment activity is seldom equally fair for all students; some students do better at oral assessments, some do better at practical or written work. For example, oral presentations favour confident speakers but disadvantage students who fear public speaking.
5. Re-consider continual assessment. "Continual assessments" are many small assessments over a longer period. Make sure that your assessments really are summative. Beware of continual assessment; it is naturally formative rather than summative.
6. Do your mapping. It is quite difficult to plan an assessment that strictly uses only one kind of evidence, and it is usually quite easy to gather at least three kinds. It is more difficult to ensure that each outcome is assessed through at least two kinds of evidence.

 

The mapped grid

The best way to monitor kinds of evidence is through a mapped grid, although this is not a requirement. The point is to ensure that each column has two marks (See diagram)

Although this example meets the minimum requirement, in practice the oral questioning and the employer's reference should both cover all outcomes. As a result, most outcomes have three or four kinds of evidence.

This kind of mapping can also help you check that assessment tools cover everything necessary:

• no gaps
• nothing over-assessed
• only what should be assessed (nothing that should not be assessed)
• the assessment complies with all specified requirements

On the point of over-assessing, consider what language, literacy and numeracy skills are required for the assessment. Don't require written work or language skills (e.g. assignments) that are beyond those specified in the unit or necessary for the performing the outcome.

 

		Outcomes
		Outcome 1	Outcome 2	Outcome 3	Outcome 4
Evidence type	Supervisor reference	X
	Oral questioning		X		X
	Practical demonstration		X	X	X
	Written test	X
	Portfolio			X

 

 

Part 2: A critique

This update on assessment modes was originally based on a paper given at a moderation meeting by Russell Docking, a validator, Curtin academic, and independent training consultant. It appeared to be a major interpretation of policy, but I've checked and he didn't get it completely right. So I've revised the update to differentiate between best practice and the minimum requirements.

Two kinds of evidence for each outcome? Russel Docking has said that the quality framework (then the AQTF) requires two independent items of evidence for each outcome, but it is not so. The AQTF states only "sufficient evidence". It is only very good practice to use more than one mode of assessment.

Extended period and range of contexts. The requirement that evidence must represent the student's abilities over an extended period and in a range of contexts, is no longer included in the AQTF, although some training packages still require it. Nevertheless it is good practice.

Assessment of theoretical knowledge. He also suggests that assessment of theory may not be used as evidence because it is not an assessment of competence (i.e. the skill specified in the outcome). He is quite agreeable to using it for formative assessment.

It would be very appropriate to argue that assessment of memorized information is not assessment of applied knowledge. In fact, an information-only assessment could be invalid if it does not relate to the outcome. That is, it is invalid because the assessment does not refer to the item assessed.

The quality framework specifically allowed assessment of the application of knowledge where relevant. Consequently, you may use an assessment of applied knowledge. In fact, the WA graded assessment system allowed knowledge tests results as evidence.

It is now appropriate to argue that assessments must have at least two assessment modes for assessment of workplace competence. Assessment of knowledge is not an assessment of competence. The difficulty is the distinction between applied knowledge and purely theoretical knowledge. The difference between knowledge and competence is less distinct in studies that are more conceptually driven, for example, higher qualifications and university preparation courses.