Writing your own assessment tools

Ross Woods, rev. 2018-19

In this course, you'll learn how to write three simple kinds of assessment tools: Tasks, observation checklists, and interview questionnaires.

Other different kinds of simple assessment tools include:

• Descriptions of acceptable performance
• Form templates and proformas
• Checklists for evaluating work samples
• Rubrics
• Recognition portfolios
• Student self-assessment materials

In some unusual cases, you need to write your own. You might be unable to use the existing assessment tools. For example, a written journal was not very helpful for a group of deaf students, so local staff devised a system of interviews that worked much better and still addressed all the same competencies.

Assessment tools are simply ways of asking students to produce specific evidence in concrete situations that can be used for assessment. They include:
  •   Instructions to the student on what to do as an assessible activity.
  •   Instructions to the assessor on what to do, if it is not obvious in the instructions to the student, and,
  •   A form to record the assessment.

Some kinds of tools also need a marking guide to ensure reliable assessment.

 

What are assessment modes?

To assess a skill of any kind, the assessor normally needs to define a kind of activity as a way of of gathering evidence of the student's skills. The kind of activity is called a mode of assessment. Exams aren't the only way, and sometimes they're totally inappropriate. In most cases, you can call them assessment tasks, that is, tasks that students will do to be assessed on.

The assessment mode is sometimes decided and written down beforehand; this often happens if the assessment mode is integral to the skill. Here are two examples:

• The competency statement might require students to be able to present their thoughts in a written report. In this case, report writing (the assessment mode) is part of the skill to be learnt.
• The only real way to assess some practical skills is for the student to demonstrate them in a real or simulated workplace.

In many cases, all you will be given is the standards and criteria. You must take responsibility to choose a suitable assessment mode when you plan the assessment. By being more creative, you might be able to make your assessments much more valid, and you may be able to give the student a choice of assessment modes.

Remember:

The skill to be learned must match:

The way it is taught,

The way it is practiced,

and

The way it is assessed

 

Some kinds of assessment modes

There are many, many different kinds:

• Major projectsStudents define their own projects and (usually) write a formal report. Projects often cover a cluster of related units.
• Tasks These are smaller tasks that reflect unit requirements.
• Demonstration of skills in a classroom environment This usually means that student shows the skill to the instructor (who is probably also the assessor) as he/she learns it. However, it can also be an umbrella term for many other kinds of assessment done in the classroom.
• Written essaysEssays are appropriate as part of the assessment requirements for some knowledge-driven units, especially if the elements are to evaluate, compare or identify etc. In some cases, staff might set the topic. In other cases, students must be able to choose their own topics to identify, refine and explore issues.
• Reading programsThese are most appropriate for knowledge-driven units, and deal more with mastering a body of knowledge than workplace competency. They are usually tutorials, in which students present and discuss their papers in the group, book reviews, or annotated bibliographies.
• Private journalsThese show how the student is thinking and maturing and why, and often involve emotional or attitudinal factors of personal development.
• Written assessments under "exam conditions." Students don't get told the questions beforehand, may not communicate with each other, and cannot use textbooks. A person responsible to the assessor supervises the assessment. However, questions and topics are based directly on the element statements, and assessment follows the given assessment criteria. Answers may be explanations of factual matter, essays where the student presents and supports a viewpoint, or multiple choice. Students may also be free to select only some of the questions, for example, "three of the following five questions."
• Electronic examinations are the same as written assessments, but the computer terminal and its software are also under exam conditions. In this way, multiple choice assessments may be graded immediately through the software or anywhere in the world over the Internet, and other kinds of essays may be sent immediately to the assessor for grading.
• Open book assessment is much like a written assessment, but students may use books. These are not an easy option, because the student must have a good grasp of the assessed material to be able to use the books efficiently. It generally produces more informed answers.
• Written examinations that students may research in the library beforehand These are not an easy option either, for the same reason as open book assessments. It also produces more informed answers.
• Written examinations that students may take home or do by email These are almost the same as essays. Authenticity may be a problem in some cases.
• Oral examination It might involve asking the student each question in a list of questions. Oral language proficiency can be appraised through a face-to-face conversation with a proficient speaker of the language, who may be the assessor.
• Oral questioningThis may be more like a discussion or a personal interview to establish the authenticity of the student's work, required knowledge and/or to establish competence. The trend is toward using a fixed set of questions that can be mapped to the requirements. You can also ask spontaneous oral questions about something the student is demonstrating, with no list of questions.)
• Oral presentation or debateStudents might be required to present their learning in a prepared speech or lecture, perhaps with an adversary against whom they must defend their ideas.
• On-job assessment Some skills require assessment in an on-job situation against a checklist of relevant competencies. The assessor can do a simple observation on the job, or assess a practicum involving reports from other credible people (usually the workplace supervisor).
• Simulations that are observed by the assessor These are most useful when a real on-the-job situation is not feasible, but the skill must be demonstrated in as near-to-real situation as practical.
• Scenario analysis Respond to a particular scenario, described in writing, on video, or in simulation. The student must identify important features, what is happening, and the correct response. The student might be required to explain the rationale behind his/her approach.
• Critical incident analysis When most of a difficult job is routine and predictable, competence can be assessed by the way student reacts in particularly difficult or unpredicted circumstances. Assessments typically look at the context, preceding events, the incident, the factors determining response, other possible responses, and consequences of the response taken.
• Assessment by portfolio of documentary evidencePortfolios normally include written work done by the student (reports, manuals, essays, data printouts, letters, brochures, self-evaluations, resume, log books, designs, drawings, etc.) and supporting documentary verification (e.g. professional licenses and memberships, reports of supervisors, instructors, and responsible authorities, references, job descriptions.)
• Standardized tests
• Program audit
• LogbooksA diary of satisfactory work done by the student and verified by the supervisor. In some cases, it can also act as a reference if the supervisor makes a recommendation of competence.
• Supervisor referenceThe supervisor states that the student has performed satisfactorily. These usually work better if the supervisor addresses specific competencies; a general reference saying that the student is a nice person and has done well does little to establish competency.
• Authenticated list of achievements Check the claims made in the students CV with credible authorities.
• Lessons or workbooks that are filled inDistance educators often use this kind of material.
• 360° appraisal The assessor gathers a self-assessment from the student and detailed appraisals from the student's supervisor, a peer, a subordinate, and a client.
• Exhibition of an object made by the studentAbility is assessed through products such as manufactured goods, artworks (e.g. painting, drawing, crafts, sculpture), etc.
• Public performanceExamples are public speaking, leading meetings, drama, singing, music, and dance.

 

Hints and recommendations

1. We strongly recommend tools that can be re-used without change. (Consider the traditional alternative; many lecturers need to write completely new examination questions every time they assess students.) You will easily be able to write reusable tools if you:
• Focus on asking students to do something in their own context.
• Don’t predetermine the exact answers you expect. For example, you might ask students to give a presentation but not determine the actual topic and outline they must use.
 
2. If the task is an ongoing responsibility with no beginning and end, you can specify a long enough period of time for workplace assessment.
 
3. If the task has a start and end, you can specify how many times the student must perform the task:
   • Major, longer-term tasks normally need to be done only once.
   • Students usually need to do smaller, short-term tasks at least twice, and you might need to specify different contexts.
    
4. Check that tools are clear enough for the student and assessor to know what they are expected to do.
 
5. Use simple language, not bureaucratic gobbledygook.
 
6. Lay them out on the page so that they are easy to read.
 
7. Check that tools match the competency standards.
 
8. For interviews, tell assessors:
• "You do not need to ask questions twice. When students answer the first question, they will probably answer many questions that are intended to come up later on. You don't need to ask a question again if the student has already answered it satisfactorily.
• "You may probe with any necessary follow-up questions.
• "If students get an answer wrong, it might be that their nervousness makes them forgetful. Talk the student around so that they have another attempt.
• "Your interview should corroborate other evidence that the student has submitted. For example, they should be able to explain the work in their portfolio.
 
9. If your assessment tools work equally well for taught students and RPL students, you will need only one set of tools.

Tip 1Use workplace forms as assessment tools. | Tip 2When your assessment tools are designed for a specific context, you might find that you need another set to assess students in another different context. | Tip 3When your assessment tools are designed for a specific context, they often don't work for RPL students coming from very different contexts. Big hint: Go back to the original training package units. | Tip 4Integrate a cluster of units into a meaningful project. Students learn better and find the whole process more satisfying. | Tip 5Practitioners often don’t like unnecessary writing and generally do better in oral interviews. Besides, if they don’t get a point the first time, you can talk them round in a circle and let them have another go. Some very capable people don’t always get it right the first time around. | Tip 6You can use discussion groups for assessments, as long as the group is small enough for you to observe what each student says.

 

Tasks

A task tells students to do something. It should contain enough instructions for students to know exactly what you expect of them, and should be the natural thing to do when performing the skill. For example, if the skill is to replace a flat tyre, the task should be to replace a flat tyre.

Tasks can cover performance criteria in three ways:

1. The task instructions can contain the assessment criteria, so that students who do the whole task well according to the instructions can be assessed as performing satisfactorily.
2. In other cases, the textbook explains the details of what is required. The assessment tools at the end of this book take this approach.
3. You can list the requirements on the same page.

 

The advantages

First, you can easily ask for more than one kind of evidence. For example, you can:

• ask for the actual performance of the task,
• tell the students what kinds of questions the assessor will ask, and
• ask for the related paperwork.

Second, the instructions to the student also tell the instructor what is required, so you don’t usually need to write a separate set of instructions to the assessor.

Third, if the task (or a series of tasks) collects all required evidence for a unit, you can record "Competent" or "Not yet competent" for that unit. (If you collect only some evidence for a unit, you can record "Satisfactory" or "Unsatisfactory" for the task.)

Fourth, reporting is easy because most of the documentation is already done in the task description. You can use a fairly standard form to report the assessment. Besides, it can be quite brief, so it’s easy to report multiple units on one form. The form should contain at least:

• The name of the student
• The name of the assessor
• A list of tasks, each of which has a tick-box
• The name of the unit(s)
• The name of the task(s)
• A place to record a result e.g. Competent, Not yet competent, Satisfactory or Unsatisfactory.
• A place for general comments on the student’s performance, which usually doubles as feedback to the student
• A date.
• A place for the assessor’s signature.

 

Observation checklists

Many assessors use use a checklist to check off all essential aspects based on their observations. See this example.

An observation checklist is a form that contains at least:

• A set of instructions for the assessor Tip 1Put them put them at the top of the form where people will see them. This is usually much better than a separate piece of paper, which people will easily lose.
• The name of the student
• The name of the assessor
• The name of the unit
• A list of criteria, each of which has a tick-box
• A place for general comments on the student’s performance, which usually doubles as feedback to the student
• A place for an assessment conclusion (e.g. Satisfactory/ Not satisfactory)
• A date.
• A place for the assessor’s signature.

Many checklists only ask for one kind of evidence, so they are often inadequate to gather all evidence for a unit. However, you can add instructions about seeing relevant paperwork and asking questions.

Tip 1As much as possible, try to get the whole form onto one piece of paper. | Tip 2Add more columns if you have to do make multiple observations. This is better than using more forms. | Tip 3 You can sometimes copy performance criteria from the units into your form, but they normally need to be edited into plain language. Include enough detail to ensure reliable assessment.

 

Interview questionnaire

An interview questionnaire is quite similar to an observation checklist. An interview questionnaire is a form that contains at least:

• A set of instructions for the assessor Tip 1Again, put them put them at the top of the form where people will see them. This is usually much better than a separate piece of paper, which people will easily lose.
• The name of the student
• The name of the assessor
• The name of the unit
• A list of questions, each of which has a tick-box.
• A place for general comments on the student’s performance, which usually doubles as feedback to the student
• A place for an assessment conclusion (e.g. Satisfactory/ Not satisfactory)
• A date.
• A place for the assessor's signature.

Sometimes it is also good to have a list of model answers, but you can only do so if all students must give the same or similar answers. Besides, you will find that some students will give excellent answers that aren't what you expected.

The simplest way to write an interview questionnaire is to go though the list of requirements and turn them into questions. The trick is to make them clear and concrete, not Explain everything in the universe. You can split up complicated questions into simple questions, ask people to explain the steps, or ask them how they would do or respond to something.

Most of the questions below are open-ended, that is, they cannot be answered with Yes or No. Even when students give an answer, you usually need to ask the reasons why. The questions below are well suited to higher qualifications.

1. Use questions to get information.
• How often do you review progress in that department?
• What strategies did you try?
• How do you know how well they worked?
2. Ask for an analysis. You can ask the student to draw conclusions or to identify causes, variables, or significant factors.
• What are the main elements in your program?
• What are the main steps in how you did that?
• Why do some succeed and some fail?
• Why did those difficulties arise?
• What conclusions did you draw?
• What are the significant factors in that approach?
• What variables affect it?
• "What kind of things does that depend on?"
3. Use questions to get students to defend their arguments.
• Why did you do it that way?
• What evidence supports that conclusion?
• How would your respond to the idea that ... ?
4. Ask the student to anticipate trends and to plan ahead.
• What needs to be done to make that idea work?
• How will that plan work in the market place?
5. Ask students to choose priorities and defend their choice.
• Of the three problems, which one will you try to solve first?
• Why did you spend more time on that option?
6. Hypothetical questions ask "What would happen if …?"
• What would have happened if John had resigned at that time?
• How would your colleagues have reacted to a reduced share price at that stage?
• What if you had a funding cut of 10%?
7. Explore implications
• What are the implications of that kind of policy change?
• How would your members react to those recommendations? How would you answer them?
8. Make generalizations
• Based on your experience, what are the major risks?
• What are the main reasons for that kind of strategy?

Tip 1When you plan a qualification, start by writing a job description of what you want your graduates to be able to do. It will make the course look much more practical to your students. | Tip 2When you write a set of questions to test required knowledge, offer it as either an interview or an assignment. | Tip 3When the list of knowledge items is quite detailed, you can replace one item with one question. But when a knowledge item only specifies a broad area of knowledge, you will normally need a series of questions to cover it. | Tip 4Knowledge is sometimes adequately demonstrated by doing the assessment task.

 

Simple mapping

The idea of mapping is to show how your assessment tools address the requirements of the competency statement.

There are several simple ways to do mapping:

• On a hard copy of your assessment tools, write references to the various parts of the unit.
• Color-code a hard copy of the accredited unit to your assessment tool.
• Color-code a spare copy of your assessment documents to the unit requirements.

Another method is to write references to the various parts of the unit into the assessment tools. The advantage is that it doesn't require an extra document but the condition is that it should not confuse anybody. Options are tiny print in the text, footnotes, or endnotes.

 

Trial your assessment tools

You cannot anticipate all the interpretations of your assessment tools that assessors and students might make. Trial your draft assessment tools to check that the content is right and they suit your group of students. Record outcomes and make any necessary corrections and improvements.

1. Decide how you will trial your assessment tools. You can simply get a colleague to check your assessment tools, such as another staff member or an industry person. You might need to trial the assessment tools with a sample group of people similar to your students to know how well they work.

2. Analyse feedback. In the feedback, people might say:

• "What do you mean by this sentence?"
• "Could be more relevant to what we actually do here."
• "Didn't work on campus. But that's where we assess it. "
• "Oops, you got this fact wrong."
• "We don't do it that way anymore. They brought out a new rule."
• "Difficult to use."
• "Good test, but it'd cost a fortune to use."
• "Took a long time to do. Could you do it some way that takes less time?"
• "The jargon confused lots of people."
• "Do we really need all that paperwork?"
• "I have someone who is very good but got stumped doing it in English."
• "The deaf student just gave up in frustration."

3. Fix anything wrong before use. If you have to make significant changes, then you should test it again.

4. File them properly. When you've finished, file a copy and make backups of the soft copies.

Tip 1Align your assessment goals with the business goals of the student’s employer. Everybody wins. | Tip 2I like to write each unit’s set of assessments to be a standalone statement. | Tip 3You can sometimes write an assessment activity that is equally relevant to multiple units. So students only have to do it once.