Empowering Online Assessments
Empowering Online Assessments adapted from https://www.oercommons.org/courses/types-of-online-assessment and http://www.oercommons.org/courses/guidelines-for-online-assessments-for-educators/view
The purpose of this resource is to provide an introduction to online assessment types and discuss the ways in which they are likely to center or empower students. Due to the pandemic, many educators and students are put in a situation where they must unexpectedly teach and learn online. Online learning spaces may present new opportunities to learning, and this resource is an attempt to take a closer look at online assessment and how it may empower student learning the construction of knowledge.
This resource first introduces the benefits of online instruction and then introduces a theoretical framework to guide educators in their design and evaluation of assessments. Next, we look at priciples in online assessment, and finally, types of online assessment are listed.
This resource has been developed by Aubree Evans, Director of Professional Learning at Branch Alliance for Educator Diversity. The content was developed from the BranchED Quality Framework and Habermas's knowledge consistuents. Additonal content was adapted from "Type of Online Assessment" by Evan Abbey at AEA Learning Online and Abbey and Guidelines for Online Assessment for Educators by the Commonwealth of Learning.
Benefits of Online Assessment
In many institutions, assessment tasks are completed by using traditional pen/ pencil and paper, but the online world of assessment offers features that are not available in face-to-face environments. Online assessment tools have several benefits over traditional pen-andpaper tests:
- Written or oral exams:
- Rubrics with radial buttons (depending on the learning management system) can be used for ease of grading and leaving feedback
- Some systems offer audio or video feedback that faculty can leave for students on assignments
- There are many online options for students to create assessment portfolios that combine text, image, video, and auditory material (whiteboards, storytelling boards, e-books, wikis, blogs, etc)
- There are also apps where students can leave feedback and comment on each others' work
- Online assessment empower authentic assessment opportunities! Students can create work and link to it on their resumes)
- Multiple-choice tests:
- Test items can be randomised when the assessment is taken, so no student will have test items appear in the same order as the student who is taking the same test on the next workstation.
- In the case of multiple-choice questions, distractors can be randomized.
- Test items can be tagged by level of difficulty.
- “Pools” of test items can be used from which the tool can randomly assign different test items to different students.
- Some types of test items can be scored by the tool that is used, relieving the teacher or teacher educator from that burden.
- Online tools can give immediate feedback to students.
Other benefits of online assessment is the availability of learning analytics. The Society for Learning Analytics and Research (SoLAR) defines this as “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (Khalil & Ebner, 2016). Learning analytics can often be easily collected using a learning management tool. Learning analytics are particularly useful for identifying students who are at risk of failing. At Purdue University, for example, educators began employing the Purdue Early Warning System, using learning analytics, to identify “at-risk” students, combining the LMS tools with Google Analytics and institutional management information systems. Data were exported to Excel or to statistical software tools to identify trends in performance, spot outliers and anticipate who may be at-risk students.
(Adapted from Guidelines for Online Assessment for Educators p. 8)
The TPACK Conundrum
f you are not familiar with the term TPACK, well... I won't be able to change that here. TPACK as a theory for technology integration really requires its own course with several dedicated lessons to fully explore. The best I can say is to summarize it as an intersection between technology, pedagogy, and content knowledge in the course of teaching; in other words, a teacher cannot integrate technology very well unless she has a mastery of 1) all 3 parts and 2) how the parts help support each other.
There is, however, a term that applies to any discussion of online activities or assessment: TPACK Conundrum. In brief, the condundrum is that online learning best practices were not developed in a top-down, systemic manner. They were developed organically from many different sides of the spectrum, and with that, they were developed by individuals limited in other areas of the TPACK integration.
A translation would be good here.
What this means is that learning platforms were created by individuals who did not have expertise in the area of assessment. Alternatively, curricular experts often have a good understanding of pedagogical needs, but that doesn't always translate into the technology itself. There have been many new online teachers who have stated, "But, I want Moodle to do this...!" with the reality being, it can only do what any tool was built to do.
So, this lesson will try to bridge the TPACK Conundrum gap, and show how different categories of LMS assessment tools (the "techie" side of the equation) match up with different assessment types in the Rick Stiggins et al categories for assessment (the "currichie" side of the equation). The thing you will note right away is that there isn't a 100% match... LMS tools, while having a definitive purpose, are a bit fuzzy when applied to pedagogical models.
So, embrace the fuzz!
Online assessment principles
Types of Online Assessment
When online learning first started, the amount of assessments were limited. Assessments often were either online selected-response questions or proctored work to be completed at the students' location and sent to the instructor. Assessment options have changed as quickly as the amount of multimedia options for teachers. There are many more ways now to provide authentic assessments, formative assessments, peer- and self-assessments... (you name it), and that makes online learning more powerful.
Listed below are short descriptions of the most common digital techniques used by educators to assess learning, for either formative or summative assessment. In the online assessment world, multiple content formats can be used. Whilst traditional pen and paper represent two-dimensional questions, online tests can incorporate digital features such as audio, video, animation and innovative item types. The opportunities this presents for different types of learning are staggering. The principles of universal design for learning (UDL) in traditional assessment can also be applied in the online assessment world. The website for the National Center on Universal Design for Learning (udlcenter.org) gives a thorough overview of the principles of UDL. A number of the learning management systems (LMSs) that institutions are using have a variety of capabilities for employing different assessment techniques.
Multiple-choice tests are a common form of online assessment. It is important to create multiplechoice tests properly, because in online assessment, the flaws of poorly written items are potentially more visible to the world when they are placed online. Students may also have to cope with technical challenges, which exacerbates the impact of poorly written questions. A great advantage of online multiple-choice tests is that they are easy to administer. In addition, item analysis enables the test-setter to eliminate poorly written items. Online multiplechoice questions are instantly marked by the software that is used to deliver them. As the questions do not measure writing ability, students are not disadvantaged if they have poor writing skills in another language. The questions can, however, measure reading ability. It is a fallacy that multiplechoice tests are “objective.” Scoring them may be objective, but humans create questions and formulate the response options, which certainly involve subjective decisions. The disadvantages of traditional and online multiple-choice tests are that they are subject to guessing and often are time-consuming to create. In the online world, however, the reuse of previous items becomes easier, and accordingly, the process becomes less time-consuming. A multitude of tools exist for creating multiple-choice question items. Most LMSs contain quiz tools. Further, a host of free and commercial tools are available online. At the time of publication, some popular tools are:
- Withoutbook: http://www.withoutbook.com
- ProProfs: http://www.proprofs.com
- That Quizz: https://www.thatquiz.org
- PollEverywhere: https://www.polleverywhere.com
Traditional true-or-false question items require students to indicate which of two potential responses is true. A student therefore has a 50 per cent chance of being correct by guessing. With an online true-or-false question, guessing can be reduced by requiring an explanation. This does impact the ease of marking, but it is an efficient way to digitally collect additional information. The use of negative marking for the incorrect answer can be created in some software systems, which reduces the calculation involved when applying the same approach in a traditional exam.
Essays are flexible and can assess higher-order learning skills. However, they are time-consuming for educators to score. If essays are submitted online, it may become easier to mark them using online rubrics, or by having an online marking scheme with prepared comments or other anticipated responses. These comments can be dragged onto the electronic essay, or new comments can be made on the essay. However, subjectivity may be an issue during marking. One of the most significant advantages of completing essay questions online is the ability to read from text as opposed to hand-written responses. Figure 1 presents a screen capture of grading an essay using an online rubric. The student would have written the essay and uploaded it to the LMS (in this example, Moodle™). The educator would then be able to use the “Manual Grading” method to mark the essay. This tool allows the educator to insert custom comments. An educator can write a number of anticipated responses and simply copy and paste these as custom comments. Alternatively, the educator can make use of the “Rubric Editor,” which is available from the “advanced grading method management” screen following the “advanced grading” link. The editor allows for naming the rubric form and for adding a description, new criteria and levels of performance. Criterion descriptions should be entered, and for each level at which performance can be set, a level definition and the marks associated with the level should be specified.
These test items require the student to fill in a word or phrase in response to a direct question, or to enter a word or phrase that was left out of a statement. The advantage of having students take this type of examination online is that the answers can be scored immediately by comparing the student response to a pre-populated answer. The disadvantage is that often students will not type the exact answer and will be marked wrong. Spelling mistakes, for example, can cause an incorrect grade, whereas the teacher would have accepted the answer.
Online games offer exciting assessment opportunities. They can provide a safe, creative environment in which students can learn to experiment, collaborate and solve problems. They can be used in almost all educational disciplines for a variety of assessment types. In 2016, the Center for Online Education produced a list of 50 websites with game resources for teachers, as shown in Figure 2. At the time of writing these guidelines, the list is available through http://www.onlinecolleges. net/50-great-sites-for-serious-educational-games/.
Table 1 gives examples of some of the gamification online tools.
Student journaling, blogging and wiki building
Many LMSs provide functionalities for student journaling, blogging or wiki building. There are also tools freely available on the Internet. Outside of your LMS, journaling is a particular useful tool for encouraging student reflection, and teacher educators may elect to assess the journal entries by using a rubric. The Centre for the Enhancement of Teaching and Learning provides detailed information on the assessment of journals, at http://ar.cetl.hku.hk/am_rj.htm (Chan, 2009). Specific guidelines for journal entries may be given (structured journal), or students can create free-form journals. Journals are typically private, visible only to teaching staff and the individual student. Journals can be onerous to mark when large numbers of students are involved.
Building a blog or wiki is an important strategy for encouraging individual or collaborative student writing. Think of a blog as a website organised by posts according to the date the posts were made. Think of a wiki as a website that organises by content (such as Wikipedia does). Depending on your desired outcome, either or both can be chosen in a learning environment.
Blogging is similar to journaling, except that there are more features for providing access to the information. One or more students can build private, semi-private or public blogs. Blogs can be assessed in the same manner as journals. Blogs are a particularly useful tool for teacher educators and teacher students when they are developing a professional online identity (POI). Many teacher education programmes are now including the development of a POI for each student teacher as an outcome of the programme.
Wikis are also tools that can be created by one or more students and can be constructed as private, semi-private or public. Wikis are particularly useful for collaborative group projects and are conducive to social constructivist learning. Wikis are non-linear, evolving, complex, networked online resources with multiple authors, and they can be used to support student collaboration and co-production. At the time of writing this document, Internet Techies (2015) has listed ten free wiki software platforms that educators may consider for developing collaborative knowledge making: http://www.clickonf5.org/7599/10-free-opensource-wiki-software-engine.
Online, digital or ePortfolios
Typically, a portfolio is a collection of student work that is organised, reflected upon and presented to show content comprehension and learning growth over a period of time. Reflections on the individual artefacts as well as the overall portfolio are critical components of this assessment tool. Portfolios enable deeper learning for students, and educators gain a better understanding of their students’ knowledge and skills.
Online portfolios can be constructed using a variety of ICT tools –– for example, Evernote, or the open-source tool Mahara (https://mahara.org/), the latter of which integrates with Moodle. These tools allow students to write documents and upload photos, audio and video. All content can be tagged and, if necessary, shared to other media tools. Evernote is versatile in that it exists in both computer and mobile app formats, and content can be synchronised across multiple computers and mobile devices. Basic websites, blogs and wikis can also be used as online portfolio tools. The most significant advantage of creating a digital portfolio is the ability to include a variety of content. ePortfolios, for example, can contain movies, audio, presentations, text, hyperlinks and animations.
The use of online ePortfolio tools is strongly aligned with the upper level of Bloom’s revised taxonomy; this sets outcomes at the “Create” level, where students are expected to be creating, composing, constructing, designing, generating, inventing and producing
Self- and peer-assessment tasks
The primary aims of self- and peer-assessment are to:
- increase student responsibility and autonomy;
- achieve a more advanced and deeper understanding of the subject matter, skills and processes;
- elevate the role and status of students as assessors;
- encourage a deeper approach to learning;
- involving students in critical reflection; and
- develop in students a better understanding of their own subjectivity and judgement.
To guide the design and evalaution of assessments that empower students, we will use a theoretical framework comprised of Habermas' Emancipatory Knowledge Constituent and Inclusive Pedagogy principle of the BranchED Quality Framework.
Habermas' 3 Knowledge Constituents
Jürgen Habermas referred to the ways that we interpret knowledge as knowledge constituents. Educators can use these knowledge constituents to probe thinking about their inclusive pedagogy practices.
The technical knowledge is about the ways in which knowledge is objectively controlled. This can apply to teaching when we think about how content and the mode of learning might be imparted from the instructor to the learners. There are structured rules that learners must follow, which are determined by the instructor.
The practical interest is about learning through social interaction with others. Learners are encouraged to ask questions and deepen understanding through communication and feedback from others.
The emancipatory interest applies technical and practical knowledge in combination with self reflection to help learners achieve freedom from domination. This interest gives students room to make decisions about their own learning and action; therefore, assessment that incorporates the emancipatory interest will need to center learners and allow individual adaptation on the assessment. Rennert-Ariev (2005:08) stated, "In order to empower students in ways congruent with an emancipatory intent, students will need to have a significant amount [of] control over how they are assessed and the conditions and contexts of the assessment."
Equity-Oriented Questions to guide Curriculum Design, Instruction, and Assessment
The purpose of assessment is to measure student learning as a product of the curriculum and instruction. We can ask the following questions to think about the inclusivity of curriculum design, instruction, and assessment.
Step 1: Curriculum Design - Since assessment is part of curriculum design, the first set of questions from the BranchED Quality Framework Inclusive Pedagogy principle guides the teacher educator to ask the following questions about the curriculum:
How are you centering learners in outcome/objective design?
How do you identify and center student assets in instructional materials and learning activities?
Step 2: Instruction - To help learners feel empowered and autonomous over their own learning, inclusive pedagogy must be incorporated into all aspects of the classroom from design to teaching. Following are questions to ask yourself if your teaching is modeling culturally-sustaining pedagogies:
How are you centering all learners in your teaching?
How are you taking intentional steps toward ensuring that all learners have access to the learning opportunities provided?
Step 3: Assessment - Due to the weight of grades and role of assessment to evaluate students' understanding, the assessment holds a lot of weight and power. Below are some questions to ask yourself if the assessment you are using is emancipatory in nature:
How are you centering all learners in assessment?
How do your feedback practices lead learners to use their assets towards inquiry and discovery?
Habermas, Jurgen. 1972. Knowledge and Human Interests. Boston: Beacon Press.
Habermas, Jurgen. 1973. Theory and Practice. Boston, MA: Beacon Press.
Rennert-Ariev, Peter. 2005. "A Theoretical Model for the Authentic Assessment of Teaching." Practical Assessment, Research, & Evaluation, 10(2):1-11.