This is a whitepaper, describing the KBA examination for the university.

TABLE OF CONTENTS

Whitepaper information

The whitepaper describes the fundamental components to deliver a good medical KBA examination, sometimes with sections referring to Ortrac articles.

Detailed section

Create a course

Manually

Go to Blueprinting/Program layout
Select a <program> under Program section
Click the button Add
Add Name
Add URL ID (needed for component paths, no blanks in the name)
Select Create from scratch (or Copy existing if possbile)
Select Portfolio in the dropdown menu Student menu
Enter a course code
Add Credits for the course
Program reports, select Show in reports
Click the button Save

API

Handled separately

Create course occasion

Go to Courses

Select a <course>

Click the button Create new +<course>

Create KBA exam

Create the examination body

Go to Course/<course occasion>
Go to Exam
Click the button Add segment and select Exam
Click on the Exam bar:
Add a <name>
(Select no Prerequisite, if not already chosen)
Click the button Save

Edit the exam

Click on the exam created above.

Click the button Edit exam

Edit exam info

Start time: Enter Scheduled start date/time
Duration: Enter <hours & minutes>
Optional: If you want password to start or submit closed, add those
Optional: No feedback
Add grading scale: 0-9
Click the Save button

Add questions

Click the tab Questions
Create Sections
Click the Browse button
Select your questions in the left field and press Add to: Exam name in the right column.
Press Submit questions

Add students with groups

Click the tab Students
Click the Add Student buttons
- Manually
- Import from the course (all/selection). Most common.
Fix the login code for student, first part of email

Publish exam

Click the tab Info
Click the Publish exam

Publish course occasion

Go to Courses/<course>/<course occasion>
Go to Settings
Click the Publish button

Alternative question types

Test of the moodle DB leads to the following problems for the Ortrac import:

SBA, inga problem (låsning på randomizerade alternativ får man titta i parsern, under fliken SBA och helt enkelt gå in per fråga och låsa, efter import
SHORT ANSWER, gick att importera dock har de ingen stem, bara en fråga och ibland inget/inga svar (STEM=ITEM)
T/F går inte att importera, då det saknas två alternativ, de har bara ett med T/F som var, samma här saknas en (STEM=ITEM)
EMQ, här kan man importera, dock så har filerna olika antal frågor, vilket leder till en förskjutning in i Answer o Answer key kolumnerna (vet inte hur man löser detta, kanske med att lägga de frågor med samma antal Frågor tillsammans i rader, eller under flikar eller så får man helt enkelt handjaga detta efter parsern gjort Excel)

Tag & Tag groups

Tagg/ar: får ju lösas separat genom att titta i parsern, under fliken Categories och uppdatera Tagggrupper

Reports

Which reports to use, in Ortrac?

Import of question database

How to proceed to import the Moodle database? Only SBA questions?

Import SBA questions with tag(s)

(+ legacy tag moodle path) automatisk or manuell

Article: Ortrac - Import an SBA question, via Excel : Orzone AB

Deadlines and questions

Next examination period?
Date to import the Moodle database
routines creating exams?
routines reports after exams?
create/publish/run exams?

General section

Benefits with online examinations

There are several benefits to using online exams, including the following:

Convenience and flexibility: Online exams can be taken from any location with an internet connection, allowing students to take the exam at a time and place that is convenient for them.
Reduced costs: Online exams can be more cost-effective than traditional exams, as they do not require the use of physical examination materials, such as paper and pens.
Improved accessibility: Online exams can be more accessible to students with disabilities, as they can use assistive technologies, such as screen readers, to complete the exam.
Enhanced security: Online exams can be more secure than traditional exams, as they can be designed with a range of security measures, such as proctored exams, to prevent cheating.
Faster grading: Online exams can be graded automatically and quickly, allowing for faster feedback and results.
Enhanced learning: Online exams can be designed to provide students with immediate feedback on their performance, allowing them to learn from their mistakes and improve their skills.

Design of a KBA examination

How to balance the amount of questions, parts and types of questions in an exam:

When designing an exam for medical students, it is important to carefully balance the number of questions, the number of parts within each question, and the types of questions that are included. Here are some tips for achieving this balance:
Determine the overall length of the exam, and the amount of time that students will have to complete it. This will help to determine the total number of questions that can be included on the exam.
Divide the exam into sections or parts, based on the different topics or concepts that will be covered. This can help to organize the questions and make it easier for students to navigate the exam.
Include a variety of question types on the exam, such as multiple choice, short answer, and essay questions. This can help to assess students' knowledge and skills in different ways, and can prevent the exam from becoming repetitive or boring.
Within each section or part of the exam, carefully balance the number and types of questions. Avoid having too many of the same type of question, and make sure that there is a good mix of easy, medium, and difficult questions.
Review and revise the exam to ensure that it is well-balanced and effective. Consider seeking feedback from other educators or experts in the field, and make any necessary changes to improve the exam.

Design of a progressive examinations

How to design o good progressive examination sequence for medical students:

Progressive examination sequences for medical students should be designed to gradually introduce them to increasingly complex aspects of medicine, allowing them to build their knowledge and skills in a structured manner. Some key considerations in designing a progressive examination sequence include:

Starting with fundamental concepts: The examination sequence should begin with core concepts and principles of medicine, such as anatomy, physiology, and pathology, that form the foundation of medical knowledge.
Building on previous knowledge: Each subsequent examination should build on the knowledge and skills acquired in previous exams, allowing students to gradually expand their understanding of the subject matter.
Incorporating clinical experience: The examination sequence should include opportunities for students to gain hands-on clinical experience, such as through patient interactions or simulations, to help them apply their knowledge to real-world situations.
Providing feedback and support: The examination sequence should include regular feedback and support from instructors to help students understand their strengths and weaknesses, and provide guidance on how to improve.
Assessing a range of skills: The examination sequence should assess a range of skills, including not only knowledge and understanding, but also practical skills, such as physical examination and diagnostic reasoning, as well as communication and professionalism.

Overall, a good progressive examination sequence should be carefully planned and structured to provide medical students with a comprehensive and well-rounded education.

Cut scores - Angoff versus Cohen methods

Angoff and Cohen are two different methods used to set cut scores on standardized tests.

The Angoff method involves having a panel of subject matter experts estimate the difficulty of each test item and the minimum score a minimally competent examinee would need to get the item correct.
The Cohen method involves having the same panel of experts estimate the probability that a randomly selected examinee who just meets the cut score would get the item correct. Both methods are commonly used in educational testing to set cut scores for passing exams.

Perform an Angoff evaluation

To perform an Angoff evaluation, the following steps can be followed:

Identify the specific test or examination that is being evaluated.
Assemble a panel of subject matter experts who are familiar with the content and format of the test and have experience in evaluating the performance of test takers.
Provide the panel members with a copy of the test, along with any relevant information about the test format and scoring.
Have each panel member independently evaluate each test question and estimate the minimum score that a minimally qualified test taker would need to achieve in order to pass the test.
Calculate the average of the panel members' estimates for each test question to determine the final minimum passing score for the test.
Review and refine the minimum passing scores as needed based on feedback from the panel members and other stakeholders.

The Angoff evaluation method is commonly used in educational assessment to help determine the difficulty of test questions and set a fair and accurate minimum passing score for a test. It is important to ensure that the panel members are knowledgeable and experienced in the relevant subject matter, and that they are provided with sufficient information and guidance to accurately evaluate the test questions.

Article: Ortrac: manage an Angoff Session

Importance of recalculation of passmarks

The recalculation of pass marks is an important process in educational assessment because it helps ensure the fairness and accuracy of test results. Pass marks, also known as cut scores, are the minimum scores that a test taker must achieve in order to pass a test or course. The pass mark is typically set by the educational institution or program based on the difficulty of the test and the desired performance level of the test takers.

However, over time, the difficulty of a test or course can change due to factors such as changes in the curriculum, new instructional methods, or the performance of previous test takers. As a result, the pass mark that was originally set for the test may no longer be appropriate. Recalculating the pass mark on a regular basis helps ensure that it continues to accurately reflect the current difficulty of the test and the performance level of the test takers. This can help ensure that test results are fair and consistent, and that students are being evaluated according to the same standards.

Negative scoring in examinations

Best practise using negative scoring in examinations

Negative scoring in examinations, also known as "deductions," involves subtracting points from a student's score for incorrect answers or penalties, such as for late submissions or cheating. While negative scoring can be a useful tool for ensuring the integrity and fairness of examinations, there are some best practices to consider when using this approach:

Be transparent about the negative scoring system: It is important to clearly communicate the negative scoring system to students before the examination, including what actions will result in deductions and how much will be deducted for each infraction.
Use negative scoring sparingly and only for serious infractions: Negative scoring should not be used for minor errors or mistakes, but only for serious infractions that have a significant impact on the fairness or integrity of the examination.
Limit the amount of points deducted: The amount of points deducted for each infraction should be reasonable and proportional to the severity of the infraction. Large deductions can unfairly penalize students, so it is important to strike a balance.
Provide opportunities for students to avoid or minimize deductions: Students should be given the opportunity to avoid or minimize deductions by submitting their work on time, following examination rules, and taking preventative measures, such as checking their work for errors.

Overall, negative scoring can be a useful tool for maintaining the fairness and integrity of examinations, but it should be used carefully and with consideration for the impact on students.

Perform a online KBA examination

A KBA exam, or Knowledge-Based Authentication exam, is a type of security measure used to verify the identity of a person by asking them questions that only they are likely to know the answers to. To perform a KBA exam online, the following steps can be followed:

The person being verified will need to provide some basic personal information, such as their name and date of birth, in order to initiate the KBA process.
The KBA system will then ask the person a series of questions, which may be based on information from public records or other sources, that only the person being verified is likely to know. Examples of such questions could include the person's address, previous employer, or high school attended.
The person being verified will need to answer the questions accurately in order to pass the KBA exam. In some cases, the system may allow for multiple attempts or allow the person to provide additional information to verify their identity if they are unable to answer the initial questions.
If the person successfully passes the KBA exam, their identity will be verified and they will be able to access the online service or resource they were trying to access.

It is important to note that KBA exams are not foolproof and can be vulnerable to fraud if the person being verified has access to the necessary personal information. As such, KBA exams are often used in combination with other security measures, such as two-factor authentication, to provide a more robust level of security.

Write a good single best answer question

To write a good single best answer (SBA) question, the following steps can be followed:

Identify the specific learning objective or concept that the SBA question is intended to assess. This should be clearly defined and aligned with the curriculum or instructional goals for the course or subject.
Develop a question or prompt that clearly and concisely asks the student to demonstrate their understanding of the learning objective or concept being assessed. The question should be well-written and free of ambiguity or confusion.
Create a list of potential answer choices that includes the correct answer and several distractors, or incorrect answers, that are plausible but not quite right. The answer choices should be clear and concise, and should not contain any clues or hints that could give away the correct answer.
Ensure that the SBA question and answer choices are developmentally appropriate for the age and ability level of the students being assessed.
Provide clear and detailed instructions for how the student should answer the SBA question, including any specific formatting or referencing requirements.
Create a scoring guide that will be used to evaluate the student's response to the SBA question. The scoring guide should include clear criteria and benchmarks for assessing the student's performance.
Review and revise the SBA question as needed to ensure that it is clear, effective, and aligned with the learning objective and curriculum.

Overall, a good SBA question should be well-written, relevant, and aligned with the learning objectives of the course or subject. It should also provide a fair and consistent evaluation of the student's performance.

How many options within a single best answer(MCQ) question is best practise?

There is no specific rule or guideline for the number of options that should be included within a multiple choice question. The number of options will depend on the complexity of the question and the amount of information or evidence available to support the various options.

In general, it is best to include a reasonable number of options that cover a range of possible answers and provide the respondent with a variety of choices. It is also important to include at least one clearly correct answer and one or more distractors (incorrect options) to challenge the respondent and test their knowledge.

Here are a few guidelines for the number of options to include in a multiple choice question:

For simple or straightforward questions, three to four options may be sufficient.
For more complex or detailed questions, five to six options may be necessary to provide a range of possible answers.
For very difficult or challenging questions, it may be necessary to include seven or more options to provide a thorough assessment of the respondent's knowledge.

Ultimately, the number of options included in a multiple choice question should be determined based on the specific goals and objectives of the quiz or test, and should be tailored to the abilities and knowledge level of the intended audience.

Write a good true or false question

To write a good true or false question, follow these steps:

Start with a clear and specific statement or fact about a particular topic.
Make sure the statement can be definitively proven true or false.
Avoid using vague or ambiguous language that could make the statement difficult to understand.
Avoid using value judgments or personal opinions in the statement.
Avoid using double negatives or other confusing language that could make the statement difficult to answer.

Here are a few examples of good true or false questions:

e Earth rotate on its axis once every 24 hours?
Is the capital of France Paris?
Do all mammals have hair or fur?
Is the square root of 16 equal to 4?
Was the American Revolutionary War fought between 1775 and 1783?

How many questions within a true question is best practise?

There is no specific rule or guideline for the number of questions that should be included within a true or false quiz or test. The number of questions will depend on the purpose of the quiz or test, the difficulty of the material, and the amount of time available for the respondent to answer the questions.

In general, it is best to include a variety of questions that cover different topics and range in difficulty to challenge the respondent and test their knowledge. It is also important to include an appropriate number of true and false questions to provide a balanced and fair assessment of the respondent's knowledge.

Ultimately, the number of questions included in a true or false quiz or test should be determined based on the specific goals and objectives of the quiz or test, and should be tailored to the abilities and knowledge level of the intended audience.

How to write a good extended matching question

To write a good extended matching question, follow these steps:

Start with a clear and specific topic or concept that you want to test the respondent's knowledge of.
Identify a range of key terms, facts, or ideas related to the topic that the respondent should be familiar with.
Organize the key terms, facts, or ideas into groups or categories that make sense in relation to the topic.
Write a stem or prompt that introduces the topic and explains the purpose of the question.
Write a set of options that includes the key terms, facts, or ideas, along with any necessary explanations or context.
Write a set of answer choices that includes one answer choice for each group or category.
Make sure the stem, options, and answer choices are clear, concise, and easy to understand.
Avoid using vague or ambiguous language that could make the question difficult to answer.

Here is an example of a good extended matching question:

STEM: Match the type of glacier with its description.

OPTIONS:

Valley glacier
Ice cap
Ice sheet
Piedmont glacier

A. A large mass of ice covering a highland area

B. A glacier that flows outward from a central ice dome

C. A glacier that flows down a valley

D. A glacier that spreads out at the foot of a mountain

ANSWER CHOICES:

In this example, the stem introduces the topic of glaciers and explains the purpose of the question, which is to match the different types of glaciers with their descriptions. The options provide a list of key terms related to glaciers, along with their definitions. The answer choices include one answer choice for each type of glacier, and the respondent must match the correct answer choice with each option to answer the question correctly.

How to write a good short answer question

A good short answer question should be specific, focused, and clear. It should ask a question that can be answered concisely, without the need for lengthy explanations or extended examples. The question should also be relevant and meaningful, and should encourage the respondent to engage with the topic in a thoughtful and thorough manner. To write a good short answer question, consider the following steps:

Identify the topic or concept that you want to ask a question about. This should be something that is relevant to the course material or subject matter, and that will be of interest to the respondent.
Determine the specific aspect of the topic or concept that you want to ask a question about. This should be a focused and narrow topic, rather than a broad or general one.
Craft a question that is clear and concise, and that can be answered in a few sentences or a short paragraph. Avoid asking multiple questions in one, and avoid using complex or obscure language.
Consider using a prompt or a scenario to provide context for the question, and to help the respondent understand what is being asked. This can help to make the question more engaging and relevant.
Review and revise the question to ensure that it is clear, focused, and relevant. Make sure that the question is not too broad or too narrow, and that it can be answered easily and effectively.

Thessialoniki - KBA examination Whitepaper Print