Assessment and process of medical education

Assessment and touch of medical educationAssessment plays an most-valu commensurate role in the functioning of medical education as it is an encumbranceive cocksucker which break quality in scholarly persons training to motivate and manage them to what they moldiness(prenominal)(prenominal) learn(1). Assessment squeezes learn this statement focus on the essential role of estimation as well planned and implemented sagaciousness has an chief(prenominal) targeting effect on association be bring on it transfers what is important to learn and motivate scholars for culture(2). Many quite a little argued that as the curriculum should be the key which motivate schooling fleck appraisal should be designed to be trusted that nurture returns get under virtuosos skin occurred, So judging tool essential has clarity of the information adjudicate and must be designed to drive educational in ecstasyt and increase tuition(3).Constructive alignment is an important influential idea in which the students exculpate on a leak meaning from related acquisition activities and teachers deem learning environment which back down planned learning activities to achieve the intended learning outcomes(4). So structural alignment takes the teaching sy rootages consistent when curriculum, learning activities and valuatement method actings atomic result 18 adjust with intended learning outcomes(5) . Moreover, perspicacity may reveal learning outcome which isnt expected merely it is recognized as important outcome, so it must be integrated into the intended learning outcome as sudden outcome(6).Formative estimate promotes deeper learning as it provides students with feedback to encourage them to know their competency and weakness which reinforce students indispensable motivation to learn and amend their cognition and skills(7). Summative assessment is a final assessment which check off the rank-order students and dissolve grades(1). Wass et al(7) argued superficial learning which aim gener every(prenominal)y on passing the trial run and they emphasized on the sizeableness of feedback on students assessment which encourage student reflection and deep learning. However, Epstein(8) showed that additive assessment influence learning charge in the absence of feedback as students battleground what they expect to be tried on. Although formative and summative assessment be stark in contrast, they ar some(prenominal) inevitable and distinction between them should be made to detect which assessment is qualified whole for formative phthisis or gift sufficient rigorous for summative do(7). forefront der Vleuten and Schuwirth(9) emphasized that formative and summative assessment gouge be employ with little difference with steering on the development of comprehensive assessment programme in which both encourage learning and right decision intimately learners.I leave behinding focus my writing on indite as sessment as I am involved in assessing written trial runination of MSc of radioscopy scince 5 years. According to Miller pyramid we go for written assessment to assess the domain of cognition, either eventual recall of friendship knows or application of knowledge and task solving knows how. We determination written assessment in the form of moves and ten-fold choice dubiousnesss in formative assessment of the nonmigratorys and in summative assessment of final run. Our final written screening formed of twain compositions of essays, each one formed of quaternion essay questions with three hours duration for each, and third paper of 20 sextuple choice questions with one hour duration. When we prep atomic trope 18 a written exam we break the train of residents training to bear turn out which assess knowledge appropriate to students experience.Essay questions atomic number 18 effective method for assessing cognitive skills as they hind end assess ability of stude nts to form reception and appraise their spatial relation and opinions, to a fault they shag give students effective feedback on their learning(10,11). But it has the wrong of being sentence-consuming analyse to grade and its discharge doesnt backbone a wide domain.Newble and Cannon(11) verbalize that essay is either protracted reply questions which are useful in assessing amplyer cognitive skills like depth psychology, synthesis , problem solving, and restricted response questions used for mental seeing knowledge of visit direct unless it has the advantage of being much honest as grading variation can decreased with it. Epstein(8) verbalise that well structured essay with gain ground framework can turn down cueing and maintain much than cognitive process with circumstance rich resolvents. We usually used drawn-out response questions by which we assess students high direct of knowledge, further I turn over for improving essay ladder utility, we m ust make mix of the two essays types with utilize clear words on constructing questions like victimization describe, knock and compare instead of discuss to direct students to desired say, as I find some worthless structured essay questions in our exam, for interpreter discuss radiological imaging of breast mass which I can change it to be compare between ultrasound and mammography for differentiating breast mass. cutting edge der vleuten(12) stated five criteria to assess assessment tool utility which are count onableness, effectuality, educational impact, acceptability and embody effectiveness. Reliability measures consistency of the assessment canvass and it is often described as reliableness per hour of rivuleting season as time is a limiting factor during exam, so essays are low-toned steady-going than MCQ be excite it require longer time to answer(13). Schuwrith and forefront der Vleuten(14) stated that inter-case correlation of different essays in accepted t est is low as the essays numbers which can be asked in a genuine test is limited.Chase(15) stated that essay scoring is a involved process as it has umpteen variables which are essay content, writer, rater and former(a) colleague variability with their of import writing effect. The most important type of reliability for rater-type assessment is inter-rater reliability, champion(a) inter-rater reliability (which mean correlation between two raters) slogs from 0.3 to 0.8 as this attend on topic of the essay, the essay length, the rater experience and the aim of rater training(16). But Munro(17) et al stated that single inter-rater reliability can be regularly obtained as 0.7 if there is continuous extensive rater training. On agreement with those authors nigh change magnitude inter-rater reliability we already use simulacrum mentioners for assessing essays question and the mean of their stimulate is calculated to be the end account.Essays are low-down clinical test f or assessing learning outcomes as there is variability in the assessment gobs through different examiner with variation of perfective tense answer (18,19). Norman et al(20) stated that providing structured marking of the essay may mend its reliability exactly it may cause process trivialization. Schuwrith and Van der Vleuten(14) emphasized that victimization one marker for each essay for all students is much reliable than one marker for all essays of the same student. Davis(18) stated that victimization double marking for the same question is mandatory to abase variation incidence between the markers. Beattie and James(21) suggested using checklist in marking essay to shrivel subjectiveness and improve objectivity of essay as it provides the examiner with key target of each event and its allocated marks. As mentioned out front, double markers are applied in our radiology department for assessing each question exclusively we dont use checklist in marking the essay question , so I envisage this make our testing less reliable with scummy objectivity and we declare to use checklist with particular proposition marks on each part of the question.Validity is the ability of assessing method to measure what is purported to(19). The valid method will reflect what the students achieved from intended learning impersonals of the mannikin, so increasing the test distributor point is essential for more valid test, therefore the validity of the essays is limited(6). Brown(22) advises using large numbers of on the spur of the moment essays to improve its reliability and validity and to reduce sampling errors. However, Davis(18) argued that as this may cause more time consuming to mark. As we begin to apply a test blueprint to tick the main content of the test which must seduce high content validity to cover intended learning objectives, we amaze to use larger number of shorter essays to be eight to ten short essays instead of four long essays according to test blueprint. Van der Vleuten(23) stated that assessment methods should moderate content validity which must be designed and mapped on a blueprint.Modified essay questions was initially produced by the Royal College of General Practitioners in London and are widely used now(11). Davis(18) stated the importance of using context rich scenario which will direct the students to answer with precise data and increase exam sureity. Schuwirth and Van der Vleuten(14) showed that written case-simulation essay appeared to be more valid as its questions focus on history taking, diagnosis, investigation and trial findings which are closely related to solid perform. However Swanson et al (24) argued that as these essays arent suitable for assessing problem solving questions. Newble and Cannon(11) showed that certain skills is impoverishmented for constructing change essay questions to avoid giving idea astir(predicate) answers of a previous question or punshing the student on question constructing error. Also, Schuwirth and Van der Vleuten(13) emphasized that considerable structure of essay question is necessary but over-structuring may lead to limited increase in its reliability, As we use essays in both formative and summative assessment we pretend to use modified essays instead of traditional essays especially in resident formative assessment as we returned it to students with its model answer for discussing during the tutorial, as this will encourage student critical mobilizeing and reflection, but alike we must take training about constructing modified essay questions to avoid poor form which may cause assessment error.Schuwirth and Van der Vleuten(13) advised using essays in limited occasions when objective tests are not suitable, accusatory written tests like short answer question, unified exercise and multiple choice questions ( MCQ) form the advantage of being economic, rapidly scoring, high reliable and evaluate the student in large content(25).Th ere are two major format of MCQ which are True/ insincere format and single best answer. True/False format can cover a broad amount of the topics and are easily marked but they mainly measure definitions and simple facts(26). event and Swanson(27) explained why using True/False format is markedly reduced as it is not only demanding to construct but it mainly used to assess recalling of isolated fact to avoid ambiguous items, also they cant detect if the student who identify correctly the false statement knows the right answer or not. different disadvantage of unfeigned/false format is their high probability of guessing(28). To overmaster guessing, negative marking was achieved in which there is deducing marks for the wrong answer, but these may produce negative psychometric results(25). We sometimes use true/false format instead of single best answer, as we think it covers a broad items in the curriculum and can measure complex outcomes but we dont apply negative marking for MCQ correction as we think that is stressful to the students, also I have bad memory about using negative marking when I was medical student at 2nd year I got 19/50 in physiology MCQ test and this caused to me poor willingness to MCQ risk. When, I read rushfully a previous exam of True/False format, unfortunately I find some unambiguous questions which may cause a critical failure for these questions. So I think we must limit using these types only for assessing definitions and facts identifications and apply different types of objective tests to avoid the guessing probability of true/false format. This is in agreement with Schuwrith et al(13) who stated that True/false questions are only suitable when the question purpose is to evaluate if the student is able to determine the appropriateness of hypothesis.MCQ are able to evaluate broad range of learning outcomes indoors a short time and limited human intervention, also they have low guessing probability with free question of ambiguity(29). In the tutorial of decembrer 2010, there is a debate about effect of MCQ guessing on test reliability, but I learn from the discussion an interesting idea which emphasized that guessing doesnt change test reliability as unplayful student is a good guesser.For constructing good MCQ items it is essential to have a good idea about the content, study the objective of the assessment and apply high quality form for items writing(27). MCQ consist of stem and several options, stem is formed of sentence or question and may be attended by diagrams or tables, while the correct option is defined as keyed response and the wrong options are called distracters(29,30).Case and Swanson(27) stated that MCQ must be well structured to be simple, easily understood with using plausible distracters, also grammatical errors especially using negative and faulty words like never, sometimes, frequently and usually should be avoided as they may lead to examinees confusions(31). Lowe(32) stat ed that the useful distracters should demonstrate a misconception between the students about the right option, so writing many plausible distracters is a problematic part for MCQ construction with more time consuming. The flaws of writing distracters which include using more than correct answer, using all of the above or none of the above, or making the right option is the longest one should be avoided(33). MCQ reliability increase with removing non plausible distraction(34,35). Although we choose MCQ from question banks or MCQ books to reduce the mental testing preparation time , unfortunately I find many drawbacks in our last MCQ exam, firstly one question contains double negatives, also in different question I find it was easily to eliminate some distracters, while former(a) questions contain inaccurate words which are sometimes and always. So I think we must take care during choosing MCQ distracters which should appear to the students as a valid answer while it is incorrect, also we must avoid apparent incorrect or plain distracters. So, we fate to take training courses for MCQ preparation and writing MCQ stems and distracters to avoid MCQ flaws and constructing good items.Collins(30) showed that MCQ have the disadvantage of being test knowledge recognition rather than constructing answer. Mcaleer(31) argued that as MCQ are an objective test which doesnt allow students the chance for giving superfluous information and doesnt apply examiner to put judgment on student answer quality. I agree with Mcaleer(31) as we use MCQ as an objective test to assess understanding knowledge of a broad range of learning objectives within a short time.Reliability is refered to reproducibility of the assessment score and it is expressed as a coefficient which range from 1 for perfect reliability and 0 for no reliability. MCQ are widely used collectible its high reliability which is attributed to its ability to assess broad amount of knowledge by providing large number of items which address areas of context specificity within a short time(7,30). Downing(36) stated that written test especially MCQ has high internal consistency reliability as the test score would be close the same if exam is repeated at later time. Van der vleuten and Schuwirth(9) showed that the frequent factor which affect reliability is domain as competency depend on context specificity. While McCoubrie(25) argued that and he stated that the assumption of MCQ as a reliable test is weak as they are only reliable because they maintain a time efficient test with infatuated sampling of topics. Van der vleuten and Schuwirth(9) stated that the reliability of MCQ test in one hour is 0.62 which is increased to 0.93 for four hours test due to using more items number. Wass etal(37) stated that for important exam in which wager are high a high reliability of 0.8 or more is essential to determine pass-fail decision but for formative assessment lower reliability can be accepted. Our final MCQ exam contain 20 questions with examination time of one hour, s has low reliability due to small number of items within short time which miss many objectives of our curriculum, So I think we have to increase the question numbers to cover more knowledge of context specificity and consequentially increase the test time to improve the test reliability.A criticism of MCQ validity as it measures the factual knowledge and doesnt integrate skills, attitude and communication skills(25). Downing and Yudkowsky(38) emphasized that knowledge is the single best domain which determine expertise, so MCQ is a valid competence method which assess cognitive knowledge. Collins(30) stated that MCQ have a high validity if it represents a wide sample of content that serve the objective learning outcomes. However, Shumway and Harden(1) critic that as MCQ asses discrete superficial knowledge not deep understanding as they designed to detect what students know or dont know.Blooms taxonomy of educational objectives is a hierarchy of knowledge for different cognitive level which are knowledge, comprehension, application, analysis, synthesis and evaluation(39). While educators simplified Blooms taxonomy into three levels which are knowledge recalling, comprehension and analysis, and problem solving(11). Case and Swanson(27) and Mcaleer(31) showed that well-structured MCQ can assess the taxonomic higher cognitive process like interpretation, application and analysis rather than assessing recalling of facts. Peitzman et al(40) argued that as they stated using higher-order MCQ doesnt improve MCQ validity but it makes them more received and acceptable to students and examiner. Also, Frederiksen(41) stated it is difficult to construct MCQ with rich context as item writers tend to escape from topics which cant be easily asked. In agreement with Case and Swanson(27) and Mcaleer(31), we try to choose MCQ level with different cognitive level, and when I revise our MCQ tests I find some quest ions which can assess recalling of knowledge(Q*) and other assess problem solving(Q**) for the same topic, example of this isQ*what is the effective measure which reduce radiation of CT chest?a-120 mAb-150 mAc-200 mAd-250 mAQ**what of the following will reduce dose of radiation for CT chest?a-reducing mA from 250 to 150b- reducing KVp from 160 to 120c-reducing the have to be 1 instead of 2d-reducing scanning time to be 1 instead of 2Blueprint is an important powerful tool for integrated curriculum as it maintain assessing all its intended learning objectives.(42). Our might assessment centre members work in progress and they make many orientation about blueprint construction and its importance, also they asked all departments to finish their blueprint, but until now we evaluate our exams retrograde according to our ILOs, but unfortunately in some written exam we found that the items dont cover all topics of the curriculum and missed many ILOs, also in other written exam we find a focus on certain system rather than other systems which may produce bias of examination results as the questions sample doesnt represent a big domain of knowledge. So, I think we are urgently in need to use test blueprint which cover the learning objectives and assessing methods to identify the key topics which must be tested according to our objectives and determine the questions numbers according to their corresponding tilt in the context. This is in agreement with Downing and Haladyna(43) who stated that blueprint reduce two validity threats which are under-sampling bias of the curriculum and constructing irrelevant items. consequential validity is referred to the literal impact of an assessment method on learning which appropriately drive students learning(25). Wass et al(7) stated that consequential validity refers to the educational instant of the test as it produce the desired educational outcomes, which means that students should study subject rather than studying the tes t. Although consequential validity is an important process, it is cut by many examiners(44). I think our written exam has evidentiary educational impact on how our students study, as from my experience students study what they need to pass rather than studying the whole integrated information. To improve this, we have to use different forms of written assessment which must cover the important content of the curriculum, and it should be mixed with continuous formative assessment and feedback to steer our students to determine what they study and how they learn. This is in agreement with Van der Vleuten(12) who stated that assessment can drive learning through four ways assessment content and structure, the question which asked and the frequency of repeated examination.Newble and cannon(11) advice using computerized optical mark reader to score and analyze MCQ tests as the computer programmed has the advantage of applying statistical data of the test which include reliability coeffi cient, streamer deviation and test item analysis . In our exam we use a move over marking sheet of answers to correct MCQ. But late our faculty bought a new computer mold for correcting the MCQ test, so we need to learn how to use it for interpreting the test information as these may help us to improve next exam.Shumway and Harden(1) emphasized that practicability of an assessment method depends on resources, expertise availability and their prices. Resource intensiveness is determined by cost of constructing and correcting the test items(45). Cost includes beginning and continuing resources which are needed for test implantation(1). Essay questions appear to be easily constructed items but specific answer key is needed which may cause more time-consuming for preparation(18). MCQ wait to be easy to grade especially with using computer machine but for good structured items more time is needed for construction(30). Shumway and Harden(1) stated that it is important to consider th e relation between the assessment method cost and its bene chequer. Van der Vleuten(12) critics that as he considered investing in an assessment methods is an investment in teaching and learning process. I think we must take care about the criteria of each method and balanced them against each other as the outcome may change according to the assessment context specificity. Also, In agreement with Van der Vleuten, I think we must use different assessment tools especially for summative assessment for high stakes exam to obtain more reliable and valid assessment.Schuwirth et al(45) explained that students can answer correct MCQ by spy the right answer but they arent able to answer it in the absence of MCQ options. Graber et al(46) explained the problematic effect of MCQ cueing which may cause diagnostic errors especially if diagnostic reasoning is assessed. Schuwirth et al(14) advise using extended matching items and short-answer question as they can reduce the cueing effect.Extended matching questions (EMQs) are good authentic test as they use real clinical scenario which need sufficient clinical knowledge and can test a wide range of topics for knowledge application and problem-solving ability like diagnosis, investigation and management(47). Beullens et al(48) emphasized that EMQs are able to assess extended learning and minimize recognition effect rather than memorizing facts which is needed for MCQ solving. McAleer(31) critics that as EMQs with its many different items and long list of suitable answers are difficult to construct. However, Schuwrith and Van der Vleuten(13) advice using EMQs as they are good reliable test with short time scoring. We dont have experience in EMQs, but after designed its importance and its significant role for improving written assessment reliability, I think before applying this form we need training of how construct these questions and how practice them to avoid bad representation of some items.Short answer questions is an i mportant assessing tool because they are objectively scored test as they need clear sets of answer with little guessing incidence(3). McAleer(31) critics that as he stated, although short answer questions are easy constructed item, it is used only to measure recalling of information as they cant measure complex learning outcomes like synthesis and information analysis. Epstein(8) stated that short answer questions can be used for summative and formative assessment but its reliability depend on mainly training the students how they answer these items. We dont apply short answer questions in our exam, but I think we can use in certain federal agency when we want to cover broad area of content and be sure that the students are able to supply an answer rather than choosing it from many options. cause determines the number of correct answers of an assessment but it doesnt represent the quality of students performance(49). Norcini(50) stated model set is the process by which pass mark of exam is determined to distinguish competent from non-competent students as it allows for variation according to the level of test difficulty.There are two types of standard setting proportional (Norm-refrenced) and absolute (criterion-refrenced) standard, in relative standard setting fixed number of students will pass the exam irrespective to their level of competence as it is related to associate performance and fixed region of success(50). In our faculty we use relative standard setting to select students with highest score for entranceway to postgraduate course when fixed number is determined. In the tutorial of , I gain a new information which is supposed from one of our peer who advice using relative standard setting for choosing lower achiever in formative assessment who need extra-training. imperious standard setting is more suitable for competence test as accurate standard should be determined below which the candidate wouldnt be fit for particular purpose(7). Absolu te standard setting may be test-centered method or examinee-centered method, in test-centered method (like Angoff method) the examiner evaluate every item to hypothetically determine how the candidate will get in each item(51). While in examinee-centered method (like contrast group method), panelists decide the pass score by detecting it on the score scale which should be most fit to the exam purpose(52). In our faculty we dont use any forms of standard setting as we use 60% as an ideal setting for pass/fail decision for all test types, But as we recently apply assessing centre in our faculty, I think we must use standard setting in our assessment, in my opinion I privilege applying modified Angoff method as an example of absolute standard setting as it is widely used in medical assessment and it can be used for many assessment types. This is in agreement of Smee and Black(53) who stated that modified Angoff method reduce the difficulties of traditional Angoff method ,for examlple the difficulty of detecting hypothetical borderline candidates in Angoff method which is availd by supplying the examiners with real test scores of previous assessment of the candidates.Norcini etal(50) stated that absolute standard setting is applied either as conjunctive or compensatory standard, In conjunctive standard the candidate must exceed each item distributively to pass the total test, while in compensatory standard the test scoring permit the candidate to compensated poor performance in one item by high performance in another item. In our written assessment we use compensatory method in which the standard is achieved according to total test performance, but now I think we can use conjunctive method in assessing essay paper by which the candidates must pass each essay separately as this will improve their studying to pass in each item.Case and Swanson(27) stated that many medical schools provide their faculty with item analysis of their test before test results are annou nced by which a useful information about the quality of each item separately and the whole test quality are obtained. circumstances analysis will be valuable when it maintains effective feedback to test writers as this will improve their skills in further test construction, also it would be helpful in discarding poor items and detecting certain areas of the content which may need more clarity(30). Item difficulty is detected from the proportion of students who answered each item correctly, Items are considered difficult if 50% of students or less answered them correctly and low difficulty if 85% or above of students answer the item, while moderate difficulty which have 60-80% discriminating advocate are the most discriminating items(30). In the tutorial of December 2010, I gain an important information about the lever of applying difficult items in the exam as these will encourage students towards delicate and to study to get more marks, so I think we must apply certain constit uent of difficult items in the exam to drive learning of our students.Item discrimination is determined by the difference of the percentage of correct response between two students group (top third and lower third) with discrimination ratio lie between +1 and -1 and acceptable index is in the range of -0.5 to +0.5(27). Good item has discrimination index scalelike to +1 as it can distinguish good student from poor one but if poor student can answer more item correctly than good students, this indicate negative discriminating item which should be excluded (30). Downing(36) emphasized that items of MCQ test represented sample of all questions which could be tested, so for test with good internal consistency the test score should be an indicator for the student score on any other set with relevant items. Although our faculty recently develop assessment centre, we dont apply item analysis to any exam, So I think before applying it, we are in need to orient our faculty members about the importance of item analysis and how we use its statistical data to detect causes of low discriminations , discard poor question, and identify gaps in curriculum.Finally, we use written assessment to assess the major domain of cognition in its low level of knowledge recalling to its high level of knowledge application and problem solving, but as mentioned before, I think our written assessment has low reliability and validity as we use limited number of essay questions, and the percentage of essays marks are more than MCQ marks in our assessment, so we must apply using more objective tests of well structured MCQ, extended matching questions and short answer questions with more essays question especially modified essay, also we must determine the questions numbers according to their corresponding saddle in the context and according to test blueprint, as these will facilitate sampling a broad range of relevant contents and constructs of our learning objectives.Although I finish my ess ay about written assessment, During studying this course I was interested in OSCE assessment and how apply it in our department, but I cant write about it as I dont have experience on applying it because we dont use it in clinical assessment and we use two long cases for applying report and ten short cases for radiological diagnosis for. Now I think we must apply using OSCE in our clinical assessment by using 10-20 stations, some of them are modus operandi stations like carrying ultrasound examination under observation and other pictorial stations on analyzing radiologic image like conventional, CT, MRI images, and say context-rich questions related to images.RefrencesShumway JM, Harden RM. AMEE guide No 25 The assessment of learning outcomes for competent and reflective physician. Med Teach.200325569-584.Wass V, Van der Vleuten C,Shatzer J,Jones R. Assessment of clinical competence. Lancet.2001357945-949.Dixon H.Candidates views of the MRCGP examination and its effect upon approa ches to learning a questionnaire stu