Skip Navigation


Occupational Medicine Advance Access originally published online on January 18, 2008
Occupational Medicine 2008 58(2):83-87; doi:10.1093/occmed/kqm151
This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrowOA All Versions of this Article:
58/2/83    most recent
kqm151v1
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrow Disclaimer
Google Scholar
Right arrow Articles by Kok, R.
Right arrow Articles by van Dijk, F. J. H.
PubMed
Right arrow PubMed Citation
Right arrow Articles by Kok, R.
Right arrow Articles by van Dijk, F. J. H.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?

© 2008 The Author(s)
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/2.0/uk/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Evaluation of a workshop on evidence-based medicine for social insurance physicians

Rob Kok1,2, Jan L. Hoving1,2, Jos H. Verbeek2,3, Frederieke G. Schaafsma2, Paul B. A. Smits2 and Frank J. H. van Dijk1,2

1 Research Center for Insurance Medicine AMC-UWV-VUmc, Amsterdam, The Netherlands
2 Coronel Institute of Occupational Health, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands
3 Finnish Institute of Occupational Health, Knowledge Transfer Team, Kuopio, Finland

Correspondence to: Rob Kok, Coronel Institute of Occupational Health, Academic Medical Center, University of Amsterdam, PO Box 22700, 1100 DE Amsterdam, The Netherlands. Tel: +31 20 56 639 03; fax: +31 20 56 692 88; e-mail: r.kok{at}amc.uva.nl


   Abstract
Top
 Abstract
Introduction
 Methods
 Results
 Discussion
 Conflicts of interest
 Appendix: Workshop 'Introduction...
 References
 
Background Evidence-based medicine (EBM), a comprehensive method to support clinical decision making by using evidence, has been instrumental in clinical specialties but not yet in insurance medicine.

Aims We developed and evaluated a workshop on EBM for Dutch social insurance physicians who perform disability evaluations.

Methods Sixty-six social insurance physicians followed a 1-day introductory workshop that focused on teaching two EBM core skills: to ask answerable questions and to search for the best evidence. All outcomes were measured before, immediately after and 3 months after the workshop by means of self-assessment. The primary outcomes were knowledge, skills, attitude and intention to apply EBM in practice. The secondary outcomes were social influence, self-efficacy and behaviour.

Results Immediately after the workshop, a marked and significant improvement was seen in self-assessed skills (mean difference 4.2, 95% CI 3.7–4.6) and in self-efficacy to apply EBM (mean difference 0.7, 95% CI 0.6–0.8). For attitude, knowledge and intention, the improvements were small. Three months after the workshop, the improvements in skills (mean difference 2.3, 95% CI 1.8–2.9) and self-efficacy (mean difference 0.5, 95% CI 0.3–0.6) remained significant.

Conclusions The workshop improved self-assessed EBM skills and self-efficacy both in the short and long term. The workshop also resulted in limited short-term improvements in self-assessed knowledge and in the intention to apply EBM in practice. The EBM approach can be successfully taught to social insurance physicians working in the field of disability evaluation.

Keywords      Continuous medical education; disability evaluation; disability [MeSH]; education (MeSH); evidence-based medicine; insurance medicine; postgraduate education


   Introduction
Top
 Abstract
 Introduction
Methods
 Results
 Discussion
 Conflicts of interest
 Appendix: Workshop 'Introduction...
 References
 
Evidence-based medicine (EBM), as a comprehensive method to support clinical decision making by using evidence, is instrumental in many clinical specialties [1]. EBM is now recognized as a medical milestone and the question has moved beyond ‘why is evidence-based medicine important?’ to ‘why is it not already a reality?’ and ‘how can we all work together to make it a reality quickly?’ [2]. Recently, the Dutch National Advisory Council on Health Research has recommended that EBM should be applied in the area of insurance medicine and disability evaluation [3]. However, it is unclear how successful EBM methods can be taught to physicians who work in this area.

Disability evaluation can best be described as the assessment of the degree of a physical, mental or emotional handicap. An important aspect of the assessment in the Netherlands is the prognosis of the limitations. Social insurance physicians in the Netherlands evaluate disabilities and assess if claimants medically qualify for benefits under the Disability Benefit Act. These physicians are all employed by the Dutch Workers Insurance Authority (UWV). Worldwide, physicians are involved in similar assessments even though national practices can vary considerably under social insurance or workers' compensation legislation [4]. For various diseases, much evidence has been gathered on the limitations they cause and the course of these limitations over time [5]. As in other medical specialties, physicians in the field of disability evaluation should keep up to date and search for the best evidence to support their decision making.

A recent national survey also showed that social insurance physicians need education in the area of EBM [6]. Although postgraduate EBM courses are available to physicians from other medical specialties, their applicability and efficacy within the context of disability evaluation are unknown. Systematic reviews on the effectiveness of postgraduate EBM teaching report improvements in knowledge of EBM methods, but evidence on other outcomes such as EBM-related skills, attitude and behaviour is less convincing [7].

The aim of this study was to evaluate the applicability and efficacy of a 1-day introductory postgraduate EBM workshop for social insurance physicians working in the field of disability evaluation. Applicability and efficacy were evaluated in terms of self-assessed knowledge, skills, attitude and intention to apply EBM. In addition, we evaluated the effect of the workshop on social influences on EBM practice, self-efficacy related to EBM and behaviour, i.e. the application of EBM.


   Methods
Top
 Abstract
 Introduction
 Methods
Results
 Discussion
 Conflicts of interest
 Appendix: Workshop 'Introduction...
 References
 
All physicians (n = 66) working in two offices of the UWV in the Amsterdam area attended a 1-day introductory workshop on EBM, tailored to the specific content area of disability evaluation. The workshops were held in June 2006 and involved 14–18 social insurance physicians at a time. The workshop tutorials included knowledge questions encountered in daily practice such as ‘what is the prognosis for a person with active rheumatoid arthritis to remain at work in the long term?’ The workshop included tutorials and practical exercises with an emphasis on two EBM core skills [8]: how to ask and write answerable questions (Patient Intervention Control Outcome) and how to search for the best evidence in PubMed. Also included was a short tutorial on another EBM core skill: how to critically appraise the literature. This tutorial focused on principles for the interpretation of statistics and various study designs. Applying the evidence, another EBM core skill, was not included in this workshop. The workshop was delivered by teachers experienced in EBM. Details of the workshop can be seen in the appendix.

The questionnaires were distributed before (T0), immediately after the 1-day workshop (T1) and after 3 months, the final questionnaire by mail (T2). All physicians completed a form which recorded information on personal and work characteristics (e.g. age, work experience and the specific types of disability evaluation); how (often) they kept up to date, searched for answers to questions in their daily practice and what resources they used such as electronic databases, guidelines, websites and hand books.

The main outcomes of the study were measured using a self-assessment questionnaire on EBM, used in a study on occupational physicians [9]. The questionnaire was modified for use in disability evaluation and included questions using the underlying construct of the attitude–social influence–(self) efficacy (ASE) model, which aims to explain change in behaviour [10]. In this model, behaviour, e.g. using the EBM approach in practice, is thought to be directly influenced by the ‘intention’ towards that behaviour, which in turn is influenced by the constructs ‘attitude’, ‘social influence’ and ‘self-efficacy’. Using the adapted pyramid model of Miller [11], questions on ‘knowledge’ and ‘skills’ related to EBM were included as primary outcomes, as these are frequently used in the evaluation of educational interventions. Other primary outcomes were attitude towards EBM and intention to apply EBM. As secondary outcomes, we included social influence on doing EBM, (self) efficacy in doing EBM and applying EBM (behaviour).

At 3-month follow-up, three open-ended questions were added to the questionnaire: (i) describe any search questions used in the past 3 months, (ii) what was the practical use of the evidence which was found and (iii) which measures could be taken to improve the use of EBM in daily practice, i.e. how could disability evaluations be more evidence based.

We calculated the mean score for the questions on attitude (10 questions), social influence (7 questions), self-efficacy (8 questions) and intention to apply EBM (7 questions) which were scored on a five-point Likert scale (ranging from ‘I fully disagree’ = 1 to ‘I fully agree’ = 5). Similarly, we calculated a sum score for the dichotomous questions on ‘behaviour’ (six questions, maximum score 6), knowledge (seven questions, maximum score 7) and skills (six questions, maximum score 6). Paired t-tests were used to compare outcomes between baseline and follow-up, i.e. directly after the workshop and after 3 months.


   Results
Top
 Abstract
 Introduction
 Methods
 Results
Discussion
 Conflicts of interest
 Appendix: Workshop 'Introduction...
 References
 
Before and after the workshop, all 66 physicians completed the questionnaire. After 3 months, 15 physicians (23%) did not complete the questionnaire despite several reminders. Reasons for non-response were unknown (n = 13), medical reasons (n = 1) and lack of motivation (n = 1). The values of the main outcomes at baseline and directly after the workshop showed similar results for the responders and the non-responders. We have therefore no strong reason to suspect that loss to follow-up reflected a major source of reporting bias.

Characteristics of the participating social insurance physicians (n = 66) can be seen in Table 1. Almost all social insurance physicians indicated a need for more information about the assessment of a patient in their routine practice and 35% had used PubMed in the past month.


View this table:
[in this window]
[in a new window]

  		Table 1. Baseline demographic characteristics of participating social insurance physicians (n = 66), their information need and electronic resources used

 
The results for the primary and secondary self-assessment outcome measures are shown in Table 2. A marked improvement was seen after the workshop on skills, i.e. from 1.5 to 5.7 (mean difference 4.2, 95% CI 3.7–4.6). After 3 months, this improvement from baseline remained significant (mean difference 2.3, 95% CI 1.8–2.9). Significant but small improvements were observed for the outcomes attitude, knowledge and intention to behaviour after the workshop, which returned to baseline levels after 3 months.


View this table:
[in this window]
[in a new window]

  		Table 2. The mean sum scores and difference in mean sum scores of the primary and secondary self-assessed outcome measures before (T0), directly after (T1) and 3 months (T2) after the EBM workshop

 
The results for the secondary outcome measures are also shown in Table 2. Social influence, self-efficacy and EBM behaviour showed small improvements from baseline directly after the workshop, but only outcomes for self-efficacy were significant (mean difference 0.7, 95% CI 0.6–0.8) and remained so after 3 months (mean difference 0.5, 95% CI 0.3–0.6) (Table 2). Scores for social influences on EBM practice and EBM behaviour in practice did not change at follow-up.

Examples of answers to the open-ended questions by social insurance physicians after 3 months are shown in Table 3. In the 3 months after the workshop, 32 of 51 respondents searched PubMed to find information for a question. Of the 19 social insurance physicians who did not perform any search, lack of time was reported as the most important reason. Examples of suggested improvements for daily practice are summarized.


View this table:
[in this window]
[in a new window]

  		Table 3. Examples of answers to open-ended questions by social insurance physicians 3 months after the EBM workshop

 

   Discussion
Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
Conflicts of interest
 Appendix: Workshop 'Introduction...
 References
 
Our results suggest that a 1-day introductory workshop on EBM methods is applicable and effective for social insurance physicians as evident from the improvements in self-assessed EBM skills and self-efficacy, directly after the workshop and after 3 months. Directly after the workshop, there was a slight increase in self-reported knowledge of EBM, positive attitude to EBM and intention to apply EBM, but differences were small.

Our study is the first to investigate the effectiveness of an EBM workshop for social insurance physicians. We were able to evaluate the workshop by measuring dimensions of the ASE model and the adapted pyramid model of Miller, describing the process of learning and implementing the EBM approach. Since we used a prospective single-arm intervention study design, it is possible that a trend in time could explain our results. However, no other events happened during the study period that could bring about a substantial increase in knowledge and skills related to EBM. Therefore, we assume that the effects found can be attributed to the workshop.

We found favourable results for the EBM workshop on two core EBM skills: asking answerable questions and finding literature in PubMed. This outcome has also been reported in studies in different settings and different medical specialties, such as internal residents, paediatric residents and occupational therapists [12–14]. Although we found a modest increase in knowledge of EBM directly after our workshop, other studies have reported more favourable and sometimes longer lasting improvements [14,15].

We found no significant improvement in attitude towards EBM directly after the workshop. Other studies evaluating the concept of attitude reported mixed results [16]. We interpret this lack of improvement as a ceiling effect in our study, e.g. attitude towards EBM was positive and remained positive.

The intention to practice EBM directly after the workshop only marginally changed after 3 months. Studies that use intention to change behaviour as an outcome measure are rare. We agree with Schaafsma et al. [17] who concluded that the construct of intention, an intermediate factor in the ASE model, may not have added value outside this model. It will be interesting to see whether behavioural change, which in our view is the ultimate goal of any EBM teaching, can be achieved in the setting of disability evaluation.

Self-efficacy is a construct related to that of skills, which might explain the parallel improvement over time of both of these measures, since most of the questions on self-efficacy dealt with the confidence experienced by social insurance physicians in practising EBM skills. The fact that skills were self-assessed only adds to this.

In accordance with some studies, we did not find a substantial effect on behaviour [14,16]. It is difficult to change behaviour and many barriers for implementing EBM in daily practice have been identified [18]. However, we should not be discouraged in developing a future EBM intervention as examples of successful EBM implementation do exist [19].

In this study, we showed that the EBM approach was successful in changing self-assessed EBM skills and self-efficacy of insurance physicians and that it could be successfully adapted to questions of physicians performing disability evaluations. In order to promote the use of EBM in daily practice, attention will need to be given to potential (organizational) barriers which limit its use. It seems particularly important to deal with lack of time, a commonly reported barrier to apply EBM both in the literature [18] and in our study. Given the short duration of our workshop and the limited workshop focus on the first two EBM steps, i.e. on asking questions and searching the literature, we see opportunities for the development of a more thorough EBM educational intervention, which will incorporate strategies on how and when to apply EBM in busy daily practice.


Key points
  • A workshop on EBM for social insurance physicians improves self-assessed EBM skills and self-efficacy both in the short and long term.
  • The EBM approach can be successfully adapted to the field of disability evaluation.
  • We see opportunities for the development of a more thorough EBM educational intervention, which will incorporate strategies on how and when to apply EBM in the busy daily practice of insurance physicians.

 


   Conflicts of interest
Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 Conflicts of interest
Appendix: Workshop 'Introduction...
 References
 
None declared.


   Appendix: Workshop ‘Introduction to EBM for social insurance physicians’.
Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 Conflicts of interest
 Appendix: Workshop 'Introduction...
References
 
The duration of the workshop was 1 day. During this day, there were theoretical sessions and practical sessions. Interactivity was stimulated in all lectures. The primary goal of the workshop was to become familiar with the first two steps in EBM and critical appraisal, the third step in EBM. The course schedule was divided in a morning session (i–iv) and an afternoon session (v–viii):

(i) Lecture: Theoretical considerations around EBM (45 min).
Goal: To get an overview of the theoretical concepts in EBM.
Key slides:
  • Definition of EBM by Sackett et al. (1).
  • Five steps in EBM.
  • The concept of background versus foreground questions.
  • EBM as a method suited for health-related questions, not for juridical or statistical questions.

(ii) Break-out sessions/practical session (30 min).
Goal: To practise how to formulate answerable questions with help of a PICO.
Clinical scenarios:
  • A case of a fireman with rheumatoid arthritis.
  • Cases from participant's own practice as insurance physicians.

(iii) Lecture: a short overview of study designs and how to recognize them (45 min).
Goal: To provide a theoretical background for critical appraisal.
  • Discussion of the appropriate designs in diagnostic, therapeutic, aetiologic and prognostic research.

(iv) Break-out sessions/practical session (30 min).
Goal: To practise critical appraisal.
This was done by:
  • Critically appraising an article.
  • Assign level of evidence.

(v) Lecture: using PubMed (30 min).
Goal: To explain the basics of PubMed, including like Boolean operators, MeSH terms, filters, etc.
  • This was done by showing an online example.

(vi) Break-out sessions/practical session (90 min).
Goal: To practise how to search in PubMed.
  • This was done in groups of 2–3 with a common case and cases from participant's own practise.

(vii) Lecture (10 min).
Goal: To show some useful (teaching) EBM resources in the Netherlands.
(viii) Questions and Answers.


   Acknowledgements
 
We would like to thank Wouter Reedijk for technical support with the data analysis. Furthermore, we would like to thank Hans Duin and Frans Vlek of the Netherlands School for Public and Occupational Health for the opportunity to evaluate this workshop.


   References
Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 Conflicts of interest
 Appendix: Workshop 'Introduction...
 References
 

  1. Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-Based Medicine. How to Practice and Teach EBM (2000) Edinburgh: Churchill Livingstone.

  2. Dickersin K, Straus SE, Bero LA. Evidence based medicine: increasing, not dictating, choice. Br Med J (2007) 334:s10.[Free Full Text]

  3. Raad voor Gezondheidsonderzoek. Advies Onderzoek Verzekeringsgeneeskunde [Advisory Report Research Insurance medicine] (2004) Den Haag: Raad voor Gezondheidsonderzoek. Publication No. 44.

  4. de Boer WEL, Besseling JJM, Willems JHBM. Organisation of disability evaluation in 15 countries. Prat Organ Soins (2007) 38:205–217.

  5. Krishnan E, Fries JF. Reduction in long-term functional disability in rheumatoid arthritis from 1977 to 1998: a longitudinal study of 3025 patients. Am J Med (2003) 115:371–376.[CrossRef][ISI][Medline]

  6. Choy J. ‘Het woord is aan de professionals’. In: Rapportage over de raadpleging van de verzekeringsgeneeskundigen en arbeidsdeskundigen over hun gewenste beeld van het sociaal medisch handelen. [Survey. The opinion of social insurance physicians and vocational experts regarding their profession] (2005) Amsterdam: Bureau SMZ, UWV. 1–17.

  7. Coomarasamy A, Khan KS. What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review. Br Med J (2004) 329:1017–1021.[Abstract/Free Full Text]

  8. Straus SE, Green ML, Bell DS, et al. Evaluating the teaching of evidence based medicine: conceptual framework. Br Med J (2004) 329:1029–1032.[Free Full Text]

  9. Schaafsma F, Hulshof C, van Dijk FJH, Verbeek J. Information demands of occupational health physicians and their attitude towards evidence-based medicine. Scand J Work Environ Health (2004) 30:327–330.[ISI][Medline]

  10. De Vries H, Mudde AN. Predicting stage transitions for smoking cessation applying the attitude-social influence-efficacy model. Psychol Health (1998) 13:369–385.[CrossRef][ISI]

  11. Smits PBA, Verbeek JHAM, van Dijk FJH, Metz JCM, ten Cate TJ. Evaluation of a postgraduate educational programme for occupational physicians on work rehabilitation guidelines for patients with low back pain. Occup Environ Med (2000) 57:645–646.[Abstract/Free Full Text]

  12. Baum KD. The impact of an evidence-based medicine workshop on resident's attitudes towards and self-reported ability in evidence-based practice. Med Educ Online [serial online] (2003) 8:4.

  13. Dinkevich E, Markinson A, Ahsan S, Lawrence B. Effect of a brief intervention on evidence-based medicine skills of pediatric residents. BMC Med Educ. (2006) 6:1.[CrossRef][Medline]

  14. McCluskey A, Lovarini M. Providing education on evidence-based practice improved knowledge but did not change behaviour: a before and after study. BMC Med Educ (2005) 5:40.[CrossRef][Medline]

  15. Taylor RS, Reeves BC, Ewings PE, Taylor RJ. Critical appraisal skills training for health care professionals: a randomized controlled trial. BMC Med Educ (2004) 4:30.[CrossRef][Medline]

  16. Forsetlund L, Bradley P, Forsen L, Nordheim L, Jamtvedt G, Bjorndal A. Randomised controlled trial of a theoretically grounded tailored intervention to diffuse evidence-based public health practice. BMC Med Educ (2003) 3:2.[CrossRef][Medline]

  17. Schaafsma F, Hulshof C, de Boer A, van Dijk FJH. Effectiveness and efficiency of a literature search strategy to answer questions on the etiology of occupational diseases: a controlled trial. Int Arch Occup Environ Health (2006) 80:239–247.[CrossRef][ISI][Medline]

  18. Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients' care. Lancet (2003) 362:1225–1230.[CrossRef][ISI][Medline]

  19. Straus SE, Ball C, Balcombe N, Sheldon J, McAlister FA. Teaching evidence-based medicine skills can change practice in a community hospital. J Gen Intern Med (2005) 20:340–343.[CrossRef][ISI][Medline]


Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us    What's this?



This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrowOA All Versions of this Article:
58/2/83    most recent
kqm151v1
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrow Disclaimer
Google Scholar
Right arrow Articles by Kok, R.
Right arrow Articles by van Dijk, F. J. H.
PubMed
Right arrow PubMed Citation
Right arrow Articles by Kok, R.
Right arrow Articles by van Dijk, F. J. H.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?