Occupational Medicine Advance Access originally published online on June 20, 2008
Occupational Medicine 2008 58(6):413-418; doi:10.1093/occmed/kqn070
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Psychometrics and validation of a screening instrument for sickness absence
1 Department of Epidemiology, Maastricht University, Maastricht, The Netherlands
2 Present address: Department of Psychosocial Research and Epidemiology, The Netherlands Cancer Institute, Amsterdam, The Netherlands
3 Present address: Dow Chemical Terneuzen, Terneuzen, The Netherlands
Correspondence to: Saskia F. A. Duijts, The Netherlands Cancer Institute, NKI-AvL, Plesmanlaan 121, 1066 CX Amsterdam, The Netherlands. Tel: +31 20 5122485; fax: +31 20 5122322; e-mail: s.duijts{at}nki.nl
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionFundingConflicts of interestReferences |
|---|
Background Absence from work due to psychosocial health complaints has considerable negative effects for employees, employers and society. A better and more effective strategy would be early identification of employees at risk for psychosocial sickness absence and early intervention to prevent sickness absence as far as possible.
Aims To assess psychometric characteristics and the external validation of a recently developed screening instrument, called ‘Werkwijzer’, for the identification of employees at risk for sickness absence due to psychosocial health complaints.
Methods Exploratory factor analysis was applied on items of the instrument. Cronbach's alpha coefficient was used to determine internal consistency of the subscales. Sum scores on the screening instrument were correlated to absenteeism data to determine their potential for predicting sickness absence. Predictive value was investigated, using objective sickness absence data as criterion measure. Sensitivity and specificity rates were calculated for external validation.
Results For both men and women, three interpretable factors were found. Sum scores on the instrument showed low correlations with sickness absence. The association between ‘being at risk’ (yes/no) and sickness absence (yes/no) was odds ratio (OR) 3.1 (95% CI 1.5–6.5) for men and OR 2.0 (95% CI 1.4–2.7) for women. Sensitivity scores were rather low, whereas specificity scores were remarkably high.
Conclusions The results of this study provide the screening instrument a theoretic and scientific basis. Predictive value is promising, when absence is treated as a dichotomous measure. Sensitivity and specificity were unvarying during the development and validation phases of the instrument.
Keywords Instrument; prediction; psychometric; sickness absence
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflicts of interestReferences |
|---|
Absence from work due to psychosocial health complaints, such as depression, fatigue, stress or work family conflicts, has considerable negative effects for employees, employers and society [1]. For the individual, reintegration after psychosocial absenteeism causes more difficulties and generally takes longer than return to work following physical problems. For both employers and society, the consequences of this type of sickness absence involve enhanced payments and reduced productivity [2]. A better and more effective strategy would be early identification of employees at risk for psychosocial sickness absence and early intervention to prevent sickness absence as far as possible [3].
At present, a randomized trial is in progress to determine the effectiveness of a preventive coaching intervention for employees at increased risk for sickness absence due to psychosocial health complaints. For this trial, it is essential to know which employees are at increased risk to allow a preventive intervention to be implemented. In a preceding study, an instrument called ‘Werkwijzer’, consisting of predictive items for both men and women to screen employees for their risk of sickness absence due to psychosocial health complaints, was developed by means of data from the Maastricht Cohort Study on ‘fatigue at work’. The items in this instrument were derived from various questionnaires, originating from different settings and all known for their reliability and unidimensionality. Furthermore, internal validation took place in this foregoing study to establish a final set of stable items. However, the confirmed set remained a diffuse enumeration of predictive factors with diverse conceptual content for which scale characteristics are not representative [4].
The objective of the present study is to gain insight in the practical use of the screening instrument by investigating several of its psychometric characteristics and by performing the external validation using data of employees approached for participation in the randomized trial.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflicts of interestReferences |
|---|
Data from the Maastricht Cohort Study on fatigue at work were used for the development, internal validation and determination of the cut-off point of the screening instrument Werkwijzer [4]. From September 2004 to May 2005, the screening instrument was sent to the home addresses of employees of three companies in the south-eastern part of The Netherlands. The responses to the screening instrument formed the basis for the identification of ‘at risk’ employees. These employees received the more extensive baseline questionnaire and respondents were randomly allocated to the ‘usual care’ control group or the ‘preventive intervention’ group. Follow-up data for the trial were collected until the end of 2006. Design of the study, sample size and selection of employees are described elsewhere [5].
One year after the first dispatch of screening instruments, responding employees of two participating companies received a second mailing of instruments, to gather self-reported sickness absence data of those at risk and those ‘not at risk’ for sickness absence. On the basis of the assessment of psychometric characteristics and the external validation, employees were subdivided into two groups. The first group contained all employees who responded to the first screening instrument (N = 3617, 42%). Information on these employees was used to examine the structure and internal consistency of the screening instrument. Next, the predictive value was assessed for respondents for whom objective sickness absence data were available for analysis (N = 2778, 77%). The second group consisted of employees who responded on both screening instruments, with an intervening period of 1 year, and were employed in participating companies that supplied objective sickness absence data (N = 1736). Of these employees, self-reported as well as objective sickness absence data were available. Employees who were randomly allocated to the intervention group in the aforementioned trial were excluded. Data of employees in the second group were used to examine the external validity of the screening instrument. Employees were excluded from the study if they were fully or partially on sick leave, suffered from chronic psychological problems at baseline, had more than one work contract and were pregnant or on maternity leave at the time the questionnaires were sent out [4].
The Werkwijzer screening instrument contains a set of predictive factors (different for men and women) associated with sickness absence due to psychosocial health complaints (Table 1). Specifically, general health factors, mental health factors, work-related factors and factors concerning domestic circumstances were included in the instrument [4]. The screening instrument is four pages long with a total of 40 items and a completion time of 
10 min. The 34 predictive items, 17 for men and 17 for women, were completed with a few general demographic items, e.g. sex and age, and some exclusion criteria, e.g. items about pregnancy and items concerning current absenteeism. Response options for 22 predictive items were on a two-point scale (yes or no); two items were scored on a three-point scale, six items on a four-point scale, one on a five-point scale and three items on a seven-point frequency scale. An overall sum score on the screening instrument was calculated for each individual employee and was compared to a determined cut-off point on the instrument, i.e. a score of 
–3.0 for men and –2.4 for women indicated an increased risk for sickness absence due to psychosocial health complaints (Table 1) [4].
|
Sickness absence data were used to assess both the predictive value and the external validity of the screening instrument. Subjective absence data were gathered by means of the second mailing of screening instruments. Employees were asked to report the frequency of sickness absence in the previous 4 months, given the six possibilities: never, once, twice, three times, four times and five times or more. In case of at least one absence report, they were asked to give information about the reason of their latest sick leave. A code was allocated, in accordance with the International Classification of Primary Care, to the reasons employees gave for their latest sickness absence [4]. This allowed identification of employees who reported sick as a result of psychosocial health complaints. Objective sickness absence data were gathered by means of absence records of employees from participating companies.
The statistical analyses included reliability and validity analyses of the screening instrument. Exploratory factor analysis was used to investigate the structure, i.e. the factor validity of the predictive items on the instrument for both men and women separately. Prior to the application of this analysis, Bartlett's sphericity test was used to assess the mutual correlation of the items in the screening instrument for both men and women. In cases of non-significant test results, i.e. no correlation, the use of factor analysis was irrelevant. The Kaiser–Meyer–Olkin measure of sampling adequacy (MSA) was applied to describe the degree of correlation. MSA adopts values between 0 and 1, with values >0.8 being excellent. The number of factors to retain was determined through both Cattell's scree plot analysis and the assessment of eigenvalues. A breakdown between the factors with relatively large eigenvalues and those with smaller eigenvalues was explored. Items failing to show salient loadings on any of the factors (<0.4) and items with substantial cross-loadings (0.4) were removed. Each factor should comprise at least three items [6]. A forced two-factor analysis was carried out on the removed items. After establishing the structure of the instrument, internal consistency reliability of the identified scales was assessed by the Cronbach's alpha coefficient. Values of 0.7–0.8 are regarded as satisfactory. Predictive value was investigated, using objective sickness absence data as criterion measure. The hypothesis was that an increase in sum score on the instrument is accompanied by an increase in absence duration or absence frequency. Finally, the external validity of the screening instrument was assessed by means of both subjective and objective sickness absence data from employees. Sensitivity and specificity rates were presented for different cut-off points on the instrument. All analyses were performed using the Statistical Package for Social Sciences version 13.0 [7].
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflicts of interestReferences |
|---|
Demographic details of the study population are described elsewhere [5]. Exploratory factor analysis was performed on 17 predictive items for both men and women (N = 3617). Bartlett's sphericity test was significant for both men (P < 0.001) and women (P < 0.001) and MSA resulted in 0.8 and 0.8, respectively. Cattell's scree plot applied to the data of both men and women showed a distinct break before factor three, suggesting that only the first two factors were meaningful enough to be retained (Figure 1).
|
In men, the first factor accounted for 16% of the variance among respondents and consisted of five mental health-related items (2–6) (Table 1). The second factor explained 8% of the variance and captured four items, all pertained to social support (7, 8, 11, 14). A third factor, retrieved from the forced two-factor solution, accounted for 17% of the variance and consisted of three mainly health-related items (1, 9, 13). The fourth factor explained 14% of the variance, but merely consisted of two items, both concerning leisure activities, i.e. items 15 and 16 (Table 1).
Examination of the factor loadings showed three items with loadings of <0.40 on all factors, i.e. items 10, 12 and 17. Both the two items included in the fourth factor and the three items with substandard loadings were excluded from further reliability analyses. Internal consistency as measured by Cronbach's alpha was 0.7, 0.1 and 0.0 for factor 1, 2 and 3, respectively (Table 2).
|
In women, the first factor accounted for 17% of the variance among respondents and consisted of six mental health-related items (3, 4, 5, 6, 7, 17) (Table 1). The second factor explained 9% of the variance and yielded three items, indicating sociodemographic characteristics (14–16). A third factor, retrieved from the forced two-factor solution, accounted for 16% of the variance and consisted of three work-related items (1, 10, 11). The fourth factor explained 13% of the variance, but consisted of just two items, both concerning autonomy at work, i.e. items 8 and 9 (Table 1).
Assessment of the factor loadings showed three items with loadings <0.40 on all factors, i.e. items 2, 12 and 13. Both the two items included in the fourth factor and the three items with substandard loadings were excluded from further reliability analyses. Internal consistency as measured by Cronbach's alpha was 0.7, –0.3 and 0.2 for factor 1, 2 and 3, respectively (Table 3).
|
Employees who were randomly allocated to the intervention group in the aforementioned trial were excluded (N = 63). Sum scores of both men and women on the screening instrument were correlated to absenteeism data to determine how powerful they are in predicting sickness absence. In men, the correlation between the sum score on the screening instrument and absence duration (in days) and absence frequency (number of spells) were both 0.21 (P < 0.001). Since there was no clear linear relation between the sum scores and absenteeism, binary logistic regression was used to determine the association [in odds ratios (ORs)] between ‘being at risk’ (yes/no) and sickness absence (yes/no) in the following year (OR 3.1, 95% CI 1.5–6.5). In women, the correlation between the sum score on the screening instrument and absence duration (in days) and absence frequency (number of spells) were both 0.14 (P < 0.001). Again, there was no clear linear relation between sum scores and absenteeism. Therefore, binary logistic regression was used to determine the association between being at risk (yes/no) and sickness absence (yes/no) in the following year (OR 2.0, 95% CI 1.4–2.7).
The external validity of the screening instrument was tested, using data of employees who responded on two screening instruments (N = 1736). The ability of the screening instrument to identify employees with or without self-reported sickness absence due to psychosocial health complaints, at 1-year follow-up, was assessed by calculating sensitivity and specificity, and their corresponding confidence intervals, of different cut-off points. A cut-off point of 10 on the screening instrument resulted in a sensitivity score of 13% for women and 50% for men and a specificity score of 96% for women and 97% for men (Table 4). The cut-off points were determined using data from the development of the screening instrument [4]. In addition to the original external validation of the screening instrument, i.e. its feasibility to predict sickness absence due to psychosocial health complaints, the instrument might also be predictive for overall sickness absence. Sensitivity and specificity were calculated, using objective sickness absence data from participating companies. Now, a cut-off point of 10 on the screening instrument resulted in a sensitivity of 8% for women and 12% for men and a specificity of 96% for women and 98% for men (Table 4).
|
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflicts of interestReferences |
|---|
For the development of the instrument, an empirical approach was used to identify predictive factors for sickness absence. In both men and women, factor analysis revealed a comprehensible structure of variables related to the dimension ‘mental health’. Analogous to the literature, mental health-related complaints are identified as a main cause for sickness absence. Although the remaining individual factors did not significantly merge into an overall construct, they are independently predictive for sickness absence. In order for the instrument to be a usable screening device for sickness absence, it should have predictive value. The present form of the instrument is suitable to predict who is not at risk or at risk for sickness absence due to psychosocial health complaints. However, the prediction of the duration or the frequency of sick leave was indefinite. Since no definite linear relation was found between the sum score on the instrument and absence duration or frequency in the following year, little value could be attached to these results. Moreover, the extent to which the screening instrument predicts sickness absence is promising, if it is intended for the dichotomous outcome measure sickness absence. After the evaluation of the predictive value, sickness absence data at 1-year follow-up were used to assess the external validity of the screening instrument. Potential cut-off points on the developed screening instrument were applied to absenteeism data of employees for the calculation of sensitivity and specificity rates. To affect those who would probably benefit most from a preventive intervention, one would prefer a screening instrument with high specificity. However, the low sensitivity scores that were found in the external validation are undeniable and responsible for the weakness of the instrument, i.e. a substantial number of false-negatively classified employees. The combination of a more sensitive instrument with this specific screening instrument therefore is a functional possibility for future implementation [4].
Although we have made an accurate examination of psychometric characteristics of the screening instrument, there still remain aspects that merit discussion. Questionnaire reduction and modification of the screening instrument, through deletion of items, were not objectives of this study. Moreover, factor analysis was applied to detect the conceptual structure of the instrument, to compare these with previous research and to enable the investigation of scale characteristics, though the identification of unambiguous factors was less successful than expected. Keeping the predictive value of the individual items in the instrument in mind, one could ask oneself if barely revealing comprehensible constructs is a prohibited objective. Sickness absence is a multivariate phenomenon, of which the occurrence and course is likely to be explained by a range of factors. The structure and content of the screening instrument, i.e. a diffuse enumeration of predictive factors from different sources, is a representation of this variety and at the same time an underpinning of not identifying obvious clusters. Also, a reason for results not being as clear and unambiguous as expected could be that different factors are associated with different measures of absence. Nevertheless, more structure in the screening instrument could have resulted in higher internal consistency reliability. In addition, factor analysis implicates a few choices, for example the minimum items within a factor, which may contribute to the current results. Consequently, numerous underlying dimensions may be unrevealed. Furthermore, the expected correlation between sum scores on the screening instrument and sickness absence failed to materialize, whereas the dichotomous comparison between at risk and not at risk employees and being absent from work (yes/no) yielded noteworthy associations. Consequently, the present form of the instrument is explicitly suitable to predict who is not at risk or at risk for sickness absence due to psychosocial health complaints. As regards the external validation of the instrument, both self-reported and objective sickness absence data were used to identify employees at increased risk for sickness absence and to determine the sensitivity and specificity rates of the instrument. Results for sensitivity were not noteworthy, but the findings for specificity were remarkably high. Both sensitivity and specificity were unvarying during the development and validation phases of the instrument, and no immense differences in rates were found when using self-reported or objective sickness absence data. These results contradict the view that self-reports cannot be considered a valid measure of absenteeism.
To conclude, the results of this study provide a theoretical and scientific basis for the recently developed screening instrument for the prediction of sickness absence due to psychosocial health complaints. Factor analysis proved to be quite valuable since the findings are a concrete underpinning of the multifactor concept that is sickness absence. The low internal consistency is the consequential manifestation of a parcel of individual predictive factors. Still, the screening instrument is capable of identifying employees at risk for sickness absence due to psychosocial health complaints. The increased risks for both men and women and the constant rates for sensitivity and specificity support this ability. With this, opportunities for a more preventive approach to sickness absence become feasible for employees, employers and society.
Key points
|
|
Funding |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflicts of interestReferences |
|---|
Health Research and Development Council (Zorg Onderzoek Nederland), The Netherlands (2200.0105).
|
Conflicts of interest |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflicts of interestReferences |
|---|
None declared.
|
Acknowledgements |
|---|
Presented at the European Congress of Work and Organizational Psychology. The Medical Ethics Committee of Maastricht University/Academic Hospital Maastricht approved the study protocol.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingConflicts of interestReferences |
|---|
- Luz J, Green MS. Sickness absenteeism from work—a critical review of the literature. Public Health Rev (1997) 25:89–122.[Medline]
- Borritz M, Rugulies R, Christensen KB, Villadsen E, Kristensen TS. Burnout as a predictor of self-reported sickness absence among human service workers: prospective findings from three year follow up of the PUMA study. Occup Environ Med (2006) 63:98–106.
[Abstract/Free Full Text] - Schröer C. Verzuim wegens overspanning. Een onderzoek naar de aard van overspanning, de hulpverlening en het verzuimbeloop. Dissertation. Maastricht: University Press Maastricht, (1993).
- Duijts SFA, Kant I, Landeweerd JA, Swaen GMH. Prediction of sickness absence: the development of a screening instrument. Occup Environ Med (2006) 63:564–569.
[Abstract/Free Full Text] - Duijts SFA, Kant I, van den Brandt PA, Swaen GMH. Advantages and disadvantages of an objective selection process of employees for preventive intervention. BMC Public Health (2007) 7:67.[CrossRef][Medline]
- Streiner DL. Figuring out factors: the use and misuse of factor analysis. Can J Psychiatry (1994) 39:135–140.[Web of Science][Medline]
- SPSS. Base 13.0 for Windows User's Guide (2004) Chicago, IL: SPSS Inc.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||