To examine a narrative multisource feedback (MSF) instrument concerning feasibility, quality of narrative comments, perceptions of users (face validity), consequential validity, discriminating capacity and number of assessors needed.

ABSTRACT
Objectives
To examine a narrative multisource feedback (MSF) instrument concerning feasibility, quality of narrative comments, perceptions of users (face validity), consequential validity, discriminating capacity and number of assessors needed.
Design
Qualitative text analysis supplemented by quantitative descriptive analysis.
Setting
Internal Medicine Departments in Zealand, Denmark.
Participants
48 postgraduate trainees in internal medicine specialties, 1 clinical supervisor for each trainee and 376 feedback givers (respondents).
Intervention
This study examines the use of an electronic, purely narrative MSF instrument. After the MSF process, the trainee and the supervisor answered a postquestionnaire concerning their perception of the process. The authors coded the comments in the MSF reports for valence (positive or negative), specificity, relation to behaviour and whether the comment suggested a strategy for improvement. Four of the authors independently classified the MSF reports as either ‘no reasons for concern’ or ‘possibly some concern’, thereby examining discriminating capacity. Through iterative readings, the authors furthermore tried to identify how many respondents were needed in order to get a reliable impression of a trainee.
Results
Out of all comments coded for valence (n=1935), 89% were positive and 11% negative. Out of all coded comments (n=4684), 3.8% were suggesting ways to improve. 92% of trainees and supervisors preferred a narrative MSF to a numerical MSF, and 82% of the trainees discovered performance in need of development, but only 53% had made a specific plan for development. Kappa coefficients for inter- rater correlations between four authors were 0.7–1. There was a significant association (p<0.001) between the number of negative comments and the qualitative judgement by the four authors. It was not possible to define a specific number of respondents needed.
Conclusions
A purely narrative MSF contributes with educational value and experienced supervisors can discriminate between trainees’ performances based on the MSF reports.
Strengths and limitations of this study
This is to our knowledge the first study reporting details of a purely narrative multisource feedback (MSF) instrument used in postgraduate training in internal medicine.
Participants were drawn from a convenience sample.
Trainees and their supervisors compared the narrative MSF to a scale based MSF based on their previous experience or knowledge concerning MSF.
INTRODUCTION
Multisource feedback (MSF) also termed 360 degrees feedback is a process in which feed-back from multiple assessors is collected. The assessment method was developed and has been used extensively in private business and industrial settings for personal development as well as for appraisal purposes.
MSF in the medical environment
In the healthcare system, the increasing demand for accountability to health authorities, funding agencies and patients, as well as the concerns about physician performance and patients’ safety has required new methods for assessment. Physicians must be competent in domains such as interpersonal and communication skills, professionalism, safety and quality, partnership and teamwork. Competences in these domains have required new assessment methods. MSF was introduced based on the empirical findings from use in the industry. The first studies on use of the method to assess physicians were published in the late 1980s and beginning of the 1990s. Since then a large number of studies as well as several systematic reviews on MSF have been published.