Unreliable self-reporting in mystery shopper study

codeine medicine pills pharmacy pharmacist patient customer consult sale

Pharmacists and pharmacy staff are not very reliable at self-reporting their own behaviour during non-prescription medicine consultations, study finds

Researchers from the University of Sydney’s Pharmacy School recently sought to examine how pharmacy staff self-report their behaviour compared to ‘actual’ behaviour when it comes to patient counselling.

They sent out 61 undergraduate pharmacy students who were trained as simulated patients (aka mystery shoppers) to visit a sample of 36 community pharmacies across metropolitan Sydney, Australia from March to October 2015. A total of 540 visits were planned during this period.

Pharmacies were allocated one of 10 minor ailment topic areas: adult cough/cold, adult pain, allergic rhinitis, asthma, diarrhoea, dyspepsia, insomnia, paediatric fever, paediatric cough/cold, and smoking cessation.

Simulated patients presented to their pre-allocated pharmacy with a scripted request for a product, speaking with the first staff member encountered – which could be either pharmacist or non-pharmacist staff – and recorded the encounter with a concealed audio-recording device.

Immediately after the visit, the simulated patients exited the pharmacy and scored the interaction against the standardised scoresheet. Following this, they returned to the pharmacy and asked the staff member they interacted with to complete a self-assessment and provided verbal feedback.

Of all visits, 240 randomly selected cases were complete and eligible for analysis.

Pharmacists and staff were assessed based on questions they asked the patient (information gathering) for example, asking who the medicine was for; how long they had had symptoms; and what symptoms were being experienced.

They were also assessed on the course of action they took, for example, did they refer the patient to a doctor; provide verbal or written advice, etc; and whether that visit had an “appropriate outcome”.

Inter-rater agreement was determined for pharmacy staff self-assessment versus researcher (from audio recordings), and simulated patient versus researcher, and was calculated through intra-class correlation (ICC).

When comparing information gathering scores as reported by pharmacy staff the median score was 63%, versus 38% reported by the researcher. Single measures ICC analysis returned a “poor” to “moderate” level of agreement.

More than 10% of pharmacy staff considered the encounter to have an appropriate outcome (“Yes”), where the researcher recorded “No”.

Meanwhile overall median scores for information gathering correlated better between the simulated patient scores (43%) and the scores reported by the researcher from the audio recording (38%). The ICC for the information gathering equated to “good” to “excellent” agreement between simulated patients and researchers.

Similarly, total scores across all measures reported by simulated patients had a median of 50%, compared to a median of 38% reported by the researcher, with the ICC for total score also equating to “good” to “excellent” agreement.

However simulated patients were unable to reach a high level of agreement with the researcher on the overall appropriateness of outcome.

Pharmacy staff self-reported their behaviour during non-prescription medicine consultations with covert simulated patients with a poor degree of reliability, resulting in significant recall bias, said the research team, led by pharmacist and researcher Jack Collins.

“Conversely, undergraduate pharmacy student simulated patients achieved a good level of agreement with the researcher, comparable to existing literature,” they said.

“This study highlights the limitations of self-reported data in pharmacy practice research, and future studies relying on this method should consider the effect of this on interpretations of findings.

“Future research should further investigate the discrepancy between staff-reported perceived behaviour and actual behaviour.”

See the full study in Research in Social and Administrative Pharmacy here

Previous ‘Nothing against pharmacists, but…’
Next Returning to normal

NOTICE: It can sometimes take awhile for comment submissions to go through, please be patient.


  1. Philip Smith

    Terrible results not just in self reporting but the health interaction.
    Makes it seem like pharmacy may as well be in supermarkets.
    Or is this just an example of paying peanuts getting monkeys?

    • Paul Sapardanis

      I recently had a health interaction with a patient in store about an allergic rhinitis nasal spray. At the end of our 4 to 5 minutes the patient pulled out their mobile phone to check the price of the recommended product elsewhere, they left. I hope this interaction doesn’t leave me becoming one of those phsrmacists in the above article

Leave a reply