Skip to main content

Patient involvement in questionnaire design: tackling response error and burden


Questionnaires capture patient perspectives succinctly and at relatively low cost, making them a popular data collection tool for health researchers. However, questionnaire data can be affected by response error and response burden. Patient involvement during questionnaire design can help reduce the effect of response error and burden. This paper describes a novel approach for patient involvement during questionnaire design, combining methods from cognitive interviewing (Think Aloud Tasks) with an open-ended follow-up discussion to collate and act on patient feedback, while also taking account of the common challenges in questionnaire design (i.e. response error and burden). The strengths and limitations of this approach are discussed, and recommendations are made for future use.

Plain english summary

Health researchers often use questionnaires to collect data. When patients fill in questionnaires, they may interpret the questions differently from how the researchers intended. This is an example of response error. Answers can also be affected by how much effort it takes to fill in the questionnaire, this is called response burden. For example, patients may pay less attention at the end of a long questionnaire. Involving patients in questionnaire design is important because it can help prevent response error and burden. This paper describes a new way of gathering patient feedback during questionnaire design, which combines techniques used in research with an open-ended discussion. It describes the strengths and weaknesses of this approach, and outlines tips for researchers and patients involved in questionnaire design.


Patient and Public Involvement (PPI) refers to “research being carried out ‘with’ or ‘by’ members of the public, rather than ‘to’, ‘about’ or ‘for’ them”. It is an integral part of healthcare policy and research guidance [1]. An example of PPI is when patient contributors provide feedback to develop new research materials such as questionnaires. Involving patients while designing new questionnaires improves questionnaire comprehensiveness and relevance, and reduces item ambiguities [2, 3].

In health research, key challenges in questionnaire design include response error and response burden [4,5,6,7]. Response error occurs when a questionnaire item or the way a respondent processes the item results in an inaccurate answer [6]. Response burden refers to the effort required to complete the questionnaire, determined by factors such as the emotional/cognitive load of items, questionnaire length, layout, or distribution method [6, 7].

In this commentary, we describe a novel approach for tackling the aforementioned challenges and facilitating PPI in questionnaire design. By drawing on methods previously used in cognitive interviewing, we aimed to identify potential sources of response error/burden and generate an initial springboard for additional discussion between researchers and patient contributors. The commentary acts as an illustrative case of health-related questionnaire design, including key insights, resources from the literature, and recommendations from the patient contributors involved during questionnaire design.

Main text

Case background

We designed an intervention targeting medication adherence in adults with asthma, to be delivered by general practice pharmacists. To explore the perspectives of adults with asthma on this new intervention, we wanted to design a questionnaire that measured intervention acceptability. Acceptability refers to how appropriate patients feel an intervention is, based on experienced or anticipated cognitive/emotional responses [8].

Designing the questionnaire

The first version of the acceptability questionnaire was drafted by researchers based on the Theoretical Framework of Acceptability (TFA), focusing on intervention content and delivery (i.e. by pharmacists) [8]. Four members of the Asthma UK Centre for Applied Research (AUKCAR) Patient Advisory Group agreed to help refine the questionnaire as patient contributors. The AUKCAR Patient Advisory Group consists of adults with lived experience of asthma who offer advice and guidance on incorporating patient perspectives into all stages of research.

Feedback was gathered over the telephone, enabling contributors from across the United Kingdom (UK) to be involved. The telephone calls were audio-recorded for our records and detailed summaries of patient feedback were written. Drawing on cognitive interviewing methodology, we began with a Think Aloud Task (TAT) based on recommendations by van Someren MW, Barnard YF and Sandberg JAC [9]. Contributors were asked to continuously verbalise their thought process as they worked through the questionnaire. This helped researchers understand how people interpreted, processed, and responded to questionnaire items [9]. Researchers and contributors then picked up on issues emerging from the TATs in an open-ended follow-up discussion, where contributors were also encouraged to bring up any additional issues they found relevant. Patient contributors were e-mailed TAT instructions and background information on the intervention ahead of time. To ensure that contributors felt comfortable with the TAT procedure, an unrelated questionnaire about job satisfaction was used to practice. We did not e-mail the acceptability questionnaire to the contributors until directly before the telephone call to ensure that the TAT was based on their first impressions. Patient contributors’ feedback was incorporated into a new questionnaire draft (see Table 1 for examples). To keep patient contributors informed, we produced a brief summary of their feedback and how it was incorporated into the new questionnaire. This was e-mailed to them, in case of further comments.

Table 1 Examples of patient contributors’ feedback and the associated changes to the questionnaire

Key insights and recommendations

Two patient contributors reflected on the questionnaire design process with the researchers to identify the strengths and limitations of our novel PPI approach (TATs paired with an open-ended discussion). Recommendations for this approach are highlighted in Table 2.

Table 2 Key recommendations from researchers and patient contributors about the novel approach (TATs and follow-up discussion) for patient involvement in questionnaire design

A key strength of the TAT is the fact that it structures the initial feedback process. Patient contributors mentioned that this may be a useful approach for people who initially struggle with open-ended questions. The TAT also facilitated researcher and contributor communication by acting as a springboard for the open-ended follow-up discussion: researchers picked up on issues identified in the TAT for further discussion with the contributors.

However, a potential limitation of the approach is that contributors may have limited experience with TATs, and this may act as a barrier to feedback. We followed recommendations made by van Someren MW, Barnard YF and Sandberg JAC [9] regarding clear TAT instructions: asking contributors to continuously say what was going through their minds with limited pauses, rather than asking for evaluation (e.g. “what do you think about…”). A practice TAT with an unrelated questionnaire also helped contributors familiarise themselves with the procedure [9]. We tried to overcome the limitations associated with the TATs with an open-ended follow-up discussion, which tackled issues that had not been identified in the TAT procedure (e.g. external factors) (see Table 1). Furthermore, employing multiple feedback methods ensured that contributors were able to offer feedback in a way that was comfortable for them.

Underpinning this entire process was the rapport between researchers and patient contributors. Contributors emphasized the importance of investing adequate time to establish rapport before gathering feedback. Contributors that feel comfortable with researchers may provide more detailed feedback, and giving contributors a positive PPI experience may encourage future research involvement. Minimising researcher interruption during both the TAT and the follow-up discussion signalled to the contributors that their time to speak was valued and respected.


As well as tackling common issues in questionnaire design (e.g. response error), the novel approach outlined in this commentary also incorporated people’s lived experiences of a long-term condition and potential study settings (e.g. general practice waiting rooms) into the questionnaire design process. Using methods previously employed in cognitive interviewing, researchers and patient contributors were able to create an initial springboard for further discussion. This unique combination of feedback methods may help researchers collate and act on patient feedback, while also taking account of the common challenges in questionnaire design (i.e. response error and burden). For patient contributors involved in questionnaire design, having a clear-cut process for feedback may also improve the PPI experience. These benefits may encourage increased PPI in questionnaire design, and reduce the risk of making questionnaires overly complex.



Asthma UK Centre for Applied Research


Patient and Public Involvement


Think Aloud Task


Theoretical Framework of Acceptability


United Kingdom


  1. 1.

    INVOLVE. Briefing notes for researchers: involving the public in NHS, public health and social care research. Eastleigh: INVOLVE; 2012.

    Google Scholar 

  2. 2.

    Wyatt K, Carter M, Mahtani V, Barnard A, Hawton A, Britten N. The impact of consumer involvement in research: an evaluation of consumer involvement in the London primary care studies Programme. Fam Pract. 2008;25:154–61.

    Article  Google Scholar 

  3. 3.

    Staniszewska S, Haywood KL, Brett J, Tutton L. Patient and public involvement in patient-reported outcome measures: evolution not revolution. Patient. 2012;5:79–87.

    Article  Google Scholar 

  4. 4.

    Pascale J, Roemer MI, Resnick DM. Medicaid underreporting in the CPSResults from a record check study. Public Opin Q. 2009;73:497–520.

    Article  Google Scholar 

  5. 5.

    Molina JG, Rodrigo MF, Losilla J-M, Vives J. Wording effects and the factor structure of the 12-item general health questionnaire (GHQ-12). Psychol Assess. 2014;26:1031–7.

    Article  Google Scholar 

  6. 6.

    Willis GB. Cognitive interviewing: a tool for improving questionnaire design. Thousand Oaks, California: Sage Publications Inc; 2005.

    Google Scholar 

  7. 7.

    Rolstad S, Adler J, Rydén A. Response burden and questionnaire length: is shorter better? A review and meta-analysis. Value Health. 2011;14:1101–8.

    Article  Google Scholar 

  8. 8.

    Sekhon M, Cartwright M, Francis JJ. Acceptability of healthcare interventions: an overview of reviews and development of a theoretical framework. BMC Health Serv Res. 2017;17:88.

    Article  Google Scholar 

  9. 9.

    van Someren MW, Barnard YF, Sandberg JAC. Practical procedures in obtaining think aloud protocols. In: The think aloud method: a practical approach to modelling cognitive processes. London, UK: Academic Press; 1994. p. 41–8.

    Google Scholar 

  10. 10.

    Horne R. Treatment perceptions and self regulation. In: Cameron LD, Leventhal H, editors. The self-regulation of health and illness behaviour. London: Routledge; 2003. p. 138–53.

    Google Scholar 

Download references


We are incredibly grateful to Elisabeth and Anthony for sharing their PPI experiences and for their continuous input into our research.


The research was funded by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care North Thames at Barts Health NHS Trust. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care.

Availability of data and materials

Not applicable.

Author information




All authors read and approved the final manuscript.

Corresponding author

Correspondence to Rob Horne.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Our two patient contributors (Elisabeth and Anthony) reviewed and approved their contribution to this publication.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Competing interests

The authors declare that they have no competing interest.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Mes, M.A., Chan, A.H.Y., Wileman, V. et al. Patient involvement in questionnaire design: tackling response error and burden. J of Pharm Policy and Pract 12, 17 (2019).

Download citation


  • Patient and public involvement (PPI)
  • Think Aloud Tasks (TATs).
  • Cognitive Interviewing.
  • Questionnaire Design.