Workforce research priorities for resilience to future health shocks—and the workforce crisis

Lessons for workforce research from previous health shocks

The most recent response to a health shock, the covid-19 pandemic, vividly showed the strengths—and weaknesses—of the UK’s healthcare research infrastructure. On one hand, the UK rapidly deployed longstanding investments in research capacity in virology, vaccine development, genomics, clinical trials, data science, and a host of other largely biomedical areas to set up initiatives such as the UK Research and Innovation (UKRI) led rolling responsive covid-19 call programme and the National Core Studies programme led by the government’s chief scientific adviser.45 On the other hand, the UK scrambled to assemble the evidence base on a wide range of non-drug interventions to reduce the spread of infection, such as masks, social distancing, or shielding,6 and to understand and communicate knowledge from less familiar disciplines such as behavioural science and ventilation engineering.

One huge research gap stands out—the healthcare workforce, for which major and rapid changes in workforce supply and deployment and service delivery were needed, but the UK lacked a strong existing evidence base and the capacity to set up and undertake rapid research concurrently with those changes. Tackling these gaps is a priority to mitigate for future health shocks.

In comparison, some other industries such as retail grocery used their research for business continuity and shared intelligence to remodel work, workforce, and supply chains.7 The construction industry responded with measures to ensure project continuity and performance,8 and the public transportation industry workforce was profoundly affected and advocated successfully for substantial government support.9

The health research response to covid-19 reflected the overwhelming predominance of the biomedical research paradigm in health research in the UK.10 The UK’s main advisory body, SAGE (Scientific Advisory Group for Emergencies) and its various subgroups included little specialist expertise in healthcare workforce research.11 None of the covid-19 national core studies focused on the healthcare workforce, and of the 645 new projects commissioned through the UKRI’s covid-19 responsive research programme, only 14 centred on the healthcare workforce (box 1). Although these cover some important topics, they do not represent a coherent programme of workforce research and did not adequately respond to the need for research and evidence. Of £327.8m ($416.7m; €383.7m) spent in the UKRI programme, just £7.2m (2.2%) was allocated to workforce research projects.12 In short, the pandemic research effort focused on a largely biomedical research agenda and largely neglected questions to do with the healthcare workforce.

However, the pandemic presented many questions about the healthcare workforce that needed urgent answers, as will future health shocks, whether a respiratory virus or not. Many concerned workforce capacity. How could staffing for expanded acute care provision be rapidly and safely implemented in the first wave of the pandemic and used to support testing and vaccination programmes later in the pandemic? How could healthcare professionals who had left the NHS or even left the professional register be safely brought back into practice? What was involved in redeploying healthcare staff to work in areas of high demand such as acute medical inpatient care when they had not worked in those areas for many years? What precautions or control of infection measures were needed to protect healthcare staff and their families from infection with SARS-CoV-2, and who was at greatest risk from infection because of factors such as job role, age, or ethnicity?

Wider questions about how to protect the mental health and wellbeing of healthcare staff during a protracted emergency included how to provide organisational resilience in all healthcare services at a time of unprecedentedly high staff sickness and absence, and how to maintain staff morale and team functioning. In addition, new models of service delivery rapidly emerged with profound workforce implications, such as how to organise and staff virtual wards, remote outpatient clinics, telephone or video based general practice consultations, and separate “hot” and “cold” acute and primary care pathways for patients with and without suspected covid-19.

Without either a strong existing research base to provide evidence to underpin decision making in most of these areas, or the research capacity and capability to respond rapidly to such needs for knowledge, policy makers, healthcare leaders, and clinicians in the UK and internationally responded as best they could to the challenges of the pandemic.13 However, better decisions could probably have been made with more and better research to support them. The key lesson is that any future shock to the healthcare system is likely to have important implications for the healthcare workforce—and mitigating or dealing with health shocks will likely always directly or indirectly mean tackling questions of workforce supply, deployment, and protection.

Setting a workforce research agenda

Since well before the pandemic, policy concerns about the healthcare workforce have been driven by a growing realisation that the NHS faces a perfect storm—high rates of workforce exit through retirement, migration, career change, and other routes,14 longstanding deficits in training and workforce supply,15 a huge reliance on international recruitment,16 alarming levels of workforce stress and burnout,17 declining labour productivity, and so on. Many other nations face similar challenges.1819

Somewhat belatedly, NHS England published a workforce plan in 2023,20 setting out proposals for increased investment in workforce supply, yet this included little to tackle the immediate acute crisis in workforce recruitment and retention. The plan did not cost the underlying assumptions about the long term growth in the NHS workforce required to meet future demand, but others have calculated that the plan would increase the NHS workforce from 1.5 million in 2021-22 to between 2.3 and 2.4 million in 2036-37 and need an additional £50bn a year (at 2022-23 prices) in NHS spending by 2036-37.21

The NHS workforce plan also underestimates the need for highly skilled and experienced supervising staff to oversee larger numbers of less skilled and lower paid workers in new roles—what has been termed the “Christmas tree” model—and the capacity and costs for providing these, as well as reinforcing assumptions not always supported by existing evidence of reduced workload in priority staff groups such as general practitioners.22

Many of the workforce research questions raised during the pandemic and outlined above are relevant and important in the longer term, beyond the setting of that particular health emergency. Substantial overlap also exists with the research needs related to the current NHS workforce crisis. The UK spends about 65% of the NHS budget on its workforce, yet it invests a tiny fraction of the public and government budget for health and care research on understanding that workforce and improving its efficiency, effectiveness, and outcomes.23

Box 2 sets out workforce research areas identified by a recent report from the membership organisation Health Services Research UK,23 based on wide consultation with the UK health services research and practice communities.24 In each area, immediate priorities for research driven by the workforce crisis are identified. For example, in relation to recruitment, retention, and workforce exit, the most pressing need is for good research on interventions to reduce the rate of wastage in the training pathway and first few years of post-qualification work for doctors and nurses,25 or for interventions (such as flexible and part time working or reform of pension arrangements) that would enable health professionals to remain in the NHS workforce for longer.

In the longer term, more in-depth research on recruitment and training policies could help to improve training outcomes and the transition to post-qualification careers, and research on changing career patterns and progression could help the NHS to design staffing structures, job roles, and remuneration to be more attractive to the current and future workforce. One neglected area is research on people who have left the healthcare workforce, for a wide range of personal and professional reasons, and how their return to work might be enabled and managed effectively.

Many of these workforce research areas overlap or intersect—for example, questions of equality, diversity, and inclusion in the workforce clearly arise in relation to many of the other topics such as recruitment and retention, bullying and harassment, management and leadership, and international workforce/migration.

Building workforce research capacity, capability, and infrastructure

The workforce research community in the UK is small and fragmented in academia and other settings. Disciplinary groupings exist in labour economics, organisational psychology, human resource management, health services research, health professions education and training, and other areas. The discipline is also fragmented by profession—for example, research groups concerned with doctors, nurses, pharmacists, allied health professionals, and so on—and by sector, with groups researching the workforce in primary care, mental health, care of elderly people, acute care, and so on. The community is small mainly because of limited investment in workforce research over many years. Most research centres lack critical mass and scale. Some welcome recent initiatives have been made by the UK funder the National Institute for Health and Care Research and others to invest in workforce research,26 but a managed step change in investment is needed to build capacity and capability and draw together researchers with relevant expertise from other disciplines and sectors. Research funders need to encourage more disciplines relevant to workforce into health, as well as draw in some of the strengths of applied health research to the broader workforce research community in areas such as implementation science, rapid evaluation, evidence synthesis, and interventional research designs.

Other areas of research infrastructure also need concerted investment. Perhaps the most pressing need is to improve access to workforce data for research. The UK has many separate and disconnected datasets: the nine professional regulatory bodies hold registrant data; the NHS Business Services Authority runs the Electronic Staff Record system for the NHS in England and Wales; the universities who train health and care professionals and the royal colleges and others involved in postgraduate training hold training and qualification data; NHS England commissions and holds national staff survey data; and many more staffing data are locked up in the staffing, rostering, and other information systems of local NHS organisations. Access to these datasets for researchers varies from difficult to impossible, and data linkage is highly problematic because the necessary data infrastructure and information governance frameworks do not exist, but these data could be extremely useful for research.

A secure workforce data environment is needed for workforce data, and this could be provided by building on the secure data environments being established for patients’ data,27 with the necessary legal provisions and confidentiality arrangements to host these workforce data, link them in fully anonymised form, and make them available for research. It can be done—for example, the General Medical Council already hosts the UK Medical Education Database (UKMED) platform for collating data on the performance of UK medical students and trainee doctors across their education and future careers.28

Some new research data should be collected to augment that contained in the proposed secure data environment. Most obviously, no significant cohort studies of the NHS workforce exist, following clinical professionals longitudinally across their career path and collecting data on workplace experiences and future intentions. Such longitudinal datasets can inform longer term evaluations of training and workforce interventions. Also, few data exist on the healthcare workforce outside the NHS, although by some estimates up to a third of people working in healthcare are outside the public sector.29

Using research and evidence on the workforce in policy and practice

More and better research also demands investment by both research funders and NHS organisations in evidence synthesis and knowledge translation to enable timely sharing of the best research evidence in a form usable by key decision makers in government and the NHS. A few examples already exist, such as guidance from the National Institute for Health and Care Excellence on safe staffing levels issued in 2014 but not since updated,30 or some of the small but growing number of reviews on workforce questions in the Cochrane Library. One solution would be that a collaboration of NHS Employers and other stakeholders such as the professional regulatory bodies come together with academic expertise to provide that kind of authoritative evidence based guidance.

Investment is needed at a national level. Skilled workforce leaders, data analysts, and researchers in individual healthcare organisations also need to be nurtured and developed to respond to pressing local needs about how best to deploy or optimise staff. Embedded researchers, often using approaches such as simulation and modelling, can provide useful and practical support on service questions.31 NHS organisations at board level need to take the use of evidence in their workforce strategies and plans more seriously.

The workforce is vital, whatever the nature of the next health shock. The UK’s response to the covid-19 pandemic vividly showed the value and impact of excellent research and how it can improve policy and clinical practice and make an important difference to the lives of patients and communities. It also showed that some areas of research, such as the workforce, have been relatively neglected and hence the UK was not well prepared to support evidence based decision making in this area.

The NHS invests substantially in its workforce—in training and developing staff and employing them to deliver care to patients. In the current period of sustained and arguably normalised crisis for the NHS workforce, many experiments in how staff are deployed and new ways of working are as yet untested. It is time for us to fund more and better research and invest in evaluations and capacity to use evidence in decision making in the NHS, both to get the best value out of our NHS workforce and to nurture their health, wellbeing, and careers. We advocate for a prospective research programme/agenda, including horizon scanning and evidence synthesis, and standing capacity to respond to new research needs quickly and flexibly.