Incident reporting

Over the past six months the number of deployed systems on the National Programme has increased from 10,116 to 13,200, an increase of over 30 per cent. There has been a similar increase in users of some 25 per cent.

In the same time frame the number of reported incidents has reduced by 15.3 per cent overall despite the increase in deployments and users. This represents a significant improvement in reliability.

Terminology

A severity one incident is considered by NHS CFH to be the loss of a system, for example, one GP practice losing system use for five minutes, that is, the whole of a deployment. This is clearly very different from the significant incident we had in July 2006 when CSC suffered significant data centre outage.

Inaccurate reporting of incidents

Inaccurate reporting by NHS trusts of severity one and severity two incidents inflates the initial number of incidents which are recorded and progressed by the National Service Desk.

For example, approximately 15 per cent of both severity one and severity two incidents which were reported to the National Service Desk over the past six months were in fact identified as being caused by local infrastructure problems. This is hardly surprising given the heterogeneous and complex nature of the infrastructure in many hospitals and GP practices.

Similarly five per cent of severity one and 10 per cent of severity two incidents were downgraded upon investigation, that is, their impact had been incorrectly assessed at the time they were originally reported.

NHS Connecting for Health appreciates that these factors, along with the high level of usage of our systems and their importance in the NHS, creates an environment in which we will continue to prosecute a robust Service Management framework which defaults to over reporting of problems in order that they can be properly aired.

We believe that this is in line with industry best practice and that it would be unfortunate if inaccurate reporting were to begin to lead to a culture of under reporting of incidents with a commensurate degradation in the quality of root cause analysis.

Picture Archiving and Communications System (PACS)

PACS has now been rolled out to most trusts across the South of England.

End-to-end service availability is a function not only of the performance of the systems delivered by Fujitsu but also local infrastructure and performance of electro mechanical devices such as CR Readers.

Fujitsu is currently undertaking detailed analysis of the run time performance of their equipment and the end to end performance of PACS services.

Significantly better than the industry average

It should be noted that not only do NHS CFH systems perform well within normal tolerance levels according to Gartner, with significant incidents being one per cent for NHS CFH systems compared to an industry average of nearly five per cent, but also that our level two incidents are running at less than four per cent compared to an industry average of approximately 10 per cent.

Fixing problems

The meantime between failure on many of NHS CFH's systems exceeds contracted levels and in almost all cases the "time to fix" for our systems is within industry standards and contracted timeframes.

Some 98.6 per cent of severity one and two incidents were fixed within the target timescales and in January 2007, the last month for which full data is available, the average amount of time lost due to systems outage (on the presumption that a user was using a system for 24/7 - clearly an exaggeration of the norm) is only five minutes.

Transparency

NHS Connecting for Health is unusual in publishing service availability data on our website.

There is a high degree of availability of our core national systems such as the Personal Demographics Service. This is now receiving approximately 1.4 million queries a day from the order of 50,000 unique users. This is the backbone of many patient interactions in the NHS today.

Isn't there a case for NHS CFH publishing the severity one and two incidents and the lessons learnt from them so that other trusts could gain from adversity?

Yes, we do this already. Details of all Higher Severity Service Incidents (HSSIs) are already published throughout NHS CFH with the intention that these are then cascaded as appropriate to the wider NHS in order to provide the ability to identify trends or to pre-empt similar incidents occurring in other areas.

In addition, analysis of HSSIs and any underlying problems is undertaken by NHS CFH staff. In conjunction with suppliers, work-around, known error and root cause analysis information is circulated in order to help prevent re-occurrence of incidents or to mitigate potential risks to the live service.

NHS CFH takes any and all opportunities to share lessons learnt. Every HSSI is thoroughly reviewed after the event and any findings are fed into all affected processes and procedures. NHS CFH also promotes and supports a number of user groups to share best practice, common experience and lessons learnt.

NHS CFH Service Management monthly HSSI summary (PDF 11Kb) 02/03/2007

NHS CFH Service Management weekly HSSI statistics (PDF 11Kb) 02/03/2007

If these problems occur at a relatively small number of sites, what are the implications when there are many more deployments?

NHS CFH already delivers systems which are ubiquitous in sectors of the NHS such as QMAS and we regularly publish updated data on its availability.

Similarly, the NHS network (N3) in general performs above contracted levels and has delivered a high degree of resilience on a value for money basis.

Are you confident the national infrastructure will cope well with increasing volumes of transactions?

This is something which we are aware of and for which there can be no absolute guarantees. Active risk management and soak testing does of course reduce the probability of such events occurring.

However, delivering systems where cost is a consideration in their purchasing arrangements means there is always a possibility, however small, of system failure.

Spine availability data (PPT 400Kb)

Should NHS CFH rely on LSPs rather than trusts to report major incidents, as it may sometimes prove difficult for some trusts to convince LSPs of the potential seriousness of the incidents?

We run a culture of openness around system issues and encourage LSP, NASPs and end-user organisations to report problems as they occur. Mature collaborative working between all parties is necessary to ensure that problems are resolved rapidly.