Varian TrueBeam: Testing Definition
Designed and implemented a repeatable method, protocol, KPIs, and reporting structure for medical hardware & software human factors testing.
Varian Medical Systems was facing new FDA requirements for safety and usability human factors testing for all medical devices and software. EchoUser was tasked to develop a new process of testing medical devices and piloted this process by determining the level of usability for Varian’s new product TrueBeam 2.0.
Safety is paramount in medical device design. Rigorous, systematic testing with real operators is key to finding and mitigating possible sources of human error.
EchoUser worked with the Varian team over the course of one year to develop, pilot, and implement an FDA-compliant, rigorous, repeatable process to measure, diagnose, classify, and report on the user safety and experience of its medical products. We wanted to test how well designed, productive, and safe their interface was for conducting tasks. We used benchmark usability tests to determine the product's level of usability.
The device selected for the initial pilot was the TrueBeam 2.0 radiotherapy machine which treats cancer by delivering powerful, targeted radiation to cancerous cells with pinpoint accuracy, directed by its complex imaging system. Our team was able to test the process from start to finish on a large, highly complex, safety-critical system encompassing both hardware and software, and both patient and operator considerations.
Both current and potential users were tested with possible scenarios to gauge the users’ perceptions of the device's current safety in the usability tests.
We measured research findings by four basic Key Performance Indicators (KPIs), which we determined to be Effectiveness, Use Safety, Efficiency, and Satisfaction. After, we identified areas that measured well in the KPIs as well as areas that could use some improvement.
The successful pilot of the Safety and User Experience Benchmarking process for the TrueBeam system EchoUser developed has now become standardized for use with a wide range of medical devices and software across Varian.