How to Test an IRT: A Primer for Users, Part I

IRT Essentials

Regulatory requirements and plain old common sense make the documented testing and validation of the electronic systems used in clinical trials of critical importance to the success of any clinical study. To put it bluntly, approval authorities around the world will reject clinical data collected by what regulators deem to be substandard means (see, e.g., 21CFRPart11, EU Annex 11, etc)

Veracity Logic is an exclusive IRT (IWR/IVR) vendor, so let’s get into the nitty-gritty of what this means for IRT systems.

First, there are two top-level categories of IRT testing:

  • Step I: Vendor verification (VV) of system functionality and the confirmation of system compliance with approved user specifications
  • Step II: User Acceptance Testing (UAT) which confirms that the system does what it was the study needs it to do

To break this down further, VV itself consists of two parts:

  • Unit testing by developers to verify the proper functioning of any new codes/components prior to release for extended internal testing of the system. This presumes the existence of a fully-validated core functionality which does not need to be re-tested on every project unless impacted by project-based changes, or, unless regarded by the vendor to be a high-risk item, deserving of project level re-testing.
  • Detailed, documented (i.e., a written plan and test scripts, etc.) internal testing, positive and negative, for the following:
    • Re-confirmation of the proper functioning of customizations/changes
    • Re-confirmation of the proper functioning of items deemed high risk

Note that to this point the term ‘verification’ has been used for the vendor’s self-testing, as opposed to the term ‘validation.’ Though in reality the vendor’s internal testing is often referred to colloquially as ‘system validation’, validation actually means something else. Here’s the difference:

  • Verification: Performed by the vendor to confirm that the system was built correctly and meets the functional specifications provided by the client.
  • Validation: Performed by the user to confirm that the system as developed meets the study’s needs. In a sense, user testing tests the approved specifications along with system functionality.

It should be obvious from this distinction that, in principle, a vendor cannot ‘validate’ a project-level IRT. Validation of the project-level IRT must be accomplished by the user. Our above reference to a vendor’s “fully validated core system” is appropriate only because in the special case of testing the system core the vendor IS the user.

At the project level, therefore, UAT – User Acceptance Testing -- is the final step in testing a project-specific IRT system. Since the purpose of UAT is to validate the project IRT it makes sense that user testing should be as independent of vendor testing as possible. Here is where some common difficulties arise.

Many clients are not experienced in creating UAT test scripts or performing UAT. What do they test for? How? How does one go about creating test cases/scripts? Do they include both negative and positive testing, or just the latter?  Do they redo the testing of the vendor or something else? Can they use a vendor’s verification documents or must they create testing materials of their own?

We’ll look at all of these conundrums in our next hot topics(s). So be sure to stay tuned!

Contact Us Today to Learn More!

Share this Post