SCTO Computerized Systems Validation Policy for R

Document development, review and version history

Development and Review

	Name	Date
Authored/Revised by
Reviewed by
Released by

Version History

Version	Date	Author	Summary of Changes
0.1	2024-04-18	Alan Haynes	Initial draft

Introduction

This policy provides the framework for validating R and R add-on packages in use by Swiss Clinical Trial Organization (SCTO) and its SCTO network members¹ for the purpose of creating R products in the context of clinical research. Validation may be executed on SCTO or CTU level, where the SCTO provides

• recommendations for R base installation validation,
• processes for R add-on package validation and
• a platform to serve as an inventory for documentation of all R add-on package validation performed by the SCTO and its network members according to this policy.

Validation activities in general ensure that computerised systems (CS) can be relied upon to consistently produce a product that meets predetermined specifications of accuracy. The formal process of validation results in a validation (documentation) package².

The validation of a computerised system consists of the following elements:

• Infrastructure
• Hardware
• Software
• Processes
• Training

The validation activities focus on verifying that the software consistently fulfills the specified requirements. Processes must be in place to ensure that the infrastructure and hardware required for the functionality of the software are qualified and changes controlled. Additionally, all users and (if applicable) support personnel must be trained to use the software and follow related business processes.

Any GxP critical system supporting activities such as statistical analysis (e.g., in the context of clinical trials) should be validated according to the following international standards and guidelines:

• ICH E6: Guideline for Good Clinical Practice
• GAMP 5: A Risk Based Approach to Compliant GxP Computerized systems
• FDA 21 CFR part 11: Electronic Records; Electronic Signatures
• FDA 21 CFR part 11: Guidance for Industry
• FDA: General Principles of Software Validation
• Eudralex 4 Good Manufacturing Practice Guidelines, Annex 11: Computerised Systems

This SCTO policy is based on the ICH E6 guideline for Good Clinical Practice. FDA regulations and guidelines (US) and Eudralex 4 (EU) represent additional best practices for validation of computerised systems in the industry and may serve as a guideline for academical purposes.

Background

R is a ‘language and environment for statistical computing and graphics’ (source: the R file). It is an official part of the Free Software Foundation’s GNU project and is released under the Free Software Foundation’s GNU Public License . CTUs use R primarily, but not exclusively, in the preparation and analysis of data from supported studies. Internal use for other reporting, visualisation and business management might also take place, according to individual CTU needs, but is not the focus of this policy.

Installation and use of R is based on several levels and sources:

1. The “official R distribution” (referred to as ‘R base installation’ in this document), which includes the R base and recommended packages (as defined in the R-foundation declaration with regards to validation and compliance, 2021) and is available via the Central R Archive Network (CRAN).
2. Distribution of contributed R add-on packages provided by the community and available via CRAN.
3. Distribution of R add-on packages provided by the community via other sources than CRAN (e.g., Bioconductor, Github, and various websites).

Additionally, R packages may be developed in-house for internal use only and not distributed via any of the channels mentioned above. If such packages or some of their functions are used in the GxP context (e.g., clinical trials), they must also be validated. This validation should, whenever feasible, follow the process outlined in this document and associated SOPs even if the R package is not shared with the SCTO community.

Scope

In Scope

• Recommendations for R base installation validation (the final process and all documentation has to be defined and prepared by each CTU according to their local processes)
• Processes for R add-on "intended for use” packages validation (incl. package risk assessment & functional testing)

Note

Processes for R add-on package validation described in this document are to be considered minimum requirements as shared by the SCTO platform. Individual organizations (CTUs) can define stricter processes.

Out of Scope

• Standards for IT infrastructure (to be defined on local level by the CTUs, see Section 5)
• R add-on package management to ensure traceability and reproducibility on R product level (to be defined on local level by the CTUs, see Section 5.3)
• Risk assessment and management of R products (to be defined on local level by the CTUs, see Section 5.4): Products are always CTU and clinical trial specific – a high-risk product may result in the need for additional validation activities for an R package, even if that package/ required function is available as “validated” on the SCTO platform.
• In general: any additional internal, organization specific, processes of CTUs.

Abbreviations & Definitions

Terminology	Definition
CTU	Clinical Trial Unit. For purposes of this document: all SCTO network members.
CRAN	Comprehensive R Archive Network
EMA	European Medicine Agency
FDA	Food and Drug Administration
HLRA	High-Level Risk Assessment
HRA	(Swiss) Human Research Act
Intended for use package	"Intended for use” packages are add-on packages that comprise the functions required for a specific product and are typically loaded directly by the user during an R session.
Product	Any deliverable produced using R via the processes below. Typical statistical products include sample size estimation, statistical analysis plan, an analysis report etc.
QMS	Quality Management System
R add-on package	Provided by members of the R community available via CRAN and other distribution repositories
R Base Installation	Provided as part of the official R distribution released by the R Foundation available via CRAN includes Base R and recommended packages
SCTO	Swiss Clinical Trial Organization
SOP	Standard Operation Procedure
Validation Package	A set of documents that is prepared for documentation of the validation process.

General R Validation Approach

The SCTO follows a risk-based approach to the validation of R. For the validation and assessment of risks, we differentiate between the following levels:

• IT infrastructure qualification (according to local CTU processes, see Section 5.1)³
• System level (R base installation, incl. recommended packages, according to local CTU processes, recommendations and examples provided by SCTO, see Section 5.2)
• Package level (R add-on packages/ functions used within those packages, according to SCTO processes, see Section 5.3)
• Product level (“intended use”/ outcome, according to by local CTU processes, see Section 5.4)

Important: To produce a product in R, under a validated environment, it must be ensured that R is running using a validated base installation version and qualified IT infrastructure with proper documentation, to ensure consistent and reliable results on all levels. The different levels build on each other:

• Qualified infrastructure provides the basis for the validation of the R base installation and any packages running on that installation.
• R base installation validation ensures that the basic installation of R (including recommended packages) provides a reliable framework on which additional packages can be installed (add-on packages) and functions from those packages executed.
• R add-on package validation ensures that all functions within an add-on package required for a specific product are sufficiently tested in relation to the package and product risk (see R Product Risk Assessment Metrics for details). This takes place by evaluating the base risk associated with an R package, and, upon necessity, performing specific function tests to ensure that functions required for a specific product (within an add-on package) produce consistent and reliable results from a statistical perspective (see R Function Testing SOP for details).

IT Infrastructure Qualification

The IT infrastructure qualification is managed at CTU level according to local IT processes. The risk associated with IT infrastructure components is usually low.

For the reliable use of R and statistical traceability it is important to note which IT infrastructure (specifically which operating system software and version) R is installed on and to ensure that the infrastructure on which R is running is qualified.

R Base Installation Validation

Every CTU must define:

• The minimal (validated) software version of the R Base Installation to be used at the CTU and a process of handling version changes.
• The R base installation validation package documenting the validation activities (see table below for recommendations).

The R base installation validation package is a set of documents that should be prepared to document the validation process. The following table lists all documentation that is typically prepared during a computerised system validation process according to global standards and guidelines. It additionally provides recommendations how this documentation may be covered when preparing an R base installation validation package. The exact content of the R base installation validation package (i.e., the required documentation) at a CTU should follow the local processes and all documentation must be prepared and approved by the CTU. For some of the documents listed below, the SCTO provides examples (as indicated under “Detailed Description”).

Document	Detailed Description
R Base installation High-Level Risk Assessment (HLRA; also known as System Risk Assessment, SRA)	The HLRA is usually created once, with the first validation of an R base installation version and is valid for all future validated versions. It must be updated in case the scope/ intended use/ content of the software changes. The HLRA usually also documents the GxP relevance of the system and therefore the need to validate the system. Example available on the SCTO platform
Vendor Assessment	Vendor assessments are usually conducted with the first validation of a software version. Since R is an open-source software managed by a consortium, a traditional vendor assessment/ audit approach is not feasible. A critical review of “R: Regulatory Compliance and Validation Issues - A Guidance Document for the Use of R in Regulated Clinical Trial Environments” may suffice as vendor assessment and should be documented together with any gaps identified that may require attention during the validation of R and its packages. The assessment of this document should be repeated with a publication of a new article version, or in case the R consortium replaces the existing document, or periodically according to each CTU’s local periodic review processes. Example available on the SCTO platform
Validation Plan & Test Plan	The Validation Plan describes the details of the planned validation process, including required documentation and acceptance criteria for productive use. The Test Plan describes the testing strategy and testing process and may contain details on the tests to be executed. The exact content of both documents depends on the CTU’s local processes. The content of the two documents may also be covered in one “Validation & Test Plan”. Example available on the SCTO platform
(User) Requirements Specification	This document is usually created for the first validation of an R base installation version and should be reviewed, and if required updated, for future validated versions. It specifies the user requirements and intended use for the R base installation. It may also contain any compliance (e.g., data integrity), regulatory (e.g., personal data protection) and safety requirements (e.g., controlled access). Example available on the SCTO platform
Base installation (Functional) Risk Assessment	This document is usually created with the first validation of an R base installation version based on the user requirements or system functions supporting these requirements and should be reviewed and, if required, updated for future validated version. Example available on the SCTO platform
Software installation plan/ instruction	Typically, either the vendor or the local IT department provides a software installation plan/ instructions to follow when installing new software. In case of R, a specific “R software installation plan/ instruction” is not required. It should be sufficient to follow the instructions on CRAN when downloading and installing the new R base installation version. Important is that the installation of the R base installation and version management follow the CTU’s local processes for validated systems and that all versions installed are documented, including the date/ time of installation (see Installation Verification Document below).
Test scripts (“User Acceptance Tests”)	Test scripts are usually created with the first validated R base installation version and should be reviewed, and if required updated and/or new test scripts created, for future validated version. The tests must verify the functions of the R base installation supporting the user requirements. Test scripts provided by R may be used as-is or adapted/ extended as required. The scope of testing must be defined according to the CTU’s local processes and defined user requirements. Example available on the SCTO platform
Installation Verification document (TEST)	The purpose of this document is to provide evidence of the installation of a specific system version on an environment/ machine that is used for testing purposes before the new system version is released to production. In case of R this may be a separate document or an entry in an R repository or other “version tracker” (depending on your local IT processes). The following information should be captured: R base installation version installed Date and time of installation Person performing the installation (may be someone from your local IT department) Outcome of installation: successful, successful with deviations, not successful If applicable: Any actions taken in addition to the regular “download and install from CRAN” process (e.g., uninstalling an older version) Note: depending on your local processes it may not be feasible or even possible to do a preliminary “test installation” and perform tests before productive use of the new R base installation version (e.g., if R is distributed remotely by your local IT department). In this case, we recommend you still execute the tests after the installation and document the test execution. Should any issues be detected during testing, please inform your local IT department immediately. Ideally you implement a process that would ensure nobody is using the new R version for productive use before the tests are completed successfully (i.e., without major issues) and the validation & test report is approved (see “Validation & Test Report” below).
Executed test scripts (“User Acceptance Tests”)	Execute the pre-defined test scripts for each new R base installation version (i.e., follow the steps described in the test scripts) and document the outcome according to the CTU’s local processes. Example available on the SCTO platform
Traceability Matrix	Traceability between requirements and tests can be achieved by creating a separate document (traceability matrix) or by linking requirements and tests within the requirements and tests themselves. It is important, that whatever means is used, it is feasible to proof that all requirements are verified with a test (or if not tested formally, a rationale why the test is not required, e.g., low risk requirements which can be considered “verified” based on the experience by the user community).
Test Report and Validation Report	The Test Report and Validation Report are usually created (or at least updated to a new version) with every new R base installation version to be released for productive use. The main purpose is to summarize the testing activities and results, including any deviations from the expected results to document all completed validation activities, including any deviations from the original plan (see Validation & Test Plan). Deviations should be justified and assessed for criticality. to document the acceptance for productive use (with or without restrictions) The exact content of this report/ these reports depends on the CTU’s local processes. Example available on the SCTO platform
Installation Verification document (PROD)	The purpose of this document is to provide evidence of the installation of a specific system version on an environment/ machine that is used for productive purposes. In case of R, this may be a separate document or an entry in an R repository or other “version tracker” (depending on your local IT processes). The following information should be captured: R base installation version installed Date and time of installation Person performing the installation (may be someone from your local IT department) Outcome of installation: successful, successful with deviations, not successful If applicable: Any actions taken in addition to the regular “download and install from CRAN” process (e.g., uninstalling an older version) See also note under “Installation Verification document (TEST)”: In case of R base installation, the installation may only be documented once and not separately on “TEST” and “PROD”. Important is, that any version changes of the R base installation are managed and documented according to your CTU’s local IT change management processes.

R Add-on Package Validation

The R Add-on Package validation package is a set of documents that should be prepared for the documentation of the validation process of an R add-on package following this policy and associated SOPs. The following table lists all documentation that is typically prepared during a computerised system validation process. It additionally provides details on how this documentation is covered for an R add-on package validation according to this policy.

Document/ Activity	Detailed Description
Validation & Test Plan	No separate document is created. Rationale: The content typically contained in a Validation & Test Plan is determined, with respect to R add on packages, in the various steps of this policy and the associated SOPs and its documentation is specified therein as follows: Determine the product associated risk (local CTU SOP) Determine the package associated risk (based on SCTO Risk Metrics) Determine, if testing is required (see Section 6) Perform and document testing, if required (SCTO Function Testing SOP) Document compliance with this policy and above-mentioned SOPs (local CTU process, e.g., in metadata or in a document)
Package High-Level Risk Assessment	The add-on package risk depends on factors described in the SCTO Risk metrics. The R add-on package high-level risk assessment should be documented according to the SCTO R-Package Risk Metrics on the SCTO designated platform.
(User) Requirements Specification	No dedicated (User) Requirements document is created. Rationale: The requirements and intended use of an R package and its functions can only be defined in relation to a product and the required outcome (“intended use”) of a specific function for that product. Therefore, the requirements can only be documented when planning the R product. Requirements shall be documented in the Statistical Analysis Plan or a Statistical Protocol or in other, similar documentation4. It is important that this “plan” indicates the intended use and what is required of the functions planned to be used within the R package.
Function Risk Assessment	The risk of separate functions within a package can only be assessed in relation to a product and the required outcome (“intended use”) of a specific function for that product (see Section 6).
Package installation plan/ instruction	There is no need for a separate installation plan or instruction. The installation follows a standard procedure using R base installation functions. One of the following procedures is followed: R add-on packages may be installed for functional testing after the high-level package risk and the product related requirements and functional risk are documented and the testing is planned, if the initial assessment shows that the required package is not yet sufficiently tested by one of the SCTO members providing input to the R add-on validation package repository. R add-on packages may be installed for productive use/ used to create R products after the functional tests are completed and no deviations were detected that would raise concerns against using the tested package functions for a given R product OR if the initial assessment shows that the required package was already sufficiently tested and the respective validation package is available in the SCTO R add-on package validation repository.
Test scripts	The process of writing and executing tests for the R add-on packages is covered in the SCTO Function Testing SOP.
Software installation verification (TEST)	No separate document is created. Rationale: The purpose of this document would be to provide evidence of the installation of a specific add-on package (version) for testing purposes. In the context of R add-on package validation it is not feasible to create a separate document. The tested R add-on package (version) functions are documented when following the SCTO Function Testing SOP.
Executed test scripts	Execute the pre-defined test scripts and document the outcome of each step as well as the overall outcome of the test script (see SCTO Function Testing SOP).
Validation & Test Report	The purpose of this document is to summarize all executed validation activities and their outcome, including testing and any defects found and/or any deviations from the original plan with a rationale. It documents the decision, if specific functions from an R add-on package (version) can be accepted for productive use or not and if there are any restrictions, specifies these restrictions (see SCTO Function Testing SOP).
Software installation verification (PROD)	No separate document is created. Rationale: The purpose of this document would be to provide evidence of the installation of a specific add-on package (version) for productive use. In case of an R add-on package and its functions the “productive use” is determined by the product upon calling library(). Therefore, it is not feasible to create a separate installation verification document. What needs to be ensured when using a specific function within an R add-on package: Documentation of package risk assessment based on the SCTO risk metrics, and of function testing according to SCTO Function Testing SOP Documentation of package use (meta-data/session info)

R Product Validation

To be covered by local SOPs for statistical analysis execution at the CTUs. The R product validation should cover at minimum:

• Assessment of risk associated with the R product
• Assess the tools used within the R product
• Documentation for reproducibility and traceability

Risk Management

Given the different validation levels, also the risk needs to be assessed and managed on all those levels, specifically on:

• System level (R base installation, recommendations provided in this policy, documentation according to local CTU processes, see Section 5.2)
• Package level (R add-on packages, see R Package Risk Assessment)
• Product level (according to local CTU processes, see Section 5.4)

Depending on the outcome of the R add-on package risk assessment, the validation activities are adapted to be stricter for high-risk, or more high-level for low-risk packages. Thus, the testing of specific functions required for a specific product will be adapted depending on the combined risks from the package risk and product risk (defined on a local level):

		R – package risk (baseline risk)
		Low	Medium	High
Product risk	Low	No function testing required	No function testing required	No function testing required
	Medium	No function testing required	No function testing required	Function testing required
	High	No function testing required	Function testing required	Function testing required

User Training

All users performing statistical analysis and/or developing packages and creating products with R need to be qualified and trained according to their CTU’s guidelines.

All users (potentially) involved in validation activities (e.g., risk assessments, testing) must additionally be trained on this policy and the associated SOPs.

Periodic Review

Every 3 years the validation status of the R add-on packages in the SCTO validated package inventory should be reviewed if packages and/or functions were not updated and re-tested within that period. The review needs to be documented, even if no updates are required.

The periodic review of the R base installation validation package has to be defined and managed according to the CTU’s processes.

Associated Documents

R Package Risk Metrics

This document describes the metrics defining the risk of R add-on packages (development version).

R Function Testing SOP

This document describes the standard process for testing specific functions within an R add-on package (development version).

References

Reference Title	Description
ICH E6: Guideline for Good Clinical Practice	International Council for Harmonisation, E6 Good Clinical Practice (1997) and Integrated Addendum to Good Clinical Practice (GCP) (2016)
The Good Automated Manufacturing Practice (GAMP) Guide 5: A Risk Based Approach to Compliant GxP Computerized Systems (ISPE/GAMP, 2 008)	Guideline for computerised systems validation in regulated environments issued by the International Society for Pharmaceutical Engineering (ISPE). Note that the content of GAMP 5 is not available online.
FDA 21 CFR part 11: Electronic Records; Electronic Signatu res	FDA regulation: Title 21—Food and Drugs, Chapter I - Food and Drug Administration, Department of Health and Human Services, Subchapter A - General, Part 11 — Electronic Records; Electronic Signatures (1997)
FDA 21 CFR part 11: Guidance for Industry	FDA guidance for Industry, Part 11, Electronic Records; Electronic Signatures - Scope and Application (2003)
General Principles of Software Validation; Final Guidance for Industry and FDA Staff (2002)	FDA guidance paper for computerised system validation.
EudraLex Volume 4: Good Manufacturing Practice (GMP) guidelines, Annex 11: Computerised Syst ems	European standard for comupterised systems validation.
R: Regulatory Compliance and Validation Issues - A Guidance Document for the Use of R in Regulated Clinical Trial Environm ents	R Foundation for Statistical Computing guidance document for the validation of R for the use in regulated environments.

Footnotes

1. for purposes of this document referred to as CTUs = Clinical Trial Units

2. A validation package is a set of documents prepared for the documentation of the validation process itself.

3. For example, standards set by the local IT department.

4. User requirements for sample size estimation or power analyses are inherently the sample size or the power or a combination of both and do not need to be documented in a separate plan.