CONTROL TEST AUTOMATION PROJECT

Project Name: CTA- Control Test Automation

Role: Senior Technical Writer/Systems Analyst

Environment: IT Risk Management Teams to cover the complete process improvement of the following risk control categories:

Application Security Assessment (ASA), Architecture Management Office (EAR), Business Technology Solutioning Support (BTSS), Credential Management (CRM), Cyber Threat Intelligence (CTI), Data Backup & Restoration (DBR), data protection Management (DP), deployment Management (DPM), Disaster Recovery management (DIR), End Use Device Security (EUDS), Enterprise Email Hygiene Platform Management (EPM), EO&T Resiliency Operations Management (ROM), Identity Lifecycle Administration (IACTM), Infrastructure Management (INFR), Keys and certificates Management (KCM), malware protection (MAP), Network Operations (AVM), Network Security (NRS), Operational Risk Management and Governance (ORMG), Release management (REM), Secure Configuration Design and Architecture (SCIM), Secure Event Detection and Monitoring (SLEM), Security Incident Response (SIR), Technology Change Management (TCM), Tecchnology Event Management (EVM), Vulnerability Management (VM).

Tools Used: MS SharePoint Office365, Visio, OneNote, Teams, PowerPoint, Jira, Confluence, Bitbucket

Project owner: Enterprise Operations & Technology Risk Management

Partnership: Modern Delivery, Risk Assurance, IT Operations, Internal Audit, ERM, Process and Control Owners

Coding Language: Python, Java

SDLC: Modern Delivery

Code Repo: Bitbucket

Platform: Control Automation v3.5.9

Technical Tools: MD Toolchain

Project Management: Agile execution

Risk Management: Aligned with SDLC processes/controls

Description: Worked on the Risk Management Team to create Business Requirements and Test automation documents.

CTA Project Details:

Project Process:

1. Planning: Control Selection, Feasibility Study, Project Plan
2. Design: Business Requirements Design/Acceptance Criteria, technical Requirements Design, System Design, Security Design, Review and approval
3. Development: Tools Selection, Script Development, Version Control
4. Testing: Unit Testing, Integration testing, User Acceptance testing, Performance Testing
5. Deployment: Deployment Plan, Environment Setup, Deployment Execution, Post-Deployment Monitoring.
6. Maintenance: Performance Monitoring, Updates and Patches, Incident Management, Knowledge Management, Training.

Project Results: A complete CTA Business Requirements, Test Scripts. Ready for automation to enhance the efficiency and consistency of risk management processes through control testing automation. By collaborating with the SDLC team and seamlessly integrating with the existing technical tools and platforms, the CTA process is naturally aligned with shared IT general controls.

The Challenge:

1. The identification of risk and management of Controls to mitigate risk within the risk management framework, as defined in Corporate Policy Enterprise Risk Policy and as per the requirements established by the Control Standard for management of Controls, including identifying, executing, testing, remediating, and reporting.

1. Identification of Controls: Controls are the actions taken to mitigate risks to processes or to the achievement of our strategic goals and business objectives. When a new process and/or risk is created or an existing process and/or risk changes, the need to identify new or modify existing Controls must be assessed.

• At least one Key Control (see details in section II. C. Control Designations) must be associated with risks that have an inherent risk rating of high or very high and require a reduced level of risk per the RCSA Standard: Multiple key Controls or a combination of Key and Non-key Controls may be required to mitigate a risk to an acceptable level.
• Controls that are required to reduce risk to an acceptable level must be documented in BWise.

1. Documenting Controls: Requirements for writing effective Control descriptions:

• Who is responsible for performing the Control (e.g., staff position, skill level, or system name if automated)
• What actions are taken in executing the Control (e.g., reconcile, review/approve, anomaly reporting/resolution) and the source of the information used to perform the Control.
• When the Control is executed (i.e., the frequency and timing of the Control); for Ad Hoc Controls, the trigger must be clearly described
• Where the evidence of Control performance and results are stored (e.g., central data repository)
• Why the Control is being executed (e.g., which risk(s) the Control mitigates)

Data Points on the control element in BWise (e.g., Control owner, key/Non-key designation, type, automation) must be documented along with the Control description.

1. Control Designations: Controls can be designated as either Key or Non-key.

• A Key Control[1] is a Control that is required to reduce risk (including the risk of non-compliance with Obligations and Corporate policies and standards not related to compliance with Governmental Obligations) to an acceptable level.

• A Non-Key Control is a Control that improves management’s ability to mitigate risk; however, a failure may not impact management’s ability to reduce risk to an acceptable level. These Controls can be considered supporting or backup Controls.

1. Shared Controls: Shared Controls are Controls that are fully owned and operated in one process and are leveraged by another process to mitigate risks.
   • Risk or process owners who want to leverage a Shared Control that is not classified as an IT General Control (“ITGC”) or a Non-key Control in BWise must first obtain written agreement from the Control owner for the association before it is updated in BWise. The agreement must indicate the Control designation and the inherent risk rating of the risk(s). The risk or process owner is responsible for obtaining and retaining[2] the agreement.
   • Agreement of the use of ITGCs and Non-key Controls is not required.
   • Agreement is not required for Shared Controls provided by a centralized process that mitigates risk at the divisional or departmental level (e.g., crisis management Control owned by a division-wide operational resiliency process), as outlined in the RSA Standard section II. Requirements for RCSA.
   • It may be necessary for divisions to have Controls in place to supplement the use of Shared Controls to mitigate their risk to an acceptable level, as determined by the process and Shared Control owner.

1. Reliance on Second or Third Line-owned Controls: Risks that are part of first line processes should generally be mitigated only by first line-owned Controls. However, there may be circumstances that warrant leveraging Control owned by the second line; in these cases, first lines may leverage second line Controls with written approval from the Control owner and the Enterprise Risk Officer (“ERO”) for the relevant risk type. Controls owned by the third line or executed by regulatory bodies cannot be relied upon by first or second lines under any circumstances.

1. Use of Controls Owned by Third Parties or Other External Parties: Controls executed by third parties or other external parties can be leveraged when divisions have a Control in place around the execution of an independent review of the third party/external party Control environment (e.g., review of System and Organization Control (“SOC”) 1 or 2 reports).

1. Control Execution

Controls with a manual component must be executed by an individual with the necessary knowledge, experience, and authority to perform the required actions, identify concerns with the Control, and communicate any deviations from expected outcomes to the Control owner. Controls must be executed as documented in the Control description; supporting processes and/or procedures may exist to provide additional instruction on Control execution. Evicence of Control execution, whether Automated, Manual, or Automated-manual Pair (“AM Pair”) must be retained, including supporting information used (e.g., reports, checklists, approvals, logs).

1. Control Testing
2. Control Testing: Control testing is the process of assessing, as of a point-in-time, if risk has been mitigated and/or if stated objectives and requirements have been met by the Control; it consists of an assessment of the design and operating effectiveness of the Control. Testing must be performed by an individual who is independent of the process and execution of the Control being tested. The division that owns the Control is responsible for testing it.

1. Test Planning: Expected test dates must be entered in BWise for Controls that require testing per the table in Section IV. Control Testing Frequency. Any deviations from the required testing frequency must be reviewed and approved in writing by the relevant Division Risk Officer (“DRO”);[3] the impacted Control IDs, proposed test dates, rationale for not testing/delaying testing, and approvals must be retained.

1. Control Testing Frequency: Testing must be performed as follows:

Control Designation	Highest Inherent Risk Rating of Associated Risks	Type of Test Required[4]	Frequency of testing
Key	Very High, High	TOD, TOE	Every 12 months
Key	Medium Low	Not required	N/A
Non-key	Very High, High, medium, Low	Not required	N/A

• New Controls for risks with an inherent risk of Very High or High must be tested withing 12 months of when the Control is created and made active in BWise.
• Existing Control designations that change from Non-key to Key must be tested within 12 months of the change.
• For Controls that require testing, the Control Design Effectiveness and Control Operating Effectiveness fields in BWise must be blank until testing is performed unless the Control is known to be ineffective.
• Existing Controls that change significantly enough that the previous test procedure is no longer sufficient to fully test the Controls must show the Control Design Effectiveness and Control Operating Effectiveness fields in BWise as blank until a test of the updated Control is performed. This test must be performed no later than 12 months after the change is made.

1. Test of Design (“TOD”) Effectiveness – A TOD involves performance of inquiry procedures with the Control owner or another individual who is knowledgeable about the Control to gain an understanding of how the Control mitigates the applicable risks, and a test of at least one execution of the Control. The TOD must include:

• An evaluation of whether the Control can prevent or timely detect and correct errors and if the Control mitigates all associated risks with an inherent risk rating of high or very high that use the Control as Key; this includes an assessment of the completeness and accuracy of inputs into the Control.
• Verification that the Control has been placed into operation and is performed by an individual with the knowledge and experience necessary to effectively execute the Control.
• A test of at least one execution of the Control to confirm that it is implemented as designed. More than one execution may need to be tested to validate the design of all aspects of the Control that are relevant to achieving the acceptable level of risk.

An Effectively designed Control is one that will not consistently mitigate the associated risks even if it operates as designed. Exceptions identified with the design of the Control result in a conclusion that the design is ineffective. The operating effectiveness is not evaluated for Controls with an ineffective TOD.

1. Test of Operating Effectiveness (“TOE”) – A TOE determines if a Control consistently executes as intended across all aspects of the Control that are relevant to achieving the acceptable level of risk.
   • Minimum sampling requirements are documented in section IV. h. Sample Sizes.
   • Commonly used sampling methodologies include random, haphazard, and judgmental.
     1. Random sampling involves the use of a random number generator to select samples for testing from the overall population; this method gives each item in the population an equal probability of being selected.
     2. Haphazard sampling approximates random sampling and does not involve applying judgment to select the test sample.

• Judgmental sampling leverages the tester’s knowledge of the Control and skews the sample toward those population items that are most critical or relevant or purposely selects samples to cover a certain aspect of the Control. It can be used in place of random or haphazard sampling or augment it (e.g., use random or haphazard sampling for some samples and judgmental sampling to include samples that fit certain criteria).

• Each test sample must be assessed against all aspects of the Control that are relevant to achieving the acceptable level of risk.
• If one or more Exceptions are identified in Control execution, the TOE conclusion must be ineffective.

1. Testing Techniques: The testing techniques used must be appropriate for the type of Control being tested and rationale for the technique(s) used must be documented.

Technique	Description
Inquiry	Interview with the Control owner, performer, or other knowledgeable individual involved in the Control to understand how the Control fits into the end-to-end process, is executed, and mitigates the related risks. This technique is leveraged during the TOD in conjunction with one of the below techniques; inquiry alone is not sufficient for the TOD or TOE.
Observation	Direct visual observation of the execution of a Control; used when examination/inspection of evidence or reperformance is not possible
Examination or Inspection of Evidence	The review of documented evidence that a Control was executed. This may include a review of configuration, logic, or code for Automated Controls.
Performance	The tester independently executes the Control and compares results to that of the Control performer to determine if it operated as intended.

1. Test Period: This sub-section applies to non-remediation testing. See section V. Control remediation for remediation testing details. The minimum test period must be at least the most recent nine-month timeframe, or the most recent execution for annual and Ad Hoc Controls (if not executed within the most recent nine-month timeframe). If changes were made to a Control, the test period can be modified to the period when the new Control was operational; however, shorter test periods limit management’s ability to assess the sustainability of Controls and must be avoided when possible. If a test period of less than nine months is used, written approval must be provided by the relevant DRO. Automated Controls can be tested as of a point-in-time and, therefore, do not require a test period unless they are tested as Manual Controls.

1. Sample Sizes: The sample size of the TOE must be based on Control frequency or, for Controls that execute on an Ad Hoc basis or multiple times on a regular cycle,[5] the population size for the test period. The table below must be used to determine the minimum sample size. The use of smaller sample sizes must be approved in writing by the relevant DRO. When conducting a TOE, if the tester obtains evidence from a partial sample that results in the conclusion that the Control is not operating effectively, testing the remaining samples is not required.

Control Frequency	Multiple Execution and Ad Hoc Controls – Test Period Population Size	Minimum Sample Size
Annual/Semi-Annual	1-2	1
Quarterly	3-5	2
Monthly	6-17	2
Weekly	18-75	5
Daily	76-250	20
Multiple Times a Day	Greater than 250	25
Automated	N/A	1 for each aspect of the Control that is relevant to achieving the acceptable level of risk

1. Reliance on Assurance Activities performed by Other Groups: Assurance activities performed by groups other than the owning division’s self-testing team can be leveraged and take the place of self-testing of a given Control for the relevant test cycle in certain situations:

Assurance Performed	Results of Assurance	Reliance Allowed?
SOX Testing	Effective or Ineffective	Yes, however, the owning division’s self-testing team is responsible for performing additional testing (TOD and TOE) to cover any risks or aspects of the Control relevant to achieving the acceptable level of risk that are not covered by the SOX Team’s test.
Second or Third Line of Defense performs oversight activities (e.g., risk deep dives, Control testing, program assessments)	Effective Control/Process	NO; TOD and TOE are required per the standard schedule
	Ineffective Control/Process	Yes
Validation of an issue[6] by Internal Audit	Effective or Ineffective	Yes

• In instances when a second team in the first line retests part or all of a Control that was originally tested by another group in the first line and finds an Exception, they must communicate details of the Exception in writing to the team that performed the original test so it can be assessed for impact.
• Regardless of which group performs the Assurance activities, the Control testing lead for the division that owns the Control is accountable for confirming that all in-scope risks and aspects of Controls in the division are fully and appropriately tested.

1. Review and Approval of Test Results: All testing workpapers and supporting evidence must be reviewed internally by at least one employee with a job level of Senior (or above) and the Control owner must concur with the results before they are considered complete.

1. Documenting test Results in BWise: Control effectiveness results and actual testing dates for tests completed during the month must be documented in BWise by month-end. BWise must also be updated to reflect the Control Design Effectiveness and Control Operating Effectiveness fields as blank for Controls that mitigate high and very high inherent risks when testing will not be completed withing the required timeframes unless the Control is known to be ineffective.

• Control Remediation

When a Control is found to be ineffective, the identified weaknesses must be remediated in accordance with the Issue Management Standard. Additional requirements for ineffective Controls include:

1. Issue Creation and Remediation – For issues opened as the result of Control testing, the testing team must review the action plan and determine if it appears reasonable to address the identified concern.

1. Remediation Testing – If the design of a Control was found to be ineffective, a retest of the TOD must be performed. If the operation was found to be ineffective, only the TOE is required unless the TOD is beyond the 12-month test requirement by the time that remediation is completed, and retesting is performed. It is acceptable to limit the test period to two months for remediation tests of Controls that execute weekly or more frequently without approval from the relevant DRO and the sample size tested must be determined by the issue owner to be sufficient to demonstrate that the remediation activities were successful and meet requirements in the Issue Management Standard. The tested samples can contribute to the full retest that is performed later. The Control Design Effectiveness and/or Control Operating Effectiveness field(s) in BWise cannot be marked as effective until a full retest, per requirements in section IV. Control Testing has been performed; however, the fields can be changed from ineffective to blank when the issue is closed.

1. Control Reporting

Control testing status and effectiveness for Controls owned in each division must be reported to the division’s Business Risk Committee (or other divisional management meeting if the division does not have a business risk committee) as part of the periodic operational risk reporting required by the Enterprise Operational Risk Policy.

As appropriate, ERM is responsible for reporting Control effectiveness at the enterprise-level to the Board or an appropriate Board committee on a quarterly basis.

1. Roles and Responsibilities

The roles and responsibilities of the first and second lines of defense for operational risk management are established in the ERP and the Enterprise Operational Risk Policy and are further delineated below for management of Controls. Execution or responsibilities may be delegated at the discretion of the responsible party.

First Line

DROs

• Managing the division’s execution of the requirements of this Standard
• Maintaining Control information in BWise, including Control description, risk and Obligation associations, and other data points (e.g., risks, frequency, automation)
• Managing and reporting on issues related to ineffective Controls in accordance with the Issue Management Standard
• Reporting testing status and effectiveness in the division’s operational risk reporting in accordance with the Enterprise Operational Risk Policy
• Reviewing and approving/rejecting deviations from testing requirements
• Acting as a point of escalation, as necessary

Process, Risk and Control Owners

• Designing, documenting, and executing Controls to:
  • Mitigate associated risks, including the risk of non-compliance with Obligations and Corporate policies and standards not related to compliance with Governmental Obligations
• For process or risk owners leveraging a Shared Control:
  • Requesting and retaining agreement from Control owners to associate a risk to a Shared Control that is not classified as an ITGC or Non-key; the request must include the Control designation and the inherent risk rating of the risk(s)
• For Shared Control owners:
  • Agreeing to or rejecting requests for risks to be associated with Controls they own that are not classified as an ITGC or Non-key
  • Identifying and communicating any supplemental Controls or requirements needed to effectively leverage the Shared Control
  • On at least a quarterly basis, communicating changes to the Shared Controls to risk owners and DROs that leverage the Controls. Changes that require communication include those that could impact how effectively the Control mitigates the associated risks (e.g., change in frequency or scope, retirement)
  • For Shared Controls not classified as an ITGC or Non-key, communicating changes in TOD and TOE results to impacted risk owners and DROs
• Identifying Control performers who have the knowledge, experience, and authority to execute the Control as intended
• Communicating significant changes to the Control Testing Lead/Control Tester so adjustments can be made to test plans, as necessary
• Retaining evidence of Control execution
• Monitoring and correcting data anomalies reported by ERM in BWise
• Resolving issues with Control design or execution in accordance with the Issue Management Standard
• Supporting testing activities and concurring with test conclusions

Control Testing Lead/Control Tester

• Performing testing on Controls and concluding on design and operating effectiveness, considering all in-scope associated risks.
• Documenting testing and retaining all related documentation.
• Recording issues identified through Control testing in accordance with the Issue Management Standard and concurring with action plans[7]
• Performing remediation testing, as necessary
• Maintaining TOD and TOE results, planned test dates, and actual test dates in BWise for all Controls owned by the division
• Confirming that other testing groups (e.g., SOX) used to perform testing on behalf of the division included all in-scope risks and tested all aspects of the Control that are relevant to achieving the acceptable level of risk and performing additional testing, if necessary
• Communicating test results to the relevant DRO(s), process, risk, and Control owners, and other relevant parties
• Communicating issues identified during self-testing or testing delays that extend beyond required timeframes to the risk owners and DROs who leverage the Control
  • This requirement excludes communication of issues identified during self-testing of Shared Controls that are not classified as ITGCs
• Supporting second line oversight activities
• If self-testing is outsourced and/or a division does not have a Control testing Lead or Control tester, the relevant DRO assumes the above responsibilities.

Second Line

EROs

• Overseeing compliance with, and consistent application of, this Standard by divisions
• Providing oversight on the use of Controls by divisions to mitigate risks, including the risk of non-compliance with Obligations and Corporate policies and standards not related to compliance with Governmental Obligations
• On a sample basis, reviewing Control documentation, including, but not limited to, clarity of Control descriptions and key/Non-key designations
• Monitoring Control effectiveness ratings for alignment with open issues
• Identifying and communicating data anomalies in BWise to divisions
• Reporting to the Chief Compliance Officer (“CCO”) on results of assessments and monitoring performed to determine compliance with Obligations
• Reviewing and approving or declining requests by the first line to leverage second line-owned Controls for mitigation of first line risks

ERM Controls and Testing Oversight

• Maintaining this Standard and providing training on the requirements of this Standard
• Maintaining BWise functionality to support Control requirements
• Executing oversight of self-testing, including reperformance of self-testing and reviews of testing teams and plans
• Performing enterprise-level reporting

1. Governance and Escalation

The [Entity] has established a governance and escalation structure as outlined in the ERP, which is supplemented below in relation to this Standard.

Interpretations

The Senior Vice President, Operational Risk and Shared Services (“SVO, ORSS”) is authorized to make interpretations of this Standard, which must be requested, in writing, by a DRO, division head, a delegate of either role, or initiated by the second line. Interpretations will be communicated in writing to all impacted EROs and DROs.

Escalations

When the first and second lines of defense have a difference in opinion or interpretation of this Standard, the matter may be escalated to the ERO for the relevant risk type for resolution with the DRO, division head, or a delegate of either role, subject to further escalation pursuant to the ERP. When first or second lines and the third line have a difference in risk judgements or decisions under this Standard, the escalation provisions of Corporate Policy 1-155, Internal Audit Charter, are applicable.

Exceptions

The SVP, ORSS is authorized to approve exceptions to the requirements in this Standard, subject to the requirements established in the Corporate Risk Policy Standard. Non-compliance with the Standard (for which no exception has been granted) must be reported to the SVP, ORSS for resolution with the appropriate DRO, division head, or a delegate of either role, which may include the recording and remediation of an issue pursuant to this Standard.

Self Testing Requirements

The Control Standard establishes minimum requirements for the self-testing of operational (non-SOX) controls

• Conducted for controls which are key to addressing a risk with a High or Very High Internet Risk Rating
• Type of testing required includes the following:
• Test of Design (TOD) Effectiveness
• Test of Operating (TOS) Effectiveness – The minimum test period must be at least the most recent nine-month timeframe, or the most recent execution for annual controls
• Process, Risk, and Control expectations are detailed in the Control Standard (4)

Testing is performed every 12 months for existing controls. Additionally, controls must be tested within 12 month of the following events:

• Creation of new controls (and made active in BWise) for risks with an inherent risk of Very High or High
• Changes to the designation of existing controls from non-key to key
• Changes which are considered significant to existing controls where the previous test procedure is no longer sufficient

Self Testing Plan

Objective

• Achieve Compliance with Control Standard (S11-109.F) Section IV. Control Testing
• Control testing is the process of assessing, as of a point-intime. If risk has been mitigated and/or if stated objectives and requirements have been met by the Control; It consists of an assessment of the design and operating effectiveness of the Control. Testing must be performed by an individual who is independent of the process and execution of the Control being tested. The division that owns the Control is responsible for testing it
• Provide reasonable assurance on effectiveness of in-scope controls

Timing

• Testing Cycle: Q2 2023 through Q1 2024
• Annual development of a test plan to be reviewed and approved by the EO+T DRO
• Quarterly reassessment of the test plan to incorporate any relevant changes to the control environment, control designation changes, etc.

Factors Considered

• Open issues
• Interdependent processes

Control Environment Summary

Total EO+T Controls: 466

181 controls In-Scope for Control testing based on Control Standard

• 170 Controls on Test Plan
• 146 Operational Controls
• 82 Controls not tied to an open issue
• 64 Controls tied to an open issue(2)
• 24 SOX Controls which cover operational risks
• SOX Control which cover operational risks (SOX team’s testing to be reviewed by EO+T Self-testing and if needed, additional testing performed to address operational risks) (1)
• 11 Controls Not on Test Plan due to the following reason(s):
• SOX Controls which do not cover operational risks (testing to be performed by the SOX team)

285 Controls Not in-Scope for Control Testing based on Control Standard

Sed tincidunt tempor leo, vitae consequat lorem ornare ultricies. Nullam fringilla ultricies pharetra. Aliquam suscipit leo vel est fringilla laoreet in eget lorem. Proin non rhoncus nulla. Pellentesque porta nunc pretium, scelerisque justo a, tincidunt tellus. Mauris eu dictum mi. Vivamus non finibus ligula.

Nulla consectetur maximus turpis a egestas. Mauris efficitur, ante non bibendum eleifend, diam massa varius ex, non vestibulum risus metus in eros. Proin eu urna vitae ex feugiat interdum. Nunc vel auctor nisi. Pellentesque a bibendum tortor. Sed scelerisque auctor odio, id scelerisque diam fermentum sit amet.

Nulla consectetur maximus turpis a egestas. Mauris efficitur, ante non bibendum eleifend, diam massa varius ex, non vestibulum risus metus in eros. Proin eu urna vitae ex feugiat interdum. Nunc vel auctor nisi.

Nulla consectetur maximus turpis a egestas. Mauris efficitur, ante non bibendum eleifend, diam massa varius ex, non vestibulum risus metus in eros. Proin eu urna vitae ex feugiat interdum. Nunc vel auctor nisi.

Nulla consectetur maximus turpis a egestas. Mauris efficitur, ante non bibendum eleifend, diam massa varius ex, non vestibulum risus metus in eros. Proin eu urna vitae ex feugiat interdum. Nunc vel auctor nisi. Pellentesque a bibendum tortor. Sed scelerisque auctor odio, id scelerisque diam fermentum sit amet.

Nulla consectetur maximus turpis a egestas. Mauris efficitur, ante non bibendum eleifend, diam massa varius ex, non vestibulum risus metus in eros. Proin eu urna vitae ex feugiat interdum. Nunc vel auctor nisi.