DR Testing and Remediation
After the implementation of a new hybrid cloud platform to support 20+ countries and 500+ business critical applications the client retained Citihub to mobilise, plan and manage the first annual DR test. The platform is physically hosted in several European locations and integrated multiple external and internal service providers.
The test was subject to intense business scrutiny on the performance of the platform and the service delivery organisation. Findings needed to be documented in detail and contractually enforceable with service providers.
How Citihub Helped
Citihub mobilised a small team that:
- Critically assessed DR policies, procedures and documentation to ensure that they reflected the ‘new’ recovery position after migration to the Hybrid Cloud;
- Identified gaps or deficiencies in the recovery plans and worked with service providers to remediate these;
- Conducted a set of progressively more mature table-top DR exercises to test documentation, procedures and the knowledge of key individuals;
- Built disaster scenarios, syndicated and agreed a set of tests with business stakeholders;
- Performed a deep-dive technical assessment of the platform and its readiness for the test so as to reduce the risk of a business impacting outage;
- Mobilised the test resources (service providers, internal IT, IT security, application teams, and business testing resources – 300+ staff in total) and coordinated the development of integrated test plans;
- Choreographed and monitored the event in real time and intervened where necessary to ensure success;
- Measured the performance of the DR test using KPIs and SLAs;
- Executed all business facing communications both pre-and-post the test;
- Documented all lessons learned along with technology, skill, documentation and process deficiencies;
- Produced remediation and improvement recommendations covering technology infrastructure, outsourced commercial arrangements, and service delivery.
Citihub’s approach delivered a structured, well planned and measurable test of the end-to-end IT solution and applications.
Recovery steps were documented in a timeline to give visibility to the business and to inform future investment decisions for priority services. Then remediation and improvement actions were prioritised so that traceable work packages could be developed and mobilised. All high probability and high impact risks were quickly addressed by implementing technology improvements or process changes.
The testing methodology took a holistic approach by examining the performance of all aspects of the IT platform including: network, data centres, servers, storage, databases, applications and IT security infrastructure. This approach was specifically designed to ensure a universal test that could identify any stacked failures resulting from technology and process dependencies.
The test strategy, governance, preparation guidelines and testing plans now form the basis of a repeatable annual DR test. This approach has now been adopted globally.
Related Case Studies
Cloud Secrets Management Solution
A top 3 US Bank asked Citihub to implement HashiCorp Vault to provide a centralized secrets management solution for their very ...
NYU partnered with Citihub to offer a course on public cloud security technologies
Citihub was recently added as an industry partner to New York University’s (NYU Tandon) Cyber Security program. Exclusive to NYU Cyber...
Ian Tivey & Jim Oulton Named Technical Directors
Ian Tivey and Jim Oulton have been promoted to Technical Directors, a role reserved for senior leaders in Citihub who provide...
In the press
Using a ‘Three Lines of Defense’ Program to Balance Development Stakeholder Needs
Using the NIST three layers of defence as a framework, Citihub’s Glen Notman outlines how to leverage agile development capabilities and underpin them...
In the press
The Balancing Act
In this podcast, we will go into the details of how the “technical” automation-for-speed perspective is shifting to a “business-centric” perspective...
Life (and work) in the time of Corona
Less than two months after starting his job at Citihub, Senior Consultant Luis Carrazana, together with the rest of New York,...
In the press
Role of Security in a Digital First Enterprise
Join Citihub’s Glen Notman as he injects practical insights on how to enable security practices in a digital enterprise.
In the press
Compliance Challenges in a Lockdown World
The ongoing coronavirus crisis has changed business norms around the world, but as organisations struggle to come to terms with large-scale...
In the press
Institutionalizing DevSecOps in the Large Enterprise
Citihub’s Chris Zanelli, joined by several industry peers, will discuss topics across DevOps & DevSecOps, Enterprise Compliance as Code, Cloud Compliance...
Military Veterans are Welcome at Citihub Digital
This Memorial Day, when the rest of the United States of America will pay tribute to the military personnel who have...