5 Layers to Successfully Organise Your Large-scale Technology Cutovers

Regardless of your specific business or technology context, the success of any large-scale technology cutover activities will depend to a great degree on an appropriate and effective organisation being put in place.

How successful your cutover approach will be depends on having a clear, concrete cutover plan.

For enterprise DevOps teams, this means clear directives and smart communication between all members of the cutover team - and release orchestration tools that provide real-time visibility so issues can be addressed as they pop up.

This blog will describe a sample organisational model that has been designed based on years of experience with tech cutovers, evolving over time to address the range of challenges, roles and processes that are required for success.

Naturally, the size of the required organisation reflects the scale and complexity of the change. The sample below has been proven to work well on some very large-scale cutover events that have been successfully managed with ICEFLO.

Sample Organisation Model (Click to Enlarge)

Organisational Structure for Cutovers at Scale

Organisational Units

In the sample organisational chart (above), we see that the 5 layers are populated by 5 “organisational units,” represented by a colour coded key:

• Technology Service Delivery

  Critical business systems are owned, operated and managed by a dedicated team of Service Delivery or IT Operations specialists. Cutovers are always overseen and often delivered by Service Delivery specialists

• Technology Projects:

  Complex technology change is planned and delivered by a dedicated team of project specialists. This includes direct participation in cutover planning, managing critical events during the cutover and oversight of the roll out period.

• 3rd Party Service Providers:

  It is highly unusual in today’s Enterprise-scale organisation to find technology changes that don’t rely to one degree or another on 3rd party suppliers. These key partners need to be embedded in your cutover planning and go live preparation.

• Business Projects:

  These are the people and teams within the Business who are tasked with planning and delivering new business projects, in collaboration with colleagues in the Technology Projects organisation

• Business Operations:

  These are the people and teams within the Business who are tasked with using, managing and operating key business systems. They are key players in the entire process, especially when creating an enterprise cutover plan that is necessarily complex and will impact multiple systems. 

To summarise, large-scale deployments necessarily involve a broad range of participants, engaging several business and technology areas across the entire business. External partners and 3rd party suppliers further complicate the organisational complexity. 

In an enterprise context, the benefits of a well-structured organisational model, and clear, reliable processes are self-evident. 

Organisation Design

The above design is typical of any large-scale cutover event and reflects the complex nature of all major transformation programmes. It is a fact of modern life that such changes are organisationally complex, with multiple parties involved, often geographically dispersed across the globe. This is why it is vital to design, build and implement an effective, well understood organisational model at the outset.

Of course, no two companies or cutover scenarios are the same but the challenges faced will be similar. The first step of any cutover is to design the organisation that is required to deliver success. The organisation itself might evolve over time, bringing in different elements as the journey to the cutover event matures. The sections that follow describe the details of each layer. 

Level 1: Executive Steering Group (ESG)

• 1.1 When to use this layer?

  The ESG layer of the deployment organisation model will typically be put in place only for very large scale deployments which have the potential to have a critical business impact. In circumstances where the ESG layer is put in place, the ESG will meet on a regular scheduled basis throughout the deployment period (say, every 4 to 8 hours). Ad-hoc meetings may also be scheduled to occur in proximity to any business-critical milestones associated with the cutover.
  For smaller-scale deployments, with limited business impact, the Deployment Control Group (DCG), Event or Lead Deployment Manager may be entrusted with the same responsibilities.

• 1.2 Who are they?

  These are the senior executives within the company who are accountable to the Board and company shareholders for the success of both the deployment and for the continuous availability of IT and Business Services to customers and staff. Typically, the ESG will be made up of Senior Executives from technology and the relevant areas of the business. Where 3rd party suppliers play a significant role in the cutover, they too may be represented by senior execs on the ESG.

• 1.3 Who do they report to?

  The ESG is where the buck stops – they don’t report to any “higher authority,” perhaps with the exception of the company CEO, Customers and external regulators.

• 1.4 What are they responsible for?

  They are collectively responsible for the business outcome. That’s it. End of story.

Level 2: Deployment Control Group (DCG)

• 2.1 When to use this layer?

  A DCG layer of the organisation will typically be put in place only for large-scale cutovers which have the potential to have significant business impact.
  In circumstances where the DCG layer is put in place, the DCG will meet on a regular scheduled basis throughout the deployment period (say, every 4 hours). Meetings may also be scheduled to occur at business critical milestones associated with the deployment.
  For smaller-scale cutovers, with limited business impact, the Event Manager or Lead Deployment Manager may be entrusted with the same responsibilities.

• 2.2 Who are they?

  These are the individuals who are tasked with overseeing the successful execution of the cutover. Typically, the DCG will be made up of Senior Managers acting as representatives of the 5 key areas of the Event Organisation.

• 2.3 Who do they report to?

  In the typical organisation model, the Deployment Control Group will report to the Executive Steering Group (if it is setup). Failing that, the DCG will report to the various senior managers aligned with the respective organisations.

• 2.4 What are they responsible for?

  They are collectively responsible for the following set of key activities:

1. The primary responsibility of the DCG is to ensure that the business services that are impacted by the cutover are available and fully operational at the end of the agreed service change window. Protection of service availability is paramount in all change scenarios.
2. The DCG provides a forum for the Event Manager to:
   • Publish regular progress reports
   • Discuss any issues encountered
   • Seek authorisation for any deviation from the cutover plan, including any decision to invoke the backout of some or all of the deployment
   • Secure additional resources, if required
   • Provide advice and authorisation to the Event Manager
3. The DCG is ultimately responsible for liaising with the ESG, if present on the organisational model

Level 3A: Event Management 

Event Managers

• 3.1 Who are they?

  These are the individuals who are tasked with the challenge of successfully delivering a complex cutover that has been designed as an Event. An Event typically involves multiple individual cutover plans, all contributing to a single business outcome.
  They will identify, mobilise, lead and manage the large number of Deployment Managers, Event Support and Event Assurance specialists - the broadest definition of the “delivery machine.”
  The Event Manager may be a business or technology professional, staff or consultant, employee or representative of a 3rd party. They will come from all areas of both the Business and Technology organisations.
  They may be specialists who are dedicated to this particular role, or more typically will be Programme Managers or other relatively senior managers who assume this role as the project lifecycle enters the critical Cutover phase of a major programme.

• 3.2 Who do they report to?

  The Event Manager reports to the DCG if this management structure is in place. In the absence of a DCG, the Event Manager will typically report to the appropriate senior execs within the wider organisation e.g. Programme Director, IT Service Delivery Director. In the typical organisation model, one or more Event Managers will be appointed, with only one person in place at any given time (working shifts in a long-running Event).

• 3.3 What are they responsible for?

  They are collectively responsible for the following set of key activities:

  • Provide a point of management and escalation for the Lead Deployment Manager
  • Prioritise allocation of available resources in support of the overall aims of the cutover event
  • Monitor issues and make judgements on whether Incident & Recovery Management processes should be invoked
  • Provide key facts and relevant metrics as input to the DCG status reporting function
  • Attend the ESG status updates as required (i.e. by invitation only)

Event Support
In larger cutovers, the core team of Deployment Managers will be fully focused on the orchestration of the tasks within the cutover plan. The following support services can be devolved to a dedicated Event Support function, alleviating this workload demand from the Deployment Managers and ensuring that the cutover is not delayed due to the overhead involved in these key support functions. The list below describes each support function in a little more detail:

Planning Support
There are a myriad of planning tools used today in an effort to maintain a current, complete and accurate view of the status of the cutover. Cutover scenarios are, by their nature, extremely dynamic; Participating teams need a responsive, bespoke tool to provide an accurate reflection of the latest facts and status at all times. In larger events, planning support individuals (or even teams) should be mobilised to operate the selected tools to provide the following accurate information:

• Status of all tasks across all active runbooks
• Forecasting of the Cutover end date and time
• Forecasting of the Backout end date & time
• Forecasting of resource demands (people, equipment etc)
• Forecasting of the impact of all issues that have arisen during the cutover

Note: ICEFLO is a Gartner-recognised solution designed specifically for this purpose

Logistics Support
In a large cutover event, there will be a significant number of people, teams and indeed organisations involved. These human resources will be geographically dispersed and will require a full range of facilities, on a 24/7 basis, including the following:

• Office premises with appropriate desktop services
• Access to buildings
• Transportation and accommodation
• Food and beverages

The Event Support team can include dedicated staff to provide the necessary logistics support. By dedicating staff to this critical set of activities, Deployment Managers can focus all of their energies on the Cutover itself, safe in the knowledge that the Event Support team are taking care of the most important resource involved in any major change – your people.

Resource Management
People are indeed the critical success factor in any Deployment. They are also one of the most expensive resources that will be deployed on any Event. The challenge is to have the right people, with the right skills and access, in the right quantity, available at the right time. On any complex cutover, even with a “static” schedule, managing geographically dispersed teams of people  is a complex challenge. When you recognise that the schedule is by its very nature dynamic, and perhaps even volatile, the need for expert resource management support becomes rather obvious.

The Resource Management support team will typically work 24/7 and will perform the following activities:

• Review and update reports on the baseline resource requirements over the course of the event – based on the “static” version of the plan
• Regularly update the resource demand profile, taking into account the dynamic changes to the schedule.
• Provide advance notification to individuals and teams of when they are required
• Highlight any resourcing constraints or gaps to the Event Management team

With these services being provided by a dedicated team, the Deployment Managers can be reasonably assured that manpower will be available when required. This removes or at least mitigates one very common cause of cutover delays i.e. tasks available but no-one available with the required skills to perform the work. It also enables the mature organisation to contemplate resource optimisation – saving considerable costs in the process.

Communications Support
On any large-scale cutover, communication is a critical success factor. The potential audience will be large in number and diverse in terms of their role and associated information needs. A dedicated Communications Support team is an essential element of any successful cutover organisation. The following activities will be performed by the Communications Support team:

• Comply with all communications policy and governance requirements of the organisation
• Compose, authorise and distribute Internal communications to staff, appropriate 3rd parties and executives
• Compose, authorise and distribute External communications where appropriate for markets, customers and partners

Reporting Support

A dedicated Reporting Support team is an essential element. Different reports for different audiences. Information gathering and processing is a time consuming activity. Deployment Managers can be relieved from this burden by having Reporting Support services. The following activities will be performed by the Reporting Support team:

• Comply with all reporting standards, policy and governance requirements of the organisation
• Gather, collate, analyse and aggregate the array of data being generated as part of the cutover
• Publish and distribute detailed cutover status reports at regular time-based intervals
• Construct exception reports in response to Issues / Incidents encountered

Level 3B: Event Assurance

Event Assurance
Event Assurance is an often overlooked but invaluable function in the overall organisation of any major Deployment. The function is defined and mobilised on the mature presumption that things can and will go wrong. While this might seem a slightly negative approach to make, experience tells us that there is no such thing as a faultless cutover. What characterises the successful cutover is how effectively the organisation handles issues or incidents when they inevitably arise.

Technical Triage
Issues will arise during any cutover, regardless of complexity or scale. One key factor that will determine the success of the cutover is how the issues that do arise are managed - ideally in an effective, timely manner.
The primary goal of the triage team is to make expert diagnosis of issues, accurate impact assessments and to support the timely resolution of issues.

The Technical Triage team will:

• Be staffed by representatives of all organisation units, bringing a breadth and depth of knowledge to this critical team
• Include an array of Subject Matter Experts, such as Architects, Technical Team leaders,Technology Vendor specialists, Business Operations specialists
• Constantly monitor open issues, identifying those that have been taking too long to be addressed
• Support and advise the individual issue owner when he/she has exhausted their diagnostic and remediation options
• Constantly review issues as they arise, perform impact assessments and intervene where appropriate
• Immediately escalate any critical issues to the Event Management team

End User Access
One of the most common scenarios that arises during these Events is where a key individual has insufficient access permissions to one Production technology or another. Project team members typically work away for months on non-Production systems. They then attempt to participate in the live cut-over, only to discover that they don’t have the relevant and necessary access permissions to perform the task at hand.

Standard, “Business-as-Usual” access requests typically have a fairly lengthy turn-around period and it is essential therefore to make specific, special case provisions for addressing the end user access “gotcha.”
This typically involves the following:

• Define an exceptions process for granting temporary, elevated access permissions to any authorised users
• Define and implement an audit trail of any and all such access permissions granted
• Mobilise a team of Subject Matter Experts to immediately execute any submitted access requests, in compliance with agreed process, policy and audit requirements
• Ensure the removal of temporary access permissions at the earliest, appropriate point in time

Incident and Recovery Management
Mature technology organisations will always have existing processes, tools and staff to handle standard Production Incidents. An Incident in this context is defined as any event or situation that is deemed to threaten the availability of a Production Business Service. Cutovers will inevitably encounter Issues - something that has unexpectedly happened and needs to be resolved by the cutover team in order to make progress. These Issues should typically be handled internally by the deployment organisation.

However, a cutover that is going significantly “off-plan” can obviously generate a condition that threatens the availability of a Business Service - and moves into the domain of Incident Management. It is vital that an effective organisational and process interface is built to connect the project cutover team to these BAU, Service Delivery processes, tools and people:

• Define and agree ownership of cutover Issues based on impact and severity
• Define clear criteria for the transition from cutover Issue to Production Incident
• Define clear ownership of the Incident once that transition has occurred
• Define clear ownership of the Recovery process once that transition has occurred

By integrating the Incident Management and Recovery Management functions into the Event Assurance function, the mature organisation effectively asserts an appropriate “Service Protection” influence and indeed control point over the project-led cutover.

Level 4: Deployment Managers

• 4.1 Who are they?

  These are the individuals who are tasked with the challenge of building and running the various runbooks. The will identify, mobilise, lead and manage the large number of subject matter experts as “task owners” who represent the “delivery machine.”
  The Deployment Manager may be a business or technology professional, staff or consultant, employee or representative of a 3rd party. They will come from all areas of both the Business and Technology organisations.
  They may be specialists who are dedicated to this particular role, or more typically will be project managers or other relatively senior professionals who assume this role as the project lifecycle enters the critical “cutover” phase.

• 4.2 Who do they report to?

  In the typical organisation model, one or more Deployment Managers will be appointed. It is the responsibility of the Deployment Manager(s) to identify, engage and successfully utilise the knowledge, expertise and manpower capacity that is required to successfully deliver business and technology transformations.
  In situations where the scope and complexity of the cutover requires multiple Deployment Managers, it is critical that one individual be assigned the role of Lead Deployment Manager. This approach simplifies command and control, communication, decision making and status reporting.

• 4.3 What are they responsible for?

  They are collectively responsible for the following set of key activities:

  • Advise on the overall strategy and approach for the cutover plan, using their specialist planning and orchestration expertise and experience
  • Assemble suitably skilled and experienced respective teams, each of which will contain multiple subject matter experts
  • Lead the construction phase of the cutover plan, comprised of several runbooks, scheduling appropriate manpower to perform the detailed planning including task detail and dependency definition
  • Plan and lead detailed plan and runbook walk-throughs, ensuring active participation by all required subject matter experts
  • Ensure that Change Governance is in place and is based upon a formally approved cutover plan.
  • Lead the actual cutover itself, managing plans, tasks, resources and issues as they arise

Level 5: Technology and Business Task Owners

• 5.1 Who are they?

  These are the individuals, teams or 3rd party providers who represent the “delivery machine”, without which no technology or business transformation would ever take place. They are the “do-ers” who implement tasks.
  They may be Business or Technology professionals, staff or Consultants, employees or 3rd parties. They will come from all areas of both the Business and Technology organisations, assembled as one cohesive but transient cutover organisation with one common aim.

• 5.2 Who do they report to?

  The Technology and Business Tasks owners will report to appointed Deployment Managers for all aspects relating to the deployment itself. Naturally, where the deployment is not a full-time, permanent role, these individuals will have other reporting lines that are typically “matrix” or “line” managed.

• 5.3 What are they responsible for?

  They are collectively responsible for the following set of key activities:

  1. Advise on the overall strategy and approach for the cutover - bringing their subject matter expertise to the task at hand.
  2. Assume ownership of a defined subset of the tasks within the overall cutover plan.
  3. Define each technical or business activity or task that has to be performed, including details such as:
     • The plain language title of the task.
     • The full technical or business detail of the task.
     • The estimated duration of the task.
     • The communication, if any, that should be sent upon completion of the task.
     • The definition of the relationship between individual tasks and other tasks within the given runbook i.e. internal dependencies.
     • The definition of the relationship between individual tasks and other tasks outside of the given runbook i.e. external dependencies.
  4. Perform a risk assessment at both a task and runbook level, assuring the runbook content and assessing the risk to operational services.
  5. Actively participate in the following key activities:
     • Runbook walkthroughs, reviewing the content, sequencing and dependencies with managers, SMEs and other peers.
     • Cutover rehearsals, verifying and enhancing the runbook on each iteration.
     • The “cutover" phase, commonly known as the “go-live”.
     • The post-deployment phase, including any lessons learned activity and associated problem management.
     • Provide key facts and relevant metrics as input to the status reporting function.

  
  Conclusion

  Regardless of the business and technology context, the success of any large-scale technology deployment will depend to a large degree on the appropriate and effective organisation being put in place. This blog describes a sample organisational model that has been designed based on years of experience, evolving over time to address the range of challenges, roles and processes that are required for success.

  
  I hope that you’ll find the overview helpful when designing your own organisational model to deliver your own technology cutover. I’d love to learn more about what you think of this blog and any ideas you might have for improving on this design. Feel free to re-use any of this content when designing your cutover organisation.

  Related Blogs

  10 top tips to achieve Technology Cutover Success

  ICEFLO: SatNav for Technology Cutovers