Why do business continuity plans fall down at failback?

By Thomas Berndorfer

‘Failback’ has become something of an dreaded issue in much of the IT community. In a disaster scenario that takes out a company’s cloud and/or IT system, getting backups or secondary cloud environments online and functional in time to meet recovery point and recovery time objectives can be hard enough, but sometimes only represents half the challenge.

The failback process of getting those users back to the primary environment can become a maddeningly difficult process depending on the size of the system and the time the team has spent using it.

Working from a secondary environment for some time can lead to the creation of new data that is difficult or impossible to reconcile manually, inevitably leading to data drift, conflict resolution problems, and duplication – all of which can lead to key data being lost.

Many business continuity planning scenarios don’t adequately prepare for this situation, often leaving IT teams to manually reconcile old and new data. Poor failback planning can lead to significant time and efficiency losses in post-disaster recovery scenarios. In our experience working with large public sector organizations, this is becoming a key factor in how they think about resilience – not just how they respond in the immediate aftermath of a disaster, but how quickly they can return to normal operations.

Why failback is complex

When an entire company moves its operations to a secondary or backup environment to preserve continuity, new data discrepancies with the original system immediately begin to emerge. New emails are written, files are edited, and contacts are changed – sometimes creating two different versions of the same data in different environments. Even in limited backups that aim to preserve minimal functionality, there is still new incoming data that needs to be processed.

This backlog creates several major problems, for instance:

Conflict resolution: there may be conflicting edits of documents across different systems, making it difficult to see which should win out.
Permissions: files created in a secondary environment may be sensitive, with complex permission structures to prevent them from being seen by team members without authorisation.
Duplication: because employees might not have all available data (depending on the solution), teams can inadvertently create duplicates of old information or spend valuable time searching for data that only exists on the primary system. Small discrepancies in metadata tags (like the name of the contact) can lead to duplication that can be very difficult to spot during the failback process.
Critical data loss: data or information might be created on the secondary system during a disaster that is vital going forward. If there are problems with failback, this information could be lost permanently or, at least, valuable time could be spent looking for it.

Some backup solutions are ill-equipped to deal with this scenario, generally being set up to synchronise one-way (from primary to secondary), with very limited means to synchronise both ways without significant data conflicts and potential losses. This can lead to scenarios where the reconciliation process overwrites or fails to transfer critical data, which can add significant friction to post-disaster recovery. When failback is poorly planned, even though the initial problem might be gone, the disruption remains.

To run through a hypothetical example, a logistics company suffers an IT outage that forces it to run its routing through a secondary system. Thankfully, it is able to continue operations during this outage – avoiding breaching its service-level obligations in the process.

However, a poorly designed failback system is unable to resolve the conflict between several vehicle codes with similar names, leaving operators with no idea about the status of a customer’s deliveries. Without a sure way of resolving data conflicts in a transition between a secondary and primary environment, similar problems are always likely to come up during failback scenarios.

What works

In order to resolve this problem, some IT teams set a strict cutover date on which the company completely stops using the secondary environment, so that the IT team can manually resolve and consolidate data conflicts. This process can work but introduces the possibility of errors and requires significant investment of time by the IT team.

The problem of failback illustrates why bi-directional synchronisation is so critical in BCP backup environments. These solutions work by creating an automated active bridge between two environments (cloud to cloud or cloud to on-premises) during normal operations that actively synchronises data and permission structures bi-directionally – changes in either environment are reflected in the other.

Because the system is bi-directional, it can be configured to automatically resolve data conflicts between the secondary and primary systems, dramatically reducing the chances of data loss through predictable rules even when a company has been using the secondary environment for an extended amount of time. This is vital in disaster scenarios, given that it can dramatically reduce failback time by reducing the need for manual management of data transfers.

Conclusion

Because cloud services are central to so many organizations’ data management strategies, businesses urgently need to ask not only how they will respond to a cloud outage, but also how they will recover afterwards. The most successful companies will be the ones that can streamline the failback process to the greatest extent.

The important questions for business leaders to ask include:

How does the organization’s business continuity strategy address failback?
Is the organization ready to transition back quickly and effectively after outages?
How can the organization manage the privacy and permissions of critical data?
How can the organization minimise data loss during post-disaster transitions?

Having well-thought-out answers to these questions will strengthen the organization’s disaster preparedness, business continuity capability, and overall resilience.