I know that artificial intelligence (AI) is all the rage today. It is the evolution of maturing computerized systems. However I thought it appropriate to look at how moving forward we should create, evaluate, and utilize these systems when they are used to operate, monitor, and control critical infrastructure systems.
I define critical infrastructure as electric grid, water distribution systems, land/sea/air transportation systems, and other control systems that serve large populations of society.
One must always remember that AI systems are designed, developed, tested, and installed by fallible humans. Failures are possible due to human involvement in design, testing, initial deployment, and in fundamental operation.
Here are some thoughts that any organization planning on deploying AI within critical infrastructure should be considering:
- When an AI controlled system fails how does your organization stop or reduce AI control of the system, assess failure, and provide a means to rapidly transfer control of the system back to human oversight and operation? Do you even know if this transfer of control is possible in the AI system that you are using? Do you know and understand what automatic failsafes and failovers are in place to protect the AI controlled system?
- Has the above situation and the various failure scenarios been tested and exercised? As a minimum this needs to be carried out after initial implementation and before each upgrade and patch deployment.
- Is there a resiliency plan in place for a situation where failed AI cannot be readily restored?
- How will your organization continuously train and prepare the people who will take over a variety of complex tasks in the event of an AI failure?
I am a proponent of the use of AI systems, but only if they are properly designed, developed, installed, tested, and thoroughly secured from breaches before their introduction.
Here are two scenarios to illustrate why the above questions need considering:
Case one:
Autonomous merchant ships already exist as a concept and are likely to become commonplace. Consider the situation where there is a failure of the AI-controlled navigation systems and a subsequent shutdown of the ship’s propulsion systems. You now have a ship dead in the water, with possible heavy seas, and a very limited human crew, if any. How would this situation be handled? This is a question that should be asked before turning entire control over to AI. Have we thought out all worse case scenarios and if a partial or total AI failure occurs do we have adequate plans to address these? Failure to resume control could be disastrous to the environment and could threaten loss of life.
Case two:
An AI system controls a manufacturing operation that extensively utilizes robotics in the shipping and packaging process. If AI controls the entirety of the process and it fails, do you have protocols in place to maintain operations without AI involvement? Can you circumvent the failed AI? Are your people trained and ready (with process and procedures in place) to assume control of systems? If not, there could be a very significant loss to the company’s bottom line.
In summary
AI systems are the natural progression of computing technology. However, those employing this technology have a responsibility to also plan for its inevitable failure. To plan for it. If not, I fear disastrous consequences could be in the offing.
The author
Dr. Jim Kennedy is an internationally known information security and business continuity lecturer, consultant, and author. Kennedy is best known as an individual that can provide senior management with a clear picture of the current state of their information security or business continuity management programs and provide a strategic vision for improvement based on risk.
