As I’ve been developing the Red Eye Monitor (REM) system, I have spent a lot of time thinking about how to think about automation. How to talk about it, how to explain it, how to do it; all of it is hard because there are more pieces to system automation than can be kept in our brain’s symbolic lookup table at one time. I believe most people will top out at 3-4 separate competing symbols and there are somewhere between 10 and 20 major areas of concern in system automation, making it very difficult to keep things in mind enough to come up with a big picture that remains comprehensive as the detail magnification levels go up and down.
I had initially phrased REM as a “Total Systems Automation” tool, because it was meant to be comprehensive, and comprehensive automation is a fairly new idea, even though many people have been working on it for a long time, it just hasn’t caught on yet, really.
Having attempted to explain what a comprehensive automation system consists of, and what benefits it has, I have now settled on a new, shinier and better name: Authoritative System Automation (ASA).
Why Authoritative System Automation?
During my many discussions with people about a comprehensive automation system, I have settled on the single-most unique and distinguishing element between this kind system and other kinds of automation, is that there is only ONE concept of what the system should look like: The authoritative information that defines what the ideal system should be.
All benefits of an ASA system comes from the fact that with only ONE blue print for the ideal system, all actual details (monitored and collected from reality) are compared to the ideal blue print, and then actions are taken to remedy any differences. This could mean deploying new code or configuration data for a now-out-of-date system after a deployment request, restarting a service that is performing below service level agreements (SLA), provisioning a new machine to decommission a machine that fails critical tests, or paging a human to fill the loop on a problem that automation is not capable or prepared to solve.
Authoritative is a super set of comprehensive, because all aspects of the system come from the authoritative data source. Authoritative data sources can only have 1 master. Systems could be split so that there are several systems that run independently, but to be an authoritative system, each of them has to respond only to the requirements as detailed by the authoritative data source that backs the authoritative system automation.
Authoritative System Automation is vastly different from typical automation practiced today, even if that automation is “comprehensive”. Any system where data resides in more than one place cannot be an authoritative system, and will need to be updated in multiple places for maintenance. Upgrades will need to track multiple places, and mismatches can occur where data is out of date in one system compared to another.
All of this reduces confidence in automation, and means that the system cannot truly be automated. It can only be mostly-automated, which is vastly different.
An Authoritative Automated System will never require or allow an independent change, unless the piece being changed has been removed or frozen from automation, the ASA will revert the change or otherwise take corrective actions, and so the independent change is essentially an attack against the ASA. A properly configured and complete ASA will correct the change, and the system will continue.
Another aspect that is a major difference between an ASA and a non-ASA automated system is that an ASA system will have EVERY piece of data stored in the ASA’s authoritative data source. No configuration data will be stored in a text file, if it is configuration data, it is in the ASA data source and then mixes with templates to become a working configuration file or data when deployed to a system.
This is a major change from most automation attempts, as they merge automated scripts, databases with information for the scripts to operate on, and configuration text and other data sources together, to create a fabric of automation.
An ASA cannot have a single piece of configuration data outside of the ASA data source, or it is no longer authoritative. If the non-ASA attempted to reconfigure something, a side effect can crop up between the new “authoritative” data update and some other source of automation or configuration data.
An ASA system must be built 100% from it’s authoritative data, and must be maintained 100% from it’s authoritative data. Any deviation, manual configuration or data embedded anywhere, breaches the goal of an ASA and creates a non-ASA.
I see Authoritative System Automation as the highest form of automation, and the only comprehensive option, as it is completely data based, and logic serves only to push the data out to reality, and to collect from reality to compare against the ideal. The benefits of an ASA system is confidence in the system doing the right thing, and is testable and verifiable because at every stage in the automation, both the intent and expected result are known and are the same.