Assessing the Real Power of Automated Controls

It is time to automate compliance and this means understanding how automated validation can work effectively to provide the continuous assurance we need for frameworks like FedRAMP, PCI, HITRUST and many other frameworks that require technical validation to reduce the burden of the assessment and leverage other operational security tools to demonstrate compliance. We're talking about moving beyond point-in-time checks to continuous assurance and proactive risk management while increasing assurance with broader visibility and less manual validation.

The buzz in compliance around automation grows daily with a lot of excitement over the promise that this offers to drive towards higher levels of assurance.  It absolutely has the potential to lead to the outcomes everyone is hoping and it is important that we examine the idea of reliance on the automation.  One main question I get after many years in the assessment space, as Coalfire discusses our own platform, Compliance Essentials and the automation capabilities is “What is required to rely on automated testing?”  With everyone so excited about the promise and so much focus on the technology and tooling, I wanted to take a few minutes to ensure this question was addressed.  Here are the key questions that need to be answered and built into any compliance automation solution for them to be provide the reliance for continuous assurance.

1. Completeness: Did the Automation Touch the Entire Population?

The first critical question is scope. Automated controls are designed to validate configurations across a specific population of systems or users. But how do we know it actually did?

• What population did it pull from to validate the configuration? Reliance comes with visibility into the data set being analyzed by the automation. What assets, cloud environments, systems, applications, and so on does the automation cover.  Additionally, if it is a modern cloud deployment, configurations are deployed from code and testing needs to look at the code as well to understand the components that are deployed from code and associated configurations.
• Is it sufficient for reliance? Just because the automation ran doesn't mean it covered enough ground. We need to determine if the sampled population meets the standards for validation, at a minimum, based on the automation targeting all relevant assets. This requires understanding the underlying architecture and potential blind spots in our data collection. However, in most cases, automation targets a much broader population, if not the full population, adding to the reliability and assurance that automation can provide.

Think about it: Current manual audits try to sample to cover all configurations while cloud environments can have hundreds, if not thousands, of systems and many are dynamically created.  With automation, validating secure configurations of what is running and how these systems are built from code can provide much higher levels of assurance by looking at the code and dynamic systems to provide much higher levels of reliability and assurance.

2. Accuracy: Are the Rules Correct and Meeting Requirements?

Automation is only as good as the rules it follows. If the logic is flawed or doesn't align with the compliance requirements, the results are meaningless.

• What are the rules that were tested along with parameters? Providing full transparency into the specific rules being applied is critical for reliance on the automation. This includes not just the rule name, but also the underlying logic and the parameters used. For example, if we're checking password complexity, what are the specific length, character type, and history parameters being validated?
• Do they meet the compliance requirements? This is crucial any frameworks specific context. Mapping automated rules directly to the relevant controls to ensure they are effectively testing compliance. This requires a clear understanding of the compliance requirements and how they translate into actionable technical checks.

Consider this: An automated rule might flag any password shorter than 8 characters. But if the compliance framework requires a minimum of 12 characters, this rule is insufficient for the specific compliance needs providing false assurance of the control status.

3. Timeliness: Is the Data Fresh Enough for Decision-Making?

Security and compliance are not static. Configurations change, new vulnerabilities emerge, and user behavior evolves. Automated controls need to reflect this dynamic landscape.

• What is the frequency and/or last time the data was pulled? Offering transparency regarding the frequency the automation runs and when the underlying data was last collected is important for reliance. Is it a daily, weekly, or ad-hoc process?
• Is it within a reasonable timeframe? The definition of "reasonable" depends on the specific framework, control and the risk associated with it. For critical security configurations, daily checks might be necessary. For less volatile settings, weekly or even monthly checks might suffice. We need to define appropriate frequencies based on risk and compliance requirements.

Imagine this: Automated controls are usually touted as providing real-time visibility and provide continuous assurance of control status, however, resources, even compute resources, cost money.  Ensuring an automation solution is optimized to run at the right time with the right scope and assessing the right rules to provide ongoing assurance of control effectiveness make the automation approach effective and sustainable.

4. Exceptions: What Happens When Things Go Wrong?

Even the most robust automation will occasionally identify exceptions – deviations from the expected configurations. How these exceptions are handled is a critical part of the control's effectiveness.

• What is the exception process if an issue is identified? We need a clearly defined workflow for handling identified exceptions. Who is notified? How are they examined? What is done to correct the exception?
• How is that documented and dispositioned? Traceability is key. Every exception needs to be documented, including the details of the issue, exception findings, the action take to correct, and the final disposition. This provides an audit trail and demonstrates the organization’s commitment to addressing exceptions.

Think about this: An automated control flags a critical security misconfiguration. If there's no clear process for notification and remediation, the issue might go unaddressed or no trackable method to understand how this issue was managed. Conversely, a repeatable approach, even automated, to address and document any corrective steps to exceptions builds trust and reliance that security issues are managed appropriately.

Moving Forward: Embracing the Power of Automation

The vision of leveraging automation through machine-readable evidence collection and automated rules for evidence validation is incredibly empowering and promising. It offers the potential for greater efficiency, enhanced accuracy, and continuous assurance. However, to realize these benefits, building automation for assurance and aligning with an organization's risk tolerance requires important elements to demonstrate the effectiveness of the automation to continue to build trust.

By focusing on completeness, accuracy, timeliness, and exception handling, we can ensure that automated validations are not just running but are truly providing the assurance we need to maintain a secure and compliant environment. Let's embrace this evolution and build a more robust and efficient future for control assessments.