Understanding Compliance Platform Capabilities: Black Box Automation Has its Limitations

Compliance automation is maturing, yet many companies are unaware of the limits. Learn what factors to consider when evaluating vendors and why institutional knowledge is key.

Compliance is hard. It is not a “black box” of opaque inputs and outputs, where systems and data are hidden and where users are oblivious to their inner workings. There has yet to be a product made that can magically produce all the evidence sufficient for testing and verification across the wide variety of today’s frameworks. On top of the raw number of controls that enterprise organizations have in place in 2022, there are also too many governance and HR issues that automation can’t and may never solve.  

We’re still in the netherworld between maturing automation and highly manual methods – see our last blog about how a majority of Coalfire’s clients are surprisingly still using spreadsheets to manage their compliance programs. Despite enterprise size or sophistication, there remains a lot of confusion and misunderstanding around the capabilities of automation solutions. 

So, how should we understand the capabilities and limitations of compliance platforms, and how should we define the most mission-critical factors for executive decision makers and the growing cadre of corporate stakeholders? While not an exhaustive list, here are some things you should consider when evaluating vendors: 

• Enterprise compliance management –Does your vendor have capabilities for multiple environments and frameworks? Can they help set the dials from the start and maintain oversight diligence based on human experience and institutional knowledge?
• Technology stack –Many automation vendors can automatically collect and test information at the cloud service level but are unable to test things at the O/S or application level. Determine what components of your scope can be automated and factor that into controls that need be tested against different technology components.
• Baseline automated tests – While any automation test may be sufficient to meet a compliance standard, it does not always translate to the proper security baseline. Do the tests performed via automation meet your security standards? If not, additional monitoring and testing should be applied to that control.
• Mappings – How does your vendor do their mapping? Do they use an open source tool? If so, be cautious as these are often not peer-reviewed, updated, and subject to auditor bias. Is it performed in-house? If so, who are their experts? How much experience do they have in the frameworks they are mapping?
• Over-promise? – If the percentage of the framework being automated seems too high, or if the price seems too low, or if anything sounds too good to be true, then it probably is. Beware of platforms that give you a green checkbox on controls where you provide documents with little substance.
• Audit partners – Evaluate the quality of audit partners. As you leverage compliance frameworks to unlock new business opportunities some potential customers will critique the quality of your third-party assessors.

In summary, compliance cannot be hacked through automation software. You still have accountability for the governance of your security and compliance program. Implementing an automation tool does not replace that. We believe in automated solutions, but compliance cannot be commoditized, not now and maybe ever. We believe in education, and sharing experiences about what works and what doesn’t. 

It takes the institutional knowledge of hundreds of assessors with thousands of client engagements under their belts to bring that level of experience to multi-framework environments and efficiently coordinate between business units and audits in real time. We believe that today’s state-of-the-art best practices bundle into a combination of SaaS technology, expert audit counsel, and oversight.  

Sometimes a new technology will gain acceptance over many years, with incremental productivity gains eventually leading to full adoption. That was the case with GRC tools until recently. Now, the gains with Compliance Automation are coming so fast, the market can barely keep up. This is a good thing – just beware of vaporware, and start your journey now before the compliance hackers get further ahead of the progress that we’re making. 

To learn more about how the Coalfire compliance team can help support your program with our new Compliance Essentials solution, please visit, our Compliance Essentials page.