merkely loop with error signs and documents
merkely loop with error signs and documents

5 reasons why your CI system is a terrible Compliance System of Record

“Why can’t we use our CI system for our Compliance System of Record (CSoR)?”

This is a question we get asked a lot when we’re talking about compliance with regulated DevOps teams. And it’s a perfectly reasonable question to ask.

If Jenkins, GitLab, GitHub, or CircleCI is the engine for your DevOps it will contain a lot of information relevant to maintaining a CSoR. However, your CI system shouldn’t form the basis for your CSoR and in this article we’ll give you 5 reasons why.

If you want to know what a successful CSoR looks like you can check out our previous article on the topic. For now, let’s consider why it shouldn’t be stored in your CI system.

Compliance System of Records diagram Carl Nygard Merkely

1. CI systems have lots of mutable data

The first objection to using a CI system concerns the single biggest challenge a CSoR should address: preventing modification and deletion of information. A compliance audit trail must stand up to scrutiny. Auditors need to know if process evidence can be modified by internal or external bad actors.

The problem with Continuous Integration systems is that they tend to have lots of mutable information. For example, it’s often trivial to delete stuff like pipeline runs and repositories. Data and evidence can also be lost in less obvious ways if pipeline reruns overwrite previous information, or if log retention policies delete old data. A robust CSoR doesn’t modify compliance data.

2. CI systems are hard to secure

A frequent counter argument to the mutable data objection is that you can apply security controls. However, “We could just lock down the CI server” isn’t the winning argument you might think.

Having a highly rigid security posture around your CI system makes it harder for developers to get work done because it imposes heavy processes around how CI changes. Things that used to be easy - like creating a new microservice or renaming a repository - now involve paperwork and risk theater. This leads to a lot of disillusionment in the tech teams and establishes security as an enemy.

But, beyond this cultural anti-pattern, this is still a poor compliance solution. By turning a compliance problem into a security problem you’re getting weak compliance and rigid pipelines - the worst of both worlds.

3. CI systems are not databases

The third reason why your CI system should not be your CSoR relates to use cases beyond audit trails. Compliance data is used in many contexts. It can be used in reports to regulators, in security monitoring systems, and in further operational automation such as deployment controls.

CI systems don’t have a structured data layout and query language in which to use this information. It is also difficult to add your own specific data into the historical record without abusing these systems.

4. CI systems provide poor change observability

CI systems are source-code centric with user interfaces often scoped at the git repository level. This makes sense for a developer working on software, but the narrow focus makes it hard to reason about change in large distributed systems. Could your CI system give you good answers to the following questions?

  • Which versions of different microservices are running in production?
  • What are the recent changes to production?
  • What was in production last Wednesday evening?
  • What deployments are missing approvals?

Beyond this, most CI systems don’t typically use reliable software identification methods like Content Addressable storage, so it’s hard to correlate the CI events with actual running software.

5. CI-based CSoR’s == DevOps lock-in

Finally, using a CI system as a System of Record relates to overall technology strategy for your development teams. Forcing everyone into a locked CI ecosystem hurts your ability to adopt better and more distributed tooling.

If teams want to use different CI systems, or would naturally benefit from doing so, basing your audit trail on distributed and incompatible systems can make audits time consuming and hard to navigate.

Tl;dr - Don’t do it!

It’s tempting, and seemingly logical, to base your CSoR in your CI system. But, as you can see, there are very good reasons to avoid it.

For one thing, there’s absolutely no guarantee that an auditor will be satisfied with a mutable CI system as the record of compliance proof. And if that wasn’t disqualifying enough, you risk undermining developer productivity with a rigid system that offers poor observability and locks in your tools. It’s a lose/lose/lose situation.

Succeeding with DevOps requires sharp tools for specific jobs - you always want to have round pegs in round holes. Building a successful CSoR in a DevOps way is no different. It shouldn’t come as a surprise that using your CI system for this task is full of inefficiency and risk.

why your ci system is a terrible compliance system of records list

Top Articles

How to do Continuous Delivery in your ITIL framework?

Continuous Compliance: A DevOps culture

We’re going to DevOpsDays Denmark 🎉

Published February 9, 2022 in
Mike Long
Mike Long

Subscribe to The Merkely Meteor Newsletter for all the latest on regulated DevOps, features, news and events

Merkely is committed to protecting and respecting your privacy. You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please visit our Privacy Policy.
Subscribe to the Merkely Meteor

More posts in technology

How to do Continuous Delivery in your ITIL framework?

Key takeaways ITIL change management framework doesn’t work for frequent software releases That’s a problem because regulated teams want continuous delivery By automating change management they can release software as standard changes Introduction There are quite a lot of articles out there on this topic, usually trying to score for SEO, that promise to show you how to do this without actually ever getting there. But it can be done and in this piece I will explain how.

Continuous Compliance: A DevOps culture

Imagine your developers are the world’s fastest relay team 🏃 When it comes to build, test, and qualify they get round the running track faster than anyone else. Unfortunately for them the finishing line is hidden somewhere outside the stadium. Welcome to regulated DevOps How did they get to be running this impossible race? Well, better tools and working practices have meant a dramatic shift from annual software releases to a world where teams have the ability to deploy multiple times every day.

Why ITIL Change Management doesn’t work for DevOps teams

Are you trying to do DevOps under regulation? If so, you’ll know the pain of change management. In this article we’ll look at how delivering software with DevOps is incompatible with old school ways of managing change with ITIL and how you can automate your change management process with a DevOps approach. As regulated industries speed up their DevOps processes they find that managing software releases with ITIL tickets and change meetings just doesn’t scale.

Subscribe to the Merkely Meteor newsletter for all the latest on regulated DevOps, how to’s, features, news and events

Merkely is committed to protecting and respecting your privacy. You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please visit our Privacy Policy.
Subscribe to the Merkely Meteor