Reduce compliance TCO by using Grafana Loki for non-SIEM logs

Compliance is a term commonly associated with heavily regulated industries such as finance, healthcare, and telecommunication. But in reality, it touches nearly every business today as governments and other regulatory agencies seek to enact tighter controls over the use of our collective digital footprint.

As a result, more and more companies need to retain a record of every single digital transaction under their control. And since these logs are often retained for years, they can translate to a massive expense for the business.

In this blog post, we’ll discuss the current solutions companies use to address this compliance challenge and where those tools fall short for companies that want to be smarter about their logs. Then we’ll show you why you should make Grafana Loki, a log aggregation system designed to be cost-effective and easy to use, a key piece of your compliance strategy.

The state of the current solutions

Companies need to keep compliance records for all sorts of reasons. For example, banks need to monitor and log every transaction for auditing purposes, while hospitals need to retain patient information for years to comply with privacy regulations.

To address these compliance use cases, many organizations utilize traditional logging platforms designed for security incidents and events management (SIEM). However, this approach carries a massive cost as the data being logged and indexed keeps growing and impacts your total cost of ownership (TCO) in unexpected ways. 

For example, SIEM platforms come with staffing requirements. You need a high number of engineers just to maintain their logging platform since these SIEM vendors tend to index every piece of log that comes through, line by line. And you also need to maintain an ever growing infrastructure footprint: To ensure the users get results returned fast (less than a minute) you need to keep adding fast read disks, which are usually the most expensive type of hard disks. Even transitioning to cloud comes with additional costs as the cloud vendors have higher costs for this sort of hardware.

And then there’s the elephant in the room: the software vendor’s licensing costs. That alone can make any CFO cringe when it comes time to sign any renewals. This is a massive challenge as these costs keep growing every year, and even more so now with high inflation across the globe impacting everyone’s bottom line. 

How companies are diversifying log management

As a result of the TCO challenges associated with SIEM platforms, we’re seeing a change in the way organizations prioritize their data. Instead of using a single platform to handle all the data, organizations are selectively picking multiple platforms, with each one addressing specific tasks.

For example, an organization might couple key security infrastructure logs with external threat intelligence data, identity providers, and key application security logs — all of which is then forwarded to their SIEM solution. 

At the same time, logs from non-critical cyber infrastructure are instead routed to alternate platforms like Loki. The key difference here is that Loki is not a full-text indexing engine. It only indexes the key fields from every log line. And for compliance use cases, these key fields are enough for the user to access compliance logs in a fast, efficient manner.

This is the smarter way to manage costs without losing the ability to prevent security incidents and obtain security intelligence on normal user behavior that picks up on any anomalies. And it also helps in managing licensing costs by prioritizing data based on actual use cases. 

This approach also helps companies make better cybersecurity investments amid growing geopolitical tensions. When organizations are smarter about how they process and store compliance data, more resources can be designated for additional network perimeter infrastructure logs that can be analyzed by the SIEM. This helps reduce the risk of external hackers damaging any organization’s reputation and also protects their customer data, which is even more important considering the damage done by high profile hacks.

Organizations that selectively filter their logs can utilize a lower-cost indexing platform for non-SIEM use cases. And since organizations are increasingly using Loki for their dev, test, and production environments already, this is where it can help reduce the overall TCO for compliance. This is especially true when it comes to handling large amounts of data consisting of digital transactions that are only required for auditing purposes.

Why Grafana Loki for compliance?

Grafana Loki is a log management software that does not index the text of the logs. Instead, entries are grouped into streams that are indexed with Prometheus styled labels.

These labels are automatically selected during the indexing process, as Loki ingests any data format. They can also be reduced or increased based on the type of logs. The flexibility of picking and choosing which labels makes it very efficient and helps Loki scale as the amount of ingestion grows or changes in data sources. Think of it like a table of contents to a very large book.

Thus, the index doesn’t grow as big as the traditional logging platforms. This efficiency helps Loki perform much faster even at scale. For example, 10 TB of logs that are ingested through Loki will create an index of only 200 MB while the rest of the raw logs are stored in object storage.

The raw logs are made available each time a search is made and the results return the actual raw logs, like this: 

So for compliance use cases, you will still have access to the raw logs for investigations or any other audit requirements that must be met. This might include regulations relating to fraud, money laundering, criminal investigations, healthcare requirements, or more.

By reducing the size of the index in Loki, the overall TCO drops significantly. And the best bit is Loki uses object storage (e.g. Amazon S3, Microsoft Azure Blob Storage, Google Cloud Storage). That means you can retain the compliance data as long as required by law while also saving money on software licensing and expensive disk storage. 

And because the size of the index in Loki is significantly smaller than the traditional logging platforms, the engineering manpower required to manage the system drops. This frees up resources to focus elsewhere in the organization’s cyber security priorities.

Best of both worlds with Grafana dashboards

Currently, despite all of Loki’s benefits, it’s not best suited for SIEM-style lookups or correlations, due to its label style indexing. And since cyber threats are always increasing, you’ll still need the SIEM-specific solutions out there for your security engineers.

So the way forward here is to selectively send logs based on the priority — to the SIEM for cyber security analytics and Grafana Loki for compliance requirements. 

And while that means using two tools, it doesn’t mean you have to constantly jump between platforms. You can use Grafana to connect to popular SIEM tools, such as Splunk or Elastic, and have both accessible from the same place. This makes it cost effective and efficient: You get the best from both solutions in a single UI where you can make required searches and correlations.

 You can try Loki easily by creating a free Grafana Cloud account and sending your compliance logs there. Or, if you want to self manage, you can run Loki within your network. Either option works! And for more on Loki, check out the latest documentation.