Observability Survey 2023

This company, which has tens of thousands of employees and generates billions in revenue every year, has cut its observability costs by roughly one-third by switching from Datadog to Grafana Labs solutions. Beyond the costs, they also appreciate the hands-on support and the relationship of mutual trust they’ve built — or, as this senior manager for observability put it, “What has attracted us to Grafana is the non-selling part.”

Still, observability hasn’t always been an easy sell. Company executives expect observability to make up 10% or less of their cloud spend, which is something the observability team has struggled to achieve. What’s ultimately won them over is the realization that they’ll never have just one observability tool, and Grafana’s “big tent” philosophy allows them to support many different data sources, which has translated to better results.

Measuring reliability comes down to KPIs focused on reducing MTTR. At the same time, they’ve improved developer productivity by tracking how much time those teams spend on critical service updates. They’ve also begun to measure the four DORA metrics — deployment frequency, mean lead time for changes, mean lead time to recover, and change failure rate — and hope to see the benefits of those efforts next year.

This company, which provides customer experience (CX) services to global organizations, needs to prioritize observability because it is committed to providing five 9s of availability (99.999% uptime). To meet that commitment, it tracks resilience and reliability metrics, as well as DORA metrics. Every week, they also review the number of incidents incurred, metrics tied to their SLOs, outage minutes, and customer impact minutes.

And while OpEx savings is an outcome of observability, the priority for this company is building customer trust through better observability practices. These “costs” are looked at in three buckets:

• Resiliency, scalability, performance. They see erosion of trust as an opportunity cost. They’re able to measure this impact through their net promoter score (NPS), which has risen from the 50s to the 60s+ thanks to their high resiliency.
• ROI on spend. By investing in observability, they’re able to decrease the signal-to-noise ratio and reduce cardinality.
• Staffing allocations. The shift to observability with Grafana Labs, which has led to more automation and maturity, has enabled this company to shift FTE staffing to higher value projects for the business.

Going forward, they’re eager to continue promoting the capabilities of their observability tools internally, in an effort to advocate for prevention over remediation across the org.

This company, which has facilitated hundreds of thousands of home loans, defines its success metrics by system uptime and the response times of its platform. So far, this focus has been on internal benchmarks, rather than UX.

The lender reviews ROI against business goals, paying particularly close attention when it comes time for contract renewals with vendors. They look at FTE savings, vendor costs, hardware savings, payback periods, and more.

One of the ancillary benefits they have cited after working with Grafana Labs has been how the partnership has allowed employees to add observability to their skill sets, which in turn boosts the overall team’s efforts. This has also helped the organization attract new employees who want to work with the best technologies.

The engineering manager for this gaming provider says customer feedback can be a good place to start when analyzing site reliability, because their users will quickly point out when something goes wrong. However, relying solely on feedback only goes so far. For example, what about the problems customers don’t tell them about? Moreover, when a user can’t log in to their account, that doesn’t automatically provide answers about what part of the application isn’t working right.

To get the best coverage possible, the observability team focuses on measuring anything that could impact the user journey. They also track incidents that won’t necessarily impact users’ in-app experience but still pose risks, including anything related to security.

They’ve also started tracking DORA metrics as they seek to improve their observability strategy, but work remains. They still struggle with discoverability across a fragmented set of tools. Each team at the company manages observability on their own, which prevents the company from having a holistic view of system health. They are also looking for more application performance monitoring (APM) capabilities to get deeper insight into the health of specific services.

Since they switched to Grafana, the engineers at this large healthcare and wellness company have reported that centralized observability has provided value, improved MTTx, and increased revenue. And that starts with practical, financial improvements.

While the organization called out cost savings as a benefit of working with Grafana Labs, the team that manages the observability platform is looking beyond the bottom line: They’re also using Grafana to outline how the team’s work impacts business outcomes.

Company executives get regular reports that show a direct link between the steps the observability team has taken with implementing Grafana and the improvements in the company’s net promoter scores (NPS). There are also indirect benefits, like a 40% spike in usage of the company’s mobile app and other portals, thanks in part to faster load times and an on-page error rate as low as 1%.

And when things do go wrong, they don’t panic. They know they can quickly surface the root cause of any issue with Grafana, which helps them get back to normal operations faster. For example, page load times had been a systemic issue that flew under the radar because the corresponding data wasn’t being surfaced. They were able to fix that with Grafana; today, they have enough visibility into anomalous data that they’ve changed their approach to root cause analysis. Instead of bunkering in war rooms for extended, investigative periods, now they simply address the issues with a 30-second Slack chat, sent directly to the responsible teams, with Grafana dashboard references.

The company even uses insights generated by working with Grafana in investor meetings to give stakeholders confidence in the actions they’re taking. Essentially, the data they’re collecting is finally getting the attention it deserves at the highest levels — and there is even a demand to expand the breadth of reporting at the executive level — because it can be presented in easily digestible formats.

Going forward, they want to help their customers begin tracking these measurements, too. The observability team would still do the heavy lifting, but by providing users with some observability basics, it could jump-start conversations about evaluating the metrics that factor into the success of a given project.

This company has been incredibly successful in the D2C space. Their observability journey, on the other hand, is just getting started. They’ve established a goal of being able to monitor end-to-end business processes, but so far their monitoring has been somewhat random, according to the company’s hosting services engineering manager.

And getting this right is a major priority since system outages lead to major losses in revenue. They have the data sources they need to address those problems more quickly, and now it comes down to tying it all together. That means tracking upstream and downstream data trends to see what business processes are being affected. They’ve also set up dashboards for developers, but they want to do more to give them increased visibility into system health.

Despite being early in the transition, they’ve already made incremental improvements to developer productivity and costs. But they’re also focused on the bigger picture. Ultimately, their observability strategy comes down to providing value back to the business. So they see more ROI in investing the time to configure, compose, and customize their Grafana setup versus relying on a more expensive turnkey solution that isn’t actually tailored to their specific needs.