How to forecast holiday data with Grafana Machine Learning in Grafana Cloud
A little over a year ago, we released Grafana Machine Learning, enabling Grafana Cloud Pro and Advanced users to easily view forecasts of their time series. We recently enhanced Grafana Machine Learning with Outlier Detection, which allows you to monitor a group of similar things, such as load-balanced pods in Kubernetes, and get alerted when something starts behaving differently than its peers. Today we want to talk about a new feature of Grafana Machine Learning that helps users improve their forecasts: Holidays.
The problem: How forecasting can fall short
The forecasting model we offer, Prophet, automatically accounts for many different features seen in time series, including yearly, weekly, and daily seasonality. This means that patterns repeating on the same day of the year, day of the week, and time of day are captured in the model and accounted for in predictions. This is in large part what allows forecasts to be used for adaptive alerting, avoiding the need to configure complex thresholds.
But what if your series exhibits repeating patterns over different periods? For example, if you have a backup running on the last day of every month, your I/O metrics may go haywire and cause alerts to spuriously fire. Or what if you have metrics measuring website traffic, which show strong variation between weekends and weekdays as well as public holidays? In that case, the forecasts on public holidays could look like regular weekdays, completely unlike your true traffic expectations.
Since you know about these events in advance, it would be nice to be able to inform the model about them, and handle them in the same way as regular seasonality, learning from past events of this type and applying those learnings to future predictions.
The solution: Holidays in Grafana Machine Learning
The Holidays feature in Grafana Machine Learning lets you do just that. It accounts for specific repeated, predictable shifts in your data by telling the model when the events have occurred in the past, and when they will occur again in the future.
We’ve added an extra tab in the Grafana Machine Learning UI where you can define holidays in one of two ways — either by adding holiday occurrences directly in the UI or simply by providing a public iCalendar address, which we’ll import and apply when we generate your forecasts. The iCalendar will be refreshed every time your forecast model is retrained (by default, every day), so you can be sure you’re getting the latest events from your calendar.
Once you’ve created your holidays, you can link them to as many forecasts as you like. All linked forecasts will use any past occurrences of your holiday to infer the behavior of future occurrences.
How Holidays work in Grafana Machine Learning
Let’s walk through a complete example of how and when to use the Holidays feature. Below we see a series showing traffic to a website, which exhibits clear weekly and daily trends. Traffic ebbs and flows over the course of a day in a regular pattern, and it is considerably higher on weekdays than on weekends. However, we can also see that on Thanksgiving and Native American Heritage Day in 2022 (highlighted), the data looked similar to a weekend as opposed to a weekday, so it’s unlikely that a regular Grafana Machine Learning forecast will have accurate predictions for this day.
Creating the forecast and viewing the in-sample predictions confirms our suspicions: The in-sample predictions during the two-day holiday period look way off, as the model predicts regular weekday values for those days. Also note how the blue predicted lines are far above the green actual lines:
Let’s create a holiday to help account for this. Fortunately for us, Google Calendar contains several pre-populated (and always updated!) calendars for public holidays in each country, so let’s grab the iCalendar URL for Holidays in the United States:
Next, in the new Holidays tab of the Grafana Machine Learning UI, we’ll click the +Holiday button to create a holiday using the above iCalendar URL and link it to our existing forecast:
Once saved, our forecast will now be automatically retrained with knowledge of all events in the holiday’s iCalendar. Taking another look at the in-sample predictions, we can see that traffic on Thanksgiving and Native American Heritage Day has definitely been adjusted for the holiday, but it’s still not perfect: The daily and weekly seasonality effects seem to be more powerful than the holiday effect.
We can make one last adjustment to deal with this. Let’s Edit the forecast and navigate to the Training Model tab and expand the Advanced algorithm settings. Here we can see two options: Separate weekly seasonality for holiday and Separate daily seasonality for holiday. By default, these fields have nothing selected, but we’ll need to select the U.S. Public Holidays option here:
This will allow us to take advantage of Prophet’s conditional seasonalities, which tell the model that the daily/weekly seasonality will be different on holidays vs. off holidays. Let’s take another look at our predictions:
Success! These in-sample forecasts are now considerably more accurate than before. Our model has successfully picked up on two key points:
- U.S. public holidays tend to have an overall lower amount of traffic than non-holidays.
- Daily and weekly seasonality is much smaller on holidays than on non-holidays.
We can also see that our forecasts for the observed public holiday in 2022 for Christmas Day on Monday, December 26th also had much more moderate predictions than the following days, so the model is clearly applying its knowledge to the future, exactly as we wanted:
Learn more about Grafana Machine Learning
Grafana Machine Learning aims to forecast your time series data regardless of its source, whether the data is stored in Prometheus, Postgres, or even Grafana Loki. But sometimes there are real world events that the model simply can’t know about in advance. The new Holidays feature allows you to incorporate that knowledge into your forecasts with no more than your preferred iCalendar. This is crucial for human-driven metrics such as in our example above, to help avoid being paged by nonsensical alerts on days you know will behave differently.
To learn more, check out our Grafana Machine Learning documentation. You can also find the Grafana Machine Learning team in the #machine-learning channel on the Grafana Labs Community Slack.
Grafana Machine Learning is available to all customers on the Grafana Cloud Pro or Advanced plans. If you’re not already using Grafana Cloud, you can sign up for a free 14-day trial of Grafana Cloud Pro.
