Evaluating Time Series Models With WAPE in Obviously AI. Introducing WAPE–Weighted Average Percentage Error metric for evaluating forecast accuracy in time series models.
At the start of this year, we introduced our no-code Time Series platform for analyzing and forecasting different types of time series datasets. The platform’s Tech Specs tab contains standard time series model evaluation metrics such as MAPE, RMSE, MAE, and MSE.
Today, we are announcing the addition of the WAPE metric in the Tech Specs tab so that you have more clarity on the models built and can make more informed decisions.
Here’s everything you need to know about the WAPE metric: what is WAPE, how to interpret WAPE, and how you can use WAPE to be more confident in your model's performance. Let’s dig right in!
What is WAPE Metric?
WAPE, or Weighted Average Percentage Error, is a statistical metric for evaluating forecast accuracy, typically for time series data. It is the sum of the absolute error between the actual and forecasted value normalized by the total demand or sales of all products or items.
While MAPE or Mean Absolute Percentage Error is commonly used for evaluating a time series model, we believe it is important to choose the metric depending on the use case. For example, WAPE is a great metric for measuring a model's performance when the dataset used has low or intermittent values. In real-world scenarios, data consists of null/low values more often than we expect.
WAPE Metric Formula
WAPE (t, n) = ∑t=1…n | At - Ft | / ∑t=1…n At
Don’t worry, we’ll explain everything.
Say you have a daily sales dataset, and you are interested in predicting the sales for the next 10 days (from the last day in your dataset). Here, t = 1 to n, where n=10
Here A represents the actual sales value, and F represents the predicted sales value. In WAPE, we take the absolute difference between the actual and forecast value for each prediction, so regardless of the real value being less or more than the predicted value, the WAPE value would always be positive.
How to Interpret Results–High WAPE, or Low WAPE
WAPE is an error metric. The lower the value of WAPE, the better.
For instance, in a population forecast problem, a low WAPE value would represent a robust prediction model. A WAPE percentage of 5 means that the population forecast was miscalculated by 5% over the entire population for a certain evaluation period. Thus, depending on your use case, your goal should always be to choose a model that gives the least WAPE value.
Is WAPE More Effective Than MAE, MAPE, or RMSE Metrics?
In certain cases, yes, WAPE is more effective.
MAPE (Mean Absolute Percentage Error), MAE (Mean Absolute Error), and RMSE (Root Mean Squared Error) are all promising forecast evaluation metrics. MAPE is commonly used to measure forecasting errors instead of MAE or RMSE because it is scale-independent, but it can be deceiving when sales reach numbers close to zero or in intermittent sales. WAPE is a measure that counters this by weighting the error over total sales.
WAPE is more effective when it is important to prioritize popular items and reduce the error effect of non-popular items over the evaluation period. If a product is analyzed over a period of 6 (n) weeks, some weeks (let’s say weeks 1, 3, and 5) would have high sales (100 items sold in each respective week) while other weeks (weeks 2, 4, and 6) might have low sales (5 items sold in each respective week). Here WAPE is a more accurate evaluation metric as it would calculate the percentage error over the total item sales across 6 weeks.
Related: Find Out If Your Machine Learning Model is Good
Trying Out WAPE With Obviously AI
Let’s explore how WAPE is calculated using Obviously AI Time Series by building a sales prediction model.
Follow the steps below.
Step 1: Select and ingest data
First, you must pick a relevant sales dataset and upload it to the platform. The platform accepts structured data from different data sources such as CSV, Dropbox, Airtable, Google Sheets, MySQL, BigQuery, and more.
For time series forecasting, you require two columns in your dataset: one date column and one prediction column. Two sample datasets are shown below.
Time Series Data Sample 1: The image below shows an average temperature dataset with a date and a temperature column.
Time Series Data Sample 2: The image below shows a population change dataset with a date and population count column.
Click on the Add Dataset (+) icon at the top left corner of the dashboard, as shown in the image below.
Select Time Series, as shown below.
Then, select your data source, as illustrated below.
Or you can select from our sample datasets like we are doing in this walkthrough under Sample Datasets.
.gif)
Step 2: Build the machine learning model
Once the dataset is uploaded, Obviously AI automatically detects the date and prediction columns. You can tweak certain attributes like Data Level (Week, Month, Year), Aggregation Function (Sum, Mean), and Seasonality. The platform also detects if the dataset is Ideal for Prediction? or not. When everything looks good, click on Start Predicting.
The platform starts building the models immediately and generates them in less than a minute.
Obviously AI implements several industry-standard and popular time series algorithms, as mentioned below:
- Holt Winters Additive
- Holt Winters Multiplicative
- ARIMA
- SARIMA
- Prophet
- XG Boost
The platform automatically chooses the best-performing model depending on your use case and the corresponding error metric.
Step 3: Evaluating the trained time series model
Once the training process is complete and the model is built, the dashboard shows a model summary from where you can click on Launch Model to start making predictions.
But before that, let’s check out what we came here for–the WAPE metric. Click on the Tech Specs tab, and there you have it, not just WAPE but all other metrics as well.
Obviously AI also allows you to try out different time series models and observe their corresponding metric scores.
Step 4: Making predictions shareable
As previously mentioned, click on Launch Model, which opens up our prediction app to start making predictions for the required forecast period (like predicting champagne sales for the next 10 weeks in this case). You can observe the prediction table and even download the prediction file in CSV format.
You can also visualize them in the interactive Prediction Plot.
Finally, in the prediction app, you can view the time series model summary under the Model Tech Specs tab.
Related: Our Ultimate Guide to Machine Learning
Putting it All Together
With the WAPE metric at your disposal, you can analyze your time series forecasting model in great detail. WAPE provides an additional measure to check the accuracy of time series predictions.
WAPE metric is calculated for all major time series problems, such as sales and revenue prediction, energy consumption prediction, stock price prediction, temperature and weather prediction, or data coming from sensors.
If you’d like to see it in action, check out Obviously AI Time Series, which lets you build state-of-the-art forecast models and run time series predictions without writing any code.
If you want to encourage us so that we continue building cool AI tools, don’t forget to vote for us on Product Hunt!
