MASE and RMSSE in ARIMA models

[nickcox896]

Suppose I have the monthly GDP growth and the quarterly GDP growth and I use an ARIMA model for each series and I want to compare
the forecast accuracy (on the test data) of the two models.

Case 1)
Both the depedent variables are modelled the same way (i.e either both are in levels or both are in first differences)

Case 2)
The depedent variables are modelled differently (i.e one is in levels and the other one is in first differences)

My thought:
In either case, I must compare the MASE and RMSSE from each model because the two dependent variables do not represent the same series (in either case). So I can not use MAE or RMSE.

Is this thought correct? Thank you.

[robjhyndman]

Just think about what the comparisons mean. If growth is measured in $, then the unscaled errors are also in $. Does it make sense to compare monthly GDP growth in $ compared to quarterly GDP growth in $? Presumably one will be roughly 3 times the other. But if growth is measured as a %, then the unscaled errors are also a %. Then monthly GDP growth and quarterly GDP growth may be more comparable.

It doesn't matter if the model includes differences or not, provided the forecasts are on the original scale.

[nickcox896]

OK, so assuming that both the series (monthly GDP growth and quarterly GDP growth) are expressed as a %, can I compare them using the RMSSE?

[robjhyndman]

You can compare anything when the errors are scaled because the results are unit-less. But taking percentages also makes them unitless.

[nickcox896]

Thanks!