Autoregressive moving-average model

"ARMA model" redirects here. For other uses, see ARMA (disambiguation).

In the statistical analysis of time series, autoregressive–moving-average (ARMA) models are a way to describe of a (weakly) stationary stochastic process using autoregression (AR) and a moving average (MA), each with a polynomial. They are a tool for understanding a series and predicting future values. AR involves regressing the variable on its own lagged (i.e., past) values. MA involves modeling the error as a linear combination of error terms occurring contemporaneously and at various times in the past. The model is usually denoted ARMA(p, q), where p is the order of AR and q is the order of MA.

The general ARMA model was described in the 1951 thesis of Peter Whittle, Hypothesis testing in time series analysis, and it was popularized in the 1970 book by George E. P. Box and Gwilym Jenkins.

ARMA models can be estimated by using the Box–Jenkins method.

Mathematical formulation

Autoregressive model

Main article: Autoregressive model

The notation AR(p) refers to the autoregressive model of order p. The AR(p) model is written as

${\displaystyle X_{t}=\sum _{i=1}^{p}\varphi _{i}X_{t-i}+\varepsilon _{t}}$

where ${\displaystyle \varphi _{1},\ldots ,\varphi _{p}}$ are parameters and the random variable ${\displaystyle \varepsilon _{t}}$ is white noise, usually independent and identically distributed (i.i.d.) normal random variables.^[1][2]

In order for the model to remain stationary, the roots of its characteristic polynomial must lie outside the unit circle. For example, processes in the AR(1) model with ${\displaystyle |\varphi _{1}|\geq 1}$ are not stationary because the root of ${\displaystyle 1-\varphi _{1}B=0}$ lies within the unit circle.^[3]

The augmented Dickey–Fuller test assesses the stability of IMF and trend components. For stationary time series, the ARMA model is used, while for non-stationary series, LSTM models are used to derive abstract features. The final value is obtained by reconstructing the predicted outcomes of each time series.

Moving average model

Main article: Moving-average model

The notation MA(q) refers to the moving average model of order q:

${\displaystyle X_{t}=\mu +\varepsilon _{t}+\sum _{i=1}^{q}\theta _{i}\varepsilon _{t-i}\,}$

where the ${\displaystyle \theta _{1},...,\theta _{q}}$ are the parameters of the model, ${\displaystyle \mu }$ is the expectation of ${\displaystyle X_{t}}$ (often assumed to equal 0), and ${\displaystyle \varepsilon _{1}}$, ..., ${\displaystyle \varepsilon _{t}}$ are i.i.d. white noise error terms that are commonly normal random variables.^[4]

ARMA model

The notation ARMA(p, q) refers to the model with p autoregressive terms and q moving-average terms. This model contains the AR(p) and MA(q) models,^[5]

${\displaystyle X_{t}=\varepsilon _{t}+\sum _{i=1}^{p}\varphi _{i}X_{t-i}+\sum _{i=1}^{q}\theta _{i}\varepsilon _{t-i}.\,}$

In terms of lag operator

In some texts, the models is specified using the lag operator L. In these terms, the AR(p) model is given by

${\displaystyle \varepsilon _{t}=\left(1-\sum _{i=1}^{p}\varphi _{i}L^{i}\right)X_{t}=\varphi (L)X_{t}\,}$

where ${\displaystyle \varphi }$ represents the polynomial

${\displaystyle \varphi (L)=1-\sum _{i=1}^{p}\varphi _{i}L^{i}.\,}$

The MA(q) model is given by

${\displaystyle X_{t}-\mu =\left(1+\sum _{i=1}^{q}\theta _{i}L^{i}\right)\varepsilon _{t}=\theta (L)\varepsilon _{t},\,}$

where ${\displaystyle \theta }$ represents the polynomial

${\displaystyle \theta (L)=1+\sum _{i=1}^{q}\theta _{i}L^{i}.\,}$

Finally, the combined ARMA(p, q) model is given by

${\displaystyle \left(1-\sum _{i=1}^{p}\varphi _{i}L^{i}\right)X_{t}=\left(1+\sum _{i=1}^{q}\theta _{i}L^{i}\right)\varepsilon _{t}\,,}$

or more concisely,

${\displaystyle \varphi (L)X_{t}=\theta (L)\varepsilon _{t}\,}$

or

${\displaystyle {\frac {\varphi (L)}{\theta (L)}}X_{t}=\varepsilon _{t}\,.}$

This is the form used in Box, Jenkins & Reinsel.^[6]

Moreover, starting summations from ${\displaystyle i=0}$ and setting ${\displaystyle \phi _{0}=-1}$ and ${\displaystyle \theta _{0}=1}$, then we get an even more elegant formulation: ${\displaystyle -\sum _{i=0}^{p}\phi _{i}L^{i}\;X_{t}=\sum _{i=0}^{q}\theta _{i}L^{i}\;\varepsilon _{t}\,.}$

Spectrum

The spectral density of an ARMA process is$${\displaystyle S(f)={\frac {\sigma ^{2}}{2\pi }}\left\vert {\frac {\theta (e^{-if})}{\phi (e^{-if})}}\right\vert ^{2}}$$where ${\displaystyle \sigma ^{2}}$ is the variance of the white noise, ${\displaystyle \theta }$ is the characteristic polynomial of the moving average part of the ARMA model, and ${\displaystyle \phi }$ is the characteristic polynomial of the autoregressive part of the ARMA model.^[7][8]

Fitting models

Choosing p and q

An appropriate value of p in the ARMA(p, q) model can be found by plotting the partial autocorrelation functions. Similarly, q can be estimated by using the autocorrelation functions. Both p and q can be determined simultaneously using extended autocorrelation functions (EACF).^[9] Further information can be gleaned by considering the same functions for the residuals of a model fitted with an initial selection of p and q.

Brockwell & Davis recommend using Akaike information criterion (AIC) for finding p and q.^[10] Another option is the Bayesian information criterion (BIC).

Estimating coefficients

After choosing p and q, ARMA models can be fitted by least squares regression to find the values of the parameters which minimize the error term. It is good practice to find the smallest values of p and q which provide an acceptable fit to the data. For a pure AR model, the Yule-Walker equations may be used to provide a fit.

ARMA outputs are used primarily to forecast (predict), and not to infer causation as in other areas of econometrics and regression methods such as OLS and 2SLS.

Software implementations

• In R, standard package stats has function arima, documented in ARIMA Modelling of Time Series. Package astsa has an improved script called sarima for fitting ARMA models (seasonal and nonseasonal) and sarima.sim to simulate data from these models. Extension packages contain related and extended functionality: package tseries includes the function arma(), documented in "Fit ARMA Models to Time Series"; packagefracdiff contains fracdiff() for fractionally integrated ARMA processes; and package forecast includes auto.arima for selecting a parsimonious set of p, q. The CRAN task view on Time Series contains links to most of these.
• Mathematica has a complete library of time series functions including ARMA.^[11]
• MATLAB includes functions such as arma, ar and arx to estimate autoregressive, exogenous autoregressive and ARMAX models. See System Identification Toolbox and Econometrics Toolbox for details.
• Julia has community-driven packages that implement fitting with an ARMA model such as arma.jl.
• Python has the statsmodelsS package which includes many models and functions for time series analysis, including ARMA. Formerly part of the scikit-learn library, it is now stand-alone and integrates well with Pandas.
• PyFlux has a Python-based implementation of ARIMAX models, including Bayesian ARIMAX models.
• IMSL Numerical Libraries are libraries of numerical analysis functionality including ARMA and ARIMA procedures implemented in standard programming languages like C, Java, C# .NET, and Fortran.
• gretl can estimate ARMA models, as mentioned here
• GNU Octave extra package octave-forge supports AR models.
• Stata includes the function arima. for ARMA and ARIMA models.
• SuanShu is a Java library of numerical methods that implements univariate/multivariate ARMA, ARIMA, ARMAX, etc models, documented in "SuanShu, a Java numerical and statistical library".
• SAS has an econometric package, ETS, that estimates ARIMA models. See details.

History and interpretations

The general ARMA model was described in the 1951 thesis of Peter Whittle, who used mathematical analysis (Laurent series and Fourier analysis) and statistical inference.^[12][13] ARMA models were popularized by a 1970 book by George E. P. Box and Jenkins, who expounded an iterative (Box–Jenkins) method for choosing and estimating them. This method was useful for low-order polynomials (of degree three or less).^[14]

ARMA is essentially an infinite impulse response filter applied to white noise, with some additional interpretation placed on it.

In digital signal processing, ARMA is represented as a digital filter with white noise at the input and the ARMA process at the output.

Applications

ARMA is appropriate when a system is a function of a series of unobserved shocks (the MA or moving average part) as well as its own behavior. For example, stock prices may be shocked by fundamental information as well as exhibiting technical trending and mean-reversion effects due to market participants.^{[citation needed]}

Generalizations

There are various generalizations of ARMA. Nonlinear AR (NAR), nonlinear MA (NMA) and nonlinear ARMA (NARMA) model nonlinear dependence on past values and error terms. Vector AR (VAR) and vector ARMA (VARMA) model multivariate time series. Autoregressive integrated moving average (ARIMA) models non-stationary time series (that is, whose mean changes over time). Autoregressive conditional heteroskedasticity (ARCH) models time series where the variance changes. Seasonal ARIMA (SARIMA or periodic ARMA) models periodic variation. Autoregressive fractionally integrated moving average (ARFIMA, or Fractional ARIMA, FARIMA) model time-series that exhibits long memory. Multiscale AR (MAR) is indexed by the nodes of a tree instead of integers.

Autoregressive–moving-average model with exogenous inputs (ARMAX)

The notation ARMAX(p, q, b) refers to a model with p autoregressive terms, q moving average terms and b exogenous inputs terms. The last term is a linear combination of the last b terms of a known and external time series ${\displaystyle d_{t}}$. It is given by:

${\displaystyle X_{t}=\varepsilon _{t}+\sum _{i=1}^{p}\varphi _{i}X_{t-i}+\sum _{i=1}^{q}\theta _{i}\varepsilon _{t-i}+\sum _{i=1}^{b}\eta _{i}d_{t-i}.\,}$

where ${\displaystyle \eta _{1},\ldots ,\eta _{b}}$ are the parameters of the exogenous input ${\displaystyle d_{t}}$.

Some nonlinear variants of models with exogenous variables have been defined: see for example Nonlinear autoregressive exogenous model.

Statistical packages implement the ARMAX model through the use of "exogenous" (that is, independent) variables. Care must be taken when interpreting the output of those packages, because the estimated parameters usually (for example, in R^[15] and gretl) refer to the regression:

${\displaystyle X_{t}-m_{t}=\varepsilon _{t}+\sum _{i=1}^{p}\varphi _{i}(X_{t-i}-m_{t-i})+\sum _{i=1}^{q}\theta _{i}\varepsilon _{t-i}.\,}$

where ${\displaystyle m_{t}}$ incorporates all exogenous (or independent) variables:

${\displaystyle m_{t}=c+\sum _{i=0}^{b}\eta _{i}d_{t-i}.\,}$

See also

• Autoregressive integrated moving average (ARIMA)
• Exponential smoothing
• Linear predictive coding
• Predictive analytics
• Infinite impulse response
• Finite impulse response

References

1. Box, George E. P. (1994). Time series analysis : forecasting and control. Gwilym M. Jenkins, Gregory C. Reinsel (3rd ed.). Englewood Cliffs, N.J.: Prentice Hall. p. 54. ISBN 0-13-060774-6. OCLC 28888762.
2. Shumway, Robert H. (2000). Time series analysis and its applications. David S. Stoffer. New York: Springer. pp. 90–91. ISBN 0-387-98950-1. OCLC 42392178.
3. Box, George E. P.; Jenkins, Gwilym M.; Reinsel, Gregory C. (1994). Time series analysis : forecasting and control (3rd ed.). Englewood Cliffs, N.J.: Prentice Hall. pp. 54–55. ISBN 0-13-060774-6. OCLC 28888762.
4. Box, George E. P.; Jenkins, Gwilym M.; Reinsel, Gregory C.; Ljung, Greta M. (2016). Time series analysis : forecasting and control (5th ed.). Hoboken, New Jersey: John Wiley & Sons, Incorporated. p. 53. ISBN 978-1-118-67492-5. OCLC 908107438.
5. Shumway, Robert H. (2000). Time series analysis and its applications. David S. Stoffer. New York: Springer. p. 98. ISBN 0-387-98950-1. OCLC 42392178.
6. Box, George; Jenkins, Gwilym M.; Reinsel, Gregory C. (1994). Time Series Analysis: Forecasting and Control (Third ed.). Prentice-Hall. ISBN 0130607746.
7. Rosenblatt, Murray (2000). Gaussian and non-Gaussian linear time series and random fields. New York: Springer. p. 10. ISBN 0-387-98917-X. OCLC 42061096.
8. Wei, William W. S. (1990). Time series analysis : univariate and multivariate methods. Redwood City, Calif.: Addison-Wesley Pub. pp. 242–243. ISBN 0-201-15911-2. OCLC 18166355.
9. Missouri State University. "Model Specification, Time Series Analysis" (PDF).
10. Brockwell, P. J.; Davis, R. A. (2009). Time Series: Theory and Methods (2nd ed.). New York: Springer. p. 273. ISBN 9781441903198.
11. Time series features in Mathematica Archived November 24, 2011, at the Wayback Machine
12. Hannan, Edward James (1970). Multiple time series. Wiley series in probability and mathematical statistics. New York: John Wiley and Sons.
13. Whittle, P. (1951). Hypothesis Testing in Time Series Analysis. Almquist and Wicksell. Whittle, P. (1963). Prediction and Regulation. English Universities Press. ISBN 0-8166-1147-5.

    Republished as: Whittle, P. (1983). Prediction and Regulation by Linear Least-Square Methods. University of Minnesota Press. ISBN 0-8166-1148-3.

14. Hannan & Deistler (1988, p. 227): Hannan, E. J.; Deistler, Manfred (1988). Statistical theory of linear systems. Wiley series in probability and mathematical statistics. New York: John Wiley and Sons.
15. ARIMA Modelling of Time Series, R documentation

Further reading

• Mills, Terence C. (1990). Time Series Techniques for Economists. Cambridge University Press. ISBN 0521343399.
• Percival, Donald B.; Walden, Andrew T. (1993). Spectral Analysis for Physical Applications. Cambridge University Press. ISBN 052135532X.
• Francq, C.; Zakoïan, J.-M. (2005), "Recent results for linear time series models with non independent innovations", in Duchesne, P.; Remillard, B. (eds.), Statistical Modeling and Analysis for Complex Data Problems, Springer, pp. 241–265, CiteSeerX 10.1.1.721.1754.
• Shumway, R.H. and Stoffer, D.S. (2017). Time Series Analysis and Its Applications with R Examples. Springer. DOI: 10.1007/978-3-319-52452-8