Wald test

In statistics, the Wald test (named after Abraham Wald) assesses constraints on statistical parameters based on the weighted distance between the unrestricted estimate and its hypothesized value under the null hypothesis, where the weight is the precision of the estimate.^[1][2] Intuitively, the larger this weighted distance, the less likely it is that the constraint is true. While the finite sample distributions of Wald tests are generally unknown,^[3]: 138 it has an asymptotic χ²-distribution under the null hypothesis, a fact that can be used to determine statistical significance.^[4]

Together with the Lagrange multiplier test and the likelihood-ratio test, the Wald test is one of three classical approaches to hypothesis testing. An advantage of the Wald test over the other two is that it only requires the estimation of the unrestricted model, which lowers the computational burden as compared to the likelihood-ratio test. However, a major disadvantage is that (in finite samples) it is not invariant to changes in the representation of the null hypothesis; in other words, algebraically equivalent expressions of non-linear parameter restriction can lead to different values of the test statistic.^[5][6] That is because the Wald statistic is derived from a Taylor expansion,^[7] and different ways of writing equivalent nonlinear expressions lead to nontrivial differences in the corresponding Taylor coefficients.^[8] Another aberration, known as the Hauck–Donner effect,^[9] can occur in binomial models when the estimated (unconstrained) parameter is close to the boundary of the parameter space—for instance a fitted probability being extremely close to zero or one—which results in the Wald test no longer monotonically increasing in the distance between the unconstrained and constrained parameter.^[10][11]

Mathematical details

Under the Wald test, the estimated ${\displaystyle {\hat {\theta }}}$ that was found as the maximizing argument of the unconstrained likelihood function is compared with a hypothesized value ${\displaystyle \theta _{0}}$. In particular, the squared difference ${\displaystyle {\hat {\theta }}-\theta _{0}}$ is weighted by the curvature of the log-likelihood function.

Test on a single parameter

If the hypothesis involves only a single parameter restriction, then the Wald statistic takes the following form:

${\displaystyle W={\frac {{({\widehat {\theta }}-\theta _{0})}^{2}}{\operatorname {var} ({\hat {\theta }})}}}$

which under the null hypothesis follows an asymptotic χ²-distribution with one degree of freedom. The square root of the single-restriction Wald statistic can be understood as a (pseudo) t-ratio that is, however, not actually t-distributed except for the special case of linear regression with normally distributed errors.^[12] In general, it follows an asymptotic z distribution.^[13]

${\displaystyle {\sqrt {W}}={\frac {{\widehat {\theta }}-\theta _{0}}{\operatorname {se} ({\hat {\theta }})}}}$

where ${\displaystyle \operatorname {se} ({\widehat {\theta }})}$ is the standard error (SE) of the maximum likelihood estimate (MLE), the square root of the variance. There are several ways to consistently estimate the variance matrix which in finite samples leads to alternative estimates of standard errors and associated test statistics and p-values.^[3]: 129 The validity of still getting an asymptotically normal distribution after plugin-in the MLE estimator of ${\displaystyle {\hat {\theta }}}$ into the SE relies on Slutsky's theorem.

Test(s) on multiple parameters

The Wald test can be used to test a single hypothesis on multiple parameters, as well as to test jointly multiple hypotheses on single/multiple parameters. Let ${\displaystyle {\hat {\theta }}_{n}}$ be our sample estimator of P parameters (i.e., ${\displaystyle {\hat {\theta }}_{n}}$ is a ${\displaystyle P\times 1}$ vector), which is supposed to follow asymptotically a normal distribution with covariance matrix V, ${\displaystyle {\sqrt {n}}({\hat {\theta }}_{n}-\theta )\,\xrightarrow {\mathcal {D}} \,N(0,V)}$. The test of Q hypotheses on the P parameters is expressed with a ${\displaystyle Q\times P}$ matrix R:

${\displaystyle H_{0}:R\theta =r}$

${\displaystyle H_{1}:R\theta \neq r}$

The distribution of the test statistic under the null hypothesis is

${\displaystyle (R{\hat {\theta }}_{n}-r)'[R({\hat {V}}_{n}/n)R']^{-1}(R{\hat {\theta }}_{n}-r)/Q\quad \xrightarrow {\mathcal {D}} \quad F(Q,n-P)\quad {\xrightarrow[{n\rightarrow \infty }]{\mathcal {D}}}\quad \chi _{Q}^{2}/Q,}$

which in turn implies

${\displaystyle (R{\hat {\theta }}_{n}-r)'[R({\hat {V}}_{n}/n)R']^{-1}(R{\hat {\theta }}_{n}-r)\quad {\xrightarrow[{n\rightarrow \infty }]{\mathcal {D}}}\quad \chi _{Q}^{2},}$

where ${\displaystyle {\hat {V}}_{n}}$ is an estimator of the covariance matrix.^[14]

Proof

Suppose ${\displaystyle {\sqrt {n}}({\hat {\theta }}_{n}-\theta )\,\xrightarrow {\mathcal {D}} \,N(0,V)}$. Then, by Slutsky's theorem and by the properties of the normal distribution, multiplying by R has distribution:

${\displaystyle R{\sqrt {n}}({\hat {\theta }}_{n}-\theta )={\sqrt {n}}(R{\hat {\theta }}_{n}-r)\,\xrightarrow {\mathcal {D}} \,N(0,RVR')}$

Recalling that a quadratic form of normal distribution has a Chi-squared distribution:

${\displaystyle {\sqrt {n}}(R{\hat {\theta }}_{n}-r)'[RVR']^{-1}{\sqrt {n}}(R{\hat {\theta }}_{n}-r)\,\xrightarrow {\mathcal {D}} \,\chi _{Q}^{2}}$

Rearranging n finally gives:

${\displaystyle (R{\hat {\theta }}_{n}-r)'[R(V/n)R']^{-1}(R{\hat {\theta }}_{n}-r)\quad \xrightarrow {\mathcal {D}} \quad \chi _{Q}^{2}}$

What if the covariance matrix is not known a-priori and needs to be estimated from the data? If we have a consistent estimator ${\displaystyle {\hat {V}}_{n}}$ of ${\displaystyle V}$ such that ${\displaystyle V^{-1}{\hat {V}}_{n}}$ has a determinant that is distributed ${\displaystyle \chi _{n-P}^{2}}$, then by the independence of the covariance estimator and equation above, we have:

${\displaystyle (R{\hat {\theta }}_{n}-r)'[R({\hat {V}}_{n}/n)R']^{-1}(R{\hat {\theta }}_{n}-r)/Q\quad \xrightarrow {\mathcal {D}} \quad F(Q,n-P)}$

Nonlinear hypothesis

In the standard form, the Wald test is used to test linear hypotheses that can be represented by a single matrix R. If one wishes to test a non-linear hypothesis of the form:

${\displaystyle H_{0}:c(\theta )=0}$

${\displaystyle H_{1}:c(\theta )\neq 0}$

The test statistic becomes:

${\displaystyle c\left({\hat {\theta }}_{n}\right)'\left[c'\left({\hat {\theta }}_{n}\right)\left({\hat {V}}_{n}/n\right)c'\left({\hat {\theta }}_{n}\right)'\right]^{-1}c\left({\hat {\theta }}_{n}\right)\quad {\xrightarrow {\mathcal {D}}}\quad \chi _{Q}^{2}}$

where ${\displaystyle c'({\hat {\theta }}_{n})}$ is the derivative of c evaluated at the sample estimator. This result is obtained using the delta method, which uses a first order approximation of the variance.

Non-invariance to re-parameterisations

The fact that one uses an approximation of the variance has the drawback that the Wald statistic is not-invariant to a non-linear transformation/reparametrisation of the hypothesis: it can give different answers to the same question, depending on how the question is phrased.^[15][5] For example, asking whether R = 1 is the same as asking whether log R = 0; but the Wald statistic for R = 1 is not the same as the Wald statistic for log R = 0 (because there is in general no neat relationship between the standard errors of R and log R, so it needs to be approximated).^[16]

Alternatives to the Wald test

There exist several alternatives to the Wald test, namely the likelihood-ratio test and the Lagrange multiplier test (also known as the score test). Robert F. Engle showed that these three tests, the Wald test, the likelihood-ratio test and the Lagrange multiplier test are asymptotically equivalent.^[17] Although they are asymptotically equivalent, in finite samples, they could disagree enough to lead to different conclusions.

There are several reasons to prefer the likelihood ratio test or the Lagrange multiplier to the Wald test:^[18][19][20]

• Non-invariance: As argued above, the Wald test is not invariant under reparametrization, while the likelihood ratio tests will give exactly the same answer whether we work with R, log R or any other monotonic transformation of R.^[5]
• The other reason is that the Wald test uses two approximations (that we know the standard error or Fisher information and the maximum likelihood estimate), whereas the likelihood ratio test depends only on the ratio of likelihood functions under the null hypothesis and alternative hypothesis.
• The Wald test requires an estimate using the maximizing argument, corresponding to the "full" model. In some cases, the model is simpler under the null hypothesis, so that one might prefer to use the score test (also called Lagrange multiplier test), which has the advantage that it can be formulated in situations where the variability of the maximizing element is difficult to estimate or computing the estimate according to the maximum likelihood estimator is difficult; e.g. the Cochran–Mantel–Haenzel test is a score test.^[21]

See also

• Chow test
• Sequential probability ratio test
• Sup-Wald test
• Student's t-test
• Welch's t-test

References

1. Fahrmeir, Ludwig; Kneib, Thomas; Lang, Stefan; Marx, Brian (2013). Regression : Models, Methods and Applications. Berlin: Springer. p. 663. ISBN 978-3-642-34332-2.
2. Ward, Michael D.; Ahlquist, John S. (2018). Maximum Likelihood for Social Science : Strategies for Analysis. Cambridge University Press. p. 36. ISBN 978-1-316-63682-4.
3. Martin, Vance; Hurn, Stan; Harris, David (2013). Econometric Modelling with Time Series: Specification, Estimation and Testing. Cambridge University Press. ISBN 978-0-521-13981-6.
4. Davidson, Russell; MacKinnon, James G. (1993). "The Method of Maximum Likelihood : Fundamental Concepts and Notation". Estimation and Inference in Econometrics. New York: Oxford University Press. p. 89. ISBN 0-19-506011-3.
5. Gregory, Allan W.; Veall, Michael R. (1985). "Formulating Wald Tests of Nonlinear Restrictions". Econometrica. 53 (6): 1465–1468. doi:10.2307/1913221. JSTOR 1913221.
6. Phillips, P. C. B.; Park, Joon Y. (1988). "On the Formulation of Wald Tests of Nonlinear Restrictions" (PDF). Econometrica. 56 (5): 1065–1083. doi:10.2307/1911359. JSTOR 1911359.
7. Hayashi, Fumio (2000). Econometrics. Princeton: Princeton University Press. pp. 489–491. ISBN 1-4008-2383-8.,
8. Lafontaine, Francine; White, Kenneth J. (1986). "Obtaining Any Wald Statistic You Want". Economics Letters. 21 (1): 35–40. doi:10.1016/0165-1765(86)90117-5.
9. Hauck, Walter W. Jr.; Donner, Allan (1977). "Wald's Test as Applied to Hypotheses in Logit Analysis". Journal of the American Statistical Association. 72 (360a): 851–853. doi:10.1080/01621459.1977.10479969.
10. King, Maxwell L.; Goh, Kim-Leng (2002). "Improvements to the Wald Test". Handbook of Applied Econometrics and Statistical Inference. New York: Marcel Dekker. pp. 251–276. ISBN 0-8247-0652-8.
11. Yee, Thomas William (2022). "On the Hauck–Donner Effect in Wald Tests: Detection, Tipping Points, and Parameter Space Characterization". Journal of the American Statistical Association. 117 (540): 1763–1774. arXiv:2001.08431. doi:10.1080/01621459.2021.1886936.
12. Cameron, A. Colin; Trivedi, Pravin K. (2005). Microeconometrics : Methods and Applications. New York: Cambridge University Press. p. 137. ISBN 0-521-84805-9.
13. Davidson, Russell; MacKinnon, James G. (1993). "The Method of Maximum Likelihood : Fundamental Concepts and Notation". Estimation and Inference in Econometrics. New York: Oxford University Press. p. 89. ISBN 0-19-506011-3.
14. Harrell, Frank E. Jr. (2001). "Section 9.3.1". Regression modeling strategies. New York: Springer-Verlag. ISBN 0387952322.
15. Fears, Thomas R.; Benichou, Jacques; Gail, Mitchell H. (1996). "A reminder of the fallibility of the Wald statistic". The American Statistician. 50 (3): 226–227. doi:10.1080/00031305.1996.10474384.
16. Critchley, Frank; Marriott, Paul; Salmon, Mark (1996). "On the Differential Geometry of the Wald Test with Nonlinear Restrictions". Econometrica. 64 (5): 1213–1222. doi:10.2307/2171963. hdl:1814/524. JSTOR 2171963.
17. Engle, Robert F. (1983). "Wald, Likelihood Ratio, and Lagrange Multiplier Tests in Econometrics". In Intriligator, M. D.; Griliches, Z. (eds.). Handbook of Econometrics. Vol. II. Elsevier. pp. 796–801. ISBN 978-0-444-86185-6.
18. Harrell, Frank E. Jr. (2001). "Section 9.3.3". Regression modeling strategies. New York: Springer-Verlag. ISBN 0387952322.
19. Collett, David (1994). Modelling Survival Data in Medical Research. London: Chapman & Hall. ISBN 0412448807.
20. Pawitan, Yudi (2001). In All Likelihood. New York: Oxford University Press. ISBN 0198507658.
21. Agresti, Alan (2002). Categorical Data Analysis (2nd ed.). Wiley. p. 232. ISBN 0471360937.

Further reading

• Greene, William H. (2012). Econometric Analysis (Seventh international ed.). Boston: Pearson. pp. 155–161. ISBN 978-0-273-75356-8.
• Kmenta, Jan (1986). Elements of Econometrics (Second ed.). New York: Macmillan. pp. 492–493. ISBN 0-02-365070-2.
• Thomas, R. L. (1993). Introductory Econometrics: Theory and Application (Second ed.). London: Longman. pp. 73–77. ISBN 0-582-07378-2.

External links

• Wald test on the Earliest known uses of some of the words of mathematics