Extending the Longley benchmark results
to 11 significant digits

by Clint Cummins rev. 8/12/97

The Longley (JASA, 1967, p.819-841) regression coefficients have for many years been a reliable benchmark for testing regression algorithms and statistical packages. However, a careful reading of the article indicates that the published coefficients are not accurate to all (8-12) reported digits. Accordingly, some new 11-digit estimates are presented here which should be accurate for benchmark comparison purposes. We also present some 11-16 digit estimates, and show that it is rare to obtain agreement beyond 11 digits between various accurate alternatives (accuracy is defined in terms of the orthogonality conditions X'e).

1. Which set of Longley coefficients is the benchmark?
2. Why are the Longley results not uniformly accurate to 10 digits?
3. What are the "truly accurate" coefficients for the Longley dataset?
4. How to use this new benchmark.
5. The benchmarks.

1. Which set of Longley coefficients is the benchmark?

The first thing which may puzzle a reader of Longley is that there are several sets of coefficient estimates reported. Which set is "THE" benchmark? The most well known results were computed with a desk calculator, using "special techniques ... to obtain additional accuracy." (p.819)

1.a. p.832 (Table 3, top row) gives "desk calculator" results with 7-12 significant digits.

1.b. p.836 (Table 10, rop row) gives "desk calculator" results with 8-12 significant digits. This is probably the set of results which is most often used as "THE" benchmark.

1.c. p.833 (Table 5, top 2 rows) gives "IBM 1401 ... BLS" results with 11-16 significant digits. Special techniques were used to make this a high accuracy solution, and it was used as the ultimate check for the deck calculator results. Unfortunately, the coefficient for the constant term was not computed.

2. Why are the Longley results not uniformly accurate to 10 digits?

2.a. The algorithms used by the computer programs which Longley wanted to test were even more inaccurate! These were primarily single precision (6-7 digit) algorithms, while the Longley dataset has large numbers of significant digits (3-6) and well-known multicollinearity problems. So, Longley only had to create a benchmark which was "sufficiently accurate [9-12 digits] to test the limits of various least squares electronic computer programs." (p.822).

2.b. The published coefficients are "truncated", rather than rounded. This is noted in the footnotes for the Tables listed above. It can also be easily seen by comparing pages 832 and 836. If rounding had been used, the coefficients on page 832 for X2, X4, and X5 would have their last digits increased by one. Accordingly, we might expect that the final digits on page 836 are also truncated, instead of rounded, and this turns out to be the case (for X3 and X5, based on independent calculations).

2.c. Results are presented in terms of a fixed number of digits to the right of the decimal point, instead of in terms of "significant digits". This means that the coefficients which are smaller in absolute value will have fewer significant digits reported. This is why the number of significant digits varies from 8-12 on page 836.

2.d. The last 2 digits of the published Constant term appear to be incorrect, for reasons unknown. Perhaps they should have had the IBM 1401/BLS program compute the constant for comparison!

3. What are the "truly accurate" coefficients for the Longley dataset?

One might expect, with modern computers, that it should be possible to obtain a benchmark set of coefficients, where each coefficient is accurate to "double precision" (15-16 digits). However, it would require an X'X inverse matrix which is accurate well beyond 15-16 digits to determine such a benchmark. Perhaps this can be done by someone with access to an extended precision (33-34 digits) compiler, but I haven't attempted this.
[update, 1998 -- exactly this was done for the NIST 15 digit benchmark, given below]

However, commonly available software can be used to determine the overall accuracy of a given vector of coefficient estimates. We can simply compute the vector of orthogonality conditions (X'e, where e is the vector of residuals computed with a set of coefficients), and see how close to zero it is. For the alternative regression algorithms that I have compared, X'e tends to be uniformly smaller for all rows for the more accurate estimates.

Using this method, I present below an 11 significant digit benchmark (TSP OLSQ-11) which outperforms the published Longley results (p.836). I also present some alternative 16 digit benchmarks which compare favorably with the best Longley results (p.833). These latter benchmarks do not usually agree to more than 11 digits, so they do not provide a 15-16 digit solution, but they do indicate the probable accuracy of the 11-digit solution.

4. How to use this new benchmark.

4.a. (precision of data storage). The benchmark results were computed with the Longley dataset stored in double precision. This is crucial for X1, which is not an integer, and would only have 6-7 significant digits in single precision. One could hardly expect a coefficient to be accurate to 11 digits when the fundamental data is not accurate to that extent. A workaround for single precision packages is to implicitly multiply X1 by 10. That is, to read in (X1*10) as an integer. I would advise this as the common benchmark, since it eliminates this potential discrepancy between packages. The benchmark coefficient for X1 is of course accordingly divided by 10.

4.b. (print coefficients to at least 11 significant digits). The package will need some method to print its regression coefficients with a large number of significant digits. Often a Fortran-style format might have to be employed to do this. Something like F28.18 should be sufficient for this purpose.

4.c. (check orthogonality conditions). If you want to compare coefficients to more than 11 digits, you should use the orthogonality conditions as an independent test of accuracy. If the package does not have a way of storing the residuals in double precision, you may have to use an alternative matrix-type package to do this (after reading in the data and coefficients of interest). Note that the orthogonality conditions are not independent of the scale of the different X variables, so some coefficients will be more important to X'e than others.

4.d. (benchmarks vs. real use of regression packages). The Longley benchmark has been treated as if the data is truly accurate to at least 11 digits. It is listed as only 3-6 digits, and the computer assumes the remaining unlisted digits are all exactly zero. While this makes sense for simple data like dummy variables, such accurate data is extremely unlikely to be encountered in practice. Economic data is rarely accurate to more than 3-4 digits, at least in terms of the concepts it is attempting to measure. There are just too many sources of error in measuring economic variables. This is the reason why many packages store data in single precision. It follows from "i.i.d. errors in variables" econometrics that the resulting coefficients cannot be more precise than the original data (roughly speaking).

Even if the underlying data were known to 11 digits, there is usually not much point in reporting coefficients to more than 3-4 digits, because the coefficient standard errors in most models are relatively large. That is, differences in the 3rd or 4th significant digit will often be insignificant (in terms of a t-statistic). This is why some regression packages could have two alternative regression algorithms. One algorithm would be very accurate (say to 11 digits), in case the base data were unusually accurate, but the algorithm would be relatively slow. The second algorithm would be faster, but less accurate (say to 7-8 digits). The speed could be important for large datasets (or for Monte Carlo experiments), and the lower accuracy would not matter for the reasons outlined above. We have done exactly this in TSP 4.4; the "FAST" option computes regressions accurate to 7-8 digits, which makes virtually no difference to the normal regression output (normally only 6 significant digits are printed).

5. The benchmarks

The benchmarks themselves, and their associated orthogonality conditions are given below. For ease of comparison, a space has been inserted where the digits of the alternative columns no longer agree.

8-12 digit benchmark coefficients

    Longley p.836           TSP OLSQ-11
(8-12 digits trunc.)   (11 digits rounded)
 -3482258.63 30      -3482258.63 46              constant
       15.061872271        15.061872271          X1 - DEFL
       -0.035819179        -0.035819179 293      X2 - GNP
       -2.02022980 3       -2.02022980 38        X3 - UNEM
       -1.033226867        -1.033226867 2        X4 - MIL
       -0.05110410 5       -0.05110410 5653      X5 - POP
     1829.1514646 1      1829.1514646            X6 - TIME
 

X'e (orthogonality conditions)

  Longley p.836  TSP OLSQ-11
  -0.28513E-01   0.49299E-03
   -2.9047       0.50145E-01
   -11105.        191.29    
   -91.352        1.5750    
   -74.478        1.2858    
   -3351.7        57.900    
   -55.732       0.96357    
 

11-16 digit benchmark coefficients

     Longley p.833                TSP 2SLS-16               TSP OLSQ-16
 (11-16 digits trunc.)       (16 digits rounded)       (16 digits rounded)
-3482258.63459 6118*   -3482258.63459 2375        -3482258.63459 5426
      15.061872271 373       15.061872271 29156         15.061872271 40337
      -0.03581917929 2       -0.03581917929 247851      -0.03581917929 259125
      -2.02022980381 6       -2.02022980381 50124       -2.02022980381 6867
      -1.03322686717 3       -1.03322686717 32705       -1.03322686717 3565
      -0.05110410565 3       -0.05110410565 401659      -0.05110410565 347529
    1829.15146461 3551     1829.15146461 1795         1829.15146461 3344
 
        NIST-15   -- from NIST
       (15 digits)
-3482258.63459 582   
      15.061872271 3733  
      -0.03581917929 25910
      -2.02022980381 683    
      -1.03322686717 359   
      -0.05110410565 35807
    1829.15146461 355    
 

*constant term for Longley p.833 was not reported; the * estimate was computed from the formula alpha_hat = y_bar - X_bar*b_hat .

X'e (orthogonality conditions)

  Longley p.833  TSP 2SLS-16   TSP OLSQ-16     NIST-15
  -0.29104E-10  -0.28886E-08   0.52387E-09   0.88476E-07
  -0.10181E-04  -0.32491E-06   0.37449E-07   0.90008E-05
  -0.94520E-01  -0.13790E-02   0.56417E-04   0.34367E-01
  -0.54684E-03  -0.13778E-04   0.21937E-06   0.28220E-03
  -0.29263E-03  -0.70376E-05   0.20959E-05   0.23060E-03
  -0.65692E-02  -0.35972E-03   0.49510E-04   0.10394E-01
  -0.45684E-05  -0.56588E-05   0.10171E-05   0.17293E-03
 

Note: if the TSP OLSQ-16 coefficients are rounded to 15 digits, their X'e is slightly worse than the X'e for NIST-15. This means NIST-15 is a good 15-digit benchmark, although it can be improved upon by using 16 digits (even though the 15-digit and alternative 16-digit benchmarks do not agree for more than 11 digits or so).

If you have any questions or comments about TSP please send an
email to info@tspintl.com. Comments or questions about this website 
should be sent to the webmaster. Lost? Please consult the site map.