You can get my USHCN C++ code (which shows the amount of fake data) here. It is simple enough to figure out and similar to the GHCN code. This downloads and runs much faster than the GHCN daily code. You can do the whole thing in a few minutes.
./get
cd into the newly created ushcn directory
ln -s ../ushcn.exe
ln-s ../ushcn-stations.txt
ln -s ../configure
./configure
./ushcn.exe US_final.tavg > US_final.csv
./ushcn.exe US_tob.tavg > US_tob.csv
./ushcn.exe US_raw.tavg > US_raw.csv
The scripts ask for tcsh to be installed in case anyone is hitting this error:
-bash: ./configure: /usr/bin/tcsh: bad interpreter: No such file or directory
it might be in /bin/tcsh
That’s a lot of code… lot of parsing. Good variable names. I haven’t tried to run it yet but I’ll give it a shot.
You might consider hosting it on github. Any changes you make will be versioned and synced there.
Does the same problem apply to GHCN?
Which data sets besides GISS use US/G/HCN data to build or calibrate their data?
Great find Steven.
Hi Steve,
Just wanted to give you a heads-up that I posted a critique of your recent work over on the Blackboard: http://rankexploits.com/musings/2014/how-not-to-calculate-temperature/
If your analysis that gridding reduces the delta is correct, that would mean that there has been selective loss of cooler station data.
It might mean that, say, there are more co-op stations in the North East than in the south (which might have more airport stations with automated reporting), and that co-op operators are more likely to stop reporting than automated systems. Regardless, unless you are silly enough to try and calculate absolute temperatures for a region (CONUS) by averaging absolute temperatures at individual points, the station drop-out should have a minimal effect. A good test would be to look at the trends of stations that did and did not drop out over a period of overlap; we did something similar to show that dropout in the GHCN network wasn’t introducing bias (in the “march of the thermometers” days of yore).
Which means that there is selective loss of cooler station data.
Actually, I got that backwards. It would mean there has been selective loss of warmer data.
As have said before, the use of estimated data replacing real measurements should not happen as this leaves a clear path open for conformation bias to set in, where it be intensional or not. Also in the field of physical science it is very poor practice to generate ‘data’, from mathematical approximations, where no data exists, when measurement could be made.
It is not as if real temperature data can not be measured, it can – easily. What has happened is that the ‘powers that be’ will not do this task. If the problem is financing the weather stations’ repair, calibration, and maintenance, then reorganize the whole project so that moneys become available for this. Oddly this is one area which is not rocket science.
I don’t understand that. Do you mean that there would have to be a loss of warmer station data for what they did to be valid?
They filled in missing data with warmer temperatures than the average, which means that they believe the missing data was warmer than the average. i.e. the US is selectively losing warmer station data over time.
I haven’t looked through the code yet to determine this myself, but did you remove the entire history of any stations that are now estimated? Try running a cumulative average of only the sites that have a complete history in order to get an accurate picture.