UNHIDING THE DECLINE

Update : There were some errors in the initial release. Please try again.

This is the initial release of my new GHCN software, titled UNHIDING THE DECLINE. It is designed to allow you to process large GHCN databases on a small computer with little memory, and make graphs very quickly.

The initial release is just for people who have python available. This includes any Mac or Linux user, and any Windows user who is willing to make the effort to install python3 and a gnu environment like cygwin or mingw. I will release a Windows exe file soon.

Open a terminal window

python3 -m pip install matplotlib
python3 -m pip uninstall numpy
python3 -m pip install numpy==1.12.1

Download these two files from dropbox :  GHCNPY.tgz  get
tar xzvf GHCNPY.tgz
cd GHCNPY

Try this command to reproduce the plot above. It will take several minutes to bring the screen up, as it has to process the data

python3 ghcn.py US.txt summer text=”USHCN”

You can generate hundreds of different graph types very quickly by simply clicking on an X-axis option (left column) and Y-axis option (right column) .  Click on the “Save Image” button to create a .png file, which is in the same directory you ran the ghcn.py script from. It also generates a csv file in that directory with all of the stats.

Other command line options include

  • spring
  • fall
  • month=[1-12]
  • date=[mmdd]
  • first_year=[yyyy]
  • last_year=[yyyy]
  • target_type=[MIN/MAX/BOTH]
  • target_min=[temperature F]      (used for stats)
  • target_max=[temperature F] .   (used for stats)
  • state=[AZ/AK/CO…..]
  • required_range=[YYYY:YYYY] . ( station must have been active in both years)

The algorithm is very simple.

  1. Average all daily temperature records for one station for one month (that is how NOAA organizes them)
  2. Average all of the 1200 station averages for that month
  3. Repeat for all 12 months, and all years from 1895 to the present.

That is it. The algorithm is an improvement over my previous software, which lumped all daily data from all stations together per month. That unfairly weighted stations which reported more days relative to stations which reported fewer days.

To update the USHCN database, change the permissions on the bash get script, and run it “./get” . That will create a new US.txt file.

If you find any bugs, let me know! There may well be some. I just started this project last weekend. I’ve been at this for 12 straight hours and need to get out on my bike!

This entry was posted in Uncategorized. Bookmark the permalink.

16 Responses to UNHIDING THE DECLINE

  1. GW Smith says:

    Fantastic! Thanks, Tony! Darn if I don’t have an HP.

  2. Mathius says:

    The picture here shows warmest year as 1939, and coolest as 2007.

    It’s offset by 3 years.

    Also, this summer maximum graph slightly disagrees with your previous ones which have shown the coolest summer to be 1992, not 2004.

    I love your work, just trying to figure out why things have changed here!

    • tonyheller says:

      I see what you are saying. It looks like the text gives the x value at the position of the label, which I offset to make it more readable. I will have to remove the offset. I’m uploading a new version which removes the labels and fixes some other problems.

  3. E.M.Smith says:

    Well, looks like the Smiths are here ;-)

    Looks like a lot of fun.

    I’m going to download it and play with it.

  4. Kris Johanson says:

    And you’re giving this away Poor Bono? This is a $250,000 grant, at least

  5. William E. Smith says:

    Interesting #climategate

    • Colorado Wellington says:

      The Smiths are out in force. I think they have something to do with the subject. Even more likely than the Russians. ?

  6. rms says:

    Note that un-installing and re-installing a specific version of numpy might “break” user’s applications that depend on a different version. If version important, setup a “virtualenv” first. See http://docs.python-guide.org/en/latest/dev/virtualenvs/

  7. Pol Knops says:

    What is the background of the original data?
    Are the daily measurements of all US weatherstations included or something like that?
    Are these data public available (in addition to the tools to make graphs etc. etc.)

  8. E.M.Smith says:

    I “gave it a go” on a Raspberry Pi Model 3 running Devuan Linux (Debian without systemd).

    The download / unpack / Python install bits all went OK. Some warnings but claimed to have installed at the end.

    I did need to make a minor change to the “get” script. Not all systems have tch, so the script interpreter shell was changed to bash.

    #!/bin/bash
    curl -O ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/ghcnd_hcn.tar.gz

    That then worked fine.

    Attempting to run the example failed. I’ve not tried debugging yet, but just FYI “has issues”.

    chiefio@Headend:~/UHTD/GHCNPY$ python3 ghcn.py US.txt summer text=”USHCN”

    ** (ghcn.py:6301): WARNING **: Error retrieving accessibility bus address: org.freedesktop.DBus.Error.ServiceUnknown: The name org.a11y.Bus was not provided by any .service files
    Traceback (most recent call last):
    File “/usr/local/lib/python3.4/dist-packages/matplotlib/backends/backend_cairo.py”, line 33, in
    import cairocffi as cairo
    ImportError: No module named ‘cairocffi’

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File “/usr/local/lib/python3.4/dist-packages/matplotlib/backends/backend_cairo.py”, line 36, in
    import cairo
    ImportError: No module named ‘cairo’

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File “ghcn.py”, line 11, in
    import matplotlib.pyplot as plt
    File “/usr/local/lib/python3.4/dist-packages/matplotlib/pyplot.py”, line 115, in
    _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
    File “/usr/local/lib/python3.4/dist-packages/matplotlib/backends/__init__.py”, line 32, in pylab_setup
    globals(),locals(),[backend_name],0)
    File “/usr/local/lib/python3.4/dist-packages/matplotlib/backends/backend_gtk3agg.py”, line 12, in
    from .backend_cairo import cairo, HAS_CAIRO_CFFI
    File “/usr/local/lib/python3.4/dist-packages/matplotlib/backends/backend_cairo.py”, line 38, in
    raise ImportError(“Cairo backend requires that cairocffi or pycairo is installed.”)
    ImportError: Cairo backend requires that cairocffi or pycairo is installed.
    chiefio@Headend:~/UHTD/GHCNPY$

    Mostly looks like maybe I need to do an apt-get install pycairo or a python3 yada yada to install it?

    No idea what that “bus address” / name issue is about.

    So going to play around a bit, but not as quickly as I’d expected… (Maybe I’ll give the C++ version a spin…)

  9. Hi Tony,
    I’m having a hoot with this!

    I got this message in the Terminal Window after running the software for a few minutes:

    RecursionError: maximum recursion depth exceeded.

    Please see the attached screen shot.

    I’m running Mac OS Sierra Version 10.12.6 and followed the instructions on your blog post. Python Version 3.6 installed.

    Hope this helps!

  10. kyle_fouro says:

    I understand your instructions for Mac but everything’s unstuck. I think I’m just too computer illiterate to recognize whatever it is I’m doing horribly wrong in terminal.

    When you find the time can you get a video up of you demonstrating the above directions?

    • Are you changing directory (cd) to the location of the program?

      You need to be in the proper “folder/location” before running the command line: python3 ghcn.py US.txt summer text=”USHCN”

      For example – cd Downloads – would bring you to your Downloads folder on Mac, then cd GHCNPY – brings you to the correct location to begin…

      Hope this helps!

  11. Stephen Lutz says:

    Two minor issues.
    1) The ‘get’ script directory location for ‘PullingBackTheCurtain’ does not work on Linux. I suggest ‘$HOME/’

    2) After updating the temperature data ghcn.py failed because the CO2 map ended at 2018. I check the Manoa Loa website and their data ends there as well.

    Also, I will take a stab at adding support for a set of stations. I would use this feature to check the validity of news statements about city temperature records.

    Thanks for all you are doing.

  12. Stephen Lutz says:

    Looking at the code, I realized I can process a defined subset of stations by merely combining just those stations daily files into one and passing that to ghcn.py instead of US.txt. Problem with CO2 data ending in 2018 still exists. I will look for another reliable source of that data and update the map in my version. I will post new data URL if I find one.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.