Python3 Script For Getting USHCN Monthly Temperatures

This script gives easy access to USHCN monthly raw, TOBS and Final adjusted temperatures.

Getting the data:
wget ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2.5/ushcn.tavg.latest.FLs.52j.tar.gz
wget ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2.5/ushcn.tavg.latest.tob.tar.gz
wget ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2.5/ushcn.tavg.latest.raw.tar.gz
tar xzvf ushcn.tavg.latest.FLs.52j.tar.gz
tar xzvf ushcn.tavg.latest.tob.tar.gz
tar xzvf ushcn.tavg.latest.raw.tar.gz

wget ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2.5/ushcn.tmax.latest.FLs.52j.tar.gz
wget ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2.5/ushcn.tmax.latest.tob.tar.gz
wget ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2.5/ushcn.tmax.latest.raw.tar.gz
wget http://cdiac.ornl.gov/ftp/ushcn_daily/ushcn-stations.txt
tar xzvf ushcn.tmax.latest.FLs.52j.tar.gz
tar xzvf ushcn.tmax.latest.tob.tar.gz
tar xzvf ushcn.tmax.latest.raw.tar.gz

wget ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2.5/ushcn.tmin.latest.FLs.52j.tar.gz
wget ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2.5/ushcn.tmin.latest.tob.tar.gz
wget ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2.5/ushcn.tmin.latest.raw.tar.gz
tar xzvf ushcn.tmin.latest.FLs.52j.tar.gz
tar xzvf ushcn.tmin.latest.tob.tar.gz
tar xzvf ushcn.tmin.latest.raw.tar.gz

Then cd into the new directory, which will have a new name every day. It will be something like ushcn.v2.5.5.20191004

Example usage:
python3 date.py USH00412679 7 1921 “EAGLE PASS”

Data for July, 1921 at Eagle Pass, Texas.

output:
Raw EAGLE PASS USH00412679 7 1921 37.29 99.12
TOBS EAGLE PASS USH00412679 7 1921 37.16 98.89
Final EAGLE PASS USH00412679 7 1921 35.92 96.66

Last two numbers in each line are Degrees C and Degrees F

You will have to know the station ID of the station you are looking for:
ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2.5/ushcn-v2.5-stations.txt

Script :
——————————————————————-

import sys

filename = sys.argv[1]
#print(filename)
#filename = filename.replace("USC", "USH")
#print(filename)
month = int(sys.argv[2])
year = sys.argv[3]
comment = sys.argv[4]

fd = open(filename + ".raw.tmax")

for line in fd :
    #print(line[12:16])
    if (line[12:16] == year) :
        #print(line)
        offset = 18 + (9 * (month - 1))
        #print(line[offset:offset+4])
        temperature_c = float(line[offset:offset+4]) / 100.0
        temperature_f = (temperature_c * 1.8) + 32.0
        print("Raw", comment, filename, month, year, "%.2f" % temperature_c, "%.2f" % temperature_f)

fd = open(filename + ".tob.tmax")

for line in fd :
#print(line[12:16])
    if (line[12:16] == year) :
        #print(line)
        offset = 18 + (9 * (month - 1))
        #print(line[offset:offset+4])
        temperature_c = float(line[offset:offset+4]) / 100.0
        temperature_f = (temperature_c * 1.8) + 32.0
        print("TOBS", comment, filename, month, year, "%.2f" % temperature_c, "%.2f" % temperature_f)

fd = open(filename + ".FLs.52j.tmax")

for line in fd :
    #print(line[12:16])
    if (line[12:16] == year) :
        #print(line)
        offset = 18 + (9 * (month - 1))
        #print(line[offset:offset+4])
        temperature_c = float(line[offset:offset+4]) / 100.0
        temperature_f = (temperature_c * 1.8) + 32.0
        print("Final", comment, filename, month, year, "%.2f" % temperature_c, "%.2f" % temperature_f)

This entry was posted in Uncategorized. Bookmark the permalink.

10 Responses to Python3 Script For Getting USHCN Monthly Temperatures

  1. G W Smith says:

    Fantastic! Thanks for sharing this, Tony. Total transparency! But they will still twist it.

  2. grilledtomoatoes says:

    I wish I were smart like this.
    Thank you for taking the time and effort to do this. If I had a vote on who should get a Nobel Prize for climate-related matters, it wouldn’t be a Swedish teenager.

  3. Luca Bertagnolio says:

    Very nice Tony!

    Just one note, in the post you tell people to download the tavg files, but the source code refers to the tmax files instead, like the

    “fd = open(filename“

    statements show.

    Luckily the same FTP site has all the tmax, tmin and tavg files on it, so changing names was all that it took!

  4. Timo Soren says:

    Luca, Tony did reference two scripts, the first does all 3 tmax tmin and tavg The second script runs via python ON those files and his example of the script was run on tmax as you obs. Tony was focusing on the newly released script and you were focusing on an earlier one!

  5. dick merrill says:

    Tony,
    What are the chances NOAA or NASA will monkey with all that original data?

  6. Bill Allen says:

    Tony,
    Thank You, this is great!

    I am planning for a lecture in January for Middle school students. I’m an engineer, not a teacher, and I plan to give these kids enough information and data to make them want to think for themselves.

    Other than screenshots of various graphs which you post in your videos, are any resources available? I want to see how traumatized they are, what they think, and to give them some hope for the future! Thanks!

  7. feathers says:

    Tony,

    I know I’m late to this post (I’m catching up, I’m 3-weeks behind), but would it be possible for you to also post a step-by-step video walking us through the steps? I’m guessing these steps are for Mac/Linux users?

  8. Marc Godard says:

    I slightly modified your code. There were a few issues I addressed and fixed. I only care about saving data in Celsius as it can be converted on demand. I also only care about raw data, but this can easily be modified for TOBS and Final.

    ——————————————————————————————————–
    import re

    fd = open(“data/ushcn-v2.5-stations.txt”)
    while True:
    station = fd.readline()
    if not station:
    break

    array = station.split()
    max_file = open(“data/us-data/” + array[0] + “.raw.tmax”)
    min_file = open(“data/us-data/” + array[0] + “.raw.tmin”)
    avg_file = open(“data/us-data/” + array[0] + “.raw.tavg”)
    station_name = station[37:67].strip()

    offset = None
    for year in range(1880, 2020):
    for month in range(1, 13):
    max_temperature_c = None
    max_temperature_f = None
    min_temperature_c = None
    min_temperature_f = None
    avg_temperature_c = None
    avg_temperature_f = None
    offset = 17 + (9 * (month – 1))
    for max_line in max_file:
    if int(max_line[12:16].strip()) == year:
    to_number = int(re.sub(“[^0-9-]”, “”, max_line[offset:offset+5].strip()))
    # print(to_number, max_line[offset:offset+5])
    if to_number != -9999:
    max_temperature_c = float(to_number) / 100.0

    for min_line in min_file:
    if int(min_line[12:16].strip()) == year:
    to_number = int(re.sub(“[^0-9-]”, “”, min_line[offset:offset+5].strip()))
    # print(to_number, min_line[offset:offset+5])
    if to_number != -9999:
    min_temperature_c = float(to_number) / 100.0

    for avg_line in avg_file:
    if int(avg_line[12:16].strip()) == year:
    to_number = int(re.sub(“[^0-9-]”, “”, avg_line[offset:offset+5].strip()))
    # print(to_number, avg_line[offset:offset+5])
    if to_number != -9999:
    avg_temperature_c = float(to_number) / 100.0

    if min_temperature_c is not None and max_temperature_c is not None and avg_temperature_c is not None:
    data_obj = {
    “stationId”: array[0],
    “month”: month,
    “year”: year,
    “min”: min_temperature_c,
    “max”: max_temperature_c,
    “avg”: avg_temperature_c,
    }

    max_file.seek(0)
    min_file.seek(0)
    avg_file.seek(0)

    print(“station:”, array[0], “finished!”)

    ———————————————————-

    I am in the process of doing this sort of script for a large amount of data, including daily data, Canadian data, Europe, and others if I find it.

    My goal is to normalize all the data into a database to play with and maybe make a website where people can get this data without going to each source.

  9. Marc Godard says:

    I made my own chart with the raw data, however, it looks different from some charts I have seen of yours. Is this them hiding the decline?

    Should I use the ghcnd data instead?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.