banner

For a full list of BASHing data blog posts, see the index page.     RSS

Fun with BOM data

Update. Alas, the data-getting described here no longer works. The BOM has changed its download policies.

Australia's Bureau of Meteorology (BOM) offers a huge range of data products on its website. Most are graphics and tables for viewing on BOM webpages, but the graphics and some of the background data are also freely available as downloads.

I've been using two such products in recent years for planning field trips. I live in northwest Tasmania, and my field study area is in the east of the island. There's no point in my travelling to that study area if the weather's been dry, because what I'm looking for are native millipedes that are very hard to find except in the days after good, solid rains.

To see what the weather's been doing in the study area, I've been checking

I've also written a shell script that allows me to view the rainfall map and my own version of the St Pauls River graph, together and offline. This post explains the script.

Rainfall map. If you do a right-click, "Save image as" on the relevant map graphic, BOM supplies its filename: IDT65312.gif. Some digging in the BOM's FTP listings shows that all the "IDT" files are listed on the "anon/gen/fwo" FTP page. A wget command to retrieve the map graphic is then

wget ftp://ftp.bom.gov.au/anon/gen/fwo/IDT65312.gif

and to save the graphic as "rain.gif" (download size 500x500 pixels) in my working folder for this project, the command is

wget -O ~/Temp/NE_rain/rain.gif ftp://ftp.bom.gov.au/anon/gen/fwo/IDT65312.gif

St Pauls River data. The BOM graph is usable, but the vertical scale will change to accommodate a higher daily rainfall total, and I'm not really interested in the accumulated-total line. To make my own graph I need the background data, and hovering over the "1 year of data" label on the graph's webpage shows that the relevant data file is in
 
"http://www.bom.gov.au/tmp/cdio/IDCJAC0009_092020_2018.zip".
 
The command to get this file and save it in the working directory as "SPR.zip" is

wget -O ~/Temp/NE_rain/SPR.zip \
http://www.bom.gov.au/tmp/cdio/IDCJAC0009_092020_2018.zip

Peering into the downloaded ZIP archive (with Xarchiver), I see that the rainfall data are in "IDCJAC0009_092020_2018_Data.csv", so I can extract just that file to my working directory with unzip :

unzip SPR.zip IDCJAC0009_092020_2018_Data.csv

The CSV is a simple one, with a header line, 8 comma-separated fields and a trailing blank line. The relevant fields for me are year (field 3), month (field 4), day (field 5) and rainfall total (field 6). An AWK command pulls out the relevant lines and reformats the fields into an ISO 8601 date followed by a space followed by the rainfall total. I save that as "SPR.txt":

awk -F"," 'NR>1 && NF==8 {print $3"-"$4"-"$5" "$6}' *.csv > SPR.txt

AWK

The next step is to write some instructions for gnuplot to plot the data listed in "SPR.txt". I've written the instructions in the file "rainrules":

set terminal gif size 900,500
set xdata time
set timefmt "%Y-%m-%d"
set output "SPR.gif"
set xrange ["2018-01-01":"2018-12-31"]
set xtics rotate
set boxwidth
set style fill solid
set yrange [0:100]
set grid
set ylabel "Rainfall (mm)"
set title "St Pauls River"
set key off
plot "SPR.txt" using 1:2 with boxes

If you've never used gnuplot, "rainrules" will look a bit complicated, but it really isn't.

There are a lot of gnuplot tutorials and tips on the Web, not to mention books about it and an official manual!

Following "rainrules", gnuplot will take the data in "SPR.txt" and plot them as I prefer in the 900x500-pixel graphic "SPR.gif", whose left-hand side looks like this:

gnuplot

The command to do this could hardly be simpler:

gnuplot rainrules

Merging and displaying the images. OK, I now have the 2 graphics I want, "rain.gif" and "SPR.gif", but it would be nice to put them together as a dated combination. I do that with the montage command in ImageMagick, saving the result as the 1400x500-pixel graphic "[date]_combo.png", where the download [date] comes from the date command:

day=$(date +"%Y-%m-%d")
 
montage rain.gif SPR.gif -tile 2x1 -geometry +0+0 "$day"_combo.png

A simple image viewer is the YAD dialog program, and a YAD command which puts the "combo" PNG more or less top left on my screen is:

yad --posx=50 --posy=50 --width=1400 --height=500 --image "$day"_combo.png

Here's a much-reduced view:

screen

Scripting. Putting it all together in a draft script, I get

#!/bin/bash
 
day=$(date +"%Y-%m-%d")
cd ~/Temp/NE_rain
wget -O ~/Temp/NE_rain/rain.gif ftp://ftp.bom.gov.au/anon/gen/fwo/IDT65312.gif
wget -O ~/Temp/NE_rain/SPR.zip \
http://www.bom.gov.au/tmp/cdio/IDCJAC0009_092020_2018.zip
unzip SPR.zip IDCJAC0009_092020_2018_Data.csv
awk -F"," 'NR>1 && NF==8 {print $3"-"$4"-"$5" "$6}' *.csv > SPR.txt
gnuplot rainrules
montage rain.gif SPR.gif -tile 2x1 -geometry +0+0 "$day"_combo.png
yad --posx=50 --posy=50 --width=1400 --height=500 --image "$day"_combo.png
rm *.csv *.gif *.txt *.zip
exit

where the final rm deletes intermediate files from my working directory "~/Temp/NE_rain" and leaves just "rainrules" and the growing archive of dated PNGs, which I can re-examine offline at my leisure.

The script's not finished, however, because occasionally a download from the BOM website will fail due to a server-side problem. To alert me to such a problem and exit the script I've added a couple of if/else statements:

#!/bin/bash
 
day=$(date +"%Y-%m-%d")
cd ~/Temp/NE_rain
wget -O ~/Temp/NE_rain/rain.gif ftp://ftp.bom.gov.au/anon/gen/fwo/IDT65312.gif
if [ $? -ne 0 ]; then
   yad --text "Failed to download NE rainfall map.\nCheck BOM website." && exit
else
wget -O ~/Temp/NE_rain/SPR.zip \
http://www.bom.gov.au/tmp/cdio/IDCJAC0009_092020_2018.zip
if [ $? -ne 0 ]; then
   rm *.gif *.zip
   yad --text "Failed to download St Pauls R data.\nCheck BOM website." && exit
else
unzip SPR.zip IDCJAC0009_092020_2018_Data.csv
awk -F"," 'NR>1 && NF==8 {print $3"-"$4"-"$5" "$6}' *.csv > SPR.txt
gnuplot rainrules
montage rain.gif SPR.gif -tile 2x1 -geometry +0+0 "$day"_combo.png
yad --posx=50 --posy=50 --width=1400 --height=500 --image "$day"_combo.png
rm *.csv *.gif *.txt *.zip
fi
fi
exit

The script is launched with a keyboard shortcut and it exits when I close the YAD window. If it stops working, my first guess will be that BOM has changed their graphic and data URLs, and if the field work isn't done by the end of this year (2018), I'll need to change "2018" to "2019' in the ZIP address...

Last update: 2018-07-11
The blog posts on this website are licensed under a
Creative Commons Attribution-NonCommercial 4.0 International License