Difference between revisions of "CalCOFI Drifter Realtime Webpage Info"
Kirk ireson (talk | contribs) |
|||
| (5 intermediate revisions by 2 users not shown) | |||
| Line 10: | Line 10: | ||
There are three main parts needed to display the data (details below):<br> | There are three main parts needed to display the data (details below):<br> | ||
1. Fetching the data (using Python script running on dub-locl)<br> | 1. Fetching the data (using Python script running on dub-locl)<br> | ||
| − | 2. Parsing the data (using PHP script running on | + | 2. Parsing the data (using PHP script running on dub-locl)<br> |
3. Displaying the data (using Javascript running on your browser)<br> | 3. Displaying the data (using Javascript running on your browser)<br> | ||
| Line 16: | Line 16: | ||
<h2>1. FETCHING THE DATA</h2> | <h2>1. FETCHING THE DATA</h2> | ||
| − | On dub-locl.icess.ucsb.edu in /localhome/kirk/ | + | On dub-locl.icess.ucsb.edu in /localhome/kirk/Scripts/CalCOFI_ARGOS_SVP_scripts_and_data/ there is a Python script 'fetch_calcofi_svp_data.py'. |
<ul> | <ul> | ||
| − | <li>There is essentially nothing to configure it, and it's well documented should you need to change anything. It is set to fetch the last | + | <li>There is essentially nothing to configure it, and it's well documented should you need to change anything. It is set to fetch via ftp from ftp.aoml.noaa.gov for the last 9 days worth of positions. |
| − | <li>It may need to be updated with drifter numbers as more are sent drifting. | + | <li>It may need to be updated with drifter numbers and temperature coefficients as more are sent drifting. |
| − | <li>It is set to run every 6 hours via a "cron job" | + | <li>It is set to run about every 6 hours (05:30, 11:30, 17:30 & 20:30 Pacific, and they should post the data at 8am Eastern Time) via a "cron job" on the kirk account. To edit this job, as kirk, at the Terminal, type 'crontab -e' and use 'vi' style editing commands, or 'crontab -l' to just show it. (To become kirk, login as drift, then in a Terminal, type 'sudo su -' (then type the drift password) to become root, then 'su kirk' to become kirk.) |
| − | <li>It | + | <li>It fetches their file, archives it (to /localhome/kirk/Scripts/CalCOFI_ARGOS_SVP_scripts_and_data/DATA_ARCHIVE/), uncompresses it, and starts the parsing script (see next section). |
</ul> | </ul> | ||
| − | + | <br> | |
<h2>2. PARSING THE DATA</h2> | <h2>2. PARSING THE DATA</h2> | ||
| − | + | This PHP script on dub-local.icess.ucsb.edu (called by the fetching script above) is /localhome/kirk/Scripts/CalCOFI_ARGOS_SVP_scripts_and_data/parse_calcofi_argos_svp_data.php and is pretty well documented. It breaks down the fairly cryptic GPS encoding and then saves it to a friendly format. It archives this file to /localhome/kirk/Scripts/CalCOFI_ARGOS_SVP_scripts_and_data/DATA_ARCHIVE/yyyymmdd_CALCOFI_SVP_LATEST_DATA.txt and copies this file to /home/kirk/public_html/drifter/realtime-SVP/CALCOFI_SVP_LATEST_DATA.txt so the website has access to it.<br> | |
| − | + | <br> | |
| − | It outputs 1 record per position report in this format: | + | <ul> |
| − | drift_no,YY-MM-DD,HH:MM:SS,milliseconds_since_Jan01_1970,orig_report_num,lat,lon,position_type(1=GPS,0=ARGOS),SST(C),batt(V) | + | <li>There is very little configuring it, and it's well documented should you need to change anything, but it does have temperature coefficients and a couple other possible configuration settings. |
| + | </ul> | ||
| + | It outputs 1 record per position report in this format:<br> | ||
| + | <pre> | ||
| + | drift_no,YY-MM-DD,HH:MM:SS,milliseconds_since_Jan01_1970,orig_report_num,lat,lon,position_type(1=GPS,0=ARGOS),SST(C),batt(V),base_time,record_num | ||
| + | </pre> | ||
e.g. | e.g. | ||
<pre> | <pre> | ||
| − | + | 93561,2007-04-03,10:34:45,1175610885000,136,+26.6980,-081.9550,0,NaN,NaN,NaN,NaN (an argos position) | |
| + | 93561,2007-04-03,10:33:37,1175610817000,137,+26.6979,-081.9557,1,NaN,12.4,2007-04-03_11:33:30,2 (a gps position) | ||
</pre> | </pre> | ||
| − | |||
| − | |||
| − | < | + | <br> |
| − | http://www.icess.ucsb.edu/drifter/realtime-SVP/ | + | <h2>3. DISPLAYING THE DATA (at [http://www.icess.ucsb.edu/drifter/realtime-SVP/ realtime page] or [http://svp.drifterdata.com http://svp.drifterdata.com]) </h2> |
| − | </ | ||
| − | + | The Google mapping code was adapted from the Iridium/Microstar realtime page at [http://www.icess.ucsb.edu/drifter/realtime/ http://www.icess.ucsb.edu/drifter/realtime/] | |
| − | |||
| − | The Google mapping code was adapted from the Iridium/Microstar realtime page at http://www.icess.ucsb.edu/drifter/realtime/ | ||
| − | |||
| − | |||
| − | |||
| − | |||
| + | The two main javascript files are ajax.js and map.js.<br> | ||
| + | <br> | ||
| + | The javascript files are compressed using http://www.icess.ucsb.edu/drifter/realtime/JavascriptCompressor.html, so changes made to the xxx__UNCOMPRESSED__.js files won't show up on the page unless they are compressed and saved without the "__UNCOMPRESSED__" portion in the name, '''OR*''' at the beginning of index.php those two types of files are commented/uncommented out as appropriate.<br> | ||
| + | <br> | ||
<h2> POTENTIAL PROBLEMS / TROUBLESHOOTING</h2> | <h2> POTENTIAL PROBLEMS / TROUBLESHOOTING</h2> | ||
| − | Prob: The data file (/home/ | + | Prob: The data file (/home/kirk/public_html/drifter/realtime-SVP/CALCOFI_SVP_LATEST_DATA.txt) is not getting saved/updated |
<ul> | <ul> | ||
<li>Is the dub-locl computer up? | <li>Is the dub-locl computer up? | ||
| − | <li>Is / | + | <li>Is /home/kirk/ mounted on dub-locl? (It should come up automatically) |
| − | <li>Has 6 hours passed since the last time it ran? It only runs every 6 hours | + | <li>Has 6 hours passed since the last time it ran? It only runs about every 6 hours (see above), but they only update the file once per day at 8am Eastern. |
| − | <li>Try fetching the data yourself via | + | <li>Try fetching the data yourself via ftp: |
</ul> | </ul> | ||
<pre> | <pre> | ||
| − | + | ftp://ftp.aoml.noaa.gov/phod/pub/daily_buoydata/7325/ | |
| − | + | login: anonymous | |
| − | + | pass: e_mail | |
| − | |||
| − | |||
</pre> | </pre> | ||
| − | + | Look for a file of today's data with the following name format: | |
| − | + | ddMMMyy.LOG-GZ, for example: 15JUL09.LOG-GZ (i.e. for july 15th, 2009)<br> | |
| − | + | <br> | |
| − | |||
| − | |||
| − | < | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | < | ||
---- | ---- | ||
| Line 178: | Line 168: | ||
<br> | <br> | ||
<b>NOTES:</b><br> | <b>NOTES:</b><br> | ||
| − | -Total bit count of all 7 lines must equal the checksum value or | + | -Total bit count (a very non trivial calculation) of all 7 lines must equal the checksum value or |
message block is considered garbled and thus ignored.<br> | message block is considered garbled and thus ignored.<br> | ||
<br> | <br> | ||
Latest revision as of 22:50, 9 July 2009
Contents
CalCOFI SVP Drifter Realtime Webpage Wiki
NOTES AND THINGS TO KNOW ABOUT THE SVP REALTIME PAGE
Jun, 2009 Kirk
Webpage: SVP Realtime Page
There are three main parts needed to display the data (details below):
1. Fetching the data (using Python script running on dub-locl)
2. Parsing the data (using PHP script running on dub-locl)
3. Displaying the data (using Javascript running on your browser)
1. FETCHING THE DATA
On dub-locl.icess.ucsb.edu in /localhome/kirk/Scripts/CalCOFI_ARGOS_SVP_scripts_and_data/ there is a Python script 'fetch_calcofi_svp_data.py'.
- There is essentially nothing to configure it, and it's well documented should you need to change anything. It is set to fetch via ftp from ftp.aoml.noaa.gov for the last 9 days worth of positions.
- It may need to be updated with drifter numbers and temperature coefficients as more are sent drifting.
- It is set to run about every 6 hours (05:30, 11:30, 17:30 & 20:30 Pacific, and they should post the data at 8am Eastern Time) via a "cron job" on the kirk account. To edit this job, as kirk, at the Terminal, type 'crontab -e' and use 'vi' style editing commands, or 'crontab -l' to just show it. (To become kirk, login as drift, then in a Terminal, type 'sudo su -' (then type the drift password) to become root, then 'su kirk' to become kirk.)
- It fetches their file, archives it (to /localhome/kirk/Scripts/CalCOFI_ARGOS_SVP_scripts_and_data/DATA_ARCHIVE/), uncompresses it, and starts the parsing script (see next section).
2. PARSING THE DATA
This PHP script on dub-local.icess.ucsb.edu (called by the fetching script above) is /localhome/kirk/Scripts/CalCOFI_ARGOS_SVP_scripts_and_data/parse_calcofi_argos_svp_data.php and is pretty well documented. It breaks down the fairly cryptic GPS encoding and then saves it to a friendly format. It archives this file to /localhome/kirk/Scripts/CalCOFI_ARGOS_SVP_scripts_and_data/DATA_ARCHIVE/yyyymmdd_CALCOFI_SVP_LATEST_DATA.txt and copies this file to /home/kirk/public_html/drifter/realtime-SVP/CALCOFI_SVP_LATEST_DATA.txt so the website has access to it.
- There is very little configuring it, and it's well documented should you need to change anything, but it does have temperature coefficients and a couple other possible configuration settings.
It outputs 1 record per position report in this format:
drift_no,YY-MM-DD,HH:MM:SS,milliseconds_since_Jan01_1970,orig_report_num,lat,lon,position_type(1=GPS,0=ARGOS),SST(C),batt(V),base_time,record_num
e.g.
93561,2007-04-03,10:34:45,1175610885000,136,+26.6980,-081.9550,0,NaN,NaN,NaN,NaN (an argos position) 93561,2007-04-03,10:33:37,1175610817000,137,+26.6979,-081.9557,1,NaN,12.4,2007-04-03_11:33:30,2 (a gps position)
3. DISPLAYING THE DATA (at realtime page or http://svp.drifterdata.com)
The Google mapping code was adapted from the Iridium/Microstar realtime page at http://www.icess.ucsb.edu/drifter/realtime/
The two main javascript files are ajax.js and map.js.
The javascript files are compressed using http://www.icess.ucsb.edu/drifter/realtime/JavascriptCompressor.html, so changes made to the xxx__UNCOMPRESSED__.js files won't show up on the page unless they are compressed and saved without the "__UNCOMPRESSED__" portion in the name, OR* at the beginning of index.php those two types of files are commented/uncommented out as appropriate.
POTENTIAL PROBLEMS / TROUBLESHOOTING
Prob: The data file (/home/kirk/public_html/drifter/realtime-SVP/CALCOFI_SVP_LATEST_DATA.txt) is not getting saved/updated
- Is the dub-locl computer up?
- Is /home/kirk/ mounted on dub-locl? (It should come up automatically)
- Has 6 hours passed since the last time it ran? It only runs about every 6 hours (see above), but they only update the file once per day at 8am Eastern.
- Try fetching the data yourself via ftp:
ftp://ftp.aoml.noaa.gov/phod/pub/daily_buoydata/7325/ login: anonymous pass: e_mail
Look for a file of today's data with the following name format:
ddMMMyy.LOG-GZ, for example: 15JUL09.LOG-GZ (i.e. for july 15th, 2009)
INFO ABOUT TIME CALCULATION
Basically each data block is for three hourly reports, and data blocks can be repeated (I
have no idea how that's determined), and there is a slight adjustment
to the time based on "fix delay" (or "Length of Acquisition", which is
usually on the order of <60 seconds).
For example (showing the second to last block in the existing file as
I write this):
07325 93575 15 27 N 3 2009-06-25 20:18:59 34.416 240.156 0.000 401652653
2009-06-25 20:21:22 3 12 62 678 94
00 01 344156 01
1198452 01 04 689
01 344155 01 1198448
01 04 661 01
344156 01 1198448 01
04 587 00
This *first line* contains ARGOS only info:
prog_num 07325 drift_num 93575 lines_in_sat_pass 15 num_sensors 27 sat_name N resolution 3 time 2009-06-25 20:18:59 xmit_lat 34.416 xmit_lon 240.156 xmit_alt 0.000 xmit_freq 401652653
The next 7 lines contain 3, hourly GPS reports.
Line 1: (2009-06-25 20:21:22 3 12 62 678 94)
BASE Date/Time 2009-06-25 20:21:22 # identical messages rcvd 3 Drogue count over last half hour 12 Battery voltage = n * 0.2 62 -> 12.4V SST(C) = x * 0.043 - 5 678 -> 24.154 Checksum 94
Line 2: (00 01 344156 01)
Msg ID 00 -> this is 1st message sent Valid Pos #1 (1=good, 0=bad) 01 -> good Lat #1 (ten-thousandths of degree) 344156 -> 34.4156 Lat #1 Ref (0 = South, 1 = North) 01 -> North
Line 3: (1198452 01 04 689)
Lon #1 (ten-thousandths of degree) 1198452 -> 119.8452 Lon #1 Ref (0 = East, 1 = West) 01 -> West Pos #1 Length of Acq (seconds = n * 4) 04 -> 16(seconds) SST #1 (C) = x * 0.043 - 5 689 -> 24.627
Line 4: (01 344155 01 1198448)
Valid Pos #2 (1=good, 0=bad) 01 -> good Lat #2 (ten-thousandths of degree) 344155 -> 34.4155 Lat #2 Ref (0 = South, 1 = North) 01 -> North Lon #2 (ten-thousandths of degree) 1198448 -> 119.8448
Line 5: (01 04 661 01)
Lon #2 Ref (0 = East, 1 = West) 01 -> West Pos #2 Length of Acq (seconds = n * 4) 04 -> 16(seconds) SST #2 (C) = x * 0.043 - 5 661 -> 23.423 Valid Pos #3 (1=good, 0=bad) 01 -> good
Line 6: (344156 01 1198448 01)
Lat #3 (ten-thousandths of degree) 344156 -> 34.4156 Lat #3 Ref (0 = South, 1 = North) 01 -> North Lon #3 (ten-thousandths of degree) 1198448 -> 119.8448 Lon #3 Ref (0 = East, 1 = West) 01 -> West
Line 7: (04 587 00)
Pos #3 Length of Acq (seconds = n * 4) 04 -> 16(seconds) SST #3 (C) = x * 0.043 - 5 587 -> 20.241 Null bits (only for padding, ignore)
NOTES:
-Total bit count (a very non trivial calculation) of all 7 lines must equal the checksum value or
message block is considered garbled and thus ignored.
-Times of GPS reports are calculated thusly:
1) Take BASE time, subtract (Msg ID*3) [that is, if message has been
repeated, it is at least 3 hours old -- I have a switch in my code
(it's turned on for now) which will ignore messages with a message
ID>0 because the original may be also present, and it's possible the
duplicate isn't an exact duplicate and then you get slightly different
reports at similar times]
2) Then, subtract 2 hours for Pos #3, and 1 hour for Pos #2 and 0 for Pos #1
3) Then subtract the associated small "Length of Acq"
Thus, the 2nd GPS time (i.e. #2) would be calculated thusly:
BASE time: 2009-06-25 20:21:22 minus 0 hours (because Msg ID is 0) minus 1 hour (because it's Pos #2) minus (pos #2 length of acq) 16 seconds equals: 2009-06-25 19:21:06
An additional wacky thing is that due to repetition of the messages,
or god knows what, the calculations of the reports you see aren't
necessarily in chronological order. So, after doing all the
calculations, I then sort them chronologically before outputting
(within the parse_argos_data.php script), so that when showing the
track lines, the points are in order.
