Difference between revisions of "CalCOFI Drifter Realtime Webpage Info"

From Locl
Jump to navigationJump to search
 
(4 intermediate revisions by 2 users not shown)
Line 10: Line 10:
 
There are three main parts needed to display the data (details below):<br>
 
There are three main parts needed to display the data (details below):<br>
 
1. Fetching the data (using Python script running on dub-locl)<br>
 
1. Fetching the data (using Python script running on dub-locl)<br>
−
2. Parsing the data (using PHP script running on website)<br>
+
2. Parsing the data (using PHP script running on dub-locl)<br>
 
3. Displaying the data (using Javascript running on your browser)<br>
 
3. Displaying the data (using Javascript running on your browser)<br>
  
Line 16: Line 16:
 
<h2>1. FETCHING THE DATA</h2>
 
<h2>1. FETCHING THE DATA</h2>
  
−
On dub-locl.icess.ucsb.edu in /localhome/kirk/scripts/ there is a script 'expect_script_to_fetch_argos_data.py'.  It uses 'pexpect', a Python version of the 'expect' scripts, which allows you to script what would normally be an interactive terminal session.  This is necessary in this case because we have to obtain the data via a telnet session.
+
On dub-locl.icess.ucsb.edu in /localhome/kirk/Scripts/CalCOFI_ARGOS_SVP_scripts_and_data/ there is a Python script 'fetch_calcofi_svp_data.py'.
 
<ul>
 
<ul>
−
<li>There is essentially nothing to configure it, and it's well documented should you need to change anything.  It is set to fetch the last 24hours worth of positions.
+
<li>There is essentially nothing to configure it, and it's well documented should you need to change anything.  It is set to fetch via ftp from ftp.aoml.noaa.gov for the last 9 days worth of positions.
  
−
<li>It may need to be updated with drifter numbers as more are sent drifting.
+
<li>It may need to be updated with drifter numbers and temperature coefficients as more are sent drifting.
  
−
<li>It is set to run every 6 hours via a "cron job" in the kirk account.  To edit this job, as kirk, at the Terminal, type 'crontab -e' and use 'vi' style editing commands, or 'crontab -l' to just show it.  (To become kirk, login as drift, then in a Terminal, type 'sudo su -' (then type the drift password) to become root, then 'su kirk' to become kirk.)
+
<li>It is set to run about every 6 hours (05:30, 11:30, 17:30 & 20:30 Pacific, and they should post the data at 8am Eastern Time) via a "cron job" on the kirk account.  To edit this job, as kirk, at the Terminal, type 'crontab -e' and use 'vi' style editing commands, or 'crontab -l' to just show it.  (To become kirk, login as drift, then in a Terminal, type 'sudo su -' (then type the drift password) to become root, then 'su kirk' to become kirk.)
  
−
<li>It saves the output of this session to /home/locl/file_sharing/SVP_CALCOFI_LATEST_DATA.txt on the ICESS network (which is accessed on dub-locl via /mnt/locl/, which should be mounted automatically if the system reboots, see /etc/fstab)
+
<li>It fetches their file, archives it (to /localhome/kirk/Scripts/CalCOFI_ARGOS_SVP_scripts_and_data/DATA_ARCHIVE/), uncompresses it, and starts the parsing script (see next section).
 
</ul>
 
</ul>
−
 
+
<br>
 
<h2>2. PARSING THE DATA</h2>
 
<h2>2. PARSING THE DATA</h2>
  
−
When someone visits the webpage (http://www.icess.ucsb.edu/drifter/realtime-SVP/), in the ajax.js file there is instruction to fetch the data. This 'fetch' is accomplished by calling 'parse_argos_data.php' in that directory.  This php script reads the file saved above (in /home/locl/), ignores all the non-data lines, and processes/cleans up the crazy ARGOS format output from the telnet session.
+
This PHP script on dub-local.icess.ucsb.edu (called by the fetching script above) is /localhome/kirk/Scripts/CalCOFI_ARGOS_SVP_scripts_and_data/parse_calcofi_argos_svp_data.php and is pretty well documented. It breaks down the fairly cryptic GPS encoding and then saves it to a friendly format. It archives this file to /localhome/kirk/Scripts/CalCOFI_ARGOS_SVP_scripts_and_data/DATA_ARCHIVE/yyyymmdd_CALCOFI_SVP_LATEST_DATA.txt and copies this file to /home/kirk/public_html/drifter/realtime-SVP/CALCOFI_SVP_LATEST_DATA.txt so the website has access to it.<br>
−
 
+
<br>
 +
<ul>
 +
<li>There is very little configuring it, and it's well documented should you need to change anything, but it does have temperature coefficients and a couple other possible configuration settings.
 +
</ul>
 
It outputs 1 record per position report in this format:<br>
 
It outputs 1 record per position report in this format:<br>
 
<pre>
 
<pre>
Line 37: Line 40:
 
e.g.
 
e.g.
 
<pre>
 
<pre>
−
75428,2007-04-03,10:33:37,1175610817000,137,+26.6979,-081.9557,1,NaN,12.4,2007-04-03_11:33:30,2   (a gps position)
+
93561,2007-04-03,10:34:45,1175610885000,136,+26.6980,-081.9550,0,NaN,NaN,NaN,NaN    (an argos position)
 +
93561,2007-04-03,10:33:37,1175610817000,137,+26.6979,-081.9557,1,NaN,12.4,2007-04-03_11:33:30,2   (a gps position)
 
</pre>
 
</pre>
−
<ul>
 
−
<li>There is very little configuring it, and it's well documented should you need to change anything.
 
  
−
<li>Its output can be seen easily:
+
<br>
−
http://www.icess.ucsb.edu/drifter/realtime-SVP/parse_argos_data.php
+
<h2>3. DISPLAYING THE DATA  (at [http://www.icess.ucsb.edu/drifter/realtime-SVP/ realtime page] or [http://svp.drifterdata.com http://svp.drifterdata.com]) </h2>
−
</ul>
 
  
−
<h2>3. DISPLAYING THE DATA  (at [http://www.icess.ucsb.edu/drifter/realtime-SVP/ realtime page]) </h2>
+
The Google mapping code was adapted from the Iridium/Microstar realtime page at [http://www.icess.ucsb.edu/drifter/realtime/ http://www.icess.ucsb.edu/drifter/realtime/]
−
 
 
−
The Google mapping code was adapted from the Iridium/Microstar realtime page at http://www.icess.ucsb.edu/drifter/realtime/
 
−
 
 
−
The two main javascript files are ajax.js and map.js.
 
−
 
 
−
The javascript files are compressed using http://www.icess.ucsb.edu/drifter/realtime/JavascriptCompressor.html, so changes made to the xxx__UNCOMPRESSED__.js files won't show up on the page unless they are compressed and saved without the "__UNCOMPRESSED__" portion in the name, OR at the beginning of index.php those two types of files are commented/uncommented out as appropriate.
 
  
 +
The two main javascript files are ajax.js and map.js.<br>
 +
<br>
 +
The javascript files are compressed using http://www.icess.ucsb.edu/drifter/realtime/JavascriptCompressor.html, so changes made to the xxx__UNCOMPRESSED__.js files won't show up on the page unless they are compressed and saved without the "__UNCOMPRESSED__" portion in the name, '''OR*''' at the beginning of index.php those two types of files are commented/uncommented out as appropriate.<br>
  
 +
<br>
 
<h2> POTENTIAL PROBLEMS / TROUBLESHOOTING</h2>
 
<h2> POTENTIAL PROBLEMS / TROUBLESHOOTING</h2>
  
−
Prob: The data file (/home/locl/file_sharing/SVP_CALCOFI_LATEST_DATA.txt) is not getting saved/updated
+
Prob: The data file (/home/kirk/public_html/drifter/realtime-SVP/CALCOFI_SVP_LATEST_DATA.txt) is not getting saved/updated
 
<ul>
 
<ul>
 
<li>Is the dub-locl computer up?
 
<li>Is the dub-locl computer up?
−
<li>Is /mnt/locl mounted on dub-locl? It should come up automatically
+
<li>Is /home/kirk/ mounted on dub-locl? (It should come up automatically)
−
<li>Has 6 hours passed since the last time it ran? It only runs every 6 hours
+
<li>Has 6 hours passed since the last time it ran? It only runs about every 6 hours (see above), but they only update the file once per day at 8am Eastern.
−
<li>Try fetching the data yourself via telnet (do these commands at a terminal prompt):
+
<li>Try fetching the data yourself via ftp:
 
</ul>
 
</ul>
 
<pre>
 
<pre>
−
telnet datadist.argosinc.com
+
ftp://ftp.aoml.noaa.gov/phod/pub/daily_buoydata/7325/
−
hansen    (at the 'Username:' prompt)
+
login: anonymous
−
aomlus    (at the 'Password:' prompt)
+
pass: e_mail
−
prv,7325,ds,,93561-93587  (at the '/' prompt, to get the previous 9 day's worth of data for drifter #93561-93587 ) 
 
−
lo  (at the '/' prompt, to log out) 
 
 
</pre>
 
</pre>
−
 
+
Look for a file of today's data with the following name format:
−
NOTE: if you want to get data from a specific time period, use julian dates and GMT hours after the comma after 'ds'.
+
ddMMMyy.LOG-GZ, for example: 15JUL09.LOG-GZ   (i.e. for july 15th, 2009)<br>
−
For example:
+
<br>
−
<pre>
 
−
prv,7325,ds,174/12-175/10,93561-93587   (for June 23rd at 12Z through June 24th at 10Z)
 
−
</pre>
 
−
 
 
−
 
 
−
Prob: The data file (/home/locl/file_sharing/SVP_CALCOFI_LATEST_DATA.txt) shows no data
 
−
<ul>
 
−
<li>There probably aren't any drifters talking in the last 24hours (look for the line "No data available or authorization failure" in the file or try manually getting the data)
 
−
</ul>
 
  
 
----
 
----
Line 180: Line 168:
 
<br>
 
<br>
 
<b>NOTES:</b><br>
 
<b>NOTES:</b><br>
−
-Total bit count of all 7 lines must equal the checksum value or
+
-Total bit count (a very non trivial calculation) of all 7 lines must equal the checksum value or
 
message block is considered garbled and thus ignored.<br>
 
message block is considered garbled and thus ignored.<br>
 
<br>
 
<br>

Latest revision as of 22:50, 9 July 2009

CalCOFI SVP Drifter Realtime Webpage Wiki

NOTES AND THINGS TO KNOW ABOUT THE SVP REALTIME PAGE
Jun, 2009 Kirk

Webpage: SVP Realtime Page


There are three main parts needed to display the data (details below):
1. Fetching the data (using Python script running on dub-locl)
2. Parsing the data (using PHP script running on dub-locl)
3. Displaying the data (using Javascript running on your browser)


1. FETCHING THE DATA

On dub-locl.icess.ucsb.edu in /localhome/kirk/Scripts/CalCOFI_ARGOS_SVP_scripts_and_data/ there is a Python script 'fetch_calcofi_svp_data.py'.

  • There is essentially nothing to configure it, and it's well documented should you need to change anything. It is set to fetch via ftp from ftp.aoml.noaa.gov for the last 9 days worth of positions.
  • It may need to be updated with drifter numbers and temperature coefficients as more are sent drifting.
  • It is set to run about every 6 hours (05:30, 11:30, 17:30 & 20:30 Pacific, and they should post the data at 8am Eastern Time) via a "cron job" on the kirk account. To edit this job, as kirk, at the Terminal, type 'crontab -e' and use 'vi' style editing commands, or 'crontab -l' to just show it. (To become kirk, login as drift, then in a Terminal, type 'sudo su -' (then type the drift password) to become root, then 'su kirk' to become kirk.)
  • It fetches their file, archives it (to /localhome/kirk/Scripts/CalCOFI_ARGOS_SVP_scripts_and_data/DATA_ARCHIVE/), uncompresses it, and starts the parsing script (see next section).


2. PARSING THE DATA

This PHP script on dub-local.icess.ucsb.edu (called by the fetching script above) is /localhome/kirk/Scripts/CalCOFI_ARGOS_SVP_scripts_and_data/parse_calcofi_argos_svp_data.php and is pretty well documented. It breaks down the fairly cryptic GPS encoding and then saves it to a friendly format. It archives this file to /localhome/kirk/Scripts/CalCOFI_ARGOS_SVP_scripts_and_data/DATA_ARCHIVE/yyyymmdd_CALCOFI_SVP_LATEST_DATA.txt and copies this file to /home/kirk/public_html/drifter/realtime-SVP/CALCOFI_SVP_LATEST_DATA.txt so the website has access to it.

  • There is very little configuring it, and it's well documented should you need to change anything, but it does have temperature coefficients and a couple other possible configuration settings.

It outputs 1 record per position report in this format:

drift_no,YY-MM-DD,HH:MM:SS,milliseconds_since_Jan01_1970,orig_report_num,lat,lon,position_type(1=GPS,0=ARGOS),SST(C),batt(V),base_time,record_num

e.g. 

93561,2007-04-03,10:34:45,1175610885000,136,+26.6980,-081.9550,0,NaN,NaN,NaN,NaN    (an argos position)
93561,2007-04-03,10:33:37,1175610817000,137,+26.6979,-081.9557,1,NaN,12.4,2007-04-03_11:33:30,2   (a gps position)


3. DISPLAYING THE DATA (at realtime page or http://svp.drifterdata.com)

The Google mapping code was adapted from the Iridium/Microstar realtime page at http://www.icess.ucsb.edu/drifter/realtime/

The two main javascript files are ajax.js and map.js.

The javascript files are compressed using http://www.icess.ucsb.edu/drifter/realtime/JavascriptCompressor.html, so changes made to the xxx__UNCOMPRESSED__.js files won't show up on the page unless they are compressed and saved without the "__UNCOMPRESSED__" portion in the name, OR* at the beginning of index.php those two types of files are commented/uncommented out as appropriate.


POTENTIAL PROBLEMS / TROUBLESHOOTING

Prob: The data file (/home/kirk/public_html/drifter/realtime-SVP/CALCOFI_SVP_LATEST_DATA.txt) is not getting saved/updated

  • Is the dub-locl computer up?
  • Is /home/kirk/ mounted on dub-locl? (It should come up automatically)
  • Has 6 hours passed since the last time it ran? It only runs about every 6 hours (see above), but they only update the file once per day at 8am Eastern.
  • Try fetching the data yourself via ftp: 
ftp://ftp.aoml.noaa.gov/phod/pub/daily_buoydata/7325/
login: anonymous
pass: e_mail

Look for a file of today's data with the following name format: ddMMMyy.LOG-GZ, for example: 15JUL09.LOG-GZ (i.e. for july 15th, 2009)

INFO ABOUT TIME CALCULATION

Basically each data block is for three hourly reports, and data blocks can be repeated (I have no idea how that's determined), and there is a slight adjustment to the time based on "fix delay" (or "Length of Acquisition", which is usually on the order of <60 seconds).

For example (showing the second to last block in the existing file as I write this):

07325 93575  15 27 N 3 2009-06-25 20:18:59  34.416  240.156  0.000 401652653
      2009-06-25 20:21:22  3         12           62          678           94
                                     00           01       344156           01
                                1198452           01           04          689
                                     01       344155           01      1198448
                                     01           04          661           01
                                 344156           01      1198448           01
                                     04          587           00


This *first line* contains ARGOS only info: 

prog_num                     07325
drift_num                    93575
lines_in_sat_pass          15
num_sensors                27
sat_name                            N
resolution                          3
time            2009-06-25 20:18:59
xmit_lat                     34.416
xmit_lon                   240.156
xmit_alt                      0.000
xmit_freq              401652653


The next 7 lines contain 3, hourly GPS reports.

Line 1: (2009-06-25 20:21:22 3 12 62 678 94)

BASE Date/Time                  2009-06-25 20:21:22
# identical messages rcvd                       3
Drogue count over last half hour               12
Battery voltage = n * 0.2                 62   -> 12.4V
SST(C) = x * 0.043 - 5                   678   -> 24.154
Checksum                                       94


Line 2: (00 01 344156 01)

Msg ID                                   00   -> this is 1st message sent
Valid Pos #1 (1=good, 0=bad)             01   -> good
Lat #1 (ten-thousandths of degree)    344156  -> 34.4156
Lat #1 Ref (0 = South, 1 = North)        01   -> North


Line 3: (1198452 01 04 689)

Lon #1 (ten-thousandths of degree)     1198452 -> 119.8452
Lon #1 Ref (0 = East, 1 = West)           01   -> West
Pos #1 Length of Acq (seconds = n * 4)    04   -> 16(seconds)
SST #1 (C) = x * 0.043 - 5                689  -> 24.627


Line 4: (01 344155 01 1198448)

Valid Pos #2 (1=good, 0=bad)                01 -> good
Lat #2 (ten-thousandths of degree)     344155  -> 34.4155
Lat #2 Ref  (0 = South, 1 = North)         01  -> North
Lon #2 (ten-thousandths of degree)     1198448 -> 119.8448


Line 5: (01 04 661 01)

Lon #2 Ref (0 = East, 1 = West)         01   -> West
Pos #2 Length of Acq (seconds = n * 4)  04   -> 16(seconds)
SST #2  (C) = x * 0.043 - 5             661  -> 23.423
Valid Pos #3 (1=good, 0=bad)            01   -> good


Line 6: (344156 01 1198448 01)

Lat #3 (ten-thousandths of degree)   344156  -> 34.4156
Lat #3 Ref  (0 = South, 1 = North)        01 -> North
Lon #3 (ten-thousandths of degree)   1198448 -> 119.8448
Lon #3 Ref (0 = East, 1 = West)         01   -> West


Line 7: (04 587 00)

Pos #3 Length of Acq (seconds = n * 4)   04   -> 16(seconds)
SST #3  (C) = x * 0.043 - 5              587  -> 20.241
Null bits                      (only for padding, ignore)



NOTES:
-Total bit count (a very non trivial calculation) of all 7 lines must equal the checksum value or message block is considered garbled and thus ignored.

-Times of GPS reports are calculated thusly:

1) Take BASE time, subtract (Msg ID*3) [that is, if message has been repeated, it is at least 3 hours old -- I have a switch in my code (it's turned on for now) which will ignore messages with a message ID>0 because the original may be also present, and it's possible the duplicate isn't an exact duplicate and then you get slightly different reports at similar times]
2) Then, subtract 2 hours for Pos #3, and 1 hour for Pos #2 and 0 for Pos #1
3) Then subtract the associated small "Length of Acq"

Thus, the 2nd GPS time (i.e. #2) would be calculated thusly:

BASE time: 2009-06-25 20:21:22
minus 0 hours (because Msg ID is 0)
minus 1 hour (because it's Pos #2)
minus (pos #2 length of acq) 16 seconds

equals:  2009-06-25 19:21:06


An additional wacky thing is that due to repetition of the messages, or god knows what, the calculations of the reports you see aren't necessarily in chronological order. So, after doing all the calculations, I then sort them chronologically before outputting (within the parse_argos_data.php script), so that when showing the track lines, the points are in order.

Retrieved from "http://wiki-locl.eri.ucsb.edu/index.php?title=CalCOFI_Drifter_Realtime_Webpage_Info&oldid=15"