Kirk's HOWTO on processing Drifter data

WARNING: THIS PAGE IS NOT DONE

Follow these steps to process 'raw' drifter data from pacificgyre to final posting on our website:

Notes:

Once proficient, it takes about 20 min to process a typical day's deployment.
Folders/files referenced here can be found on the ICESS network at /home/locl/users/kirk/ (hereafter referred to as '..../kirk/') in the data or user subfolders.
You should copy the kirk folder contents to your local computer and run the scripts there instead of on the server.

Folder: .../kirk/user/0_Info

Purpose: Track drifter usage history

On deployment day: be sure to fill out .../kirk/user/0_Info/drifter_hist.xls (drifter history) with the performance of the drifters to track if there are any consistent problems (Note that this file is now out of date).

Folder: .../kirk/user/1_get_data_from_web

Purpose: Fetch the appropriate data from pacificgyre.com
Edit data_dates.m with the start/stop date/times to get.
Notes:
- Use the http://pacificgyre.com website (log: ohlmann, pass: drifterguy) if you need to get an estimation of when the drifters went in and came out of the water (look at in-water flag values), or go to http://www.icess.ucsb.edu/drifter/realtime/BETA.php and enter custom times.
- Keep adding date/times to the bottom of this file (or in the appropriate "deployment type" area (e.g. SBCLS), and commented out), then copy these uncommented deployment times to the top of it to fetch the data. Note, when in the "deployment type" groups, I number the 'dates' sequentially, but when copying one or two of them to the top to fetch, you need to make sure those 'dates' numbering start at 1.
- You don't have to be exact on the times (cleanup happens in following steps), just be sure you have a time at least before or at start and a time after or at ending deployment.
Run get_data.m
- Be sure the IRIDIUM value in it is set correctly
- Data will be saved in '..../kirk/data/Drifter_Anal/2_data/yyyymmdd' folder as a yyyymmdd_drifter_dat.log file.
- Now, move that folder into a 'deployment type' folder (e.g. .../kirk/data/Drifter_Anal/2_data/SBCLS)
Go to the folder and make a copy of this yyyymmdd_drifter_dat.log file, naming the copy yyyymmdd_drifter_dat.ORIG.log
Notes:
- If IRIDIUM is set, you may want to check the settings of REMOVE_GPS_01 (to set GPS fix quality flags=1 (bad), to have in-water flag value of 0 so they effectively get skipped) and CONVERT_GPS_2_TO_2 (to set GPS fix quality flags=2 (2D), to have in-water flag value of 2 to aid in eliminating them in the next section) in the convert_Iridium_reports_to_Microstar_format.m file.
- Data format (column numbers):
```
     0          1            2          3         4       5      6      7       8     9    10
Device_Name,Device_Time,Network_Time,latitude,longitude,satVis,satTrk,recStat,secFix,RSSI,Subm_flag
```
e.g.
```
15723122,2008-10-20 16:19:43,2008-10-20 09:19:00,34.406943,-119.691051,13,9,57348,18,-65,0
```
Folder: .../kirk/user/2_clean_up

Purpose: To visually fix these two problems: the drifters sometimes get redeployed and so tracks need to be split, also the in-water flags are sometimes wrong (e.g. due to faulty sensor or water in bucket).
Edit top of clean_up_drifter_data.m with the correct pathroot, area and deploy_date, then start the script.
Proceed to set in-water flags to correct values and split tracks as necessary, when done, press 'SAVE' button, and then the 'EXIT' (which will not save the data) button.
NOTES: (yup lots of em, but trust me it's not that hard, and much MUCH better than trying to edit the file by hand!)
- Here's an overview of what the interface looks like:
- Buttons along the bottom:
  - RESTART: will reload the file and erase any changes you made
  - EXIT: will quit the GUI (without saving)
  - Show point numbers: will reveal next to each point on the map its consecutive number
  - ←PREV: go to previous drifter
  - NEXT→: go to next drifter
  - SAVE: save your changes
- Colors of locations:
  - Green: in-water flag is 1 (i.e. in-water)
  - Red: in-water flag is 0 (i.e. out-of-water, will be skipped in processing)
  - Pink: in-water flag is 2 (i.e. this report is flagged as bogus or questionable and will be skipped in processing)
- Right side bar:
  - first column is consecutive number count for this drifter.
  - second column is first digit of drifter number (change this if/when drifter is redeployed)
  - third column is time of report (GMT)
  - fourth column is in-water flag value
  - scroll bar on right side is for, well, scrolling around different times
- Zoom in and scroll around as you need to (you will probably do A LOT of this).
- So, again the goal is to edit the boxes, in the 2nd and 4th columns on the right side, to change either the drifter number (for redeployments) and/or in-water flag (so only when it is actually drifting gets processed) for each point:
  - You can click in a box to edit it, and/or use the 'TAB' key to jump from one column to next, then down consecutive reports.
  - Ensure points where the drifter was out of the water have an in-water flag of 0 (becomes red point on map), or if it was drifting have the in-water flag of 1 (becomes green point on map). If any points need to be flagged as bogus (e.g. drifter looked like it was caught in kelp), or any other reason, use an in-water flag value of '2' (which will turn the point pink on map), and these points will be not processed.
  - To help you determine what index numbers a point is (to edit its drifter number and/or in-water flag), you can use the 'Show point numbers'. And/or you can change the in-water value (so that it changes color, highlighting it), just be sure to reset it to the correct value when done.
  - It's pretty obvious when a drifter has been picked up as its trajectory will deviate (sometimes not so dramatically if the boat doesn't move afterwards) from what it was doing.
  - Most drifters have a initial digit of '1' for drifter number, but some are '0' (see second column). When a drifter is redeployed, just increment this number and all subsequent numbers (e.g. change them all from a 1 to a 2). Sometimes a drifter is deployed more than once, so just continue this, so for the 2nd redeployment change the initial digit from 1 to 3. In doing this, when you save this file, you have essentially created 'new' (since their drifter numbers are now different) drifters, so the processing scripts don't think of them as one combined track.
  - WARNING!: when you press the NEXT→ button, if you've changed any drifter numbers, a "new" drifter will created (you will see at the top change from, for example "Drifter 3 of 13", to "Drifter 4 of 14" when you press it). Then, when you click the $larr;PREV button you will not see those points you reassigned, instead they will be, in this example, Drifter #14 at the end. So, if you've started changing drifter numbers for a drifter, just be sure you're done changing all of them for that drifter before pressing the NEXT button.
  - Here's an example of both in-water flags and redeploy numbers changing for one drifter. This initial shot is how it looked when loaded (after zooming in):
  - Here's how it looks after correctly changing in-water flags to '0' when it was out of the water, and changing the drifter numbers to '2' for when it was redeployed:
LEFT OFF HERE!
Go to the folder and:
1. Delete the file yyyymmdd_drifter_dat.log [you made a copy of it already, yyyymmdd_drifter_dat.ORIG.log]
2. Rename the file yyyymmdd_drifter_dat.MOD.log file, naming it yyyymmdd_drifter_dat.log. The format of the "MOD" file which this script output is different than the "ORIG" one fetched.
The output format is:
```
#1 drifter ID
#2 year
#3 month
#4 day
#5 hour
#6 minute
#7 second
#8 lat
#9 long
#10 satelites availiable
#11 satelites used
#12 com flag
#13 Secs to fix
#14 Mobitex RSSI
#15 in-water flag
```
e.g.
```
15723122 2008 10 20 16 19 43 34.406943 -119.691051 13 09 57348 018 -065 0
```
folder: 3_process_drifter_data

Purpose: This is the step which takes the data and calculates all the derived products (e.g. dz/dt, u & v velocities, etc.) and outputs a standard formatted file of good data (optionally in interpolated round 10 minute steps).

Edit top portion of 'process_drifter_data.m'.
Mostly just need to set one of these:

SBCIS_SBCSS = 0;%UCSB inner shelf or small-scale CODAR area work
SBC_FULL = 0; %all of Santa Barbara Channel for Large Scale Deployments or SCR
UCSD = 1; %San Diego Imperial Beach
HBDP = 0; %Huntington Beach or Dana Point (DPIO) area
SAR = 0; %Santa Ana River

GEODAS = 1; %1=use bathy from GEODAS CD (manditory if UCSD or SAR is set above), 0=use original brian chinn sbc_bathy

Run 'process_drifter_data.m' to process all drifters from a single deployment
Notes:

'process_all.m' will go thru and run 'process_drifter_data.m' on all subfolders.
Will generate 'yyyymmdd_drifter#_summary.csv' files with this format:

         #1 drifter ID
         #2 year
         #3 month
         #4 day
         #5 hour
         #6 minute
         #7 second
         #8 lat
         #9 lon
    *   #10 u vel (cm/s)
    *   #11 v vel (cm/s)
    *   #12 along velocity (cm/s)
    *   #13 across (offshore) velocity (cm/s)
    *   #14 depth (m)
    *   #15 dz/dt (cm/s)
    *   #16 serial time
        #17 satellites available
        #18 satellites used
        #19 com flag/Receiving status
        #20 Secs to fix
        #21 Mobitex RSSI

    * denotes a derived product

folder: 3b_plot_drifter_data

Purpose: Generate the 4 different types of plots: All tracks, all tracks animated, individual tracks, and summary plots.

Edit 'deployment_settings.m' to set max depth, dz/dt, lat/lon and velocity limits on the associated plots.

Edit top portion of 'do_plots.m', mostly, to set any of these:

do_all = 0;   %0 = do plot, 1 = skip
do_all_anim = 1;
  SHOW_CODAR_TRACKS = 0; %show interpolated 'simulated drifter' tracks based on CODAR data (uses '*_compare.mat' files) for anim plot
do_individ_plots = 0;
do_summaries = 0;
SHOW_CODAR = 0; %draw the hourly codar vectors on the anim plot

folder: 4_make_web

Purpose: Generate a "stats" page (e.g. start/stop times, average speeds, average depth, number of drifters, etc.)

Run 'make_stats_page.m' (generates all 'stats.php' pages for a deployment area)
Copy this deployment's folder (e.g. 20050425) to root folder of deployment type (e.g. SDSS) on the website, but only the *.png, stats.php & *.gif files (and 'anim' subfolder).
Copy the '*_summary.log' files to the appropriate deployment type subfolder at E:\User\Carter\data\Drifter_Anal\2_data\AllData
Right-click on that folder you just copied those files to, and chose 'compress to *.zip'.
Move that generated *.zip folder to U:\public_html\drifter\data\data.
On web, update 'mod_date.txt' with correct (today's) modified date.
Check <a href="http://drifterdata.com">website</a> looks good.
Syncronize E:\User\Carter with /home/data318/kirk
Have a beer.

HOWTO Drifter Data Processing

Contents

Kirk's HOWTO on processing Drifter data

Follow these steps to process 'raw' drifter data from pacificgyre to final posting on our website:

Folder: .../kirk/user/0_Info

Folder: .../kirk/user/1_get_data_from_web

Folder: .../kirk/user/2_clean_up

folder: 3_process_drifter_data

folder: 3b_plot_drifter_data

folder: 4_make_web

Navigation menu

Page actions

Page actions

Personal tools

Navigation

Search

Tools