HOWTO Drifter Data Processing

Kirk's HOWTO on processing Drifter data

WARNING: THIS PAGE IS NOT DONE

Follow these steps to process 'raw' drifter data from pacificgyre to final posting on our website:

Notes:

• Once proficient, it takes about 20 min to process a typical day's deployment.
• Folders/files referenced here can be found on the ICESS network at /home/locl/users/kirk/ (hereafter referred to as '..../kirk/') in the data or user subfolders.
• You should copy the kirk folder contents to your local computer and run the scripts there instead of on the server.

Folder: .../kirk/user/0_Info

Purpose: Track drifter usage history

1. On deployment day: be sure to fill out .../kirk/user/0_Info/drifter_hist.xls (drifter history) with the performance of the drifters to track if there are any consistent problems (Note that this file is now out of date).
   
   

   Folder: .../kirk/user/1_get_data_from_web

   Purpose: Fetch the appropriate data from pacificgyre.com

2. Edit data_dates.m with the start/stop date/times to get.
   Notes:
   
   • Use the http://pacificgyre.com website (log: ohlmann, pass: drifterguy) if you need to get an estimation of when the drifters went in and came out of the water (look at in-water flag values), or go to http://www.icess.ucsb.edu/drifter/realtime/BETA.php and enter custom times.
   • Keep adding date/times to the bottom of this file (or in the appropriate "deployment type" area (e.g. SBCLS), and commented out), then copy these uncommented deployment times to the top of it to fetch the data. Note, when in the "deployment type" groups, I number the 'dates' sequentially, but when copying one or two of them to the top to fetch, you need to make sure those 'dates' numbering start at 1.
   • You don't have to be exact on the times (cleanup happens in following steps), just be sure you have a time at least before or at start and a time after or at ending deployment.
3. Run get_data.m
   • Be sure the IRIDIUM value in it is set correctly
   • Data will be saved in '..../kirk/data/Drifter_Anal/2_data/yyyymmdd' folder as a yyyymmdd_drifter_dat.log file.
   • Now, move that folder into a 'deployment type' folder (e.g. .../kirk/data/Drifter_Anal/2_data/SBCLS)
     
4. Go to the folder and make a copy of this yyyymmdd_drifter_dat.log file, naming the copy yyyymmdd_drifter_dat.ORIG.log
   Notes:
   
   • If IRIDIUM is set, you may want to check the settings of REMOVE_GPS_01 (to set GPS fix quality flags=1 (bad), to have in-water flag value of 0 so they effectively get skipped) and CONVERT_GPS_2_TO_2 (to set GPS fix quality flags=2 (2D), to have in-water flag value of 2 to aid in eliminating them in the next section) in the convert_Iridium_reports_to_Microstar_format.m file.
   • Data format (column numbers):

        0          1            2          3         4       5      6      7       8     9    10
   Device_Name,Device_Time,Network_Time,latitude,longitude,satVis,satTrk,recStat,secFix,RSSI,Subm_flag
   

   e.g.

   15723122,2008-10-20 16:19:43,2008-10-20 09:19:00,34.406943,-119.691051,13,9,57348,18,-65,0
   

   
   

   Folder: .../kirk/user/2_clean_up

   Purpose: To visually fix these two problems: the drifters sometimes get redeployed and so tracks need to be split, also the in-water flags are sometimes wrong (e.g. due to faulty sensor or water in bucket).

5. Edit top of clean_up_drifter_data.m with the correct pathroot, area and deploy_date, then start the script.
6. Proceed to set in-water flags to correct values and split tracks as necessary, when done, press 'SAVE' button, and then the 'EXIT' (which will not save the data) button.
   NOTES: (yup lots of em, but trust me it's not that hard, and much MUCH better than trying to edit the file by hand!)
   
   • Here's an overview of what the interface looks like:
     
   • Buttons along the bottom:
     • RESTART: will reload the file and erase any changes you made
     • EXIT: will quit the GUI (without saving)
     • Show point numbers: will reveal next to each point on the map its consecutive number
     • ←PREV: go to previous drifter
     • NEXT→: go to next drifter
     • SAVE: save your changes
   • Colors of locations:
     • Green: in-water flag is 1 (i.e. in-water)
     • Red: in-water flag is 0 (i.e. out-of-water, will be skipped in processing)
     • Pink: in-water flag is 2 (i.e. this report is flagged as bogus or questionable and will be skipped in processing)
   • Right side bar:
     • first column is consecutive number count for this drifter.
     • second column is first digit of drifter number (change this if/when drifter is redeployed)
     • third column is time of report (GMT)
     • fourth column is in-water flag value
     • scroll bar on right side is for, well, scrolling around different times
   • Zoom in and scroll around as you need to (you will probably do A LOT of this).
   • So, again the goal is to edit the boxes, in the 2nd and 4th columns on the right side, to change either the drifter number (for redeployments) and/or in-water flag (so only when it is actually drifting gets processed) for each point:
     • You can click in a box to edit it, and/or use the 'TAB' key to jump from one column to next, then down consecutive reports.
     • Ensure points where the drifter was out of the water have an in-water flag of 0 (becomes red point on map), or if it was drifting have the in-water flag of 1 (becomes green point on map). If any points need to be flagged as bogus (e.g. drifter looked like it was caught in kelp), or any other reason, use an in-water flag value of '2' (which will turn the point pink on map), and these points will be not processed.
     • To help you determine what index numbers a point is (to edit its drifter number and/or in-water flag), you can use the 'Show point numbers'. And/or you can change the in-water value (so that it changes color, highlighting it), just be sure to reset it to the correct value when done.
     • It's pretty obvious when a drifter has been picked up as its trajectory will deviate (sometimes not so dramatically if the boat doesn't move afterwards) from what it was doing.
     • Most drifters have a initial digit of '1' for drifter number, but some are '0' (see second column). When a drifter is redeployed, just increment this number and all subsequent numbers (e.g. change them all from a 1 to a 2). Sometimes a drifter is deployed more than once, so just continue this, so for the 2nd redeployment change the initial digit from 1 to 3. In doing this, when you save this file, you have essentially created 'new' (since their drifter numbers are now different) drifters, so the processing scripts don't think of them as one combined track.
     • WARNING!: when you press the NEXT→ button, if you've changed any drifter numbers, a "new" drifter will created (you will see at the top change from, for example "Drifter 3 of 13", to "Drifter 4 of 14" when you press it). Then, when you click the $larr;PREV button you will not see those points you reassigned, instead they will be, in this example, Drifter #14 at the end. So, if you've started changing drifter numbers for a drifter, just be sure you're done changing all of them for that drifter before pressing the NEXT button.
     • Here's an example of both in-water flags and redeploy numbers changing for one drifter. This initial shot is how it looked when loaded (after zooming in):
       
     • Here's how it looks after correctly changing in-water flags to '0' when it was out of the water, and changing the drifter numbers to '2' for when it was redeployed:
       
7. Again, when done cleaning up, be sure to press the SAVE button to write your changes out to file.
8. Take note of the maximum latitude and longitude limits (you'll want them for the plotting portion below).
9. Go to the folder containing the file you read-in and:
   1. Delete the file yyyymmdd_drifter_dat.log [you made a copy of it already, yyyymmdd_drifter_dat.ORIG.log]
   2. Rename the file yyyymmdd_drifter_dat.MOD.log file (output of the clean_up_drifter_data.m script), naming it yyyymmdd_drifter_dat.log (so its read in next step). The format of the "MOD" file which this script output is different than the "ORIG" one fetched.
10. The output format is:

    #1 drifter ID
    #2 year
    #3 month
    #4 day
    #5 hour
    #6 minute
    #7 second
    #8 lat
    #9 long
    #10 satelites availiable
    #11 satelites used
    #12 com flag
    #13 Secs to fix
    #14 Mobitex RSSI
    #15 in-water flag
    

    e.g.

    15723122 2008 10 20 16 19 43 34.406943 -119.691051 13 09 57348 018 -065 0
    

    
    

    folder: 3_process_drifter_data

    Purpose: This is the step which takes the data and calculates all the derived products (e.g. dz/dt, u & v velocities, etc.) and outputs a standard formatted file of good data (optionally in interpolated round 10 minute steps).

11. Edit top portion of process_drifter_data.m to set as appropriate:
    

    %SET **ONE** OF THESE!
    %THIS WILL DETERMINE WHICH BATHY DATASET GETS LOADED!
    SBCIS_SBCSS = 0;%UCSB inner shelf or small-scale CODAR area work (SBCIS or SBCSS)
    SBC_FULL = 1; %all of Santa Barbara Channel for Large Scale Deployments (SBCLS, SCROF or SBNS or MSD)
    UCSD = 0; %San Diego Imperial Beach (SDSS or TJROF)
    HBDP = 0; %Huntington Beach (HB) or Dana Point (DPIO) area
    SAR = 0; %Santa Ana River (SAROF)
    SF = 0; %San Francisco (SF)
    EFS = 0; %Eastern Florida Shelf (EFS)
    
    %if this is true, then doesn't matter what areas above are set, or GEODAS setting below. set to 1 for areas w/o bathy (e.g. PHILEX) 
    % data. will skip along/across shore, dz/dt and depth calculations.
    SKIP_ALONG_ACROSS = true 
    TEN_MIN_ROUNDING = true % before doing calculations, first round drifter times to 10 minute steps (will have "_TEN_MIN_" in filename)
    SMOOTH_DATA = false; % run smooth.m (2 point running mean or average) on the u and v velocities (and only these, i.e. not lat/lon) to reduce noise (usually NOT)
    

12. Run 'process_drifter_data.m' to process all drifters from a single deployment.
    Notes:
    
    • Will generate 'yyyymmdd_drifter#_summary.log' files (or 'yyyymmdd_drifter#_TEN_MIN_summary.log' if TEN_MIN_ROUNDING is set) with this format:

               #1 drifter ID
               #2 year
               #3 month
               #4 day
               #5 hour
               #6 minute
               #7 second
               #8 lat
               #9 lon
          *   #10 u vel (cm/s)
          *   #11 v vel (cm/s)
          *   #12 along velocity (cm/s)
          *   #13 across (offshore) velocity (cm/s)
          *   #14 depth (m)
          *   #15 dz/dt (cm/s)
          *   #16 serial time
              #17 satellites available
              #18 satellites used
              #19 com flag/Receiving status
              #20 Secs to fix
              #21 Mobitex RSSI
      
          * denotes a derived product
      

    • Iridium drifters will still have this format, but columns 17-21 will be filled with NaNs.

    folder: 3b_plot_drifter_data

    Purpose: Generate the 4 different types of plots: All tracks, all tracks animated, individual tracks, and summary plots.

13. Edit deployment_settings.m:
    1. Copy/paste an 'elseif' block from an existing one:
       e.g.

       elseif strcmp(folder,'20050307')
          z_rate = [-0.2 0.2];
          max_z = -65;
          lon_lim = [-117.4 -117.06666];
          lat_lim = [32.4 32.7];
          vel_rate = [-60 60];
       

    2. Note that I try to keep these grouped by project
    3. Edit the first line of the elseif to the date of the deployment
    4. Try to capture the lat/lon limits, or use what you noted from the clean_up section above
    5. We'll get to the others in a sec
14. Edit do_plots.m:
    1. Set pathname to your working folder.
    2. Set do_all and do_summaries to 1, and do_all_anim and do_individ_plots to 0.
    3. Set any other values in the "USER SETTINGS" section if needed, but not usually.
15. Run do_plots.m
    • There are now 5 summary plots and one yyyymmdd_all.png plot.
    • Use what you see to tweak the values on the deployment_settings.m file (note: if you didn't use bathy in the processing, it doesn't matter what the z_rate or max_z are set to)
    • When all looks good, you'll want to eventually set the do_all_anim and do_individ_plots to 1 to make those plots.

    
    

    
    

    
    
    
    
    
    
    
    
    

    LEFT OFF HERE!

    
    

    
    

16. Edit 'deployment_settings.m' to set max depth, dz/dt, lat/lon and velocity limits on the associated plots.
17. Edit top portion of 'do_plots.m', mostly, to set any of these:
    

    do_all = 0;   %0 = do plot, 1 = skip
    do_all_anim = 1;
      SHOW_CODAR_TRACKS = 0; %show interpolated 'simulated drifter' tracks based on CODAR data (uses '*_compare.mat' files) for anim plot
    do_individ_plots = 0;
    do_summaries = 0;
    SHOW_CODAR = 0; %draw the hourly codar vectors on the anim plot
    
    

    
    

    folder: 4_make_web

    Purpose: Generate a "stats" page (e.g. start/stop times, average speeds, average depth, number of drifters, etc.)

18. Run 'make_stats_page.m' (generates all 'stats.php' pages for a deployment area)
19. Copy this deployment's folder (e.g. 20050425) to root folder of deployment type (e.g. SDSS) on the website, but only the *.png, stats.php & *.gif files (and 'anim' subfolder).
20. Copy the '*_summary.log' files to the appropriate deployment type subfolder at E:\User\Carter\data\Drifter_Anal\2_data\AllData
21. Right-click on that folder you just copied those files to, and chose 'compress to *.zip'.
22. Move that generated *.zip folder to U:\public_html\drifter\data\data.
23. On web, update 'mod_date.txt' with correct (today's) modified date.
24. Check <a href="http://drifterdata.com">website</a> looks good.
25. Syncronize E:\User\Carter with /home/data318/kirk
26. Have a beer.