HOWTO Drifter Data Processing

Kirk's HOWTO on processing Drifter data

WARNING: THIS PAGE IS NOT DONE

Follow these steps to process 'raw' drifter data from pacificgyre to final posting on our website:

Notes:

• Once proficient, it takes about 20 min to process a typical day's deployment.
• Folders/files referenced here can be found on the ICESS network at /home/locl/users/kirk/ (hereafter referred to as '..../kirk/') in the data or user subfolders.
• You should copy the kirk folder contents to your local computer and run the scripts there instead of on the server.

Folder: .../kirk/user/0_Info

Purpose: Track drifter usage history

1. On deployment day: be sure to fill out .../kirk/user/0_Info/drifter_hist.xls (drifter history) with the performance of the drifters to track if there are any consistent problems (Note that this file is now out of date).
   
   

   Folder: .../kirk/user/1_get_data_from_web

   Purpose: Fetch the appropriate data from pacificgyre.com

2. Edit data_dates.m with the start/stop date/times to get.
   Notes:
   
   • Use the http://pacificgyre.com website (log: ohlmann, pass: drifterguy) if you need to get an estimation of when the drifters went in and came out of the water (look at in-water flag values), or go to http://www.icess.ucsb.edu/drifter/realtime/BETA.php and enter custom times.
   • Keep adding date/times to the bottom of this file (or in the appropriate "deployment type" area (e.g. SBCLS), and commented out), then copy these uncommented deployment times to the top of it to fetch the data. Note, when in the "deployment type" groups, I number the 'dates' sequentially, but when copying one or two of them to the top to fetch, you need to make sure those 'dates' numbering start at 1.
   • You don't have to be exact on the times (cleanup happens in following steps), just be sure you have a time at least before or at start and a time after or at ending deployment.
3. Run get_data.m
   • Be sure the IRIDIUM value in it is set correctly
   • Data will be saved in '..../kirk/data/Drifter_Anal/2_data/yyyymmdd' folder as a yyyymmdd_drifter_dat.log file.
   • Now, move that folder into a 'deployment type' folder (e.g. .../kirk/data/Drifter_Anal/2_data/SBCLS)
     
4. Go to the folder and make a copy of this yyyymmdd_drifter_dat.log file, naming the copy yyyymmdd_drifter_dat.ORIG.log
   Notes:
   
   • If IRIDIUM is set, you may want to check the settings of REMOVE_GPS_01 (to set GPS fix quality flags=1 (bad), to have in-water flag value of 0 so they effectively get skipped) and CONVERT_GPS_2_TO_2 (to set GPS fix quality flags=2 (2D), to have in-water flag value of 2 to aid in eliminating them in the next section) in the convert_Iridium_reports_to_Microstar_format.m file.
   • Data format (column numbers):

        0          1            2          3         4       5      6      7       8     9    10
   Device_Name,Device_Time,Network_Time,latitude,longitude,satVis,satTrk,recStat,secFix,RSSI,Subm_flag
   

   e.g.

   15723122,2008-10-20 16:19:43,2008-10-20 09:19:00,34.406943,-119.691051,13,9,57348,18,-65,0
   

   
   

   Folder: .../kirk/user/2_clean_up

   Purpose: To visually fix these two problems: the drifters sometimes get redeployed and so tracks need to be split, also the in-water flags are sometimes wrong (e.g. due to faulty sensor or water in bucket).

5. Edit top of clean_up_drifter_data.m with the correct pathroot, area and deploy_date, then start the script.
6. Proceed to set in-water flags to correct values and split tracks as necessary, when done, press 'SAVE' button, and then the 'EXIT' (which will not save the data) button. NOTES: (yup lots of em)
   
   • Here's an overview of what the interface looks like:
     File:Clean up overview.png
   • Buttons along the bottom:
     • RESTART: will reload the file and erase any changes you made
     • EXIT: will quit the GUI (without saving)
     • Show point numbers: will reveal next to each point its consecutive number
     • ←PREV: go to previous drifter
     • NEXT→: go to next drifter
     • SAVE: save your changes
   • Colors of locations:
     • Green: in-water flag is 1 (i.e. in-water)
     • Red: in-water flag is 0 (i.e. out-of-water, will be skipped in processing)
     • Pink: in-water flag is 2 (i.e. this report is flagged as bogus or questionable and will be skipped in processing)
   • Right side bar:
     • first column is consecutive number count for this drifter.
     • second column is first digit of drifter number (change this if/when drifter is redeployed)
     • third column is time of report (GMT)
     • fourth column is in-water flag value
     • scroll bar on right side is for, well, scrolling around different times
   • Zoom in and scroll around as you need to.
   • So, again the goal is to edit the boxes, in the 2nd and 4th columns on the right side, to change either the drifter number (for redeployments) and/or in-water flag (so only when it is actually drifting gets processed) for each point:
     • You can click in a box to edit it, and/or use the 'TAB' key to jump from one column to next, then down consecutive reports.
     • Ensure points where the drifter was out of the water have an in-water flag of 0 (becomes red point on map), or if it was drifting have the in-water flag of 1 (becomes green point on map). If any points need to be flagged as bogus (e.g. drifter looked like it was caught in kelp), or any other reason, use an in-water flag value of '2' (which will turn the point pink on map), and they will be not processed.
     • To help you determine what index numbers a point is (to edit its drifter number and/or in-water flag), you can use the 'Show point numbers'. And/or you can change the in-water value (so that it changes color, highlighting it), just be sure to reset it to the correct value when done.

   
   

   
   

   
   
   
   
   
   
   
   
   

   LEFT OFF HERE!

   
   

   
   

7. Go to the folder and:
   1. Delete the file yyyymmdd_drifter_dat.log [you made a copy of it already, yyyymmdd_drifter_dat.ORIG.log]
   2. Rename the file yyyymmdd_drifter_dat.MOD.log file, naming it yyyymmdd_drifter_dat.log. The format of the "MOD" file which this script output is different than the "ORIG" one fetched.
8. The output format is:

   #1 drifter ID
   #2 year
   #3 month
   #4 day
   #5 hour
   #6 minute
   #7 second
   #8 lat
   #9 long
   #10 satelites availiable
   #11 satelites used
   #12 com flag
   #13 Secs to fix
   #14 Mobitex RSSI
   #15 in-water flag
   

   e.g.

   15723122 2008 10 20 16 19 43 34.406943 -119.691051 13 09 57348 018 -065 0
   

   
   

   folder: 3_process_drifter_data

   Purpose: This is the step which takes the data and calculates all the derived products (e.g. dz/dt, u & v velocities, etc.) and outputs a standard formatted file of good data (optionally in interpolated round 10 minute steps).

9. Edit top portion of 'process_drifter_data.m'.
    Mostly just need to set one of these:
   

   SBCIS_SBCSS = 0;%UCSB inner shelf or small-scale CODAR area work
   SBC_FULL = 0; %all of Santa Barbara Channel for Large Scale Deployments or SCR
   UCSD = 1; %San Diego Imperial Beach
   HBDP = 0; %Huntington Beach or Dana Point (DPIO) area
   SAR = 0; %Santa Ana River
   
   GEODAS = 1; %1=use bathy from GEODAS CD (manditory if UCSD or SAR is set above), 0=use original brian chinn sbc_bathy
   

10. Run 'process_drifter_data.m' to process all drifters from a single deployment
    Notes:
    
    
    • 'process_all.m' will go thru and run 'process_drifter_data.m' on all subfolders.
    • Will generate 'yyyymmdd_drifter#_summary.csv' files with this format:

             #1 drifter ID
             #2 year
             #3 month
             #4 day
             #5 hour
             #6 minute
             #7 second
             #8 lat
             #9 lon
        *   #10 u vel (cm/s)
        *   #11 v vel (cm/s)
        *   #12 along velocity (cm/s)
        *   #13 across (offshore) velocity (cm/s)
        *   #14 depth (m)
        *   #15 dz/dt (cm/s)
        *   #16 serial time
            #17 satellites available
            #18 satellites used
            #19 com flag/Receiving status
            #20 Secs to fix
            #21 Mobitex RSSI
    
        * denotes a derived product
    

    
    

    folder: 3b_plot_drifter_data

    Purpose: Generate the 4 different types of plots: All tracks, all tracks animated, individual tracks, and summary plots.

11. Edit 'deployment_settings.m' to set max depth, dz/dt, lat/lon and velocity limits on the associated plots.
12. Edit top portion of 'do_plots.m', mostly, to set any of these:
    

    do_all = 0;   %0 = do plot, 1 = skip
    do_all_anim = 1;
      SHOW_CODAR_TRACKS = 0; %show interpolated 'simulated drifter' tracks based on CODAR data (uses '*_compare.mat' files) for anim plot
    do_individ_plots = 0;
    do_summaries = 0;
    SHOW_CODAR = 0; %draw the hourly codar vectors on the anim plot
    
    

    
    

    folder: 4_make_web

    Purpose: Generate a "stats" page (e.g. start/stop times, average speeds, average depth, number of drifters, etc.)

13. Run 'make_stats_page.m' (generates all 'stats.php' pages for a deployment area)
14. Copy this deployment's folder (e.g. 20050425) to root folder of deployment type (e.g. SDSS) on the website, but only the *.png, stats.php & *.gif files (and 'anim' subfolder).
15. Copy the '*_summary.log' files to the appropriate deployment type subfolder at E:\User\Carter\data\Drifter_Anal\2_data\AllData
16. Right-click on that folder you just copied those files to, and chose 'compress to *.zip'.
17. Move that generated *.zip folder to U:\public_html\drifter\data\data.
18. On web, update 'mod_date.txt' with correct (today's) modified date.
19. Check <a href="http://drifterdata.com">website</a> looks good.
20. Syncronize E:\User\Carter with /home/data318/kirk
21. Have a beer.