HOWTO Drifter Data Processing

Kirk's HOWTO on processing Drifter data

WARNING: THIS PAGE IS NOT DONE

Follow these steps to process 'raw' drifter data from pacificgyre to final posting on our website:

Notes:

• Once proficient, it takes about 20 min to process a typical day's deployment.
• Folders/files referenced here can be found on the ICESS network at /home/locl/users/kirk/ (hereafter referred to as '..../kirk/') in the data or user subfolders.
• You should copy the kirk folder contents to your local computer and run the scripts there instead of on the server.

Folder: .../kirk/user/0_Info

Purpose: Track drifter usage history

1. On deployment day: be sure to fill out .../kirk/user/0_Info/drifter_hist.xls (drifter history) with the performance of the drifters to track if there are any consistent problems (Note that this file is now out of date).
   
   

   Folder: .../kirk/user/1_get_data_from_web

   Purpose: Fetch the appropriate data from pacificgyre.com

2. Edit data_dates.m with the start/stop date/times to get.
   Notes:
   
   • Use the http://pacificgyre.com website (log: ohlmann, pass: drifterguy) if you need to get an estimation of when the drifters went in and came out of the water (look at in-water flag values), or go to http://www.icess.ucsb.edu/drifter/realtime/BETA.php and enter custom times.
   • Keep adding date/times to the bottom of this file (or in the appropriate "deployment type" area (e.g. SBCLS), and commented out), then copy these uncommented deployment times to the top of it to fetch the data. Note, when in the "deployment type" groups, I number the 'dates' sequentially, but when copying one or two of them to the top to fetch, you need to make sure those 'dates' numbering start at 1.
   • You don't have to be exact on the times (cleanup happens in following steps), just be sure you have a time at least before or at start and a time after or at ending deployment.
3. Run get_data.m
   • Be sure the IRIDIUM value in it is set correctly
   • Data will be saved in '..../kirk/data/Drifter_Anal/2_data/yyyymmdd' folder as a yyyymmdd_drifter_dat.log file.
   • Now, move that folder into a 'deployment type' folder (e.g. .../kirk/data/Drifter_Anal/2_data/SBCLS)
     

   
   Notes:
   

   • If IRIDIUM is set, you may want to check the settings of REMOVE_GPS_01 and CONVERT_GPS_2_TO_2 in the convert_Iridium_reports_to_Microstar_format.m file.
   • Data format (column numbers):

         #1 drifter ID
         #2 year
         #3 month
         #4 day
         #5 hour
         #6 minute
         #7 second
         #8 lat
         #9 long
        #10 satelites availiable
        #11 satelites used
        #12 com flag
        #13 Secs to fix
        #14 Mobitex RSSI
        #15 in-water flag
   

   
   

   Folder: .../kirk/user/2_clean_up

   Purpose: Two things: the drifters sometimes get redeployed and so tracks need to be split, also the in-water flags are sometimes wrong (e.g. due to faulty sensor or water in bucket). This step is to fix both of these problems.

4. Edit clean_up_drifter_data.m with the correct area and deploy_date at the top.
   
   
   
   
   
   
   
   
   
   LEFT OFF HERE!

   
   

   
   

   folder: 3_process_drifter_data

   Purpose: This is the step which takes the data and calculates all the derived products (e.g. dz/dt, u & v velocities, etc.) and outputs a standard formatted file of good data (optionally in interpolated round 10 minute steps).

5. Edit top portion of 'process_drifter_data.m'.
    Mostly just need to set one of these:
   

   SBCIS_SBCSS = 0;%UCSB inner shelf or small-scale CODAR area work
   SBC_FULL = 0; %all of Santa Barbara Channel for Large Scale Deployments or SCR
   UCSD = 1; %San Diego Imperial Beach
   HBDP = 0; %Huntington Beach or Dana Point (DPIO) area
   SAR = 0; %Santa Ana River
   
   GEODAS = 1; %1=use bathy from GEODAS CD (manditory if UCSD or SAR is set above), 0=use original brian chinn sbc_bathy
   

6. Run 'process_drifter_data.m' to process all drifters from a single deployment
   Notes:
   
   
   • 'process_all.m' will go thru and run 'process_drifter_data.m' on all subfolders.
   • Will generate 'yyyymmdd_drifter#_summary.csv' files with this format:

            #1 drifter ID
            #2 year
            #3 month
            #4 day
            #5 hour
            #6 minute
            #7 second
            #8 lat
            #9 lon
       *   #10 u vel (cm/s)
       *   #11 v vel (cm/s)
       *   #12 along velocity (cm/s)
       *   #13 across (offshore) velocity (cm/s)
       *   #14 depth (m)
       *   #15 dz/dt (cm/s)
       *   #16 serial time
           #17 satellites available
           #18 satellites used
           #19 com flag/Receiving status
           #20 Secs to fix
           #21 Mobitex RSSI
   
       * denotes a derived product
   

   
   

   folder: 3b_plot_drifter_data

   Purpose: Generate the 4 different types of plots: All tracks, all tracks animated, individual tracks, and summary plots.

7. Edit 'deployment_settings.m' to set max depth, dz/dt, lat/lon and velocity limits on the associated plots.
8. Edit top portion of 'do_plots.m', mostly, to set any of these:
   

   do_all = 0;   %0 = do plot, 1 = skip
   do_all_anim = 1;
     SHOW_CODAR_TRACKS = 0; %show interpolated 'simulated drifter' tracks based on CODAR data (uses '*_compare.mat' files) for anim plot
   do_individ_plots = 0;
   do_summaries = 0;
   SHOW_CODAR = 0; %draw the hourly codar vectors on the anim plot
   
   

   
   

   folder: 4_make_web

   Purpose: Generate a "stats" page (e.g. start/stop times, average speeds, average depth, number of drifters, etc.)

9. Run 'make_stats_page.m' (generates all 'stats.php' pages for a deployment area)
10. Copy this deployment's folder (e.g. 20050425) to root folder of deployment type (e.g. SDSS) on the website, but only the *.png, stats.php & *.gif files (and 'anim' subfolder).
11. Copy the '*_summary.log' files to the appropriate deployment type subfolder at E:\User\Carter\data\Drifter_Anal\2_data\AllData
12. Right-click on that folder you just copied those files to, and chose 'compress to *.zip'.
13. Move that generated *.zip folder to U:\public_html\drifter\data\data.
14. On web, update 'mod_date.txt' with correct (today's) modified date.
15. Check <a href="http://drifterdata.com">website</a> looks good.
16. Syncronize E:\User\Carter with /home/data318/kirk
17. Have a beer.