HOWTO Drifter Data Processing

Kirk's HOWTO on processing Drifter data

WARNING: THIS PAGE IS NOT DONE

Follow these steps to process 'raw' drifter data from pacificgyre to final posting on our website:

Notes:

• Once proficient, it takes about 20 min to process a typical day's deployment.
• Folders/files referenced here can be found on the ICESS network at /home/locl/users/kirk/ (hereafter referred to as '..../kirk/') in the data or user subfolders.
• You should copy the kirk folder contents to your local computer and run the scripts there instead of on the server.

Folder: .../kirk/user/0_Info

1. On deployment day: be sure to fill out .../kirk/user/0_Info/drifter_hist.xls (drifter history) with the performance of the drifters to track if there are any consistent problems (Note that this file is now out of date).
   
   

   Folder: .../kirk/user/1_get_data_from_web

2. Edit data_dates.m with the start/stop date/times to get.
   Notes:
   
   • Use the http://pacificgyre.com website (log: ohlmann, pass: drifterguy) if you need to get an estimation of when the drifters went in and came out of the water (look at in-water flag values), or go to http://www.icess.ucsb.edu/drifter/realtime/BETA.php and enter custom times.
   • Keep adding date/times to the bottom of this file (or in the appropriate "deployment type" area (e.g. SBCLS), and commented out), then copy these uncommented deployment times to the top of it to fetch the data. Note, when in the "deployment type" groups, I number the 'dates' sequentially, but when copying one or two of them to the top to fetch, you need to make sure those 'dates' numbering start at 1.
   • You don't have to be exact on the times (cleanup happens in following steps), just be sure you have a time at least before or at start and a time after or at ending deployment.
3. Run get_data.m
   • Be sure the IRIDIUM value in it is set correctly
   • Data will be saved in '..../kirk/data/Drifter_Anal/2_data/yyyymmdd' folder as a yyyymmdd_drifter_dat.log file.
   • Now, move that folder into a 'deployment type' folder (e.g. .../kirk/data/Drifter_Anal/2_data/SBCLS)
     

   
   Notes:
   

   • If IRIDIUM is set, you may want to check the settings of REMOVE_GPS_01 and CONVERT_GPS_2_TO_2 in the convert_Iridium_reports_to_Microstar_format.m file.
   • Data format (column numbers):

         #1 drifter ID
         #2 year
         #3 month
         #4 day
         #5 hour
         #6 minute
         #7 second
         #8 lat
         #9 long
        #10 satelites availiable
        #11 satelites used
        #12 com flag
        #13 Secs to fix
        #14 Mobitex RSSI
        #15 in-water flag
   

   
   

   Folder: .../kirk/user/2_clean_up

4. Edit clean_up_drifter_data.m with the correct area and deploy_date at the top.
   
   
   
   
   
   
   
   
   
   LEFT OFF HERE!

   
   

   
   

   folder: 3_process_drifter_data

5. <input type="checkbox">Edit top portion of 'process_drifter_data.m'.
    Mostly just need to set one of these:
   

   <span class="mygeneral1-identifier">SBCIS_SBCSS </span><span class="mygeneral1-symbol">=</span><span class="mygeneral1-space"> </span><span class="mygeneral1-number">0</span><span class="mygeneral1-symbol">;</span><span class="mygeneral1-comment">%UCSB inner shelf or small-scale CODAR area work</span>
   <span class="mygeneral1-identifier">SBC_FULL </span><span class="mygeneral1-symbol">=</span><span class="mygeneral1-space"> </span><span class="mygeneral1-number">0</span><span class="mygeneral1-symbol">;</span><span class="mygeneral1-space"> </span><span class="mygeneral1-comment">%all of Santa Barbara Channel for Large Scale Deployments or SCR</span>
   <span class="mygeneral1-identifier">UCSD </span><span class="mygeneral1-symbol">=</span><span class="mygeneral1-space"> </span><span class="mygeneral1-number">1</span><span class="mygeneral1-symbol">;</span><span class="mygeneral1-space"> </span><span class="mygeneral1-comment">%San Diego Imperial Beach</span>
   <span class="mygeneral1-identifier">HBDP </span><span class="mygeneral1-symbol">=</span><span class="mygeneral1-space"> </span><span class="mygeneral1-number">0</span><span class="mygeneral1-symbol">;</span><span class="mygeneral1-space"> </span><span class="mygeneral1-comment">%Huntington Beach or Dana Point (DPIO) area</span>
   <span class="mygeneral1-identifier">SAR </span><span class="mygeneral1-symbol">=</span><span class="mygeneral1-space"> </span><span class="mygeneral1-number">0</span><span class="mygeneral1-symbol">;</span><span class="mygeneral1-space"> </span><span class="mygeneral1-comment">%Santa Ana River</span>
   
   <span class="mygeneral1-identifier">GEODAS </span><span class="mygeneral1-symbol">=</span><span class="mygeneral1-space"> </span><span class="mygeneral1-number">1</span><span class="mygeneral1-symbol">;</span><span class="mygeneral1-space"> </span><span class="mygeneral1-comment">%1=use bathy from GEODAS CD (manditory if UCSD or SAR is set above), 0=use original brian chinn sbc_bathy</span>
   

6. <input type="checkbox">Run 'process_drifter_data.m' to process all drifters from a single deployment
   Notes:
   
   
   • 'process_all.m' will go thru and run 'process_drifter_data.m' on all subfolders.
   • Will generate 'yyyymmdd_drifter#_summary.csv' files with this format:

            #1 drifter ID
            #2 year
            #3 month
            #4 day
            #5 hour
            #6 minute
            #7 second
            #8 lat
            #9 lon
       *   #10 u vel (cm/s)
       *   #11 v vel (cm/s)
       *   #12 along velocity (cm/s)
       *   #13 across (offshore) velocity (cm/s)
       *   #14 depth (m)
       *   #15 dz/dt (cm/s)
       *   #16 serial time
           #17 satellites available
           #18 satellites used
           #19 com flag/Receiving status
           #20 Secs to fix
           #21 Mobitex RSSI
   
       * denotes a derived product
   

   
   

   folder: 3b_plot_drifter_data

7. <input type="checkbox">Edit 'deployment_settings.m' to set max depth, dz/dt, lat/lon and velocity limits on the associated plots.
8. <input type="checkbox">Edit top portion of 'do_plots.m'.
      Mostly, to set any of these:
   

   <span class="mygeneral1-identifier">do_all </span><span class="mygeneral1-symbol">=</span><span class="mygeneral1-space"> </span><span class="mygeneral1-number">0</span><span class="mygeneral1-symbol">;</span><span class="mygeneral1-space">   </span><span class="mygeneral1-comment">%0 = do plot, 1 = skip</span>
   <span class="mygeneral1-identifier">do_all_anim </span><span class="mygeneral1-symbol">=</span><span class="mygeneral1-space"> </span><span class="mygeneral1-number">1</span><span class="mygeneral1-symbol">;</span>
   <span class="mygeneral1-space">  SHOW_CODAR_TRACKS </span><span class="mygeneral1-symbol">=</span><span class="mygeneral1-space"> </span><span class="mygeneral1-number">0</span><span class="mygeneral1-symbol">;</span><span class="mygeneral1-space"> </span><span class="mygeneral1-comment">%show interpolated 'simulated drifter' tracks based on CODAR data (uses '*_compare.mat' files) for anim plot</span>
   <span class="mygeneral1-identifier">do_individ_plots </span><span class="mygeneral1-symbol">=</span><span class="mygeneral1-space"> </span><span class="mygeneral1-number">0</span><span class="mygeneral1-symbol">;</span>
   <span class="mygeneral1-identifier">do_summaries </span><span class="mygeneral1-symbol">=</span><span class="mygeneral1-space"> </span><span class="mygeneral1-number">0</span><span class="mygeneral1-symbol">;</span>
   <span class="mygeneral1-identifier">SHOW_CODAR </span><span class="mygeneral1-symbol">=</span><span class="mygeneral1-space"> </span><span class="mygeneral1-number">0</span><span class="mygeneral1-symbol">;</span><span class="mygeneral1-space"> </span><span class="mygeneral1-comment">%draw the hourly codar vectors on the anim plot</span>
   

   
   

   folder: 4_make_web

   1. Run 'make_stats_page.m' (generates all 'stats.php' pages for a deployment area)
   2. Copy this deployment's folder (e.g. 20050425) to root folder of deployment type (e.g. SDSS) on the website, but only the *.png, stats.php & *.gif files (and 'anim' subfolder).
   3. Copy the '*_summary.log' files to the appropriate deployment type subfolder at E:\User\Carter\data\Drifter_Anal\2_data\AllData
   4. Right-click on that folder you just copied those files to, and chose 'compress to *.zip'.
   5. Move that generated *.zip folder to U:\public_html\drifter\data\data.
   6. On web, update 'mod_date.txt' with correct (today's) modified date.
   7. Check <a href="http://drifterdata.com">website</a> looks good.
   8. Syncronize E:\User\Carter with /home/data318/kirk
   9. Have a beer.