HOWTO Drifter Data Processing

Kirk's HOWTO on processing Drifter data

Follow these steps to process 'raw' drifter data from pacificgyre to final posting on our website:

Notes:

• Once proficient, it takes about 20 min to process a typical day's deployment.
• Folders/files referenced here can be found on the ICESS network at /home/locl/users/kirk/ (hereafter referred to as '..../kirk/') in the data or user subfolders.
• You need to copy the kirk folder contents to your local computer and run the scripts there instead of on the server.
• These instructions should work on PC, Linux or Mac, but are specific to Mac.

Folder: .../kirk/user/0_Info

Purpose: Track drifter usage history

1. On deployment day: be sure to fill out .../kirk/user/0_Info/drifter_hist.xls (drifter history) with the performance of the drifters to track if there are any consistent problems (Note that this file is now out of date).
   
   

   Folder: .../kirk/user/1_get_data_from_web

   Purpose: Fetch the appropriate data from pacificgyre.com

2. Edit data_dates.m with the start/stop date/times to get.
   Notes:
   
   • Use the http://pacificgyre.com website if you need to get an estimation of when the drifters went in and came out of the water (look at in-water flag values), or go to http://www.icess.ucsb.edu/drifter/realtime/BETA.php and enter custom times.
   • Keep adding date/times to the bottom of this file (or in the appropriate "deployment type" area (e.g. SBCLS), and commented out), then copy these deployment times to the top of the file, and uncomment. Note: under the "deployment type" groups, I number the 'dates' sequentially, but when copying one or two of them to the top to fetch, you need to make sure those 'dates' numbering are sequential and start at 1.
   • You don't have to be exact on the times (cleanup happens in following steps), just be sure you have a time at least at or before start of deployment and a time at or after ending deployment.
3. Run get_data.m
   • Be sure the IRIDIUM value in it is set correctly!
     If IRIDIUM is set, you may want to check the settings of REMOVE_GPS_01 (to set GPS fix quality flags=1 (bad), to have in-water flag value of 0 so they effectively get skipped) and CONVERT_GPS_2_TO_2 (to set GPS fix quality flags=2 (2D), to have in-water flag value of 2 to aid in eliminating them in the next section) in the convert_Iridium_reports_to_Microstar_format.m file.
   • Data will be saved in ..../kirk/data/Drifter_Anal/2_data/yyyymmdd folder as a yyyymmdd_drifter_dat.log file.
   • Now, move that folder into a 'deployment type' folder (e.g. .../kirk/data/Drifter_Anal/2_data/SBCLS, and will hereafter called the "working folder")
     
4. Go to this working folder and make a copy of the yyyymmdd_drifter_dat.log file, naming the copy yyyymmdd_drifter_dat.ORIG.log
   Notes:
   
   • Data format (column numbers):

        0          1            2          3         4       5      6      7       8     9    10
   Device_Name,Device_Time,Network_Time,latitude,longitude,satVis,satTrk,recStat,secFix,RSSI,Subm_flag
   

   e.g.

   15723122,2008-10-20 16:19:43,2008-10-20 09:19:00,34.406943,-119.691051,13,9,57348,18,-65,0
   

   
   

   Folder: .../kirk/user/2_clean_up

   Purpose: To visually fix these two problems: the drifters sometimes get redeployed and so tracks need to be split, also the in-water flags are sometimes wrong (e.g. due to faulty sensor or water in bucket).

5. Edit top of clean_up_drifter_data.m with the correct pathroot, area and deploy_date, then start the script.
6. Proceed to set in-water flags to correct values and split tracks as necessary, when done, press 'SAVE' button, and then the 'EXIT' (which will not save the data) button.
   NOTES: (yup lots of em, but trust me it's not that hard, and much MUCH better than trying to edit the file by hand!)
   
   • Here's an overview of what the interface looks like:
     
   • Buttons along the bottom:
     • RESTART: will reload the file and erase any changes you made
     • EXIT: will quit the GUI (without saving)
     • Show point numbers: will reveal next to each point on the map its consecutive number
     • ←PREV: go to previous drifter
     • NEXT→: go to next drifter
     • SAVE: save your changes
   • Colors of locations on map:
     • Green: in-water flag is 1 (i.e. in-water)
     • Red: in-water flag is 0 (i.e. out-of-water, will be skipped in processing)
     • Pink: in-water flag is 2 (i.e. this report is flagged as bogus or questionable and will be skipped in processing)
   • Right side bar:
     • 1st column = consecutive number count for this drifter.
     • 2nd column = first digit of drifter number (change this if/when drifter is redeployed)
     • 3rd column = time of report (GMT)
     • 4th column = in-water flag value
     • scroll bar on right side is for, well, scrolling around to the different times
   • Zoom in and scroll around as you need to (you will probably do A LOT of this).
   • So, again the goal is to edit the boxes, in the 2nd and 4th columns on the right side -- to change either the drifter number (for redeployments) and/or in-water flag (so only when it is actually drifting gets processed) for each point:
     • You can click in a box to edit it, and/or use the 'TAB' key to jump from one column to next and then down consecutive reports.
     • Ensure points where the drifter was out of the water have an in-water flag of 0 (becomes red point on map), or if it was drifting have the in-water flag of 1 (becomes green point on map). If any points need to be flagged as bogus (e.g. drifter looked like it was caught in kelp), or any other reason, use an in-water flag value of '2' (which will turn the point pink on map), and these points will be not processed.
     • To help you determine what index number a point is (to edit its drifter number and/or in-water flag), you can use the 'Show point numbers'. And/or you can change the in-water value (so that it changes color, highlighting it), just be sure to reset it to the correct value when done.
     • It's pretty obvious when a drifter has been picked up as its trajectory will deviate (sometimes not so dramatically if the boat doesn't move afterwards) from what it was doing.
     • Most drifters have a initial digit of '1' for drifter number, but some are '0' (see second column). When a drifter is redeployed, just increment this number and all subsequent numbers (e.g. change them all from a 1 to a 2). Sometimes a drifter is deployed more than once, so just continue this, so for the 2nd redeployment, change the initial digit from 1 to 3. In doing this, when you save this file, you have essentially created 'new' (since their drifter numbers are now different) drifters, so the processing scripts don't think of them as one combined track.
     • WARNING!: when you press the NEXT→ (or ←PREV) button(s), if you've changed any drifter numbers, a "new" drifter will created (you will see at the tilte above the map change from, for example "Drifter 3 of 13", to "Drifter 4 of 14" when you press it). Then, if you were to click the ←PREV button you will not see those points you reassigned, instead they will be, in this example, Drifter #14 at the end. So, if you've started changing drifter numbers for a drifter, just be sure you're done changing all of them for that drifter before pressing the NEXT button.
     • Here's an example of both in-water flags and redeploy numbers changing for one drifter. This initial shot is how it looked when loaded (after zooming in):
       
     • Here's how it looks after correctly changing in-water flags to '0' when it was out of the water, and changing the drifter numbers to '2' for when it was redeployed (the track which heads due north):
       
7. Again, when done cleaning up, be sure to press the SAVE button to write your changes out to file!
8. Take note of the maximum latitude and longitude limits (you'll want them for the plotting portion below), then press 'EXIT'.
9. Go to your working folder and:
   1. Delete the file yyyymmdd_drifter_dat.log [you made a copy of it already as yyyymmdd_drifter_dat.ORIG.log]
   2. Rename the file yyyymmdd_drifter_dat.MOD.log file (output of the clean_up_drifter_data.m script), to yyyymmdd_drifter_dat.log (so it's read in next step).
   3. The format of the "MOD" file which this script output (different than the "ORIG" one fetched) is:

      #1 drifter ID
      #2 year
      #3 month
      #4 day
      #5 hour
      #6 minute
      #7 second
      #8 lat
      #9 long
      #10 satelites availiable
      #11 satelites used
      #12 com flag
      #13 Secs to fix
      #14 Mobitex RSSI
      #15 in-water flag
      

      e.g.

      15723122 2008 10 20 16 19 43 34.406943 -119.691051 13 09 57348 018 -065 0
      

   Folder: .../kirk/user/3_process_drifter_data

   Purpose: This is the step which takes the data and calculates all the derived products (e.g. dz/dt, u & v velocities, etc.) and outputs a standard formatted file of good data (optionally in interpolated round 10 minute steps).

10. Edit top portion of process_drifter_data.m to set these as appropriate:
    

    %SET **ONE** OF THESE!
    %THIS WILL DETERMINE WHICH BATHY DATASET GETS LOADED!
    SBCIS_SBCSS = 0;%UCSB inner shelf or small-scale CODAR area work (SBCIS or SBCSS)
    SBC_FULL = 1; %all of Santa Barbara Channel for Large Scale Deployments (SBCLS, SCROF or SBNS or MSD)
    UCSD = 0; %San Diego Imperial Beach (SDSS or TJROF)
    HBDP = 0; %Huntington Beach (HB) or Dana Point (DPIO) area
    SAR = 0; %Santa Ana River (SAROF)
    SF = 0; %San Francisco (SF)
    EFS = 0; %Eastern Florida Shelf (EFS)
    
    %if this is true, then doesn't matter what areas above are set, or GEODAS setting below. set to 1 for areas w/o bathy (e.g. PHILEX) 
    % data. will skip along/across shore, dz/dt and depth calculations.
    SKIP_ALONG_ACROSS = true 
    TEN_MIN_ROUNDING = true % before doing calculations, first round drifter times to 10 minute steps (will have "_TEN_MIN_" in filename)
    SMOOTH_DATA = false; % run smooth.m (2 point running mean or average) on the u and v velocities (and only these, i.e. not lat/lon) to reduce noise (usually NOT)
    

11. Run process_drifter_data.m to process all drifters from a single deployment.
    Notes:
    
    • Will generate 'yyyymmdd_drifternum_summary.log' files (or 'yyyymmdd_drifternum_TEN_MIN_summary.log' if TEN_MIN_ROUNDING is set) with this format:

               #1 drifter ID
               #2 year
               #3 month
               #4 day
               #5 hour
               #6 minute
               #7 second
               #8 lat
               #9 lon
          *   #10 u vel (cm/s)
          *   #11 v vel (cm/s)
          *   #12 along velocity (cm/s)
          *   #13 across (offshore) velocity (cm/s)
          *   #14 depth (m)
          *   #15 dz/dt (cm/s)
          *   #16 serial time
              #17 satellites available
              #18 satellites used
              #19 com flag/Receiving status
              #20 Secs to fix
              #21 Mobitex RSSI
      
          * denotes a derived product
      

    • Iridium drifters will still have this format, but columns 17-21 will be filled with NaNs.

    Folder: .../kirk/user/3b_plot_drifter_data

    Purpose: Generate the 4 different types of plots: All tracks, all tracks animated, individual tracks, and summary plots.

    IMPORTANT!: See ImageMagick note below!
12. Edit deployment_settings.m:
    • Copy/paste an 'elseif' block from an existing one:
      e.g.

      elseif strcmp(folder,'20050307')
         z_rate = [-0.2 0.2];
         max_z = -65;
         lon_lim = [-117.4 -117.06666];
         lat_lim = [32.4 32.7];
         vel_rate = [-60 60];
      

    • Note that I try to keep these grouped by project
    • Edit the first line of the elseif to the date of the deployment
    • Try to capture the lat/lon limits, or use what you noted from the clean_up section above
    • We'll get to the others in a sec
13. Edit do_plots.m:
    1. Set pathname to your working folder.
    2. Set do_all and do_summaries to 1, and do_all_anim and do_individ_plots to 0.
    3. Set any other values in the "USER SETTINGS" section if needed, but not usually.
14. Run do_plots.m
    • There are now 5 summary plots and one yyyymmdd_all.png plot in your working folder.
    • Use what you see to tweak the values on the deployment_settings.m file (note: if you didn't use bathy in the processing, it doesn't matter what the z_rate or max_z are set to)
    • When all looks good, you'll want to eventually set the do_all_anim and do_individ_plots to 1 to make those plots.

    
    

    Folder: .../kirk/user/4_make_web

    Purpose: Generate "stats.php" pages (e.g. start/stop times, average speeds, average depth, number of drifters, etc.) for each deployment in a project [optional-ish]

15. Run make_stats_page.m and select the project folder (e.g. SBCLS).

    Website Setup

    Purpose: Copy the files so the website shows them, and other web maintenance.

16. Copy this deployment's folder (e.g. 20050425) to root folder of deployment type (e.g. SBCLS) on the website, which is found at /home/kirk/public_html/drifter/.
17. Copy all the *_summary.log files to the appropriate deployment type subfolder inside .../kirk/dataDrifter_Anal/2_data/AllData/
18. Right-click on that folder you just copied those files to, and chose 'Compress'.
19. Move that generated *.zip folder to inside /home/kirk/public_html/drifter/data/data/.
20. On web, update /home/kirk/public_html/drifter/mod_date.txt with today's date.
21. Check that the webpage looks good.
22. Have a beer.

---

IMPORTANT NOTE ABOUT IMAGEMAGICK

In order for the plotting codes to work, you must have ImageMagick (a free suite of powerful software for image manipulation from the command line) installed.
Here's instructions on installing on a Mac via MacPorts (since the scripts are currently setup for ImageMagick programs to be in that path):

1. Download the MacPorts*.dmg install file
2. Double-Click on the *.dmg file, then double-click on the MacPorts-xxx.pkg file to launch the Installer.
3. Follow the directions (mostly by clicking on 'Continue').
4. Download Porticus
5. Double click on the Porticus*.dmg file and drag-drop Porticus to your Applications folder
6. Launch Porticus
7. On the left side, select 'All Ports'
8. On the right side, scroll down to 'ImageMagick' (NOTE: it will be towards the top since packages starting with a capital letter come first)
9. Highlight it, then click the 'Install' button.
10. All MacPorts stuff gets installed to your /opt/local/ directory, with the programs in /opt/local/bin/.