Unravelling


Unravelling

An unravel script does different steps for us to unpack a data file, and extract it to OMG format. An unravel script is handy, but everyone who uses an unravel script should still know what it does. I will explain the basic steps for unravelling a Simrad .all file. Be advised that each dataset might require some additional commands or tweaking. It is just a skeleton of the complete process.

The raw data is stored in binary files which have a certain (proprietary) structure. The OMG commands are able to read these raw data files and extract the required information to an OMG-format. So in fact the raw data gets reorganized to our desires. The results are several other files (the raw data file remains untouched off course) of which the .merged file will be used later in the pipeline.

First a proper directory structure is created. You should store raw data in a folder called raw/sub_dir/. You will have create the raw/sub_dir manually, use the mkdir command to make a new folder. Then folders merged/sub_dir, decnav/sub_dir and if you do some things with backscatter, ss/sub_dir are created by the unravel script. The logic behind this folder comes clear in the following steps.

It is the omg convention to name the sub directories to the Julian day. You get the Julian day simply by typing julian yyyy-mm-dd on the command line. Finally it will look as follows:

projectname

raw

JDXXX

merged

JDXXX

decnav

JDXXX

ss

JDXXX


Simrad - Unzipping

Often the data raw data is compressed. It’s unzipped by:

gunzip raw_data_file.all.gz




During "real" field trips, for example the research work of the Ocean Mapping Group in the Arctic, we use an unravel script to go from .all files to .merged files. Nevertheless I would encourage you to do it once by hand. Every step in an unravel script is explained in the paragraph manual here below. You can view and tweak the unravel script by viewing it in vi. Type in commandline: vi new_unravel


Unravel Script

Downloadable at Johnnie's site.. link



Manual

Simrad has a proprietary file format, the .all files. Unravelling is, unlike Reson data, done with just one command, the RT (read telegram) command:

RT –v4 –force_swap –em3002 –prefix raw/sub_dir –suffix suffix.all –line linenumber.all –out merged/subdir/

-v4

version 4, always use this

-force_swap


Linux and windows computer have different byte orders. The force_swap command takes this into account. Although the latest versions of RT are automated I believe this command isn’t really necessary anymore, still wouldn’t hurt using it.

-em3002

specify your sonar; em3002 em1002, elac2 …

-prefix

Give it the Julian day

The operator can configure the raw data filenames, in most cases they look like this:

number_date_time_ship.all --> 0001_20060714_142506_Titanic.all

The prefix, suffix and line options are use to name and organize the files RT will create

-suffix

_ship.all e.g. _Titanic.all

-line

number_date_time e.g. 0001_20060714_142506

-out

Direct to the merged/sub_dir folder

If you have successfully unravelled the data you can join back to the final steps which are the same for Reson and Simrad. Note that when you are using a script you might have to adjust the sonar type or add the water column option if applicable (-unpack_watercol).

appendNav

We choose to decimate the position fixes and get rid of the redundant data.

appendNav –skip 10 –comp decnav/sub_dir/filename.decnav merged/sub_dir/filename.nav

-skip x

skipping every x fixes

-comp

define the path and filename of the new decnav file

merged/

direct to the merged-file

newMergeAtt

Merge the attitude. To perform ray tracing you will need to add sound velocity profiles (svp). The svp are ascii and need a be formatted to OMG-standard. If your not using a svp use the skip_raytrace option. In that case you can even choose to specify the sound velocity, otherwise the program will use the sound velocity provide in the .all-file.

newMergeAtt –in merged/sub_dir/filename.merged

Always add either:

-svp file1.svp file2.svp …..

or:

-skip_raytrace

Other options:

-velocity xxxx.x

-interpolate –nearest_int_time

-interpolate –last_observed

-interpolate –next_observed

-rules rules.txt

specify time based rules on how to interpolate

mergeNav

Merge the navigation data to the bathymetry:

mergeNav –use_params –delay x.x merged/sub_dir/filename.merged

-use_params

now it incorporates the .param file with the offsets. Instead of such a file you can also specify it separately. Offsets of nav source w.r.t. reference point:

-ahead x.x

-right x.x

-below x.x

-declin

if you want to add the magnetic declination

-delay

delay in seconds (result from patch test, is not already applied in acquisition software)

Now you have unravelled everything and are you ready to clean it in swathed with:


swathed merged/JD???/filename(s).merged