This chapter describes steps that unravel the data and do some math to get actual x-y-z-values. The raw data is stored in binary files which have a certain (proprietary) structure. The OMG commands are able to read these raw data files and extract the required information to an OMG-format. So in fact the raw data gets reorganized to our desires. The result are several other files (the raw data file remains untouched off course) of which the .merged file will be used most later in the pipeline.

The unravel route is different for Reson (.xtf) and Simrad (.all). Both are describe here, though the final commands are used in the same way. This chapter describes the route for Simrad bathy data and Reson bathy data, the backscatter data doesn’t really require processing. The next chapter will cover the commands to use for gridding up bathy and backscatter data.

Once you get familiar with the steps you can decide to use a unravel script. With a script you can unravel complete datasets automatically. But when using a script bear in mind that you’ll have to tweak it a bit for each sonar or survey. Follow this link to Jonnies website for some unravel scripts.

First we need to create a proper directory structure. Store the raw data in a folder called raw/sub_dir/. Create folders merged/sub_dir, decnav/sub_dir and if you plan to do some thing with backscatter, ss/sub_dir. Use the mkdir command. The logic behind these folder comes clear in the following steps. It is the omg convention to name the sub directories to the Julian day. You get the Julian day simply by typing julian yyyy-mm-dd on the command line. The unravel script will create such a structure for you, though you will have create the raw/sub_dir manually.
Finally it will look as follows:

projectname raw JDXXX merged JDXXX decnav JDXXX ss JDXXX The steps below describe the route to follow for Simrad .all raw data files and Reson .xtf files. Reson xtf-data requires a few more steps to follow but the final steps are for both systems the same.

---

Simrad

Simrad I) Unzipping

Often the data raw data is compressed. Unzip it:

gunzip raw_data_file.all.gz

Simrad II) RT

Simrad has a proprietary file format, the .all files. Unraveling is, unlike Reson data, doen with just one command, the RT (read telegram) command:

RT –v4 –force_swap –em3000 –prefix raw/sub_dir –suffix suffix.all –line linenumber.all –out merged/subdir/

-v4

version 4, always use this

-force_swap

linux and windows computer have different byte orders. The force_swap command takes this into account. Although the latest versions of RT are automated I believe this command isn’t really necessary anymore, still wouldn’t hurt using it.

-em3000

specify your sonar; em3002 em1002, elac2 etc etc etc

-prefix

give it the Julian day

The operator can configure the raw data filenames, in most cases they look like this:

number_date_time_ship.all --> 0001_20060714_142506_Titanic.all

The prefix, suffix and line options are use to name and organize the files RT will create

-suffix

_ship.all e.g. _Titanic.all

-line

number_date_time e.g. 0001_20060714_142506

-out

direct to the merged/sub_dir folder

If you have successfully unraveled the data you can join back to the final steps which are the same for Reson and Simrad. Note that when your using a script you might have to adjust the sonar type or add the watercolumn option if applicable (-unpack_watercol).

---

Reson

The xtf-format is used by a handful of sonar manufactures. Reson too, uses this format. The xtf-format is kind of generic but the third party software such as QINSy program which data will be stored in the file. The xtf file has few packets, each packet contains designated information. For example a packet with ID number 2 is preserved for bathy data, ID 3 for attitude and ID100 for raw navigation. You can view the contents of a xtf file with:

xtfTool –vi desired_packet_id_number –in raw/JDXXX/filename.xtf

Just typing xtfTool gives the usage with descriptions of other options that help viewing the xtf data.

Reson I) Unzipping

Often the data raw data is compressed. Unzip it:

gunzip raw_data_file.xtf.gz

Reson II) xtf2omg_reson

Convert the Reson xtf data to OMG format.

xtf2omg_reson –sonar 8101 –fq –xtf raw/JDxxx/line_number.xtf –omg merged/ JDxxx/line_number

-sonar 8101

declare the sonar model

-fq

Reson flags the data online, with this option you choose to use this flags too.

-xtf

direct to the raw xtf files

-omg

direct to the merged folder where the data shall be stored

Go to the merged/sub_dir folder. You’ll see it created a merged file and a empty merged.ss_data file. The merged.ss_data file insn' used for the bathymetery.

Reson III)xtf2nav

This command extracts the navigation data from the xtf-file which can be store on different places. The xtf-file has a navigation field in the so called XTF bathy header, and in some occasions it has a special Reson navigation packet (ID 100). xtf2nav will use either of the two but when only the XTF bathy header is present you will also have to define the navigations origin. The XTF bathy header contains fields for the ship and the sensor, both are associated with navigation. Now you need to specify which navigation data to use. This another step where the xtfTool comes in handy. First check is your data has Reson navigation packets:

xtfTool –vi 100 –in filename.xtf |more

-vi 100

verbose packet ID 100 (=Reson nav)

|more

the more command is used to pipe the data once at the time on the screen

Now check which navigation field is used in the bathy header (ship or sensor):

xtfTool –vi 2 –in filename.xtf |more

Examine the ship and sensor X&Y values and decide which make sense to use. Do the same for the heading values. Note that in some cases both ship and sensor use the same values. In such a case it doesn’t matter what you choose of course.

Your ready to run xtf2nav:

xtf2nav –nav ship –hdg ship –xtf raw/sub_dir/filename.xtf –omg merged/sub_dir/line_number

-nav

define to use navigation of ‘ship’ or ‘sensor’

-hdg

define to use heading of ‘ship’ or ‘sensor’

-xtf

direct to the raw data

-omg

direct to the merged path and provide a filename

xtf2nav will create OMG-navigation files. When your dataset contained Reson navigation packets it created a .100_nav file. In other cases a .nav_2 file. In the following processing steps the software will fall back to navigation data stored in files with a .nav extension. As a result you will have to rename either of the two to a nav-file.
So if your dataset does not have the Reson navigation package:

cp merged/sub_dir/filename.nav_2 merged/sub_dir/filename.nav

Otherwise:

cp merged/sub_dir/filename.nav_100 merged/sub_dir/filename.nav

Before you run a script you will always have to check whether to use heading and navigation from the ship or sensor and which nav file your going to use. Then modify the script!

Reson IV) xtf2orientation

Add the orientation data to OMG format:

xtf2orientation –xtf raw/sub_dir/filename.xtf –omg merged/sub_dir/filename.orientation

Reson V) xtf2param

OMG uses a different axis convention, recall:

+x towards bow

+y towards starboard

+z down

Use the xtf2param command to convert to OMG convention. The program outputs a ascii file with the new offsets. You can also choose to add a waterline offset with –wl x.x

xtf2param –switch_axes y x –switch_axes x y –flip_axis z –xtf raw/sub_dir/filename/xtf –omg merged/sub_dir/filename.param

-switch_axes y x

the xtf y-axis becomes the OMG x-axis

-switch_axes x y

the xtf x-axis becomes the OMG y-axis

-flip_axis z

......duh….

-xtf

direct to the raw data

-omg

direct to the merged path and provide a filename

So far the conversions from xtf to omg data, the next steps are for both Reson and Simrad the same.

---

I)appendNav

We choose to decimate the position fixes and get rid of the redundant data.

appendNav –skip 10 –comp decnav/sub_dir/filename.decnav merged/sub_dir/filename.nav

-skip x

skipping every x fixes

-comp

define the path and filename of the new decnav file

merged/…..

direct to the merged-file

II) newMergeAtt

Merge the attitude. To perform raytraceing you will need to add sound velocity profiles (svp). The svp are ascii en need a be formatted to OMG-standaard. If your not using a svp use the skip_raytrace option. In that case you can even choose to specify the sound velocity, otherwise the program will use the sound velocity provide int the xtf- or all-file.

newMergeAtt –in merged/sub_dir/filename.merged

Always add either:

-svp file1.svp file2.svp …..

or:

-skip_raytrace

Other options:

-velocity xxxx.x

-interpolate –nearest_int_time

-interpolate –last_observed

-interpolate –next_observed

-rules rules.txt

specify time based rules on how to interpolate

III) mergeNav

Merge the navigation data to the bathymetry:

mergeNav –use_params –delay x.x merged/sub_dir/filename.merged

-use_params

now it incorporates the .param file with the offsets. Instead of such a file you can also specify it separately. Offsets of nav source w.r.t. reference point:

-ahead x.x

-right x.x

-below x.x

-declin

if you want to add the magnetic declination

-delay

delay in seconds (result from patch test, is not already applied in acquisition software)

Now you have unraveled everything and you ready to clean it in swathed with:

swathed merged/sub_dir/filename(s).merged

---

Next 2) Gridding and cleaning
Back
Home