IMC Raster Mode - processing optimizations for windows

[prometheus]

Raster AT5 processing optimization 101:

If you are working on a massive imagery project and the IMC is either crashing or you had to shell out some serious money to buy more RAM, there are actually a lot of optimizations in Windows that can be done in order to process imagery much faster. The information below is quite detailed, but nonetheless it will help you with large raster image projects.

Down sampling source images
Down sampling source images for lower res at5s.

The Issue:
The IMC crashes occur when someone attempts to down sample high res imagery 1 meter imagery to a low resolution at5 range and the machine does not have adequate computing resources.

Example:
There are 2 copies maintained during the interpolation when converting imagery to the AT5 format:
1)   the cache
2)   the workspace

Cache will not exceed the uncompressed size on disk.  The workspace, depending on image gaps, might. 
The workspace is limited by the extents of the source imagery. Hence even though compressed imagery saves disk space, it will still have to be uncompressed for processing.


Bytes per Source Pixel (Num Copies x Bytes Per Pixel) = 8
Bytes per Output Pixel (horizontal multiplier x vertical multiplier x Num Copies x Bytes Per Pixel) = 8 * X * Y

All numbers grow at a 4 times growth rate for output pixels:
1x1x2x4 = 8 bytes
2x2x2x4 = 32 bytes
4x4x2x4 = 128 bytes
8x8x2x4 = 512 bytes
16x16x2x4 = 2,048 bytes
32x32x2x4 = 8,192 bytes
64x64x2x4 = 32,768 bytes
128x128x2x4 = 131,072 bytes
256x256x2x4= 524,288 bytes
512x512x2x4 = 2,097,152 bytes
1024x1024x2x4 = 8,388,608 bytes
2048x2048x2x4 =  33,554,432 bytes
4096x4096x2x4 =  134,217,728 bytes
8192x8192x2x4 =  536,870,912 bytes


The Solution:
The easy way around the problem is to down sample your source imagery to a lower resolution prior to processing in the IMC. So if you already down sampled your 1 meter imagery to something like 256 meter imagery (GeoTIFF, PNG, JP2), you would notice a huge speed improvement when processing the 256 meter source imagery as 256 meter at5 because instead of trying to interpolate (and potentially uncompressing the compressed images if the source is compressed) the 1 meter imagery to 256 meter images to 256 meter at5s, you are converting 256 meter resolution images to 256 meter at5 or even lower resolution at5s. If the Windows  system has less RAM you will need to down sample your source images much sooner. This trick is completely dependent on your system resources.

File formats
Compressed image formats take up less disk space. The IMC has to decompress the image so memory usage may far exceed file size with compressed images compared to uncompressed images (see “Down sampling source images” section for more info) . This is really important when you are trying to run (for example) 1 meter source imagery that is compressed while trying to process a significantly lower resolution at5 dataset such as 256 meter at5. The most commonly used formats are:
•   GeoTIFF - uncompressed without loss, supports transparency
•   PNG - compressed without loss, supports transparency
•   JPG - compressed with loss, does NOT support transparency
•   JP2 - compressed with loss, supports transparency if alpha channel is present

The IMC supports all free image formats   input as long as the projection/Datum is Geographic/WGS84 or Lowrance/Mercator Meter (Mercator/Lowrance). Please keep in mind, projections are ignored if they are stored in the file headers. The IMC only supports un-rotated world files with the same name as the image file.

Alpha Transparency and stacking
Keep in mind GeoTIFF and PNG support alpha channel (image transparency) in processing so if you have alpha (transparency) with overlapping images, they will still stack seamlessly using this alpha channel (color model RGBa for Red, Green, Blue, alpha). The image stacking order is quite simple; the highest resolution for the same space is always stacked on top so it will appear as the final output in at5. The image alpha channel allows transparent collars to be ignored and show neighboring images to come through.

If you have two images in the same block (area) or partial block with the same resolution, the filename (alphanumeric order in windows OS) will determine what image file is stacked on top, that file appears as the final image. JPG does not support transparency so if you are working with JPG and you use a single pixel value for edges, you can always set that to transparent in an application to re-export images, but any image editing in Photoshop for example may cause the pixels to shift from a single color to numerous “off-color” pixel values, which will ruin your final export because you will no longer have a single pixel you can filter by. The most common transparent pixel selection tool for jpg to filter is the magic wand for pixel selection (photoshop and Paint.net both have a magic wand tool for pixel selection). If the alpha channel of a PNG image is converted to a single color (even black or white) when the image is converted to a JPG for example, this color will still appear in the final at5 instead of remaining transparent like a format that supports alpha such as geoTIFF or PNG. JP2 has an alpha channel but you may experience data loss in that alpha due to a “lossy” compression scheme.

Image size clipping
Also, you probably don’t want to exceed image sizes of 2000x2000 (something smaller is preferred). If you tried to use giant GEOTIFFs of 4GB each, obviously all of that imagery would need cached for every at5 processed, which will make your processing time take forever (if it even finishes without crashing due to the downsampling issue described above). Most computers cannot handle data sizes that large because there is not enough RAM, virtual memory, or disk cache.

If you clip your source images too small (to an excessively small size like 10x10 pixels), the number of read/writes would increase processing time because there would be exponentially more images which requires more hard disk seek time to traverse.

Example:
Source images for each GEOTIFF as 15,000x15,000 as 300 files
Converting to PNG with each PNG at 1500x1500 as 3000 files will save a lot of processing time
Converting to PNG with each PNG at 10x10 as 450000 files will cause the operating system to perform a lot more read/writes which will slow down processing on hard disk thus removing any benefits achieved with smaller file size.

Shrinking the at5 data/card sizes
There are a lot of tricks that can be used to shrink your overall data foot print with filtering, but it is difficult when you are wanting continuous land coverage.

For example:
If land areas near lakes are desired, lake polygons can be used as a filter to shrink the overall size of the raster at5 dataset with the IMC advanced options>atlas options>Raster options>Raster Filter Shapefile. If a shapefile is used for each lake, the output would only be raster at5s that covered each lake for each resolution instead of continuous coverage across the entire area where imagery is present, at all at5 resolutions processed. Imagery for 5 lakes in Montana take up a lot less space than the entire state of Montana.

The real key is to only process large resolution at5 files where you absolutely need coverage (like 1 meter imagery near lakes in a country) and then use continuous coverage at lower resolutions to fill in the difference (for example 128 meter resolution at5s for the entire country). This will allow you to have continuous coverage while saving valuable disk space. Where there is no 1 meter source imagery available for 1 meter at5 through 4 meter at5, there will still be 128 meter source resolution at 8 meter at5 through 512 meter at5 for a range of 1 meter at5 through 512 meter at5.

Please note:
Currently produced Lowrance, Simrad, and B&G units officially support up to 32GB so your final regions cannot exceed the 32GB chip. Technically speaking you could use EXT2 or EXT3 on a 64GB chip (which is backwards compatible XCSD), but we DO NOT OFFICIALLY SUPPORT EXTENDED Capacity on our units yet so you should not attempt this).

Windows, network, and system resourcing
Please keep in mind that if your RAM fills up in Windows and you start dumping an enormous amount of data into your virtual RAM (hard drive swap file), your process will most likely die a very slow and painful death.

Windows and RAM limitations

•   If a Windows computer has for example 8GB of RAM, this does not mean 8GB worth of imagery at 1 meter can be processed. Windows carves out a system partition of RAM and a user partition of RAM. Applications share the user portion. In the 8GB example, the end user only has roughly 5GB of usable RAM to put towards applications.
•   Any other application can rob RAM resources from the IMC via the user partition of RAM, including apps running through windows explorer.
•   Further you are limited to roughly 4GB total if you are running 32bit (x86) windows. You will need to run 64bit (x64) windows when using the IMC for large image processing.
•   Lastly, if you have the RAM (and the guts), you can disable virtual memory in all windows drives and run off pure RAM, but if it ever gets completely full your system will lock and the fun will be over till you reboot.

Other thoughts and considerations
•   RAMdisk is a possibility because the source data is already loaded in RAM, but it probably won’t be feasible for really large datasets.
•   Running the process from a solid state drive (with the OS on the solid state drive) can also help for increased performance
•   15k or 10k platters are much more efficient than the standard 7200rpm hard drives which are still better than the 5400rpm laptop hard disks.
•   Disk cache on the drive also helps with transfer rate. If you have 50MB of hard disk cache verses only 10MB of hard disk cache, that is a considerable difference.
•   It is not recommended to process imagery across a network because if latency occurs after the file read, the final output at5 file may become pure black tiles instead of storing anything useful, hence there may not be any output data.
•   Fiber channel storage arrays and other local SAN/cloud technologies can avoid the latency but testing is required.

Black tiles and Lowrance Elite HDI/CHIRP display issues
You will want to process small sample areas for testing to ensure you can make imagery appear at lower zoom levels on elite and get rid of black borders in the maps with proper polygon filtering.

Splitting your process into multiple processes
The most obvious and time saving trick is to process one instance of the IMC for each resolution of at5 while running other instances of the IMC at other at5 resolutions at the same time. Running one instance of IMC per CPU core depending on computer resources or less seems to be successful. MAKE SURE DIFFERENT WORK DIRECTORIES ARE USED to prevent one process from over-writing another process in the same work folder. By splitting up each resolution of at5 to a separate instance of the IMC, If any single process crashes, only that process will be lost, not all rsolutions unless the computer locks up due to lack of resources.

Example:
On computer A, with a 4 core and plenty of RAM: (3 processes running at the same time).
launch insightmapcreator.exe and run 1 meter min resolution through 1 meter at5 max resolution, load 1 meter.sap, click build.
launch insightmapcreator.exe and run 2 meter min resolution through 2 meter at5 max resolution, load 2 meter.sap, click build.
launch insightmapcreator.exe and run 4 meter min resolution through 4 meter at5 max resolution, load 4 meter.sap, click build.

Computer B, with a 6 core and plenty of RAM: (6 processes running at the same time).
launch insightmapcreator.exe and run 8 meter min resolution through 8 meter at5 max resolution, load 8 meter.sap, click build.
launch insightmapcreator.exe and run 16 meter min resolution through 16 meter at5 max resolution, load 16 meter.sap, click build.
launch insightmapcreator.exe and run 32 meter min resolution through 32 meter at5 max resolution, load 32 meter.sap, click build.
launch insightmapcreator.exe and run 64 meter min resolution through 64 meter at5 max resolution, load 64 meter.sap, click build.
launch insightmapcreator.exe and run 128 meter min resolution through 128 meter at5 max resolution, load 128 meter.sap, click build.
launch insightmapcreator.exe and run 256 meter min resolution through 256 meter at5 max resolution, load 256 meter.sap, click build.

Computer C, a 6 core with plenty of RAM: (5 processes running at the same time).
launch insightmapcreator.exe and run 512meter min resolution through 512 meter at5 max resolution, load 512 meter.sap, click build.
launch insightmapcreator.exe and run 1024 meter min resolution through 1024 meter at5 max resolution, load 1024 meter.sap, click build.
launch insightmapcreator.exe and run 2048meter min resolution through 2048 meter at5 max resolution, load 2048 meter.sap, click build.
launch insightmapcreator.exe and run 4096 meter min resolution through 4096 meter at5 max resolution, load 4096 meter.sap, click build.
launch insightmapcreator.exe and run 8192 meter min resolution through 8192 meter at5 max resolution, load 8192 meter.sap, click build.

[siiverlce]

Thank you for this fantastic information! I greatly appreciate it, and it helped me speed up my processing time.

I was only able to use one recommendation: Splitting your process into multiple processes.

I am new to creating/processing/making custom maps, and I am new to IMC, Lowrance, and SASPlanet. I use SASPlanet to create the KML file, and I use IMC to take SASPlanet's info to convert to Lowrance.

How can I down sample source images or any of your other recommendations? I do not understand how to do your other suggestions.

Any help in this matter is greatly appreciated. Thank you for posting this information! It is very helpful!

[gregory]

If using an application that has raster data at specified resolutions (like SASPlanet), you don't want to downsample the data as much as you want to download source data at an input resolution that closely matches your output resolution.  Underzooming or Overzooming imagery will affect the color composition making imagery appear more gray as you underzoom and more vibrant as your overzoom.  Using imagery that is designed for that level (if possible) generally has a more appropriate color composition.

[the_ace]

Modern machines often have more than 32 cores but a low clock speed.
Is there any way to force IMC to use more than one core for a single instance?

[gregory]

Raster Mode is multithreaded as of v0.7.4.24.  Request access to the beta for versions past v0.7.4.0.

[SkunkApe]

I have a question regarding the tip below:

"Shrinking the at5 data/card sizes
There are a lot of tricks that can be used to shrink your overall data foot print with filtering, but it is difficult when you are wanting continuous land coverage.

For example:
If land areas near lakes are desired, lake polygons can be used as a filter to shrink the overall size of the raster at5 dataset with the IMC advanced options>atlas options>Raster options>Raster Filter Shapefile. If a shapefile is used for each lake, the output would only be raster at5s that covered each lake for each resolution instead of continuous coverage across the entire area where imagery is present, at all at5 resolutions processed. Imagery for 5 lakes in Montana take up a lot less space than the entire state of Montana.

The real key is to only process large resolution at5 files where you absolutely need coverage (like 1 meter imagery near lakes in a country) and then use continuous coverage at lower resolutions to fill in the difference (for example 128 meter resolution at5s for the entire country). This will allow you to have continuous coverage while saving valuable disk space. Where there is no 1 meter source imagery available for 1 meter at5 through 4 meter at5, there will still be 128 meter source resolution at 8 meter at5 through 512 meter at5 for a range of 1 meter at5 through 512 meter at5.

Please note:
Currently produced Lowrance, Simrad, and B&G units officially support up to 32GB so your final regions cannot exceed the 32GB chip. Technically speaking you could use EXT2 or EXT3 on a 64GB chip (which is backwards compatible XCSD), but we DO NOT OFFICIALLY SUPPORT EXTENDED Capacity on our units yet so you should not attempt this)."

I'm using Reefmaster to create at5 maps. The maps I create are sourced from high resolution bathymetric survey imagery. For best display results I use highest resolution possible when exporting the map project which is great when viewing the structure, but not so great because all the area between the wreck sites is being mapped in high resolution. Wreck site can be separated by significant distances, so there's a lot of Gigabytes being wasted on large areas with no areas of interest within. I've was told it was a 'no-no' to have more than one .at5 map project on one sd card, even if the individual project files are segregated in their own folders.

I'm unclear on the steps to include select areas of interest with enhanced min resolution (.0125 for example) in a large area map created with decreased min resolution (like 2 for example). 

Here's a typical scenario for me:
I have an area of 100 square miles. There are six wreck sites within that 100 square mile area. As mentioned before, this entire are would be mapped at .0125 and and the final file size would be quite large if not exceed the 32gb limit of the sd. So the other day I created a map for that entire area with 1 as the min resolution setting. That was good, fast and much smaller over file size (and to be expect blurry map.  Then I create a separate projects in reefmaster for each of the six wrecks and exported them individually, each with a unique Atlas Description.



 I copy each project's .at5 files into their own folders (do not mix project files together) and paste the seven individual projects on the sd card. On the chartplotter, the large area map displays and give me a good high level visual reference of where all my wreck sites are (marked with waypoint icons). Then when I get on the site, I have to select my Chart -> Options and switch the from the large area overview map then the small map of the wreck I'm over. That process is repeated for each wreck I visit. I would be happy doing that except when I switch to small wreck maps, my aesthetically pleasing large over map goes away and replaced with large vector blocks that could be orange or some random color. It doesn't affect the integrity of the small wreck map but often if I'm drifting a wreck I tend end up drifting out of the bounds of my small high resolution wreck map into the orange colored vector blocks since I had to get rid of the big map to run the small map.

How is it possible to include higher resolution imagery with lower resolution imagery in the same .at5 map? Seems to me the only way to come close to that is to have multiple projects on one card and when if I chose the hi-res wreck site project, the low res "filler" map goes away and is replaced by giant vector blocks.

Thanks for all the hard work and valuable posts. I appreciate it.

Matt. 

[gregory]

There's no issue with having separate projects on a card if they have separate description strings.  The issue with having multiple raster projects that have the same description string is that both of those projects may have the same raster block and, in those cases, only one of the blocks will display.  If the different projects are on different resolutions, there is no issue.

Most people recommend separate storage folders because of file name collision (2 files with the same name).

Having separate processing folders is required as the IMC cleans up previous runs when processing.

This said, the quoted section isn't describing how to combine two projects together.  It's describing a variable resolution presentation where only sections of your raster chart are high resolution and the other sections are back filled from lower resolution (NOS only).  In practice, there can be thousands of high resolution areas on a single description string if you have shapefile extents for each of them.