Raster AT5 processing optimization 101:
If you are working on a massive imagery project and the IMC is either crashing or you had to shell out some serious money to buy more RAM, there are actually a lot of optimizations in Windows that can be done in order to process imagery much faster. The information below is quite detailed, but nonetheless it will help you with large raster image projects.
Down sampling source images
Down sampling source images for lower res at5s.
The IMC crashes occur when someone attempts to down sample high res imagery 1 meter imagery to a low resolution at5 range and the machine does not have adequate computing resources.
There are 2 copies maintained during the interpolation when converting imagery to the AT5 format:
1) the cache
2) the workspace
Cache will not exceed the uncompressed size on disk. The workspace, depending on image gaps, might.
The workspace is limited by the extents of the source imagery. Hence even though compressed imagery saves disk space, it will still have to be uncompressed for processing.
Bytes per Source Pixel (Num Copies x Bytes Per Pixel) = 8
Bytes per Output Pixel (horizontal multiplier x vertical multiplier x Num Copies x Bytes Per Pixel) = 8 * X * Y
All numbers grow at a 4 times growth rate for output pixels:
1x1x2x4 = 8 bytes
2x2x2x4 = 32 bytes
4x4x2x4 = 128 bytes
8x8x2x4 = 512 bytes
16x16x2x4 = 2,048 bytes
32x32x2x4 = 8,192 bytes
64x64x2x4 = 32,768 bytes
128x128x2x4 = 131,072 bytes
256x256x2x4= 524,288 bytes
512x512x2x4 = 2,097,152 bytes
1024x1024x2x4 = 8,388,608 bytes
2048x2048x2x4 = 33,554,432 bytes
4096x4096x2x4 = 134,217,728 bytes
8192x8192x2x4 = 536,870,912 bytes
The easy way around the problem is to down sample your source imagery to a lower resolution prior to processing in the IMC. So if you already down sampled your 1 meter imagery to something like 256 meter imagery (GeoTIFF, PNG, JP2), you would notice a huge speed improvement when processing the 256 meter source imagery as 256 meter at5 because instead of trying to interpolate (and potentially uncompressing the compressed images if the source is compressed) the 1 meter imagery to 256 meter images to 256 meter at5s, you are converting 256 meter resolution images to 256 meter at5 or even lower resolution at5s. If the Windows system has less RAM you will need to down sample your source images much sooner. This trick is completely dependent on your system resources.
Compressed image formats take up less disk space. The IMC has to decompress the image so memory usage may far exceed file size with compressed images compared to uncompressed images (see “Down sampling source images” section for more info) . This is really important when you are trying to run (for example) 1 meter source imagery that is compressed while trying to process a significantly lower resolution at5 dataset such as 256 meter at5. The most commonly used formats are:
• GeoTIFF - uncompressed without loss, supports transparency
• PNG - compressed without loss, supports transparency
• JPG - compressed with loss, does NOT support transparency
• JP2 - compressed with loss, supports transparency if alpha channel is present
The IMC supports all free image formats input as long as the projection/Datum is Geographic/WGS84 or Lowrance/Mercator Meter (Mercator/Lowrance). Please keep in mind, projections are ignored if they are stored in the file headers. The IMC only supports un-rotated world files with the same name as the image file.
Alpha Transparency and stacking
Keep in mind GeoTIFF and PNG support alpha channel (image transparency) in processing so if you have alpha (transparency) with overlapping images, they will still stack seamlessly using this alpha channel (color model RGBa for Red, Green, Blue, alpha). The image stacking order is quite simple; the highest resolution for the same space is always stacked on top so it will appear as the final output in at5. The image alpha channel allows transparent collars to be ignored and show neighboring images to come through.
If you have two images in the same block (area) or partial block with the same resolution, the filename (alphanumeric order in windows OS) will determine what image file is stacked on top, that file appears as the final image. JPG does not support transparency so if you are working with JPG and you use a single pixel value for edges, you can always set that to transparent in an application to re-export images, but any image editing in Photoshop for example may cause the pixels to shift from a single color to numerous “off-color” pixel values, which will ruin your final export because you will no longer have a single pixel you can filter by. The most common transparent pixel selection tool for jpg to filter is the magic wand for pixel selection (photoshop and Paint.net both have a magic wand tool for pixel selection). If the alpha channel of a PNG image is converted to a single color (even black or white) when the image is converted to a JPG for example, this color will still appear in the final at5 instead of remaining transparent like a format that supports alpha such as geoTIFF or PNG. JP2 has an alpha channel but you may experience data loss in that alpha due to a “lossy” compression scheme.
Image size clipping
Also, you probably don’t want to exceed image sizes of 2000x2000 (something smaller is preferred). If you tried to use giant GEOTIFFs of 4GB each, obviously all of that imagery would need cached for every at5 processed, which will make your processing time take forever (if it even finishes without crashing due to the downsampling issue described above). Most computers cannot handle data sizes that large because there is not enough RAM, virtual memory, or disk cache.
If you clip your source images too small (to an excessively small size like 10x10 pixels), the number of read/writes would increase processing time because there would be exponentially more images which requires more hard disk seek time to traverse.
Source images for each GEOTIFF as 15,000x15,000 as 300 files
Converting to PNG with each PNG at 1500x1500 as 3000 files will save a lot of processing time
Converting to PNG with each PNG at 10x10 as 450000 files will cause the operating system to perform a lot more read/writes which will slow down processing on hard disk thus removing any benefits achieved with smaller file size.
Shrinking the at5 data/card sizes
There are a lot of tricks that can be used to shrink your overall data foot print with filtering, but it is difficult when you are wanting continuous land coverage.
If land areas near lakes are desired, lake polygons can be used as a filter to shrink the overall size of the raster at5 dataset with the IMC advanced options>atlas options>Raster options>Raster Filter Shapefile. If a shapefile is used for each lake, the output would only be raster at5s that covered each lake for each resolution instead of continuous coverage across the entire area where imagery is present, at all at5 resolutions processed. Imagery for 5 lakes in Montana take up a lot less space than the entire state of Montana.
The real key is to only process large resolution at5 files where you absolutely need coverage (like 1 meter imagery near lakes in a country) and then use continuous coverage at lower resolutions to fill in the difference (for example 128 meter resolution at5s for the entire country). This will allow you to have continuous coverage while saving valuable disk space. Where there is no 1 meter source imagery available for 1 meter at5 through 4 meter at5, there will still be 128 meter source resolution at 8 meter at5 through 512 meter at5 for a range of 1 meter at5 through 512 meter at5.
Currently produced Lowrance, Simrad, and B&G units officially support up to 32GB so your final regions cannot exceed the 32GB chip. Technically speaking you could use EXT2 or EXT3 on a 64GB chip (which is backwards compatible XCSD), but we DO NOT OFFICIALLY SUPPORT EXTENDED Capacity on our units yet so you should not attempt this).
Windows, network, and system resourcing
Please keep in mind that if your RAM fills up in Windows and you start dumping an enormous amount of data into your virtual RAM (hard drive swap file), your process will most likely die a very slow and painful death.
Windows and RAM limitations
• If a Windows computer has for example 8GB of RAM, this does not mean 8GB worth of imagery at 1 meter can be processed. Windows carves out a system partition of RAM and a user partition of RAM. Applications share the user portion. In the 8GB example, the end user only has roughly 5GB of usable RAM to put towards applications.
• Any other application can rob RAM resources from the IMC via the user partition of RAM, including apps running through windows explorer.
• Further you are limited to roughly 4GB total if you are running 32bit (x86) windows. You will need to run 64bit (x64) windows when using the IMC for large image processing.
• Lastly, if you have the RAM (and the guts), you can disable virtual memory in all windows drives and run off pure RAM, but if it ever gets completely full your system will lock and the fun will be over till you reboot.
Other thoughts and considerations
• RAMdisk is a possibility because the source data is already loaded in RAM, but it probably won’t be feasible for really large datasets.
• Running the process from a solid state drive (with the OS on the solid state drive) can also help for increased performance
• 15k or 10k platters are much more efficient than the standard 7200rpm hard drives which are still better than the 5400rpm laptop hard disks.
• Disk cache on the drive also helps with transfer rate. If you have 50MB of hard disk cache verses only 10MB of hard disk cache, that is a considerable difference.
• It is not recommended to process imagery across a network because if latency occurs after the file read, the final output at5 file may become pure black tiles instead of storing anything useful, hence there may not be any output data.
• Fiber channel storage arrays and other local SAN/cloud technologies can avoid the latency but testing is required.
Black tiles and Lowrance Elite HDI/CHIRP display issues
You will want to process small sample areas for testing to ensure you can make imagery appear at lower zoom levels on elite and get rid of black borders in the maps with proper polygon filtering.
Splitting your process into multiple processes
The most obvious and time saving trick is to process one instance of the IMC for each resolution of at5 while running other instances of the IMC at other at5 resolutions at the same time. Running one instance of IMC per CPU core depending on computer resources or less seems to be successful. MAKE SURE DIFFERENT WORK DIRECTORIES ARE USED to prevent one process from over-writing another process in the same work folder. By splitting up each resolution of at5 to a separate instance of the IMC, If any single process crashes, only that process will be lost, not all rsolutions unless the computer locks up due to lack of resources.
On computer A, with a 4 core and plenty of RAM: (3 processes running at the same time).
launch insightmapcreator.exe and run 1 meter min resolution through 1 meter at5 max resolution, load 1 meter.sap, click build.
launch insightmapcreator.exe and run 2 meter min resolution through 2 meter at5 max resolution, load 2 meter.sap, click build.
launch insightmapcreator.exe and run 4 meter min resolution through 4 meter at5 max resolution, load 4 meter.sap, click build.
Computer B, with a 6 core and plenty of RAM: (6 processes running at the same time).
launch insightmapcreator.exe and run 8 meter min resolution through 8 meter at5 max resolution, load 8 meter.sap, click build.
launch insightmapcreator.exe and run 16 meter min resolution through 16 meter at5 max resolution, load 16 meter.sap, click build.
launch insightmapcreator.exe and run 32 meter min resolution through 32 meter at5 max resolution, load 32 meter.sap, click build.
launch insightmapcreator.exe and run 64 meter min resolution through 64 meter at5 max resolution, load 64 meter.sap, click build.
launch insightmapcreator.exe and run 128 meter min resolution through 128 meter at5 max resolution, load 128 meter.sap, click build.
launch insightmapcreator.exe and run 256 meter min resolution through 256 meter at5 max resolution, load 256 meter.sap, click build.
Computer C, a 6 core with plenty of RAM: (5 processes running at the same time).
launch insightmapcreator.exe and run 512meter min resolution through 512 meter at5 max resolution, load 512 meter.sap, click build.
launch insightmapcreator.exe and run 1024 meter min resolution through 1024 meter at5 max resolution, load 1024 meter.sap, click build.
launch insightmapcreator.exe and run 2048meter min resolution through 2048 meter at5 max resolution, load 2048 meter.sap, click build.
launch insightmapcreator.exe and run 4096 meter min resolution through 4096 meter at5 max resolution, load 4096 meter.sap, click build.
launch insightmapcreator.exe and run 8192 meter min resolution through 8192 meter at5 max resolution, load 8192 meter.sap, click build.