For this month’s Geospatial Frequently Asked Question (G-FAQ), I explore a topic that perhaps some of us have given little consideration to: file compression. Specifically, I look at file compression of TIFFs which are commonly large datasets that we have to store for both short and long time periods. Take for instance the minimum order of 25 square kilometers (sq km) of 50-centimeter (cm) WorldView-2 imagery with 8-bit depth, this dataset alone is 400 megabytes (mb). Now consider the enormity of a 1-foot aerial collection over an entire county, and surely the topic of file compression now makes a bit more sense. I have compressed my archival copies of client orders for some time now to reduce the amount of hard drive space needed for long-term storage; and thus, to save a bit of money.
With this in mind, the January G-FAQ will address this core set of questions:
- What is the best compression format for TIFFs?
- What freeware applications are offered for file compression?
- Is there a format that is better for short versus long-term storage?
In past G-FAQs, I have spent some time introducing the principles behind the topics discussed; this month, however, I will leave the principles of file compression to the experts–if you want to read more about the topic, check out this through summary. Suffice it to say that compression saves space by looking for repeating patterns in words, numbers and images; and then by replacing them with shorter data strings in the compressed file. Some compressed file formats you might be familiar with are .bin (Mac only), .gz, .rar, .tar (does not actually compress, it just bundles multiple files together into one) and .zip–with .zip far and away the most commonly used.
Freeware File Compression Software
While there are a plethora of freeware programs that can create compressed files, here is a list of some good options:
But my favorite freeware program is 7-Zip which is also featured in this month’s Free For All. You can read more about 7-Zip’s features here. On the 7-Zip website, they show test results where .7z appears to be the best format for file compression, and that is part of the inspiration for this G-FAQ.
The compression tests that you see below were completed on a Sony Vaio Laptop with 12 gb of RAM, a 64-bit Windows 7 Professional operating system and an Intel Core i7-3520M CPU running at 2.90 Ghz. During the tests, my laptop was plugged in and all programs were shut down except for those that normally run in the background such as anti-virus software and NVIDIA settings. The TIFF file used for the tests was a 16-bit depth 50-cm natural color image collected by WorldView-2 and covering 5 sq km. The file’s original size was 122,611,083 kb and I had 7-Zip version 9.20 installed for the tests.
In 7-Zip, you have the ability to tweak a variety of options when you create a compressed file. Here is a short summary of each:
- Archive Format: this is the type of compressed file you will create, for example .7z or .zip
- Compression Level: this sets the compression method from the fastest to ultra
- Compression Method: specifies the algorithm used to compress your data
- Dictionary Size: a larger dictionary usually increases compression ratios but makes the process slower
- Word Size: specifies the length of words used to try and identify patterns in the compression process, a larger word size usually increases compression ratios
- Solid Block Size: only valid for .7z format, a solid block usually increases compression ratios
- Number of CPU Threads: specifies the number of computing requests to send to your processor at once, more threads usually means faster compression and more memory use
TIFF File Compression Results and Conclusions
I tweaked these options and tested 12 different combinations to get a sense of the best performing compression format. The results of these tests are presented in the chart below.
Before I present you with my conclusions following from the compression tests, let me explain the meaning of each column (or variable) in the chart above:
- Compression Specs: this details the compression options used in each test, i.e. archive format, compression level, compression method, dictionary size, word size, solid block size and number of CPU threads; the baseline compression method is a .zip file with the default options in 7-Zip
- Memory Used: the amount of RAM used to compress the TIFF file, in megabytes (mb)
- Compress Size: the final size of the compressed file, in kilobytes (kb)
- % Original: calculated as, (Compressed File Size) / (Original File Size)
- % Baseline: calculated as, (Compressed File Size) / (Baseline Compressed File Size)
- Compress Time: time to compress the TIFF file, in seconds (s)
- % Baseline Time: calculated as, (Time to Compress) / (Time to Compress Baseline)
- Addl Time: additional time calculated as, (Time to Compress Baseline)–(Time to Compress), in seconds (s)
- Addl Compress: additional compression calculated as, (Baseline Compressed File Size)–(Compressed File Size), in kilobytes (kb)
- Compress / S: compression per second calculated as, (Compress Size) / (Compress Time), in kilobytes (kb)
- Addl Compress / Addl S: additional compression per additional second of compression time versus baseline calculated as, (Addl Compress) / (Addl Time), in kilobytes (kb); N/A means that this calculation was not possible for a given entry
Now that I have explained the variables used in this chart, let’s turn our attention to some conclusions that can be drawn from this data.
Best choice for long-term compression
- When you are storing data for a long time, the compression ratio is usually the most important consideration as smaller files mean less storage space, and then less money spent on storage.
- As such, the clear winner for long-term compression is .7z format using the ultra LZMA compression method. Increasing word, dictionary and block size will also increase compression ratios.
- Increased compression comes with the drawback of extra time to compress and uncompress files.
Best choice for short-term compression
- For short-term compression, the file size is important but then so is the speed to compress as you will not keep the data around for the long haul.
- The best balance between speed and compression ratio is achieved by using .7z format with LZMA compression, a 64kb dictionary, an 8 word size and a non-solid block.–This option created files that were 15% smaller than the baseline ZIP while only taking 3 additional seconds to compress.
- One drawback is that .7z format is not recognized by all software if you plan to share the compressed data. That said, 7-Zip is a freeware program so perhaps that is not a big issue!
Additional conclusions
- The .gzip format yielded the worst (lowest) compression ratios.
- The traditional .zip format was surprisingly robust in these tests as even the most compressed file was only about 20% smaller than the baseline.
- The most important influence on compression ratio was not the file format but rather the compression method. LZMA is by far the best performer, saving an additional 7 to 10% for each compression format.
Until our next edition of G-FAQ, happy GIS-ing!
Do you have an idea for a future G-FAQ? If so, let me know by email at brock@apollomapping.com.
Find Out More About This Topic Here:
Brock Adam McCarty
Map Wizard
(720) 470-7988
brock@apollomapping.com
Have you had a chance to test out our new online imagery search engine, Image Hunter?
If not, give it a test drive now as all orders placed with Image Hunter will receive a 5% discount! (more…)
Are you a regular user of Landsat 7 data but are tired of the scan line errors?
Perhaps you are frustrated by the inability to collect Landsat data within the time frame you need?
Or maybe you are working on a regional land-use model that needs low-cost and recent imagery over an entire country?
Whatever your reason is for needing low-cost, medium resolution imagery, DEIMOS-1 is the solution. With a price point below 30-cents per square kilometer (sq km), DEIMOS-1 offers three-band (i.e. green, red and near-infrared) 22-meter resolution as both a new collection and from a growing archive.
Since its launch in 2009, DEIMOS-1 has already amassed an archive that covers more than 1.375 billion sq km. Many locations have data that spans the entire growing season, making it an effective replacement for the failing Landsat 7 mission. And for those who need imagery collected on a regular basis over a wide-area, DEIMOS-1 can deliver with its 625-km wide footprint.
For more information on the medium resolution satellite, DEIMOS-1, you can refer to our website or send us an email at sales@apollomapping.com.
Pléiades 1A has been in orbit since December 2011 and if you have not had a chance to check out any sample imagery, take a few moments and have a look at the gallery on our website. If you work with high resolution imagery, you should consider Pléiades 1A for your next geospatial project.
A variety of Pléiades 1A products are available from both a growing archive and as a new collection, including 50-centimeter (cm) pansharpened imagery and 50-cm panchromatic–2-meter (m) 4-band multispectral bundles. We are happy to discuss the technical specifications, pricing and tasking options available with this new satellite constellation as Pléiades 1A will be joined in space by its twin, Pléiades 1B, later this year.
Working with Pléiades 1A for the past months, we have noticed that:
- The imagery archive is already growing and is well targeted.
- Astrium is able to deliver on their Pléiades 1A tasking feasibilities.
- The data is crisp with excellent color saturation so that it is on par with competitors’ 50-cm satellites.
Tracking Pléiades 1A Collections Over Europe
In our October edition of The Geospatial Times, we examined the coverage collected by Pléiades 1A over North America since its launch. In this edition, we turn our attention to Europe where you find large area collections dotting the continent over major urban areas as well as over much of Russia. This time, we filtered for data with a 15% or less cloud cover specification as that is the standard for tasking Pléiades 1A; and then for images collected on and after June 4, 2012 (i.e. the official start date for commercial sales).
- 2,961,532 sq kmof imagery collected over Europe since June 4, 2012
- 1,425,741 sq kmof this imagery has 15% or less cloud cover (48.1% of the total collected)
- On average, 264,026 sq kmof imagery with 15% or less cloud cover are imaged every 30 days over Europe
- For reference, Germany covers 348,610 sq km
Pléiades 1A Sample Imagery
This month we feature stunning sample imagery collected by Pléiades 1A over Boa Vista, Brazil on October 8, 2012. We Photo Enhanced this 50-cm natural color imagery to improve its colors and sharpness and the results speak for themselves.
Our Contact Details For More Information About Pléiades 1A
The Apollo Mapping sales team can answer any questions you might have about Pléiades 1A. We can be reached anytime at (303) 993-3863 or sales@apollomapping.com.
More sample images and technical information about Pléiades 1A can be found on our website here.
- In my continued look at the renewable energy world, I turn my attention to advancements in solar technologies. Of all the articles I review each day, solar energy is far and away the most common topic. As such, I will offer a high level summary of as many of the best articles I found recently and then provide links to each so you can find out more. Multiple efficiency records were set recently: (1) Alta Devices set the record for cells made of gallium arsenide (GaAs) at 28.3% by including semiconductor materials which created luminescence; (2) Heliatek has created an organic solar cell with 10.7% efficiency and a team of Phillips 66 and Solarmer created a polymer-based organic cell with a 9.31% rating; (3) researchers at the University of Toronto set the record for collodial quantum dot (CQD) films at 7%, these materials are attractive as they are made of inexpensive materials and harvest energy from the entire solar spectrum; (4) an IBM team crushed the old record for photovoltaic cells made of earth-abundant materials by 10% with an 11.1% efficiency; (5) researchers at EPFL have developed a cell made of amorphous/monocrystalline silicon with a 21.4% rating; and finally (6) NRELhas developed a black-silicon cell with a 18.2% efficiency, this is a nice technology as it simplifies panel design thus lowering costs.Here are several new design ideas that could improve the efficiency of a solar panel, regardless of the material(s) they are made of. A University of Houston team developed a nano-particle coating that will keep solar panels cleaner and thus increase their overall efficiency. While this sounds novel, a dirty solar panel can have a 30% reduction in efficiency! Scientists at the University of Arizona believe solar efficiencies could be doubled by using a mirror to reflect and concentrate the photons in sunlight. V3Solar claims its cone-shaped, spinning solar cell can generate 20 times more electricity than a flat panel. Bandgap Engineering feels that adding nano-silicon wires to solar cells is the solution to increased efficiency as this would increase the chance a photon is absorbed. Finally, researchers at Rensselaer Polytechnic Institutedeveloped tunable-refractive nano-coatings from silicon and titanium which increase solar panel efficiency by reducing the reflectivity of its surface.To conclude this look at solar technologies, I couldn’t help but chuckle to find out that ‘Keeping up with the Joneses’ appears to apply to solar panels as well: a Yale and New York University study has shown that homes are more likely to have panels if their neighbors do.
-
-
- Do you have a few extra moments in your day? If so, check out this short video (some of you may have seen it before–sorry!) that shows the changing European landscape since 1,000 AD. To say that European borders shifted a bit in the last millennium is an understatement!
A video showing the changing European borders since 1,000 AD.
- Do you have a few extra moments in your day? If so, check out this short video (some of you may have seen it before–sorry!) that shows the changing European landscape since 1,000 AD. To say that European borders shifted a bit in the last millennium is an understatement!
-
- Have you ever Googled: GIS and dogs? If not, give it a try and buried in the long list of hits, you will find research completed by Ashley Ignatius from the University of Minnesota where GIS was used to estimate the distance dogs travel when they are lost. Ashley looked at 199 cases in southern Minnesota where the dog owner was reunited with their furry companion (thankfully!) and the results were surprising. Over half of the dogs were found within a quarter mile from where they were lost–which is a lesson to all dog owners out there! If you want to read more of the study, you can find it here.
- Are you an iPhone user? Well, I am definitely a dedicated iPhone owner having happily owned one since the 3rd version. For our readers that share my passion for iPhones, I am sure that many of you are as disappointed as I am with the new Apple Maps application. To put it bluntly, the new app is terrible. I have used it to search for five locations in the past month, all of them in Boulder, and all of them failed! What I noticed is that you need to be very specific with your address request, spelling everything correctly and including city names. I am guessing that the management team at Apple never really considered the amount of effort, money, time and technology that Google poured into Google Earth. I suppose as an outsider, it would be easy to underestimate this effort but oh to return to the days of a working Google Maps app!
- Many city governments of all sizes and shapes have GIS departments that play an important role in planning, taxation, environmental management and much more. City GIS departments also have a wealth of valuable geospatial data that can typically be accessed for free or for a reasonable cost. In this, our first look at city GIS websites, we start with Alabama’s largest city, Birmingham which offers its data in an online mapping application. The online application offers a wide array of data layers including QuickBird imagery from 2007, parcels boundaries, school districts and natural features. Several of the layers are scale sensitive, such as water points which are only visible at less than 1:2,500 scale. While the website does not allow for data downloads (at least that I could find), it will let you ‘print’ out a PDF with your visible layers–the QuickBird imagery did not print however given licensing rules.
Brock Adam McCarty
Map Wizard
(720) 470-7988
Over the last decade, there has been much uproar over the use of plastic bags at grocery and retail stores. Often referred to as single-use bags, they are known to take an extraordinarily long time to break down in landfills. There isn’t a commonly agreed upon length of time, but estimates vary between 500 and 1,000 years. To make these estimates, scientists use respirometry tests. The test gives newspapers a life-cycle of 2-5 months, whereas a banana peel takes a few days. When the test is applied to traditional plastic bags you find at grocery stores, there is no decomposition or carbon dioxide production. This is caused by its polyethylene composition, a man-made polymer that bacteria don’t recognize as food. Plastic bags are only broken down in a process called photodecomposition which will eventually lead to its dissolution into microscopic granules–hence the 500-1,000 year estimate.
In 2007, San Francisco was the first city to ban plastic bags. Before the ban, there were an estimated 180 million bags distributed to shoppers in San Francisco every year. According to an EPA study in 2001, there are an estimated 500 billion to one trillion plastic bags consumed every year throughout the world. Of these bags, between 1-3% end up outside the landfill stream resulting in numerous problems, including clogged drains and consumption by fish, reptiles and animals which results in an early, painful death.
This past May, Los Angeles became the largest city in the world to ban plastic bags. On the docket was also a ban for paper bags, but council members backed away from that thinking it was too severe. That would have required patrons to provide their own reusable bags, or to find other methods of transporting their purchases. That is a fight for another day.
Not everyone is for the ban on plastic bags however. Many people who work for the companies that make the bags fear losing their jobs. Advocates for the industry think there should not be a ban, but a plan to increase reuse and recycling of the bags. Targeting the ordinance in San Francisco, the Save the Plastic Bag Coalition says that paper and compostable bags are “significantly worse for the environment” than plastic bags; and that attempts to impose a 10-cent fee on each paper or compostable bag is too low to encourage people to use reusable bags.
The Golden Gate Bridge is a marvel of American infrastructure and ingenuity. Leading into one of the most progressive cities in the country, and perhaps the world, you will not see cargo trucks bringing bundles of plastic bags for dispersal at retail outlets any longer. Image captured November 3, 2012 by the 50-cm satellite, WorldView-2. Image courtesy DigitalGlobe and PhotoEnhanced by Apollo Mapping.
But as we all know, there are always two sides to every story. The majority opinion seems to run with the push for greater environmental regulations. And while opponents on both sides admit that government intervention isn’t always necessary in regulations, advocates for the banning of plastic bags think that intervention is warranted because of the significant economic and environmental devastation they cause. So for now, the issue is not “in the bag,” but the majority are in favor of limiting the destruction that plastic bags can cause.
Justin Harmon
Staff Writer
Researchers from universities all over the world got together to study the impacts of school redistricting. They used GIS to make the most logical, and fair, districts which ensure students are given the best opportunity for education, as well as to have the minimum travel time from home to school. School redistricting is often done in response to overcrowding, decline in enrollments as well as considerations for schools that are planned to be opened or closed. The biggest dilemma in school redistricting is balancing the need for overall efficiency vis-à-vis individual convenience. There is also an element of geography in redistricting as contiguous districts are always preferred.
An issue that is unique to American schools is the distribution of grade levels. Some districts may have K-8 schools, while others may do K-5, or K-6. This mixture of grades can cause confusion when redistricting occurs; for instance, it can create a situation where siblings are split up, making it hard for parents to juggle different school schedules.
In their study, the researchers had two purposes. First, to model a school districting problem so that it accurately addresses the issues that face the majority of school districts, providing a mathematical algorithm to solve these problems. And second, to implement the model with an intuitive and interactive interface that allows users to tweak the model to address specific problems they face. The two features that are unique to all school redistricting models are: the geographic units where students live must be assigned to schools; and the students assigned to schools cannot exceed available space.
The researchers developed seven criteria that should be associated with any satisfactory school district. (1) Each block is assigned to one school. (2) School assignments must not need exceed each school’s capacity. (3) Each school district must be contiguous. (4) No school boundary can cut across geographic boundaries such as railroads or rivers. (5) There should be a maximum distance traveled that applies to all students. (6) All students on the same block must go to the same school unless the corresponding school doesn’t have that grade. (7) And all new districts must maintain some semblance to the current district boundaries to avoid creating any secondary issues.
Using Philadelphia as a case study, the researchers found that as capacities for grades became more stringent, walking distances increased. Also, if students weren’t uniformly distributed by race (which was usually the case), there was a trade-off between achieving a racial balance and keeping contiguous boundaries. The biggest issue the researchers had with their model was the lack of uniform grade levels which could create complex district boundaries to assure students maintained their current grade structuring. As such, a major conclusion of the study was implementation of a uniform grade structure such as K-5, 6-8 and 9-12. This would make school redistricting less complex and more productive from a modeling, implementation and geographic perspective–but most importantly, less stressful on parents and students alike.
Justin Harmon
Staff Writer