Tag Archives: ArcGIS

ArcGIS vs QGIS Clipping Contest Rematch

Round 2 in which ArcGIS throws in the towel.

(Please note: This post is about clipping in ArcGIS version 10.0. The functionality has been improved, and problems mentioned have been fixed in later versions of ArcGIS)

This is a follow-up to my previous post where I matched up ArcGIS and QGIS in a clipping contest. One of the commenters on that post expressed some concern that there might be “…something else going on…” with my test, and I agreed. It was unfathomable to me that an ESRI product could be out-done by such a wide margin. Knowing that ArcGIS often has problems processing geometries that are not squeaky clean, I began my investigation there. I ran the original contour layer through ArcToolbox’s Check Geometry routine, and sure enough, came up with 5 “null” geometries. I deleted those bad boys, and ran it through ArcToolbox’s “Repair Geometry” routine, and then ET GeoWizard’s “Fix Geometry” routine for good measure (These may or may not be identical tools, I do not know). No new problems were found with either tool.

I wanted to give ArcGIS  a fighting chance in this next round, but also wanted to level the playing field a bit. I did a restart of my Dell m2400 (see the specs in the previous post), exited out of all my desktop widgets, and turned off every background process I could find. I also turned of Background Processing in the Geoprocessing Options box. The only thing running on this machine was ArcGIS 10, and the only layers loaded were the contour lines and the feature I wanted to clip them to. I ran the “Arc Toolbox > Analysis Tools > Extract > Clip” tool and watched as it took 1 hour 35 minutes and 42 seconds for ArcGIS to go through the clipping process before ending with the message:
ERROR 999999: Error executing function
Invalid Topology [Topoengine error.]
Failed to execute (Clip)

ArcClipFail

Now granted, this is much better than the 12 hours it took the first time I ran it, but still, no cigar in the end.

Giving QGIS a chance to show it’s stuff, I used Windows version 1.5.0 to run a clip on the same files, on the same machine. QGIS took all of 6 minutes and 27 seconds to produce a new, clean contour layer.

QGIS - Contours v2

I ran this through the same geometry checks as the original contour layer, and came up with no problems.

My goal here is not to jump all over ESRI and do a dance in the end zone. I would really like to figure out what’s going on. As I’ve said before, I’ve had problems in the past with ArcGIS producing bad geometries with its Clipping process (and other tools, too). But the fact that another product can handle the same set of circumstances with such ease baffles me.

I’ve put about as much time as I can into this test, and taken it as far as I can. If you would like to give it a go, feel free to download the files I used through his link:
www.donmeltz.com/_files/ContourClipTest.zip

(Note: This is a 878MB file, and is not completely uploaded as of this posting. Check back later if the link does not work for you right now)

If any of you have better results than I did, or find any faults with my files or process, please let me know and I WILL make a note of them here. Thank you.

ArcGIS–QGIS Faceoff

Is QGIS a viable alternative to ArcGIS?

(Please note: This post is about clipping in ArcGIS version 10.0. The functionality has been improved, and problems mentioned have been fixed in later versions of ArcGIS)

I’ve never enjoyed working with contours. They seem to bog down my system more than any other layer type I work with. However, most of my clients are so used to looking at USGS Topo maps they expect to see them on at least one of the maps I produce for them. I recently worked on a project covering a five-town area in the Catskill Mountain region. The large area covered, and the ruggedness of the topography was proving exceptionally troublesome in processing their contours. So much so that I decided to look at other options to get the work done. I’ve used a variety of GIS tools over the years, but do most of my paying work exclusively in ArcGIS. It’s what I’m most familiar with, it does (nearly) everything I need it to do, and therefore provides my clients with the most efficient use of my time. However, in this situation that was not the case.

The one geoprocessing operation that frustrates me most often (in ArcGIS) is the Clip operation. It seems to take more time than most other geoprocessing tools, and often results in bad geometries. This happens so often, I usually resort to doing a union, and then deleting the unwanted areas of the Union results. For some reason this works much faster, and with more reliable results than doing a Clip.

Since what I wanted to do here was a clip on a contour layer, I was in for double trouble. Yes, I could have clipped the original DEM I wanted to produce the contours from first, then generated contours from the clipped DEM. But that wouldn’t have led to anything to write about. So, here’s a short comparison of how ArcGIS handled the process versus QGIS:

The hardware and software used:

ArcGIS 10, SP2

  • Windows 7, 64 bit
  • Dell Precision m2400 laptop
  • Intel Core 2 Duo CPU, 3.06GHz
  • 8 GB RAM

QGIS 1.4.0

  • Ubuntu 11.4
  • Dell Inspiron 600m laptop
  • Intel Pentium M CPU, 1.60 Ghz
  • 1GB RAM

A fair fight?

I started out with ArcGIS, and loaded up my 20’ contour lines and a 1 mile buffer of the study area to which I wanted to clip them. I began the clip operation 3 times. The first two times I had to cancel it because it was taking too long, and I needed to get some real work done. Curious to see how long it would really take, I let the process run overnight. The progress bar kept chugging away “Clip…Clip…Clip…Clip…”, and the Geoprocessing results window kept updating me with its progress, so I assumed it would complete eventually. In the morning, I looked in the Geoprocessing results window and found it had run for over 12 hours before throwing an error, never completing the clip operation. The error message said something about a bad geometry in the  output. Really, no surprise there.

ArcGIS - ClipContour1

(Yes, those are lines in the picture above, not polygons. They’re very densely packed)

QGIS gets to play

The next day I decided to give QGIS a shot at it. I copied the two shapefiles over to my 6 year old lappy. (The contour.shp file was 1.3GB) fired up QGIS, and ran the Clip operation on the two files.

QGIS - Contours Screenshot-Clip

This time it took all of 17 minutes and 21 seconds to get a new contour layer.

Clip Results - QGIS

So, who’s the winner here? Was it a far contest?

My take-away is, ESRI really needs to do some work on its Clip geoprocessing tool. As I said earlier, it is slow, and results in bad geometries more often than any of their other geoprocessing tools I use.

Addendum June 11, 2011: See the follow-up post here:
http://bit.ly/k0Fm7D

Find duplicate field values in ArcGIS using Python

As ESRI is making it’s move away from VB Script and towards Python, I’m also slowly updating my stash of code snippets along the way. One of those little pieces of code I use quite often is one that identifies duplicate field names in a layer’s attribute table. I find this particularly helpful when I’m cleaning up tax parcel data, looking for duplicate parcel-ID numbers or SBL strings. Since I’ve been working a lot with parcel data lately, I figured it was time to move this code over to Python, too. So, here it is in step-by-step fashion…

1 – Add a new “Short Integer” type field to your attribute table (I usually call mine "Dup").

2 – Open the field calculator for the new field

3 – Choose "Python" as the Parser

4 – Check "Show Codeblock"

5 – Insert the following code into the "Pre-Logic Script Code:" text box making sure you preserve the indents:


uniqueList = []
def isDuplicate(inValue):
  if inValue in uniqueList:
    return 1
  else:
    uniqueList.append(inValue)
    return 0


6 – In the lower text box, insert the following text, replacing "!InsertFieldToCheckHere!" with whichever field you want to check the duplicates for:


isDuplicate( !InsertFieldToCheckHere! )


This will populate your new field with 0’s and 1’s, with the 1’s identifying those fields that are duplicates.

 

FindDups-Python

Generating Vertical Buffers

One of the more popular analyses I’m asked to perform for my clients is a viewshed analysis. Beyond simply identifying what areas of a town are visible from roads or other public viewpoints, I’m often asked to help identify, and sometimes rank, areas that are most worthy of protection. One way to help a town identify and evaluate these high priority vistas, is to identify prominent ridgelines and the areas around them that are susceptible to inappropriate development.

One way to mitigate the impact of development on highly visible ridgelines is to make sure new buildings do not break the horizon line – the point where the ridge visibly meets the sky. Since most local zoning codes restrict building heights to 35-40 feet (in my client towns, anyway), producing a vertical buffer of 40 feet helps to identify the areas susceptible to such an intrusion.

The steps to produce such a vertical buffer are not overly complex, but I have not found them readily available online. So, for your benefit (and my easy reference) I outline the process here.
(Note: I outline these steps specifically using ArcGIS 10. I’m sure it’s possible using other tools, but this is what I use most often in my daily work)

Begin with:

A DEM of your study area

This DEM must be an integer raster for one of the following steps, so start off by using the raster calculator.
(Arc Toolbox > Spatial Analyst Tools > Map Algebra > Raster Calculator)
Use the INT( “DEM” ) expression, where “DEM” is the elevation raster that you want to convert from a floating point to an integer raster.

1-DEM

Prominent ridgelines

Ridgelines are often generated using watershed boundaries, with extensive field checking to identify the more prominent features. For these calculations, the ridgelines must be in a raster format that includes the elevation of each raster cell. To convert a line-type ridgeline to a raster, first buffer the ridgeline to produce a polygon feature (I typically use a 20 foot buffer) and then use Extract by Mask.
(Arc Toolbox > Spatial Analyst Tools > Extraction > Extract by Mask)
Use the Integer DEM produced in the first step as the input raster, and the polygon buffer of the ridgelines for the feature mask data.

2-Ridgelines

Generate the Euclidean Allocation of the Ridgeline Raster

This is where the need for an integer raster comes into play.
(Arc Toolbox > Spatial Analyst Tools > Distance > Euclidean Allocation)
Simply use the Ridgeline raster generated in the previous step as the input raster, and choose the elevation value as the Source field. This process will generate a new raster that covers the entire study area, with each cell holding the elevation value of the ridgeline raster that is nearest to it.

3-EucoAllo

Generate the Vertical Buffer

Use the Raster Calculator to generate a vertical buffer of the ridgeline.
(Arc Toolbox > Spatial Analyst Tools > Map Algebra > Raster Calculator)
The expression will look something like this: “IntegerDEM” >= (“RidgelineEucoAllo”-12) where “Integer DEM” is the DEM produced in step 1, “RidgelineEucoAllo” is the Euclidean Allocation raster produced in step 3, and “12” is the height of the buffer you want to produce. In this case, the raster measurements are in meters, so using a buffer value of 12 results in about a 39 foot vertical buffer. This allows me to identify areas where there is the potential for a new building built to the maximum allowable height to break the horizon line.

4-VerticalBuffer

Once the buffer is generated, there is usually some cleanup required. As you can see from these results, buffers are generated in areas not connected to the ridgeline areas we want to focus on, so I’ll delete these before moving on to the next steps in this town’s viewshed analysis.

I hope this is helpful, and as always, I welcome any comments and feedback.