Find duplicate field values in ArcGIS using Python

As ESRI is making it’s move away from VB Script and towards Python, I’m also slowly updating my stash of code snippets along the way. One of those little pieces of code I use quite often is one that identifies duplicate field names in a layer’s attribute table. I find this particularly helpful when I’m cleaning up tax parcel data, looking for duplicate parcel-ID numbers or SBL strings. Since I’ve been working a lot with parcel data lately, I figured it was time to move this code over to Python, too. So, here it is in step-by-step fashion…

1 – Add a new “Short Integer” type field to your attribute table (I usually call mine "Dup").

2 – Open the field calculator for the new field

3 – Choose "Python" as the Parser

4 – Check "Show Codeblock"

5 – Insert the following code into the "Pre-Logic Script Code:" text box making sure you preserve the indents:


uniqueList = []
def isDuplicate(inValue):
  if inValue in uniqueList:
    return 1
  else:
    uniqueList.append(inValue)
    return 0


6 – In the lower text box, insert the following text, replacing "!InsertFieldToCheckHere!" with whichever field you want to check the duplicates for:


isDuplicate( !InsertFieldToCheckHere! )


This will populate your new field with 0’s and 1’s, with the 1’s identifying those fields that are duplicates.

 

FindDups-Python

Generating Vertical Buffers

One of the more popular analyses I’m asked to perform for my clients is a viewshed analysis. Beyond simply identifying what areas of a town are visible from roads or other public viewpoints, I’m often asked to help identify, and sometimes rank, areas that are most worthy of protection. One way to help a town identify and evaluate these high priority vistas, is to identify prominent ridgelines and the areas around them that are susceptible to inappropriate development.
One way to mitigate the impact of development on highly visible ridgelines is to make sure new buildings do not break the horizon line – the point where the ridge visibly meets the sky. Since most local zoning codes restrict building heights to 35-40 feet (in my client towns, anyway), producing a vertical buffer of 40 feet helps to identify the areas susceptible to such an intrusion.
The steps to produce such a vertical buffer are not overly complex, but I have not found them readily available online. So, for your benefit (and my easy reference) I outline the process here.
(Note: I outline these steps specifically using ArcGIS 10. I’m sure it’s possible using other tools, but this is what I use most often in my daily work)
Begin with:

A DEM of your study area

This DEM must be an integer raster for one of the following steps, so start off by using the raster calculator.
(Arc Toolbox > Spatial Analyst Tools > Map Algebra > Raster Calculator)
Use the INT( “DEM” ) expression, where “DEM” is the elevation raster that you want to convert from a floating point to an integer raster.
1-DEM

Prominent ridgelines

Ridgelines are often generated using watershed boundaries, with extensive field checking to identify the more prominent features. For these calculations, the ridgelines must be in a raster format that includes the elevation of each raster cell. To convert a line-type ridgeline to a raster, first buffer the ridgeline to produce a polygon feature (I typically use a 20 foot buffer) and then use Extract by Mask.
(Arc Toolbox > Spatial Analyst Tools > Extraction > Extract by Mask)
Use the Integer DEM produced in the first step as the input raster, and the polygon buffer of the ridgelines for the feature mask data.
2-Ridgelines

Generate the Euclidean Allocation of the Ridgeline Raster

This is where the need for an integer raster comes into play.
(Arc Toolbox > Spatial Analyst Tools > Distance > Euclidean Allocation)
Simply use the Ridgeline raster generated in the previous step as the input raster, and choose the elevation value as the Source field. This process will generate a new raster that covers the entire study area, with each cell holding the elevation value of the ridgeline raster that is nearest to it.
3-EucoAllo

Generate the Vertical Buffer

Use the Raster Calculator to generate a vertical buffer of the ridgeline.
(Arc Toolbox > Spatial Analyst Tools > Map Algebra > Raster Calculator)
The expression will look something like this: “IntegerDEM” >= (“RidgelineEucoAllo”-12) where “Integer DEM” is the DEM produced in step 1, “RidgelineEucoAllo” is the Euclidean Allocation raster produced in step 3, and “12” is the height of the buffer you want to produce. In this case, the raster measurements are in meters, so using a buffer value of 12 results in about a 39 foot vertical buffer. This allows me to identify areas where there is the potential for a new building built to the maximum allowable height to break the horizon line.
4-VerticalBuffer
Once the buffer is generated, there is usually some cleanup required. As you can see from these results, buffers are generated in areas not connected to the ridgeline areas we want to focus on, so I’ll delete these before moving on to the next steps in this town’s viewshed analysis.
I hope this is helpful, and as always, I welcome any comments and feedback.

Linux OpenGeo Apache Server–Tweaked

Over the past week I’ve probably generated enough material for a half-dozen blog posts. However, since I have to get some billable hours in and invoices sent out this week, I’ll just post a short update for now on one aspect of project “Linux OpenGeo Apache Server”. In my prior post I described the steps I took to get an Ubuntu based Apache/GeoServer up and running. Since then I’ve tweaked the process, and with a lot of help, been able to get everything in working order.
One of the roadblocks I faced was how to get Apache/Tomcat and GeoServer working together. Working individually was easy, together was not. As I progressed with my education in JavaScript, I ran into some cross-domain issues due to the fact my website and GeoServer were hosted on different machines. Even when I put them both on the same machine, the problems persisted due to the fact the webserver  and the the map server use different ports (Apache – 80, and GeoServer – 8080). The solution turned out to be setting up a proxy server. Sounds easy. It was not.
After banging my head on the problem to the point of near concussion, a few online cohorts came to my rescue; in particular, @spara. She was generous enough to write up a script that installs Apache and the OpenGeo Suite, and configures Tomcat and the rest in a way that makes them all work together. She’s posted the code on github for anyone to use. Note: My previous post installed a full LAMP system. This one is limited to just Apache and the OpenGeo Suite, with some tweaks to the Tomcat servlet configuration.
I’ve tested her script on a few clean Ubuntu 10.10 setups, but I haven’t been able to get it to work as a single copy/paste yet. For me, it’s still a multi-step process. But, it does work and it is about as painless as it gets. What works consistently for me, is performing the OpenGeo Suite install first, and then running the Apache install/Tomcat configuration separately.

OpenGeo Suite

So far, the only process that works consistently for me is to first sudo to root, then setup the repository and install the OpenGeo suite. Start by typing the following into a terminal window:


sudo su


… hit enter, and enter your password.
Then copy and paste the following into terminal:


wget -qO- http://apt.opengeo.org/gpg.key | apt-key add -
echo “deb http://apt.opengeo.org/ubuntu lucid main” >> /etc/apt/sources.list
apt-get update
apt-cache search opengeo
apt-get install opengeo-suite


After OpenGeo finishes installing, it will ask for a proxy URL (just leave it blank and hit enter unless you know what that means), a user name, and a password. Set these up as you wish. When all that’s done, move on the the next step.

Apache/Tomcat

Here’s the script for the Apache install/Tomcat setup, just copy and paste this into a terminal window:



sudo apt-get install -y apache2
sudo ln -s /etc/apache2/mods-available/proxy.conf /etc/apache2/mods-enabled/proxy.conf
sudo ln -s /etc/apache2/mods-available/proxy.load /etc/apache2/mods-enabled/proxy.load
sudo ln -s /etc/apache2/mods-available/proxy_http.load /etc/apache2/mods-enabled/proxy_http.load
sudo chmod 666 /etc/apache2/sites-available/default
sudo sed -i '$d'  /etc/apache2/sites-available/default
sudo sh -c "echo ' ' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyRequests Off' >> /etc/apache2/sites-available/default"
sudo sh -c "echo '# Remember to turn the next line off if you are proxying to a NameVirtualHost' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPreserveHost On' >> /etc/apache2/sites-available/default"
sudo sh -c "echo ' ' >> /etc/apache2/sites-available/default"
sudo sh -c "echo '<Proxy *>' >> /etc/apache2/sites-available/default"
sudo sh -c "echo '    Order deny,allow' >> /etc/apache2/sites-available/default"
sudo sh -c "echo '    Allow from all' >> /etc/apache2/sites-available/default"
sudo sh -c "echo '</Proxy>' >> /etc/apache2/sites-available/default"
sudo sh -c "echo ' ' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPass /geoserver http://localhost:8080/geoserver' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPassReverse /geoserver http://localhost:8080/geoserver' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPass /geoexplorer http://localhost:8080/geoexplorer' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPassReverse /geoexplorer http://localhost:8080/geoexplorer' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPass /geoeditor http://localhost:8080/geoeditor' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPassReverse /geoeditor http://localhost:8080/geoeditor' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPass /geowebcache http://localhost:8080/geowebcache' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPassReverse /geowebcache http://localhost:8080/geowebcache' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPass /dashboard http://localhost:8080/dashboard' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPassReverse /dashboard http://localhost:8080/dashboard' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPass /recipes http://localhost:8080/recipes' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPassReverse /recipes http://localhost:8080/recipes' >> /etc/apache2/sites-available/default"
sudo sh -c "echo ' ' >> /etc/apache2/sites-available/default"
sudo sh -c "echo '</VirtualHost>' >> /etc/apache2/sites-available/default"
sudo chmod 644 /etc/apache2/sites-available/default


This installs Apache, and configures it and Tomcat so it appears as though your website and all of the OpenGeo Suite apps are using the same port. The last thing to do is test everything as I outlined in my previous post.

Test the Apache2 default website:

on the host computer – http://localhost – It works!
and via a remote computer (substitute your server’s domain here) – http://24.105.210.45/ – It works!

Test the OpenGeo Suite:

Open the dashboard – http://localhost:8080/dashboard/
Launch GeoExplorer
Save the default map and exit GeoExplorer

Test GeoServer by loading a GeoExplorer map:

Open the default map on a remote computer (again, substitute your server’s domain here) – http://24.105.210.45:8080/geoexplorer/viewer#maps/1
If everything is set up correctly, you should see something like this:
OpenGeoExplorerView
There are probably a few more tweaks that can be made to this process, but I don’t know enough about how Ubuntu works yet to make those changes. I do know that this process works most of the time. A restart between processes helps sometimes, and trying them in a different order occasionally works, too. If anyone can come up with a better, more streamlined approach, please feel free to let me know about it. For now, I’ll just have to live with a little complexity.
And in honor of the day, a little more cow bell, too –

Linux OpenGeo Apache Server

Moving the GeoSandbox to Full OpenSource

I’ve run into a couple of roadblocks recently, regarding my experiments with my GeoSandbox. I want to be able to play with some of of the JavaScript libraries available, so I can continue my education on those fronts. Some cross domain issues arose when I started putting JavaScript in my web pages as my website is on a different server than is my GeoServer. So, that means learning some more about setting up web servers and how proxy servers work. Also, my old Dell 600m is starting to feel the effects of the increased demands placed on it as the GeoSandbox becomes more popular, and more complex. That means an eventual move onto a cloud server. Since cloud servers running an open source OS are much less expensive than those running Windows, I felt the need to begin transferring my entire setup over to open source tools.
For now, I’m just going to do a short outline of what it took to get a basic Ubuntu/Apache/OpenGeo Suite operating on my home server. Once I get a better handle on how Ubuntu and Apache work I’ll add some posts about that.

Starting with a clean install of Ubuntu 10.10

Download and install Ubuntu 10.10 Desktop

I downloaded Ubuntuu 10.10 Desktop Edition from their website:
http://www.ubuntu.com/desktop/get-ubuntu/download
Yes, I could have used the Server edition, but Coming from a Windows world, It’s easier for me to work with some kind of GUI than to go 100% command line. After downloading the .iso, I burned it to a CD, installed it on a clean hard drive, and then installed all updates.

Turn Ubuntu Desktop into an Apache Server

Install LAMP with a single command

In case you didn’t know, LAMP stands for “Linux Apache MySQL PHP”. This may or may not be more than I need for my purposes, but a one-line install looked like the easiest way to go, so:
http://nick.onetwenty.org/index.php/2010/12/02/installing-lamp-server-on-ubuntu-10-10-desktop/
Then I opened a terminal window and entered:
sudo apt-get install lamp-server^

Add in the ability to serve maps

Install OpenGeo Suite

This was the most difficult part for me to figure out. Instructions can be found here:
http://projects.opengeo.org/suite/wiki/LinuxPackages
Being somewhat new to Ubuntu, I missed what the first step under “Repository Setup” meant (sudo to root). Once I figured that out, things went smoothly. The steps are:
In terminal, sudo to root:
sudo su
Import the OpenGeo gpg key:
wget -qO- http://apt.opengeo.org/gpg.key | apt-key add -
Add the OpenGeo repository:
echo "deb http://apt.opengeo.org/ubuntu lucid main" >> /etc/apt/sources.list
Update the package list:
apt-get update
Search for OpenGeo packages:
apt-cache search opengeo
Install the OpenGeo Suite package:
apt-get install opengeo-suite
Restart Ubuntu

See if everything works

The last step was to see if everything was working properly, which it was.

Test the Apache2 default website:

on the host computer – http://localhost – It works!
and via a remote computer (substitute your server’s domain here) – http://24.105.210.45/ – It works!

Test the OpenGeo Suite:

Open the dashboard – http://localhost:8080/dashboard/
Launch GeoExplorer
Save the default map and exit Geoexplorer

Test GeoServer by loading a GeoExplorer map:

Open the default map on a remote computer (again, substitute your server’s domain here) – http://24.105.210.45:8080/geoexplorer/viewer#maps/1
In my case, everything worked as expected. After this, I continued trying to set up a custom website to serve my GeoServer maps, but ran into a few problems. I’ll be switching the GeoSandbox back and forth between the Windows and Apache server as I continue my climb up the learning curve, so don’t be surprised if some of the links appear to be broken on occasion.

GeoSandbox 2.0

This Week’s Project

I decided it was time to take the GeoSandbox to the next level. As usual, I had multiple goals in mind for this project. Primarily, I wanted to learn how to use and write JavaScript. It’s been a long time since I’ve tried to write any kind of meaningful code, but I figured if I wanted to keep up with the big kids, it was time to take the plunge. I also wanted to get my map viewer into a state where I could make it useful enough to display my GIS projects, and help promote my business.

My Experience with Writing Code

My first exposure to computer programming was back in 1980 when I took a computer language class as an undergrad. I remember trying to write some code for a fictional car dealership inventory system, and saving it on a 8” floppy disc. I believe it was in Basic, but I really can’t remember. I never did get it to work. Since then I’ve played around with various versions of Visual Basic, but never got very far past the “Hello World” types of programs. I took a GIS class as a grad student back in 2001 where I used MapObjects to make a functional GIS program. That was fun, and I did get that to work, but I haven’t written any real code since then. Yes, I do read, use, and modify scripts in ArcGIS. That keeps me in tune with the basics of programming, but I don’t think it’s the same as writing real code.

The Goal

I wanted to make the following three enhancements to the GeoSandbox:

  • Change the buttons in the toolbar
  • Make the map window dynamically change size with changes to the browser window size
  • Have the map window change its contents through links in order to showcase different projects

The first two items were fairly easy. I was able to read through the GeoExplorer.js script, and just delete the buttons I didn’t want showing on the toolbar. I also made some changes to the About box, and figured out how to generate a proper Google API key that allows the embedded 3D viewer to show. Dynamically resizing the map window was just a matter of doing a search, copying some code, and pasting it in the right place on the web page. Yes, I had to modify the <iframe>  ID in the code, but that was simple enough.
I had it in my head that I wanted to make four maps available on the page, and to allow users to switch between these maps using a set of links along the top of the map. Seemed simple enough.
No, it wasn’t.
Setting up the webpage and the GeoServer maps was easy. Here’s what the page looks like:
GeoSandbox2-0
And you can access the page using this link: www.donmeltz.com/maps

Changing iFrames

What I found out along the way was, banging your head against an <iframe> can really hurt.
I started out by putting a single <iframe> on the page. Since the src attribute of said <iframe> specifies the URL of the document to show in the <iframe>, I figured I’d just change the src attribute to point to the various GeoServer maps I wanted to show. That didn’t seem to work for me. What I found was, there are many different ways to change the <iframe> src. I must have tried them all, and I got most of them to work. The problem was, they didn’t work in every browser I tried. Here’s a list of the browsers I used in my tests:

  • Firefox 3.1.15
  • Internet Explorer 9 beta
  • Chrome 9.0.597.107
  • Safari 5.0.3
  • Opera 10.10

Learning About iFrames

First, I’d like to mention the Dynamic Web Coding website has the most comprehensive and understandable information on scripting <iframes> that I’ve found: http://www.dyn-web.com/tutorials/iframes/
What I haven’t figured out yet is – Why do all of the examples used on that website (and many others I visited) work for them, but not for me? It just boggles my mind. In a nutshell, here are the various things I tried.
I started out by setting up some bare-bones test sites where I stripped out all of the css styling, and deleted all but two links in order to make it easy to modify and test the code, and see what I was doing. I have links to all of these test pages below each of the following paragraphs in case you want to check them out. You can right-click on each page and view the source if you want. Click on the upper part of the page, NOT on the <iframe> map.
(Please note, I have greatly simplified the syntax in these examples so they fit on one line. If you want to see the actual code, please visit the Dynamic Web Coding site, or right click on my test maps and view the source.)
I began by trying to set the <iframe> target directly in the link like this:
<a href=http://www.Maps.html#maps/2 target="mapiframe">link</a>
This worked in IE and Firefox, but only if I hit the browser refresh button after I clicked the link. Yes, I also went through many iterations of adding a variety of refresh functions to the code, but that only worked in one or two instances. And in some cases it made things even worse. Here’s a link to my test page for this trial: http://www.donmeltz.com/mapstest1.html
Next up was an attempt to set the iframe source (scr attribute) using some JavaScript code as in:
document.getElementById('MapFrame').src="http://Maps.html#maps/2"
I tried using both a function and embedding the code in the link using the onClick event. Results were similar to the previous attempt, and again, inserting a refresh function did not help. There are other variations of this, such as using “window.getElementById, etc. which I also tried, but to no avail. Here’s the link to my test page for this trial:
http://www.donmeltz.com/mapstest2.html
I then moved on to trying to assign the contents of the iframe using code like this:
contentWindow.location.assign("http://Maps.html#/2")
which worked in IE (again, with a browser refresh) but not in anything else. The trial page: http://www.donmeltz.com/mapstest3.html
What came closest to getting the task done was working with the <div> that holds the <iframe> instead of manipulating the <iframe> directly, as in:
<div id="MapContainer">
MapContainer.innerHTML = "<iframe id="MapFrame" src="http://Maps.html#maps/2">

This one worked for me in every browser except Firefox: http://www.donmeltz.com/mapstest4.html
In the end, what I wound up doing was placing four different iframes on the page. I then set the style.display property of the iframe to “none” (hide) or “block” (show) in order to hide or show each one based on which link is clicked on the web page and looks something like this:
document.getElementById("MapFrame").style.display="block";
Yes, “block” means “show”. Intuitive, I know. Here’s a link to my test page that shows how this one works: http://www.donmeltz.com/mapstest_adinfinitum.html
This works in every browser I’ve tried, but it does slow down the load times significantly. All four maps load when the page loads. It does make switching between maps faster once everything is loaded, though.

I Got It

So, where did all of this hacking get me? I got what I wanted. A workable map viewer that shows a few samples of my work, and an education in JavaScript. Even though I spent many hours, days even, working with this, I still think it was worthwhile. I’ve learned a lot, even though it doesn’t seem like I got very far. I can read JavaScript code, and understand it a lot better now than I did when I started. I’d still like to figure out if I’m doing something wrong. It seems like these options should work, as it looks like they work on other peoples pages. Yes, I’m still banging my head against the <iframe> once in a while 🙂
For now though, I think I’ll just kick back with a bottle of homebrew and a little Jimmy Cliff.

Mapping My Twitter Followers – No Code Needed

The Challenge

I’ve been pondering what new data sets to add to the GeoSandbox for a while now. It’s been a couple of weeks since the last addition, and I felt things were getting a little stale. Once again, a single tweet set my mind in motion. This time it was @briantimoney who said “2011 is the year tweet maps replace coffeeshop-finding as the go-to demo scenario in geo.” An RT by @billdollins cemented the statement in my head, and there was no going back after that. Obviously if the GeoSandbox is to remain the cutting-edge tool that it is, I’d have to move quickly.

The Process

All I wanted to do was to make a push-pin style map of the people who follow me on Twitter. Simple as that. I did not want to get into learning APIs and writing any kind of code, and I didn’t want to get into auto-updating, or anything like that. KISS. A little searching led to the following workflow:

MyTweeple – Export to CSV – Import to Fusion Table – Visualize on map –
Export to KML – Convert to shapefile – Put it into GeoServer.

More extensive searching might reveal a more efficient process, but this is what I came up with. From the time I first read the tweet to the point the data was in the GeoSandbox took about one hour.

My Tweeple to CSV

Once I signed into the MyTweeple site, I chose the Tools tab, and then the Export All (csv) option. This allowed me to save a csv spreadsheet containing up to 5,000 of my followers (I only have ~560). There are a lot of extraneous columns that can be done away with. All I really wanted was the name and address columns. Some sorting and editing of the address column can help out, too. Not everyone includes an address in their bio, and some have addresses that obviously won’t geocode properly. These were deleted.

MyTweeple

Fusion Table to KML

I’ve heard a lot of talk on the inter-tubes lately about Google Fusion Tables, so I’ve been looking for something to give this new tool a test drive. This was my chance, and it did not disappoint. All I had to do was go to the Fusion Table page, select New Table>Import Table, and browse to the MyTweeple.csv file I saved earlier.

FusionTable

I double checked a few addresses, and then clicked on the Visualize>Map link. The geocode routine ran for a few minutes, and produced a map showing all of the people who follow me on Twitter. It was like magic.

KMLmap

Clicking the Export to KML link allowed me to save the KML file to my computer, where I then converted it to a shapefile (I used the KML to Layer tool in ArcToolbox, but I’m sure there are many other ways to do this). From there it was just a matter of adding the shapefile to my GeoServer as I’ve outlined in previous posts.

The Results

GeoSandboxTwitterers

So there you have it. My somewhat convoluted way of showing my Twitter followers in the GeoSandbox.

GeoSandbox grows up to be a “Real boy”

A quick update on the status of my little experiment.
This morning I got it in my head that I wanted to access my GeoSandbox in ArcMap through WMS, and allow others access to it through the same mechanism. Through much trial and error, and some timely help from @mcbride_bryan, I was able to accomplish my goal. If you want to try loading any of the data on my GeoSandbox into your GIS client of choice, you should be able to do so through this URL:
http://donmeltz.dyndns.org:8080/geoserver/wms?service=wms&version=1.1.1&request=GetCapabilities
Here’s a screen-shot with all the current layers loaded:
GeoSandboxWMSinArcGIS
Thanks also to @tpstigers for verifying that he could access it in his GIS app of choice, QGIS running in Ubuntu:
GeoSandboxWMSinQGIS
I’m really surprised at how fast these layers load. They seem to load much faster through the WMS than they do in the webpage.
So, I am happy to report that my little GeoServer toy is well on its way to becoming a “Real Boy”

Setting up my GeoSandbox

It’s been 3 weeks since I set up my little GeoServer experiment and started populating it with various datasets. I thought I’d outline the things I’ve been working on, and what I plan on looking at next. I couldn’t have come as far as I have without the generous help of some of my Twitter friends. I’ve noted below each section those who’ve helped me out. I apologize if I’ve overlooked anyone.

Workspace > Store > Layer

My initial step in getting a useful map up and running was to load some vector data. The first thing I’m sure every GeoServer noob learns is, importing data into GeoServer is a three-step process. The first thing I did was set up a Workspace, which is essentially a folder in my “C:Users<username>.opengeodata_dirdata<foldername>” directory. Then I told GeoServer where to look for this folder. Second step was to set up a Store. A Store can hold a single layer, or multiple layers depending on how you import the data. Why a Workspace AND a Store? I have no idea, but that’s what you have to do, so that’s what I did. Third step was to add a layer or layers to the Store. After that, you can add those layers to a GeoExplorer map, and publish it or embed it in a webpage.

EPSG

Somewhere along the way, GeoServer will ask for a spatial reference system (SRS) for the data you’re adding in the form “EPSG:XXXX”. I figured out very quickly that looking up the EPSG for a shapefile using the built-in GeoServer tool is a PITA. A couple of tools were pointed out to me:
Prj2EPSG (http”//prj2epsg.org/search) and Spatial Reference (http://spatialreference.org)
Although the data I use comes to me in many different projections, I use a limited number of spatial references in my output. I now keep a Post-It note on my desk with the 5 EPSG codes I use most often.
(Thank you @mcbride_bryan, @spara)

Styling

Symbolizing vector layers in GeoServer is a little cumbersome, but I was able to figure it out. and it’s not too much of a chore for simple layers. Options are limited, and I’ve been pointed to a few other tools when it comes to styling layers. uDig, Atlas Styler, and Arc2Earth all look promising and I’ll be checking them out in the near future.
(Thank you @mcbride_bryan, @spara)

Rasters

One of my more frustrating episodes was an attempt to get raster data to display in my GeoExplorer map. The GeoServer GUI and documentation led me to believe I could just load an ArcGrid (binary) dataset into GeoServer. I wasn’t able to get that to work, and have been told by some that it’s not supported. Next, I tried a jpeg. I assumed by including a world file (jgw) everything would be hunky-dory. I assumed wrong. It was pointed out to me that a projection file (prj) needs to be included, too. That got the raster to display, but brought my poor little server to its knees. Finally, I settled on a GeoTiff. By using jpeg compression within the GeoTiff, I was able to keep the file size reasonable, and GeoServer handled it just fine.
(Thank you @geowolf, @storm72)

Zoom Levels

For the life of me I could not figure out how to zoom in close enough to make my newly loaded raster data useful. I know I had seen many GeoServer/GeoExplorer maps able the zoom in nice and close, but it just wasn’t happening for me. Finally, it was pointed out that changing the default base layer to something other than Google Terrain fixes that problem. Worked like a charm. Why would the Google Terrain base map limit zooming in close? I have no idea. Just one of those things.
(Thank you @manicmapper)

HTML and CSS

After getting these things working, I decided to dress up the web page that holds my GeoExplorer map a little bit. I have Dreamweaver CS4, and I use that to manage a couple of websites, but it’s really much more than I need, especially for a little single page website. I looked around and found Aptana Studio. After using it for an hour or so, I’ve come to really like it. It color codes and formats the HTML and CSS so it’s readable, and I think the preview feature works better than what Dreamweaver has. I might also give Komodo Edit a try, and see how that compares.
(Thank you @mcbride_bryan)

What’s Next?

Probably some more experimenting with the various SLD editors, brushing up on my HTML and CSS skills, and diving into JavaScript, which I’m embarrassed to say, I haven’t really looked at in over 5 years.

What’s it look like?

Here it is as of January 31, 2011:
GeoSandbox 2011-01-31
And a link to the site so you can see it now:
Don’s GeoSandbox

Serving Maps for Free

The Goal

This weekend I started down the road to fulfilling one of my New Year Resolutions: Get some maps online. And by maps online, I don’t mean posting links to KML files, screen shots, or PDF exports from ArcGIS. I mean full-fledged interactive maps viewable by anyone with a browser. On top of that, I had one more goal. I wanted to do this with as little extra cost to me as possible. This is going to be a learning experience for me, not a paying job.

The Map Server

I’ve put a lot of thought into this. It’s something that’s been on my mind for a long time, and I’ve looked into a few options. Choosing a map server was relatively easy. There appear to be three major players in the map server arena. ESRI’s ArcGIS Server, GeoServer, and MapServer. There are others mentioned on Wikipedia’s Web Map Service page, but these are the three that seem to be discussed most often.  ArcGIS Server was out of the question for obvious reasons. Thousand$ of obviou$ reason$. GeoServer and MapServer are two Open Source options, but GeoServer seems to be the more popular option among the open source crowd. I’m not one to pick an option basedGeoServerStatus solely on its popularity, but in this case, popularity also means more discussion in the forums, which in turn means it’s easier to find a solution to any problems that might arise during my own learning process. So, GeoServer it was. You can download GeoServer directly from GeoServer.org, but I decided to go the OpenGeo Suite Community Edition route. What led me to that decision was their great White Paper on The OpenGeo Architecture, which is the most concise, easy to read explanation of how all the pieces fit together that I’ve found anywhere on the web.

The Web Server

The most difficult decision in this whole process was how to host this beast. Yes, I know, I could just install GeoServer on my laptop and play with it there, but I wanted to make these maps public. Isn’t that the whole point of a server, be it geo or otherwise? I have been looking at Amazon Web Services, and signed up for a AWS Free Usage Tier account when they started offering them. The problem with AWS is, it’s just so darn complicated to get IIS7Managersomething set up. I am getting better at finding my way around there, but I still get lost once in a while. Plus, the free accounts require using a Linux operating system. I’ve been experimenting with Ubuntu at home, but I’m much more comfortable working in a Windows world. And Windows usually means paying for cloud computing space (e. g. $58/month at Rackspace). If you’re interested in looking at the AWS option, I highly recommend you take a look at what @spara has done: Install GeoServer on Amazon EC2 without leaving the browser.

The Home Server

I decided to merge this project with another I’ve been thinking about for some time now: setting up a home-based web server. I have an old Dell 600m laptop that’s just taking up space in my office. I have an extra copy of a Windows 7 Home Premium upgrade disc that comes with IIS7. Both were essentially free (to me anyway, your costs may vary). MyHomeWebServerThe only unknown was, would I be able to open this up for public viewing? The main roadblock to setting up a public server using a home internet connection is the use of dynamic IP addressing and port blocking by some (most?) ISPs. I was able to work around both of these obstacles by using DynDNS’s services, and setting my router and software settings appropriately. I wont go into the intricacies here, but if you do a Google search on either of these terms, a wealth of information will flood your browser.

the steps I followed to get things up and running:

  • Wiped the hard drive on the Dell 600m laptop clean
  • Installed the original XP OS, and then the Win 7 Home Premium upgrade
  • Installed all updates through Windows Update
  • Downloaded and installed the OpenGeo Suite
  • Played around with GeoServer and GeoExplorer enough to get a basic map embedded on a web page
  • Got a free Dynamic DNS host name at DynDNS.com
  • Installed DynDNS Updater
  • Modified my router settings to make sure the proper ports were open and forwarding to my laptop server
  • Tested, changed settings, tested, changed settings, tested, tested, tested
  • Scratched head, changed settings, tested, swore a few times, had a beer…
  • Installed and set up Internet Information Services (IIS7)
  • Set up a website using the web page with the embedded map I made with GeoServer/GeoExplorer
  • Went through the whole “testing-scratch head-drink a beer” routine again
  • Celebrated the results

Here’s a snapshot of the results on day 1:

Don's GeoSandbox RC1

And a Link to  what it looks like now:

Don’s GeoSandbox

I’ll try to keep this link up-to-date, but this is all a test, and it is running on a home computer, so don’t be surprised if it’s not available 24/7.
Some of you that have seen this before this post asked the question”Great, but where’s the data?”
Well, that’s coming. Please be patient with me people :-). Baby steps. Baby steps.

Addendum 01/12/2011

I now have a few shapefiles uploaded and viewable on the site. Next step will be to dress up the web page a little bit.

ArcGIS 10 Annoyances – Picture Properties Bug Not Fixed

Here’s another annoyance I just ran into today. When I try to resize an inserted graphic in a map layout using the Picture Properties dialog box, the new values are not saved. In fact, when I open the Picture Properties dialog of a graphic that has been resized, it reverts to its original size, even without saving it. Here’s what I’m trying to do.
I insert a logo into a map layout (Sorry @PetersonGIS, it’s required by my client 🙂 and attempt to resize it. In this case, the original is 10” high by about 18” wide. I set the height to 3”, check preserve Aspect Ratio, and click OK:
PicturePropertiesDialog
Nothing changes:
LogoWrongSize
I can resize the graphic by selecting it in layout view, and then dragging one of the anchor points. Here’s what I want the logo to look like:
LogoCorrectSize
So why am I complaining? Well, I want the logo border to line up exactly with the border of the white rectangular graphic it’s placed over. I can get it close freehand, but I have to set a guide and snap to that in order to get the size just right. Also, if I open the properties dialog again to change anything, even if it’s not related to resizing, I have to go through this process all over again. That’s very annoying.
A quick search on the ESRI website reveals this is a known bug:
ArcGIS Resource Center – Bug NIM059805
Now I’m not one to complain about a company having a few bugs in a newly released piece of software, especially something with as many changes to it as ArcGIS 10 has. What bothers me about this bug is, it apparently will not be fixed in the recently announced Service Pack 1!
This, ladies and gents, is going to be a major annoyance.