Tag Archives: GeoSandbox

Resizing my Ubuntu Server AWS Boot Disk

AKA: Building a Bigger GeoSandbox

(Note: This article has been updated to make it clear that expanded EBS volumes will = additional charges from AWS. Something that is not clearly stated in the AWS documentation.)

If you’ve been reading my last few blog posts, you know I’ve been experimenting with various Ubuntu server configurations using Amazon Web Services (AWS) to serve web-maps and spatial data. As my procedures have evolved, the micro-instances I started working with have outgrown their usefulness. Lately, I’ve been testing GeoWebCache, and seeing how that works with GeoServer and the rest of the OpenGeo Suite. As anyone who’s ever delved into the map-tile world knows, tile caches start gobbling up disk space pretty quick once you start seeding layers at larger scales. I had to figure out a way to expand my storage space if I wanted to really test out GeoWebCache’s capabilities without bringing my server to its knees.

The Ubuntu AMIs I’ve been using all start out with an 8GB EBS volume as the root drive with an additional instance-store volume that can be used for “ephemeral” storage. That “ephemeral” storage means, whatever is in there is lost every time the instance is stopped. Supposedly, a reboot will not clear out the ephemeral storage, but a stop and then start, will. There are procedures you can set up to save whatever is in the ephemeral instance-store volume before you stop it, but I was looking for something a bit easier.

A medium instance AMI includes a 400GB instance-store volume, but it still starts out with the same 8GB root drive that a micro instance has. So, what to do? How do I expand that 8GB disk so I can save more data without losing it every time I stop the system?

A little searching led to a couple of articles that described what I wanted to do. As usual, though, I ran into a couple of glitches. So, for my future reference and in case it might be of some help to others, the following paragraphs describe my procedure.

The two articles this post was compiled/aggregated/paraphrased from are:

The standard “Out of the Box” Ubuntu AMI disk configuration

First, connect to the server using WinSCP, SecPanel, or some other means as described in one of my previous posts. Then open a terminal (or PuTTY) window, and enter:

df -h

You should see something like this:

The first line (/dev/xvda1) is the EBS root disk, and it shows 8.0 GB, with about 3.1 GB being used. The last line (/dev/xvdb) is the instance-store “ephemeral” space that’s wiped clean on every stop.

Note: The Ubuntu AMIs use “xvda1” and “xvdb” as device identifiers for the attached disks and storage space, while the AWS console uses “sda1” and “sdb”. In this case, “xvda1” equals “sda1”. Keep this in mind as you’re navigating back and forth between the two.

Step One: Shut It Down

First, look in the AWS console, and make a note of what availability zone your server is running in. You will need to know this later on. The one I’m working on is in “us-east-1d”. Then, using the AWS console stop the EC2 instance (Do not terminate it, or you will wind up rebuilding your server from scratch). Then move to the “Volumes” window, choose the 8GB volume that’s attached to your server, and under the “More…” drop-down button, choose “Detach Volume”. It will take some time for the detach action to complete.

Step Two: Make A Copy

Next, with the same volume chosen, and using the same “More…” button, create a “Snapshot” of the volume. I recommend you give this (and all your volumes) descriptive names so they’re easier to keep track of.

Step Three: Make It Bigger

Once the snapshot is done processing, it will show up in the “Snapshot” window. Again, giving the snapshot a name tag helps tremendously with organization. Choose this snapshot, and then click on the “Create Volume” button.

In the Create Volume dialog, enter the size you want the new root disk to be. Here, I’ve entered 100 GB, but I could enter anything up to the nearly 400GB of storage space I have left in my Medium Instance. Also in this dialog, choose the availability zone to create the volume in. Remember earlier in this post when I said to note the availability zone your server is running in? This is where that little piece of information comes into play. You MUST use the same availability zone for this new, larger volume as your original server volume used. Click the “Yes, Create” button, and a new larger volume will be placed in your list of volumes.

Step Four: We Can Rebuild It

Next, attach the new larger EBS volume to the original Ubuntu server instance. Go back to the Volume window, choose the newly created larger volume, click the “More…” button, and choose “Attach Volume”.

In this dialog box, make sure the correct instance is showing in the “Instance” drop-down. In the “Device” text box, make sure the device is listed as it is shown above. It should be “/dev/sda1”. Note: This will not be the default when the dialog opens. You must change it!

Clicking on the “Yes, Attach” button will begin the attachment process, which will take a minute or two to complete. Once it’s done, you can spin up the server with the new root drive and test it out.

Step Five: Start It Up Again

Choose the server, and under “Instance Actions”, choose “Start”. Once started, connect to the server using your preferred client. Open a terminal or PuTTY window, and once again enter:

df -h

You should now see something like this:

Notice the differences from the first df command. Now the root disk (/dev/xvda1) will show a size of 99GB, or whatever size you might have made your new volume.

More Room To Play

Now I can adjust my root disk size to suit the task at hand. I can store more spatial data in my GeoServer data directory, and seed my map tiles down to ever larger scales. Knowing how to shuffle and adjust these volumes opens up a slew of other possibilities, too. I can imagine setting up a separate volume just to hold spatial data and/or tiles, and using that to copy or move that data back and forth between test servers.

Be mindful though, this extra space is not free. The larger EBS volume does not replace the space on the ephemeral instance-store volume, it is an addition to it. There will be additional charges to your AWS account for the larger EBS volume based on it’s size. This fact is not made clear in the AWS documentation. So, I recommend you increase the size of the EBS root disk as much as you need, but no more.

Oh the possibilities…

GeoSandbox 2.0

This Week’s Project

I decided it was time to take the GeoSandbox to the next level. As usual, I had multiple goals in mind for this project. Primarily, I wanted to learn how to use and write JavaScript. It’s been a long time since I’ve tried to write any kind of meaningful code, but I figured if I wanted to keep up with the big kids, it was time to take the plunge. I also wanted to get my map viewer into a state where I could make it useful enough to display my GIS projects, and help promote my business.

My Experience with Writing Code

My first exposure to computer programming was back in 1980 when I took a computer language class as an undergrad. I remember trying to write some code for a fictional car dealership inventory system, and saving it on a 8” floppy disc. I believe it was in Basic, but I really can’t remember. I never did get it to work. Since then I’ve played around with various versions of Visual Basic, but never got very far past the “Hello World” types of programs. I took a GIS class as a grad student back in 2001 where I used MapObjects to make a functional GIS program. That was fun, and I did get that to work, but I haven’t written any real code since then. Yes, I do read, use, and modify scripts in ArcGIS. That keeps me in tune with the basics of programming, but I don’t think it’s the same as writing real code.

The Goal

I wanted to make the following three enhancements to the GeoSandbox:

  • Change the buttons in the toolbar
  • Make the map window dynamically change size with changes to the browser window size
  • Have the map window change its contents through links in order to showcase different projects

The first two items were fairly easy. I was able to read through the GeoExplorer.js script, and just delete the buttons I didn’t want showing on the toolbar. I also made some changes to the About box, and figured out how to generate a proper Google API key that allows the embedded 3D viewer to show. Dynamically resizing the map window was just a matter of doing a search, copying some code, and pasting it in the right place on the web page. Yes, I had to modify the <iframe>  ID in the code, but that was simple enough.
I had it in my head that I wanted to make four maps available on the page, and to allow users to switch between these maps using a set of links along the top of the map. Seemed simple enough.

No, it wasn’t.

Setting up the webpage and the GeoServer maps was easy. Here’s what the page looks like:

GeoSandbox2-0

And you can access the page using this link: www.donmeltz.com/maps

Changing iFrames

What I found out along the way was, banging your head against an <iframe> can really hurt.

I started out by putting a single <iframe> on the page. Since the src attribute of said <iframe> specifies the URL of the document to show in the <iframe>, I figured I’d just change the src attribute to point to the various GeoServer maps I wanted to show. That didn’t seem to work for me. What I found was, there are many different ways to change the <iframe> src. I must have tried them all, and I got most of them to work. The problem was, they didn’t work in every browser I tried. Here’s a list of the browsers I used in my tests:

  • Firefox 3.1.15
  • Internet Explorer 9 beta
  • Chrome 9.0.597.107
  • Safari 5.0.3
  • Opera 10.10

Learning About iFrames

First, I’d like to mention the Dynamic Web Coding website has the most comprehensive and understandable information on scripting <iframes> that I’ve found: http://www.dyn-web.com/tutorials/iframes/
What I haven’t figured out yet is – Why do all of the examples used on that website (and many others I visited) work for them, but not for me? It just boggles my mind. In a nutshell, here are the various things I tried.

I started out by setting up some bare-bones test sites where I stripped out all of the css styling, and deleted all but two links in order to make it easy to modify and test the code, and see what I was doing. I have links to all of these test pages below each of the following paragraphs in case you want to check them out. You can right-click on each page and view the source if you want. Click on the upper part of the page, NOT on the <iframe> map.
(Please note, I have greatly simplified the syntax in these examples so they fit on one line. If you want to see the actual code, please visit the Dynamic Web Coding site, or right click on my test maps and view the source.)

I began by trying to set the <iframe> target directly in the link like this:

<a href=http://www.Maps.html#maps/2 target="mapiframe">link</a>

This worked in IE and Firefox, but only if I hit the browser refresh button after I clicked the link. Yes, I also went through many iterations of adding a variety of refresh functions to the code, but that only worked in one or two instances. And in some cases it made things even worse. Here’s a link to my test page for this trial: http://www.donmeltz.com/mapstest1.html

Next up was an attempt to set the iframe source (scr attribute) using some JavaScript code as in:

document.getElementById('MapFrame').src="http://Maps.html#maps/2"

I tried using both a function and embedding the code in the link using the onClick event. Results were similar to the previous attempt, and again, inserting a refresh function did not help. There are other variations of this, such as using “window.getElementById, etc. which I also tried, but to no avail. Here’s the link to my test page for this trial:
http://www.donmeltz.com/mapstest2.html

I then moved on to trying to assign the contents of the iframe using code like this:

contentWindow.location.assign("http://Maps.html#/2")

which worked in IE (again, with a browser refresh) but not in anything else. The trial page: http://www.donmeltz.com/mapstest3.html

What came closest to getting the task done was working with the <div> that holds the <iframe> instead of manipulating the <iframe> directly, as in:

<div id="MapContainer">
MapContainer.innerHTML = "<iframe id="MapFrame" src="http://Maps.html#maps/2">

This one worked for me in every browser except Firefox: http://www.donmeltz.com/mapstest4.html

In the end, what I wound up doing was placing four different iframes on the page. I then set the style.display property of the iframe to “none” (hide) or “block” (show) in order to hide or show each one based on which link is clicked on the web page and looks something like this:

document.getElementById("MapFrame").style.display="block";

Yes, “block” means “show”. Intuitive, I know. Here’s a link to my test page that shows how this one works: http://www.donmeltz.com/mapstest_adinfinitum.html
This works in every browser I’ve tried, but it does slow down the load times significantly. All four maps load when the page loads. It does make switching between maps faster once everything is loaded, though.

I Got It

So, where did all of this hacking get me? I got what I wanted. A workable map viewer that shows a few samples of my work, and an education in JavaScript. Even though I spent many hours, days even, working with this, I still think it was worthwhile. I’ve learned a lot, even though it doesn’t seem like I got very far. I can read JavaScript code, and understand it a lot better now than I did when I started. I’d still like to figure out if I’m doing something wrong. It seems like these options should work, as it looks like they work on other peoples pages. Yes, I’m still banging my head against the <iframe> once in a while :-)

For now though, I think I’ll just kick back with a bottle of homebrew and a little Jimmy Cliff.

GeoSandbox grows up to be a “Real boy”

A quick update on the status of my little experiment.

This morning I got it in my head that I wanted to access my GeoSandbox in ArcMap through WMS, and allow others access to it through the same mechanism. Through much trial and error, and some timely help from @mcbride_bryan, I was able to accomplish my goal. If you want to try loading any of the data on my GeoSandbox into your GIS client of choice, you should be able to do so through this URL:

http://donmeltz.dyndns.org:8080/geoserver/wms?service=wms&version=1.1.1&request=GetCapabilities

Here’s a screen-shot with all the current layers loaded:

GeoSandboxWMSinArcGIS

Thanks also to @tpstigers for verifying that he could access it in his GIS app of choice, QGIS running in Ubuntu:

GeoSandboxWMSinQGIS

I’m really surprised at how fast these layers load. They seem to load much faster through the WMS than they do in the webpage.

So, I am happy to report that my little GeoServer toy is well on its way to becoming a “Real Boy”

Setting up my GeoSandbox

It’s been 3 weeks since I set up my little GeoServer experiment and started populating it with various datasets. I thought I’d outline the things I’ve been working on, and what I plan on looking at next. I couldn’t have come as far as I have without the generous help of some of my Twitter friends. I’ve noted below each section those who’ve helped me out. I apologize if I’ve overlooked anyone.

Workspace > Store > Layer

My initial step in getting a useful map up and running was to load some vector data. The first thing I’m sure every GeoServer noob learns is, importing data into GeoServer is a three-step process. The first thing I did was set up a Workspace, which is essentially a folder in my “C:Users<username>.opengeodata_dirdata<foldername>” directory. Then I told GeoServer where to look for this folder. Second step was to set up a Store. A Store can hold a single layer, or multiple layers depending on how you import the data. Why a Workspace AND a Store? I have no idea, but that’s what you have to do, so that’s what I did. Third step was to add a layer or layers to the Store. After that, you can add those layers to a GeoExplorer map, and publish it or embed it in a webpage.

EPSG

Somewhere along the way, GeoServer will ask for a spatial reference system (SRS) for the data you’re adding in the form “EPSG:XXXX”. I figured out very quickly that looking up the EPSG for a shapefile using the built-in GeoServer tool is a PITA. A couple of tools were pointed out to me:
Prj2EPSG (http”//prj2epsg.org/search) and Spatial Reference (http://spatialreference.org)
Although the data I use comes to me in many different projections, I use a limited number of spatial references in my output. I now keep a Post-It note on my desk with the 5 EPSG codes I use most often.
(Thank you @mcbride_bryan, @spara)

Styling

Symbolizing vector layers in GeoServer is a little cumbersome, but I was able to figure it out. and it’s not too much of a chore for simple layers. Options are limited, and I’ve been pointed to a few other tools when it comes to styling layers. uDig, Atlas Styler, and Arc2Earth all look promising and I’ll be checking them out in the near future.
(Thank you @mcbride_bryan, @spara)

Rasters

One of my more frustrating episodes was an attempt to get raster data to display in my GeoExplorer map. The GeoServer GUI and documentation led me to believe I could just load an ArcGrid (binary) dataset into GeoServer. I wasn’t able to get that to work, and have been told by some that it’s not supported. Next, I tried a jpeg. I assumed by including a world file (jgw) everything would be hunky-dory. I assumed wrong. It was pointed out to me that a projection file (prj) needs to be included, too. That got the raster to display, but brought my poor little server to its knees. Finally, I settled on a GeoTiff. By using jpeg compression within the GeoTiff, I was able to keep the file size reasonable, and GeoServer handled it just fine.
(Thank you @geowolf, @storm72)

Zoom Levels

For the life of me I could not figure out how to zoom in close enough to make my newly loaded raster data useful. I know I had seen many GeoServer/GeoExplorer maps able the zoom in nice and close, but it just wasn’t happening for me. Finally, it was pointed out that changing the default base layer to something other than Google Terrain fixes that problem. Worked like a charm. Why would the Google Terrain base map limit zooming in close? I have no idea. Just one of those things.
(Thank you @manicmapper)

HTML and CSS

After getting these things working, I decided to dress up the web page that holds my GeoExplorer map a little bit. I have Dreamweaver CS4, and I use that to manage a couple of websites, but it’s really much more than I need, especially for a little single page website. I looked around and found Aptana Studio. After using it for an hour or so, I’ve come to really like it. It color codes and formats the HTML and CSS so it’s readable, and I think the preview feature works better than what Dreamweaver has. I might also give Komodo Edit a try, and see how that compares.
(Thank you @mcbride_bryan)

What’s Next?

Probably some more experimenting with the various SLD editors, brushing up on my HTML and CSS skills, and diving into JavaScript, which I’m embarrassed to say, I haven’t really looked at in over 5 years.

What’s it look like?

Here it is as of January 31, 2011:

GeoSandbox 2011-01-31

And a link to the site so you can see it now:

Don’s GeoSandbox

Serving Maps for Free

The Goal

This weekend I started down the road to fulfilling one of my New Year Resolutions: Get some maps online. And by maps online, I don’t mean posting links to KML files, screen shots, or PDF exports from ArcGIS. I mean full-fledged interactive maps viewable by anyone with a browser. On top of that, I had one more goal. I wanted to do this with as little extra cost to me as possible. This is going to be a learning experience for me, not a paying job.

The Map Server

I’ve put a lot of thought into this. It’s something that’s been on my mind for a long time, and I’ve looked into a few options. Choosing a map server was relatively easy. There appear to be three major players in the map server arena. ESRI’s ArcGIS Server, GeoServer, and MapServer. There are others mentioned on Wikipedia’s Web Map Service page, but these are the three that seem to be discussed most often.  ArcGIS Server was out of the question for obvious reasons. Thousand$ of obviou$ reason$. GeoServer and MapServer are two Open Source options, but GeoServer seems to be the more popular option among the open source crowd. I’m not one to pick an option basedGeoServerStatus solely on its popularity, but in this case, popularity also means more discussion in the forums, which in turn means it’s easier to find a solution to any problems that might arise during my own learning process. So, GeoServer it was. You can download GeoServer directly from GeoServer.org, but I decided to go the OpenGeo Suite Community Edition route. What led me to that decision was their great White Paper on The OpenGeo Architecture, which is the most concise, easy to read explanation of how all the pieces fit together that I’ve found anywhere on the web.

The Web Server

The most difficult decision in this whole process was how to host this beast. Yes, I know, I could just install GeoServer on my laptop and play with it there, but I wanted to make these maps public. Isn’t that the whole point of a server, be it geo or otherwise? I have been looking at Amazon Web Services, and signed up for a AWS Free Usage Tier account when they started offering them. The problem with AWS is, it’s just so darn complicated to get IIS7Managersomething set up. I am getting better at finding my way around there, but I still get lost once in a while. Plus, the free accounts require using a Linux operating system. I’ve been experimenting with Ubuntu at home, but I’m much more comfortable working in a Windows world. And Windows usually means paying for cloud computing space (e. g. $58/month at Rackspace). If you’re interested in looking at the AWS option, I highly recommend you take a look at what @spara has done: Install GeoServer on Amazon EC2 without leaving the browser.

The Home Server

I decided to merge this project with another I’ve been thinking about for some time now: setting up a home-based web server. I have an old Dell 600m laptop that’s just taking up space in my office. I have an extra copy of a Windows 7 Home Premium upgrade disc that comes with IIS7. Both were essentially free (to me anyway, your costs may vary). MyHomeWebServerThe only unknown was, would I be able to open this up for public viewing? The main roadblock to setting up a public server using a home internet connection is the use of dynamic IP addressing and port blocking by some (most?) ISPs. I was able to work around both of these obstacles by using DynDNS’s services, and setting my router and software settings appropriately. I wont go into the intricacies here, but if you do a Google search on either of these terms, a wealth of information will flood your browser.

the steps I followed to get things up and running:

  • Wiped the hard drive on the Dell 600m laptop clean
  • Installed the original XP OS, and then the Win 7 Home Premium upgrade
  • Installed all updates through Windows Update
  • Downloaded and installed the OpenGeo Suite
  • Played around with GeoServer and GeoExplorer enough to get a basic map embedded on a web page
  • Got a free Dynamic DNS host name at DynDNS.com
  • Installed DynDNS Updater
  • Modified my router settings to make sure the proper ports were open and forwarding to my laptop server
  • Tested, changed settings, tested, changed settings, tested, tested, tested
  • Scratched head, changed settings, tested, swore a few times, had a beer…
  • Installed and set up Internet Information Services (IIS7)
  • Set up a website using the web page with the embedded map I made with GeoServer/GeoExplorer
  • Went through the whole “testing-scratch head-drink a beer” routine again
  • Celebrated the results

Here’s a snapshot of the results on day 1:

Don's GeoSandbox RC1

And a Link to  what it looks like now:

Don’s GeoSandbox

I’ll try to keep this link up-to-date, but this is all a test, and it is running on a home computer, so don’t be surprised if it’s not available 24/7.
Some of you that have seen this before this post asked the question”Great, but where’s the data?”
Well, that’s coming. Please be patient with me people :-). Baby steps. Baby steps.

Addendum 01/12/2011

I now have a few shapefiles uploaded and viewable on the site. Next step will be to dress up the web page a little bit.