Open Source Spatial Data Processing Suite

Gluing Things Together

As I described in a previous post, Boundless no longer maintains publicly accessible versions of Boundless Suite nor the suite formerly known as OpenGeo in their repositories. The Boundless Suite (now known as Boundless Server) is available on their GitHub page, but requires building from a cloned version of that GitHub repository. I’ve used Apache Ant and Git to build a few things in the past, usually with limited success. I looked through the steps involved there and quickly decided to try another approach. I figured – Why not try assembling all of the pieces included in Boundless Suite on my own? They are all open source projects after all. The main benefits of the Boundless/OpenGeo Suite are/were that the components have been tested and verified to work together, and then packaged together into a compact installation process. Why can’t I do some of that myself?

The Core Components

(PRNewsFoto/Boundless)

The pieces that could go into a full suite of spatial tools is nearly endless. The specifics will depend on the needs of the person using them. For some, using Python, R, PostGIS, and maybe Leaflet or another javascript library to post a map online is all they need. All of those are in my toolbox already, and will eventually make their way into my homemade suite. For me however, the three main elements of an Open Source Spatial Data Processing Suite are:

  • A database to store and retrieve geographic data
  • A desktop client to manage, process, and otherwise manipulate the data
  • A server to share the data publically and facilitate displaying it in a web map

Specifically:

In the past it was easy. I would start with a fresh Ubuntu server. I’d add the OpenGeo repository to my /etc/apt/sources.list, import the GPG key, update the cache, and then enter one line into a terminal window:

apt-get install suite-dashboard suite-geoserver suite-geowebcache suite-composer suite-docs suite-quickview suite-gs-gdal suite-gs-wps suite-wpsbuilder suite-gs-geopkg postgresql-9.3-postgis-2.1

Then I’d restart the tomcat server: service tomcat8 restart navigate to http://donmeltz.com:8080/dashboard/ and I’d get this:

Boom. Done.

Not so simple anymore. Let’s start with the easiest piece – The desktop.

The Desktop Side – QGIS

I still want to be able to use some of the plugins Boundless makes available for QGIS. These plugins are tested with the latest Long Term Release. The Boundless repositories do not include plugins for QGIS 3.x, so QGIS 2.18 it is. Navigate to https://qgis.org/en/site/forusers/download.html and download/install QGIS 2.18 onto your laptop or desktop computer.

Add the GeoServer Explorer Plugin:

This Plugin allows you to connect directly to a GeoServer through QGIS, manipulate some of the configuration settings of the server, add layers stored on the server to QGIS, and upload layers from QGIS to the server.

  • In QGIS, under the Plugins menu, open the “Manage and Install Plugins…” dialog.
  • Add the repository: http://qgis.boundlessgeo.com/plugins.xml
  • Scroll through the list of “Not installed” Plugins and install “GeoServer Explorer”. Make sure there’s a check mark next to it in the Plugins window.
  • Also while you’re at it, make sure the “DB Manager” is installed and checked “On” in the Plugins window.

To be clear, the Boundless Suite install does not include QGIS as the two are meant to be installed on different computers. However Boundless does provide a customized version of QGIS called Boundless Desktop that is preconfigured with the GeoServer Explorer plugin.

The Server Side – GeoServer and PostGIS

I start with a fresh install of Ubuntu. Even though the latest Ubuntu 18.04 release is a Long Term Support (LTS) version, it is still fairly new, and I’ve found the repositories are not yet populated with all of the software packages I like to use. So, I’m sticking with 16.04 LTS for now. All of the commands that follow are designed to work with 16.04, and appear to install everything correctly.

I have both a home server (which is what I used here) and a couple of servers running on Amazon Web Service (AWS). In any case, I need to be able to access the server through PuTTY, WinSCP, VNC (if there’s a display involved), or some other method in order to open a terminal window. I typically have the following ports open: 22, 80, 8080, and 5432.

GeoServer

Sticking with Long Term Support versions of software, I installed GeoServer 2.12.3. I tried the latest stable release (2.13.1), but found the Boundless GeoExplorer Plugin would not connect to that version.

GeoServer requires a Java Runtime Environment and a Tomcat Application Server to run. As the website docs explain, “The Oracle JRE is preferred, but OpenJDK has been known to work adequately.” Wanting to keep this as simple as possible, I stuck with OpenJDK.

  • Install the OpenJDK java 8 runtime environment, in a terminal window:

sudo apt-get install openjdk-8-jre

  • Install Tomcat 8. GeoServer requires Tomcat 7.0.65 or later that implements Servlet 3. Using the Ubuntu 16.04 repositories will install Tomcat 8.0.

sudo apt-get install tomcat8

  • Install GeoServer. Change the current directory to your Download directory and download the GeoServer file

cd ~/Downloads

wget http://sourceforge.net/projects/geoserver/files/GeoServer/2.12.3/geoserver-2.12.3-war.zip

  • Unzip the downloaded file and move it into the Tomcat webapps directory

sudo apt-get install unzip

unzip geoserver-2.13.1-war.zip

sudo mv ~/Downloads/geoserver.war /var/lib/tomcat8/webapps/

  • In order to allow the QGIS GeoServer Explorer plugin to publish layers directly to GeoServer, the GeoServer “Importer” extension has to be installed. Download the Importer extension zip file:

wget https://sourceforge.net/projects/geoserver/files/GeoServer/2.12.3/extensions/geoserver-2.12.3-importer-plugin.zip

  • And since this zip file contains multiple files we’ll unzip it directly into the proper directory:

sudo unzip geoserver-2.12.3-importer-plugin.zip -d /var/lib/tomcat8/webapps/geoserver/WEB-INF/lib/

  • Restart Tomcat

sudo service tomcat8 restart

  • Note for future reference – starting, stopping, restarting Tomcat:

sudo service tomcat8 start

sudo service tomcat8 stop

sudo service tomcat8 restart

You should now be able to access GeoServer by going to:
http://<your server ip>:8080/geoserver

PostgreSQL/PostGIS

We’ve got a desktop client. We’ve got a remote server. Now we need a place to store some data that’s accessible to both. So… Install PostgreSQL and PostGIS (again in a terminal window):

  • Add the appropriate repository to sources.list (in this case, for “xenial”, which means Ubuntu 16.04):

sudo add-apt-repository "deb http://apt.postgresql.org/pub/repos/apt/ xenial-pgdg main"

  • Add keys:

wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -

sudo apt-get update

  • Install the software packages:

sudo apt-get install postgresql-10

sudo apt-get install postgresql-10-postgis-2.4

sudo apt-get install postgresql-10-pgrouting

  • To get the command line tools shp2pgsql, raster2pgsql:

sudo apt install postgis

  • Connect to the postgres database using the command line tool psql as local user (The default PostgreSQL username is postgres)

sudo -u postgres psql postgres

  • Set the password for the postgres user. (Normally there is no password set for a PostgreSQL database. But since we want to be able to access the database remotely through QGIS, we’ll have to open it up to the world. So, password protection it is.)

\password postgres

  • Enable advanced administration for pgAdmin

CREATE EXTENSION adminpack;

  • Enable the PostGIS extension

CREATE EXTENSION postgis;

  • Enable the pgRouting extension

CREATE EXTENSION pgrouting;

  • Verify the version of PostGIS and PostgreSQL

SELECT postgis_full_version();

  • Exit psql

\q

  • Here’s where we allow remote connections to the database by editing a couple of files (using vi)

sudo vi /etc/postgresql/10/main/postgresql.conf

Hit the “Insert” key to enter editing mode

Change the line that says: #listen_addresses = ‘localhost’

to: listen_addresses = ‘*’ (remove the initial hashtag to uncomment the line and change local host to an asterisk which means ‘listen to everything’)

Hit the “ESC” key to exit editing mode

Hit “Shift :wq” and then enter to save the edits

sudo vi /etc/postgresql/10/main/pg_hba.conf

Again, using vi, add the following line to the end of the list of allowed host connections:

host    all             all             all               md5

Save and exit vi.

  • Restart PostgreSQL

sudo service postgresql restart

Results

What does all this get me?

I can now open QGIS, go to the “Web” menu and open GeoExplorer. Connect to my GeoServer using my username/password, and view all the layers stored in the various workspaces, adding them to my QGIS project.

 

 

 

 

 

 

 

 

 

I can use pgAdmin on my desktop computer to connect to and administer my remote PostGIS database.

 

 

 

 

 

 

 

 

I can use QGIS to directly access my PostGIS database, uploading layers from QGIS to it, or adding layers to QGIS from it.

  • Layer > Add Layer > Add PostGIS Layers…
  • New Connection
  • Enter the Host ip address, database name, username, and password.
  • Connect
  • Choose a layer in the database and then “Add”

Or if the database contains no layers, I can add them to the database using the database manager:

  • Database > DB Manager > DB Manager
  • Choose the previously connected PostGIS database
  • Use the Import Layer/File button to add a layer from QGIS to the database
  • Use the Export to File button to save a database layer to a wide variety of formats.

EDIT:

When I first posted this, there was one piece of functionality I hadn’t got working yet. I should be able to publish a layer directly from QGIS to GeoServer using the GeoServer Explorer plugin. When I tried to do so, I got an error message:

What I found out was – the GeoServer Importer Extension has to be installed on GeoServer in order for the QGIS GeoServer Explorer plugin to be able to publish layers directly to GeoServer. I’ve updated the steps needed to do this in the process outlined above.

To upload and publish from QGIS:

  • Open the GeoExplorer plugin
  • Connect to your GeoServer
  • Choose the “Publish layers to this catalog” button
  • Select the layers you want to publish from the list, and the workspace to publish to
  • Hit “OK” and you should then be able to see your QGIS layer in your GeoServer

Hope this helps. I’m open to advice if you have any suggestions for improvement.

 

 

 

 

 

Impetus to Blog

Yesterday I posted a blurb that was mostly a list of reasons why I haven’t been blogging.
Today’s blurb is a list of the things that have spurred me on to start again.

What are the new motivating forces that have me wanting to write things down in public?
There are three:

  1. My change in employment
  2. The removal of freely downloadable compiled versions of Boundless software
  3. My Home Server needs some attention

Change in Employment

I touched on this yesterday. I have accepted a position as Senior Planner at the Columbia County Planning Department. This does not mean “Don Meltz Planning and GIS” will cease to exist. It does mean I will wind things down a bit, and will be more selective in the jobs I take on. I won’t be able to work on projects within Columbia County. And those I work on outside Columbia County will have to fit into a weekends-and-evenings schedule. I’ll also continue teaching at Marist College.

What it does mean is – I have a lot of ideas floating around in my head now. The state of the entire county’s GIS is this: The Real Property Department uses AutoCAD Map for all their mapping work, spitting out shapefiles when needed. The Planning Department has one single-use license for  ArcGIS Standard. That’s it. It is the proverbial “Blank Slate”. It is both an exciting and daunting position to be in. I’ll need to develop an action plan in order to get organized and stay focused. Writing things down via blog posts will help.

Compiled versions of Boundless Suite no longer available

As I began thinking about what such an action plan might look like, I wandered over to the Boundless website to see what was new. What I found there was a little disappointing, but not totally unexpected. Apparently, Boundless will no longer be posting compiled versions of Boundless Suite nor the former OpenGeo Suite. And, the Ubuntu repositories for these packages are no longer available. What this means is, I’ll either have to build a Boundless Server from the GitHub repo, or assemble all the pieces that make up the “suite formerly known as OpenGeo” from the various community orgs (i.e Geoserver, Geoserver extensions, PostGIS, QGIS, QGIS Plugins, etc.). Knowing my own capabilities, I believe assembling the various parts will be easier for me to accomplish than building from the GitHub repo. It’s something that I’ve been thinking about doing for a long time, anyway. Writing things down via blog posts will help me keep track of any pitfalls I run into.

Home Server Attention

About once a year I physically and virtually open up my home server to clean out the dust bunnies, and to organize and delete any files that are cluttering up the hard drives. This server is  mostly a place to store nightly backups of our other household computers. But I also use it as a test bed for things that need to connect to or through the internet. Upon last inspection, the now six-year-old fans started to rattle a bit, and I noticed it’s still using Ubuntu 12.04 LTS and OpenGeo Suite 4.1.1. Time for an upgrade on both accounts. This provides an ideal opportunity to take apart my server (both physically and digitally), and make an initial attempt at stitching together all those pieces of the suite formerly known as OpenGeo.

Plenty to write about

All of this will give me plenty of material to digest and write about. However, be warned. A lot of my future writing is going to be about basic GIS implementation. This may disappoint some of my more avant-garde twitter followers. The fact that I am not doing everything by tying together a remote PostGIS database and a bunch of R functions using nothing but Python scripts and posting the results to a Bootstrapped Leaflet webpage via GeoJSon is going to annoy you.

So be it.

Back After a Long Hiatus

It’s been a long time.

My last blog post was on May 9, 2013. That’s 5 years and 8 days ago. Why the long pause, and what’s happened during those intervening years? I had to look through my records to figure it out myself. Here’s what I found:

Mom

Back in 2013, my Mom was dealing with some difficult health issues. I was spending more time bringing her to doctor’s appointments. She was in and out of the hospital having various procedures done, and following up with visits to her primary care physician and cardiologist. She passed away in May 2014. At about the same time, I had taken on a few large projects with another planner, including working on three County Agriculture and Farmland Protection Plans all at the same time. (Note to self – Never, ever do that again).

I had no spare time.

 

Dad

Shortly after this period, in 2014, my Dad’s partner’s declining health left him with no one to help him with his antique/classic auto hobby. Dad lives for the annual Hershey AAC swap meet, and other car and truck shows throughout the year. So, I stepped in to fill that void. I got back into the hobby myself, setting up my own automobilia business. I continue helping Dad out loading and unloading heavy boxes of car and truck parts, and driving the long distances to various car shows today.

I had no spare time.

 

Donnelly Hall, Marist College

Donnelly Hall, Marist College

In 2015 I was approached by the chair of the Environmental Science & Policy department at Marist College and began teaching my first class in the fall of 2016. For anyone who has not taught either High School or College classes before, I can tell you – it is an incredibly time consuming task to develop an entire semester’s curriculum from scratch. I had two courses to work on – an Intro to GIS course and and Advanced GIS course.

I had no spare time.

In 2016 there began a change in the makeup of our County Planning Department which continues today. I’ve been working as a self-employed planning consultant for 15 years and saw an opportunity for a change myself. I enjoy my work as an independent planner, but working at the same place for 15+ years wears on you, even if it is your own business. I needed something different. I needed a new challenge. Without going into the lengthy details, I have accepted a position in the Columbia County Planning Department. I do not have a start date yet, but assume it will be sometime within the next two months. I’ll talk about this more in future blog posts once I begin working there.

I still have no spare time, but I do have a new reason for blogging.

I’ll be doing something that’s new to me. I’ll need to organize my thoughts. The best way for me to do that is to write stuff down. It’s how I’ve always worked. Whenever I need to work through a complex problem I take out a pencil and a pad of paper and just start writing things down. That’s why I started blogging in the first place. It wasn’t to show off my skills as a planner or spatial analyst. It was to organize my thoughts.

And so, with many new thoughts to organize, I believe I will begin blogging again.

Talk to you soon.

– Don

Willamette River Historical Stream Channels Poster

Willamette Stream Channels Poster

Willamette Stream Channels Poster

Today’s MOTD is something that was going around the Twitter-sphere a few weeks ago. I put a link to it on my desktop, and re-found it tonight while organizing my files. Even though it’s not brand new, it is special enough to be noted here.

It is a spectacular piece of cartographic artwork. It’s so unique it almost doesn’t look like a map, but that is exactly what is is.

As the publication website says, this is a lidar derived digital elevation model of the Willamette River. Lidar (light-radar) is a technology that uses light to measure distances, and can be used to produce 3D images of the earths surface (i.e. a digital elevation model)

Basically, this is a topographic map. The brightest white areas are the lowest elevations, which are the existing stream channels. Older, historic stream channels show up as depressions in the landscape, shown in various shades of light-blue.

For more information about how this map was made, and to purchase a 17″ x 38″ poster, visit the Oregon Department of Geology and Mineral Resources website.

Delhi Kiosk Map

One of the projects I’m working on involves an economic redevelopment plan for a five town – two village area in the Central Catskills. One of our recommendations to them involoves developing a unified way-finding and mapping system that will help visitors find their way around and through the area. While doing some research on the subject, one of our team members found this cool map recently displayed in the Town of Delhi, NY. Delhi (pronounced “Dell-high”) is also a Catskill Mountain region town, but outside our study area.

DelhiKioskMap

Instead of going into detail about how the map was made, I’ll just point you to the bovinabrownbats website and their post titled: A new map for Delhi!

I will tell you, it was not done in ArcGIS, nor was it done using any opensource GIS tools. However, I do think someone proficient in Photoshop might be able to come up with a 2-dimensional version, if they tried really hard.

Grateful Dead Archive Online – Map

This is how my day went today:

  1.  A friend on Facebook (John Baumbach) posted a link to a page listing the 20 Essential Grateful Dead Shows. He mentioned how he had been to two of them.
  2. This got me thinking about one of my most memorable concerts. I did some searching, and found an archive page with a recording of the Bob Weir concert during my freshman year at Franklin Pierce College on March 4, 1978.
  3. Surprised to find this online, I thought: “Maybe there’s an archive of old Grateful Dead concerts, too”.
  4. A little more searching led me to the Grateful Dead Archive Online website.
  5. …where I then found this cool map, showing many or most of the venues and concerts the Dead played from 1965 to 1995. (I don’t think it’s possible to map them all, is it? I could be wrong.)
  6. I wound up listening to old Grateful Dead tunes the rest of the day.

Now I’ve never been much of a Dead Head. One of the reasons is, the college I attended during my undergrad years had a student body that was probably 75% Dead Head. I’ve got nothing against the Dead. I’ve actually grown to like their music more in recent years. The problem at that time was, most of the students at FPC were Dead fans because they liked the idea of being a Dead Head more than they really liked their music. Somehow, that just turned me off to the whole Grateful Dead scene.

I was talked into going to one Dead concert during my time at FPC, mainly because my friends needed a ride, I had a car with gas, and they offered to pay for my ticket. To the best of my recollection, this is the only Grateful Dead concert I ever attended.

I like this map for a few reasons.

  • It’s simple
  • It looks like it’s all done with JavaScript, jQuery, Google API. Again, simple
  • I like how the pop-ups work, leading to off-map webpages with more information (something I want to incorporate into another project I’m working on)
  • The content is mostly crowd-sourced. There are tons of fan-submitted recordings of the concerts, posters, pictures, and other paraphernalia.
  • Thanks to Michael Terner (@MT_AppGeo) for pointing me to this 2010 article in the Atlantic, describing how The Dead were early innovators in social networking and open-sourcing their music as a way to promote themselves.

Map of Grateful Dead Shows:

(Courtesy, Special Collections, University Library, University of California Santa Cruz. Grateful Dead Archive Online Collection.)

From the GDAO Help page: GDAO is built upon open-source software including the Omeka web-publishing platform, the SOLR search server, the Djatoka Java based image server, and the Seadragon image viewer.

OpenGeo Suite 3.0 on a micro AWS

The Problem: I want to run the latest 3.0 version of OpenGeo Suite on a free (or really cheap) micro instance on Amazon Web Services

OpenGeo announced the release of version 3 of the OpenGeo Suite Monday (Oct.3). I’ve been using the 3.0-beta1 Linux version since it was announced on July 27. There are some interesting improvements to the Suite, which is one reason I made the jump before the final release came out. It now includes PostgreSQL 9.2 and PostGIS 2.0, both of which I wanted to look into.

I had been using previous versions of OpenGeo Suite on a micro instance AWS ubuntu server. This configuration was obviously not optimal. Redraws in GeoExplorer where slow, and I could tell the system was struggling at times. CPU usage went up to 100% quite often, but it did work. Performance was acceptable enough for the kind of experimenting and testing I wanted to do.

With the 3.0 upgrade, however, something pushed it over the edge. Everything installed OK. I was able to upload my usual test data, and get a website with a web-map up and running. However, it would not last. It just wasn’t as stable as previous versions. Zooming and panning the map would crash the tomcat servlet within minutes. Even just letting it run with no interaction would lead to a crash within a few hours.

A few pointers from the folks at OpenGeo, and some investigation of the logs, led me to believe it was a memory issue. AWS Micro instances only have 613MB of memory.

The Answer: Add a swap file to overcome the memory limitations of a micro instance

AWS micro instance Ubuntu servers do not come set up with any swap space. Fortunately, it’s fairly easy to add a swap file to the server, and use that as your swap space. Here are the steps:

1. Create a storage file (Adjust the “count=” line to your liking. This example will make a 1 GB swap file)

sudo dd if=/dev/zero of=/swapfile bs=1024 count=1048576

2. Turn this new file into a swap area

sudo mkswap /swapfile

3. Set the file permissions appropriately

sudo chown root:root /swapfile

sudo chmod 0600 /swapfile

4. Activate the swap area every time the system reboots by adding the following line to the end of the “/etc/fstab” file:
(use your text editor of choice. vi works for me.)

/swapfile swap swap defaults 0 0

5. Reboot the server

6. Verify the swap file is activated

free –m

free-m

I’ve had my OpenGeo Suite test box running 24/7 for nearly two months now, with nary a crash. And I can honestly say, it is surprisingly perky.

Resizing my Ubuntu Server AWS Boot Disk

AKA: Building a Bigger GeoSandbox

(Note: This article has been updated to make it clear that expanded EBS volumes will = additional charges from AWS. Something that is not clearly stated in the AWS documentation.)

If you’ve been reading my last few blog posts, you know I’ve been experimenting with various Ubuntu server configurations using Amazon Web Services (AWS) to serve web-maps and spatial data. As my procedures have evolved, the micro-instances I started working with have outgrown their usefulness. Lately, I’ve been testing GeoWebCache, and seeing how that works with GeoServer and the rest of the OpenGeo Suite. As anyone who’s ever delved into the map-tile world knows, tile caches start gobbling up disk space pretty quick once you start seeding layers at larger scales. I had to figure out a way to expand my storage space if I wanted to really test out GeoWebCache’s capabilities without bringing my server to its knees.

The Ubuntu AMIs I’ve been using all start out with an 8GB EBS volume as the root drive with an additional instance-store volume that can be used for “ephemeral” storage. That “ephemeral” storage means, whatever is in there is lost every time the instance is stopped. Supposedly, a reboot will not clear out the ephemeral storage, but a stop and then start, will. There are procedures you can set up to save whatever is in the ephemeral instance-store volume before you stop it, but I was looking for something a bit easier.

A medium instance AMI includes a 400GB instance-store volume, but it still starts out with the same 8GB root drive that a micro instance has. So, what to do? How do I expand that 8GB disk so I can save more data without losing it every time I stop the system?

A little searching led to a couple of articles that described what I wanted to do. As usual, though, I ran into a couple of glitches. So, for my future reference and in case it might be of some help to others, the following paragraphs describe my procedure.

The two articles this post was compiled/aggregated/paraphrased from are:

The standard “Out of the Box” Ubuntu AMI disk configuration

First, connect to the server using WinSCP, SecPanel, or some other means as described in one of my previous posts. Then open a terminal (or PuTTY) window, and enter:

df -h

You should see something like this:

The first line (/dev/xvda1) is the EBS root disk, and it shows 8.0 GB, with about 3.1 GB being used. The last line (/dev/xvdb) is the instance-store “ephemeral” space that’s wiped clean on every stop.

Note: The Ubuntu AMIs use “xvda1” and “xvdb” as device identifiers for the attached disks and storage space, while the AWS console uses “sda1” and “sdb”. In this case, “xvda1” equals “sda1”. Keep this in mind as you’re navigating back and forth between the two.

Step One: Shut It Down

First, look in the AWS console, and make a note of what availability zone your server is running in. You will need to know this later on. The one I’m working on is in “us-east-1d”. Then, using the AWS console stop the EC2 instance (Do not terminate it, or you will wind up rebuilding your server from scratch). Then move to the “Volumes” window, choose the 8GB volume that’s attached to your server, and under the “More…” drop-down button, choose “Detach Volume”. It will take some time for the detach action to complete.

Step Two: Make A Copy

Next, with the same volume chosen, and using the same “More…” button, create a “Snapshot” of the volume. I recommend you give this (and all your volumes) descriptive names so they’re easier to keep track of.

Step Three: Make It Bigger

Once the snapshot is done processing, it will show up in the “Snapshot” window. Again, giving the snapshot a name tag helps tremendously with organization. Choose this snapshot, and then click on the “Create Volume” button.

In the Create Volume dialog, enter the size you want the new root disk to be. Here, I’ve entered 100 GB, but I could enter anything up to the nearly 400GB of storage space I have left in my Medium Instance. Also in this dialog, choose the availability zone to create the volume in. Remember earlier in this post when I said to note the availability zone your server is running in? This is where that little piece of information comes into play. You MUST use the same availability zone for this new, larger volume as your original server volume used. Click the “Yes, Create” button, and a new larger volume will be placed in your list of volumes.

Step Four: We Can Rebuild It

Next, attach the new larger EBS volume to the original Ubuntu server instance. Go back to the Volume window, choose the newly created larger volume, click the “More…” button, and choose “Attach Volume”.

In this dialog box, make sure the correct instance is showing in the “Instance” drop-down. In the “Device” text box, make sure the device is listed as it is shown above. It should be “/dev/sda1”. Note: This will not be the default when the dialog opens. You must change it!

Clicking on the “Yes, Attach” button will begin the attachment process, which will take a minute or two to complete. Once it’s done, you can spin up the server with the new root drive and test it out.

Step Five: Start It Up Again

Choose the server, and under “Instance Actions”, choose “Start”. Once started, connect to the server using your preferred client. Open a terminal or PuTTY window, and once again enter:

df -h

You should now see something like this:

Notice the differences from the first df command. Now the root disk (/dev/xvda1) will show a size of 99GB, or whatever size you might have made your new volume.

More Room To Play

Now I can adjust my root disk size to suit the task at hand. I can store more spatial data in my GeoServer data directory, and seed my map tiles down to ever larger scales. Knowing how to shuffle and adjust these volumes opens up a slew of other possibilities, too. I can imagine setting up a separate volume just to hold spatial data and/or tiles, and using that to copy or move that data back and forth between test servers.

Be mindful though, this extra space is not free. The larger EBS volume does not replace the space on the ephemeral instance-store volume, it is an addition to it. There will be additional charges to your AWS account for the larger EBS volume based on it’s size. This fact is not made clear in the AWS documentation. So, I recommend you increase the size of the EBS root disk as much as you need, but no more.

Oh the possibilities…

Serving Maps – in the Cloud – for Free (part 3)

It was not my intention to make this a 3-part blog post series, but here it is anyway.
(If you want to catch up, you can read Part 1 and Part 2 first).

As I continued to work on, and tweak my new AWS Ubuntu server, I decided I might as well add website serving capabilities to it as well. That would allow me to embed my new web-maps into a customizable web page, allowing a more interactive experience, and a more professional appearance to anyone visiting them. The first step in that direction is to:

Install Apache Server

This is the easy part. Connect to the server with WinSCP/PuTTY or SecPanel/FileZilla as I explained in part 1, and enter this command:

sudo apt-get install lamp-server^

That’s it. Just follow the prompts, and enter a password when it asks. When it’s done installing, there will be a new directory called /var/www on the server. Just copy the servers AWS Public DNS string into a web browser address bar and hit enter. You should see the famous Apache default index.html file:

Voilà. A real cloud based web server, just like the big boys.

Now, how do I connect to this one? It’s possible to use the same procedure as I did with OpenGeo/GeoServer. However, I really want to make things easier on the webmaster (aka, Me). I want to be able to use regular old FTP to access the website, which will allow me to use a wider variety of tools, like DreamWeaver (Yes, I said it. DreamWeaver) to edit and manage the website files.

Enable Password Authorization

The default setting for the AWS Ubuntu AMIs (and I believe, all AMIs) is to require key pairs for authenticating users. Password authentication is turned off. To turn it on, the /etc/ssh/sshd_config file has to be edited. The easiest way to do that, is to use VI. VI is scary. It runs in the terminal window. It has a black screen, with multi-colored text that makes the text look like code. I’m not going to try to teach anybody how to use VI because, well, I just learned how to use it yesterday myself, and I only know about 5 commands. However, if you want to follow along, I’ll outline the exact steps I took to edit the sshd_config file in order to allow users to login using passwords.

In the terminal or PuTTY window, open the sshd_config file by entering:

sudo vi /etc/ssh/sshd_config

Then:

  • Enter INSERT mode by typing a (Yes, that’s the lower case letter a)
  • Using the arrow keys on the keyboard, scroll down to the line that reads
    PasswordAuthentication no
  • Right arrow over to the end of the line and backspace twice to erase no
  • Type yes
  • press the escape key on the keyboard (ESC. This exits edit mode, and allows typing in commands)
  • Type :w and then enter (Yes, that’s a colon before the w. This saves the file)
  • Type :q and then enter (Again, a colon before the q. This exits VI)

That’s it. Passwords are allowed for login now. However, when I tried to apply a password to the default ubuntu user, it did not work. There might be a way around this, but I haven’t found one yet.

What to do?

Add a New User

Back in the Terminal/PuTTY window, type:

sudo adduser NewUser

Where NewUser is whatever you want it to be. Enter a password, and fill in the other information if you want to. Everything but the password is optional. Restart Ubuntu, either by entering
sudo reboot
in terminal, or by using the AWS Management Console.

Now, that allows the NewUser to login using the AWS Public DNS string, and his/her password using regular old FTP (actually, SFTP on port 22 if you have the security settings set as in Part 1). In FileZilla:

NewUser can now add and delete folders, and move files back and forth in the /home/NewUser directory. But the whole purpose of adding this new user is to enable uploading and editing in the /var/www folder, where the website files are stored. So…

Give NewUser Access to the www Folder

To give NewUser access to the website’s root folder, enter this command in the PuTTY/Terminal window:

sudo chgrp NewUser /var/www

Then, to give NewUser the ability to add, delete, and edit folders and files in the website’s root folder, enter this command in the Terminal/PuTTY window:

sudo chmod 775 /var/www

CAVEAT: I am not a professional systems administrator. I have done a little bit of research into how the root folder of a website should be set up, and what level of access should be granted to various types of users. And I can tell you, there is no definitive answer. All I know is, these settings work for me. How you set your permissions for various users on your web server are completely up to you.

One Last Tip

Through this entire 3 part blog series, I’ve been using the AWS Public DNS string to access the AWS server, and that works just fine. However, it’s a bit cumbersome to continually open up the AWS console copy the PublicDNS, and paste it into a web browser. Plus, if you ever terminate a server and spin up a new instance, the Public DNS changes. So that means any links you’ve posted leading to it are now broken.

The answer? Elastic IP

The best thing about Elastic IPs is, they’re FREE. They’re also very easy to set up. Just click on the Elastic IPs link on the left side of the AWS Management Console (EC2 tab), and click the Allocate New Address button. Then Associate the new IP address to your server, and you’re good to go.

Now, what used to look like this:
ec2-107-21-252-45.compute-1.amazonaws.com

Looks like this:
107.21.252.45

Just remember to Release the address if you ever disassociate it from your server. The Elastic IPs are free if you use them. If you don’t use them, Amazon charges you for them.

GeoSandbox – In the Cloud

So, After about 5 days of work, and 3 days of blogging (a record for me) I now have what I was after. A custom web map served from a cloud-based geo-web-server. You can check it out at:

http://www.ubugeocloud.com/maps/index.html

Now I’ve got a real sandbox to play in.

Serving Maps – in the Cloud – for Free (part 2)

(Note: This is the second part of a 3 – part blog post about setting up the OpenGeo Suite on a AWS Ubuntu server. Links to the other parts are at the bottom of this post)

Starting Fresh with a New AMI

At the end of my last post, I had my AWS Ubuntu-micro-server running smoothly, but the OpenGeo GeoExplorer was not very stable. It was crashing often, and for no apparent reason. I followed up with a few suggestions about data directory permissions, and swap-file space, but to no avail (Thank you @spara and @jeffbarr). I had been tweaking things quite a bit on that server, (The whole purpose of this exercise is to learn how things work, right?) so I decided to wipe the slate clean and start from scratch.

I began by looking for a different ami. A bit of searching led me to the Ubuntu Cloud Portal – AMI Locator, which facilitates searching and filtering all of the Ubuntu AMIs available. At the bottom of the table, I chose “Zone: us-east-1”, and “Name: oneric”.

UbuntCloudPortal

I then clicked on the ami-a562a9cc link, (a 32-bit ebs server) which then opened up the Request Instances Wizard that I talked about in the last post.

Following everything I outlined in part-1, I wound up with a shiny new Ubuntu server connected to my Windows machine through WinSCP and PuTTY.

WinSCP-PuTTY

In the PuTTY window, I entered the the following commands to make sure the new server was up to date:

sudo apt-get update
sudo apt-get upgrade

Here’s a hint: The PuTTY window does not have any menus or toolbars, and control-v does not work for pasting text. If you copy the above commands, and then simply right-click in the PuTTY window, the commands will be pasted in. Hitting enter will then run them.

Install the OpenGeo Suite

Next up, is getting the OpenGeo Suite installed. I’ve described this process in other posts, but here it is in short form. Just remember to substitute <YourAWSPublicDNS> with your actual Public DNS string, which looks something like this: ec2-75-101-170-100.compute-1.amazonaws.com.

  • In the PuTTY window (or terminal if you’re using some form of Linux), sudo to root:

sudo su

  • Then enter these commands. I’ve found they work best if they’re entered one at a time:

wget -qO- http://apt.opengeo.org/gpg.key | apt-key add -
echo "deb http://apt.opengeo.org/ubuntu lucid main" >> /etc/apt/sources.list
apt-get update
apt-cache search opengeo
apt-get install opengeo-suite

  • Back in the AWS Management Console, choose the server instance, go up to the “Instance Actions” button, and click Reboot
  • Once it’s finished rebooting, test the OpenGeo Suite
    • In a browser window, go to: http://<YourAWSPublicDNS>:8080/dashboard/
    • Launch GeoExplorer
    • Click the Login button on the right end of the toolbar.
      • Default Login credentials are User: admin, Password: geoserver
    • Make any changes to the map you want
    • Save the map (There is a save map button on the toolbar)
    • …and exit GeoExplorer

The map should now be publicly viewable at:

http://<YourAWSPublicDNS>/geoexplorer/viewer#maps/1

Here’s what mine looks like:

GeoExplorer

Now I have a real cloud-based web-map- server up and running. But wait. There’s more. The next step to making this a truly useful map server, is to add some custom data to it.

Upload some Data

Using WinSCP, I added a new folder under the /home/ubuntu directory.

  • Travel to the “/home/ubuntu” directory on the remote side
  • Right click > New > Directory…
  • Name the new folder, and make sure permissions are set to
    Owner: RWX, Group: R-X, and Other: R-X, (Octal: 0755), otherwise, upload and GeoServer access will not work

GeoDataDirectory

    • In the Local panel, I made my way to where I store GIS data on my workstation lappy. This particular folder holds all the shapefiles I plan on using with any of my OpenGeo Suite/GeoServer boxes, and they’re all in Web Mercator projection (EPSG: 3857).
    • Highlighting the files I want to upload on the Local side, I then drag and drop them into the new remote folder
    • Upload promptly ensues

Next up, is…

Loading this new data into GeoServer

  • Open up the OpenGeo Suite dashboard once more at: http://<YourAWSPublicDNS>:8080/dashboard/
  • Click on the GeoServer link, and Login

Loading data into GeoServer is another complicated process, so I won’t go into those details here. The process for importing data into a PostGIS database is well documented on the OpenGeo website. Importing shapefiles is not much different.

Now I have some custom data on my server. I can add styles to it, set up a new map using GeoExplorer, and post it for the world to see.

Here’s a look at a map I put together just for testing purposes:

And the link:
http://107.21.252.45:8080/geoexplorer/viewer#maps/2

I’m pretty happy with the way this turned out. Everything seems to be working OK so far. The new instance is much more stable than my first try. It hasn’t crashed once, even though I felt like I was pushing it to the limit with all the uploading, styling, and layout editing I was doing in GeoExplorer.

Now, if it were only 5 o’clock, I’d be able to celebrate with a beer. What’s that? It’s 4:30?
Close enough! :-)

Link to part 1
Link to part 3