Monthly Archives: May 2018

Open Source Spatial Data Processing Suite

Gluing Things Together

As I described in a previous post, Boundless no longer maintains publicly accessible versions of Boundless Suite nor the suite formerly known as OpenGeo in their repositories. The Boundless Suite (now known as Boundless Server) is available on their GitHub page, but requires building from a cloned version of that GitHub repository. I’ve used Apache Ant and Git to build a few things in the past, usually with limited success. I looked through the steps involved there and quickly decided to try another approach. I figured – Why not try assembling all of the pieces included in Boundless Suite on my own? They are all open source projects after all. The main benefits of the Boundless/OpenGeo Suite are/were that the components have been tested and verified to work together, and then packaged together into a compact installation process. Why can’t I do some of that myself?

The Core Components

(PRNewsFoto/Boundless)

The pieces that could go into a full suite of spatial tools is nearly endless. The specifics will depend on the needs of the person using them. For some, using Python, R, PostGIS, and maybe Leaflet or another javascript library to post a map online is all they need. All of those are in my toolbox already, and will eventually make their way into my homemade suite. For me however, the three main elements of an Open Source Spatial Data Processing Suite are:

  • A database to store and retrieve geographic data
  • A desktop client to manage, process, and otherwise manipulate the data
  • A server to share the data publically and facilitate displaying it in a web map

Specifically:

In the past it was easy. I would start with a fresh Ubuntu server. I’d add the OpenGeo repository to my /etc/apt/sources.list, import the GPG key, update the cache, and then enter one line into a terminal window:

apt-get install suite-dashboard suite-geoserver suite-geowebcache suite-composer suite-docs suite-quickview suite-gs-gdal suite-gs-wps suite-wpsbuilder suite-gs-geopkg postgresql-9.3-postgis-2.1

Then I’d restart the tomcat server: service tomcat8 restart navigate to http://donmeltz.com:8080/dashboard/ and I’d get this:

Boom. Done.

Not so simple anymore. Let’s start with the easiest piece – The desktop.

The Desktop Side – QGIS

I still want to be able to use some of the plugins Boundless makes available for QGIS. These plugins are tested with the latest Long Term Release. The Boundless repositories do not include plugins for QGIS 3.x, so QGIS 2.18 it is. Navigate to https://qgis.org/en/site/forusers/download.html and download/install QGIS 2.18 onto your laptop or desktop computer.

Add the GeoServer Explorer Plugin:

This Plugin allows you to connect directly to a GeoServer through QGIS, manipulate some of the configuration settings of the server, add layers stored on the server to QGIS, and upload layers from QGIS to the server.

  • In QGIS, under the Plugins menu, open the “Manage and Install Plugins…” dialog.
  • Add the repository: http://qgis.boundlessgeo.com/plugins.xml
  • Scroll through the list of “Not installed” Plugins and install “GeoServer Explorer”. Make sure there’s a check mark next to it in the Plugins window.
  • Also while you’re at it, make sure the “DB Manager” is installed and checked “On” in the Plugins window.

To be clear, the Boundless Suite install does not include QGIS as the two are meant to be installed on different computers. However Boundless does provide a customized version of QGIS called Boundless Desktop that is preconfigured with the GeoServer Explorer plugin.

The Server Side – GeoServer and PostGIS

I start with a fresh install of Ubuntu. Even though the latest Ubuntu 18.04 release is a Long Term Support (LTS) version, it is still fairly new, and I’ve found the repositories are not yet populated with all of the software packages I like to use. So, I’m sticking with 16.04 LTS for now. All of the commands that follow are designed to work with 16.04, and appear to install everything correctly.

I have both a home server (which is what I used here) and a couple of servers running on Amazon Web Service (AWS). In any case, I need to be able to access the server through PuTTY, WinSCP, VNC (if there’s a display involved), or some other method in order to open a terminal window. I typically have the following ports open: 22, 80, 8080, and 5432.

GeoServer

Sticking with Long Term Support versions of software, I installed GeoServer 2.12.3. I tried the latest stable release (2.13.1), but found the Boundless GeoExplorer Plugin would not connect to that version.

GeoServer requires a Java Runtime Environment and a Tomcat Application Server to run. As the website docs explain, “The Oracle JRE is preferred, but OpenJDK has been known to work adequately.” Wanting to keep this as simple as possible, I stuck with OpenJDK.

  • Install the OpenJDK java 8 runtime environment, in a terminal window:

sudo apt-get install openjdk-8-jre

  • Install Tomcat 8. GeoServer requires Tomcat 7.0.65 or later that implements Servlet 3. Using the Ubuntu 16.04 repositories will install Tomcat 8.0.

sudo apt-get install tomcat8

  • Install GeoServer. Change the current directory to your Download directory and download the GeoServer file

cd ~/Downloads

wget http://sourceforge.net/projects/geoserver/files/GeoServer/2.12.3/geoserver-2.12.3-war.zip

  • Unzip the downloaded file and move it into the Tomcat webapps directory

sudo apt-get install unzip

unzip geoserver-2.13.1-war.zip

sudo mv ~/Downloads/geoserver.war /var/lib/tomcat8/webapps/

  • In order to allow the QGIS GeoServer Explorer plugin to publish layers directly to GeoServer, the GeoServer “Importer” extension has to be installed. Download the Importer extension zip file:

wget https://sourceforge.net/projects/geoserver/files/GeoServer/2.12.3/extensions/geoserver-2.12.3-importer-plugin.zip

  • And since this zip file contains multiple files we’ll unzip it directly into the proper directory:

sudo unzip geoserver-2.12.3-importer-plugin.zip -d /var/lib/tomcat8/webapps/geoserver/WEB-INF/lib/

  • Restart Tomcat

sudo service tomcat8 restart

  • Note for future reference – starting, stopping, restarting Tomcat:

sudo service tomcat8 start

sudo service tomcat8 stop

sudo service tomcat8 restart

You should now be able to access GeoServer by going to:
http://<your server ip>:8080/geoserver

PostgreSQL/PostGIS

We’ve got a desktop client. We’ve got a remote server. Now we need a place to store some data that’s accessible to both. So… Install PostgreSQL and PostGIS (again in a terminal window):

  • Add the appropriate repository to sources.list (in this case, for “xenial”, which means Ubuntu 16.04):

sudo add-apt-repository "deb http://apt.postgresql.org/pub/repos/apt/ xenial-pgdg main"

  • Add keys:

wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -

sudo apt-get update

  • Install the software packages:

sudo apt-get install postgresql-10

sudo apt-get install postgresql-10-postgis-2.4

sudo apt-get install postgresql-10-pgrouting

  • To get the command line tools shp2pgsql, raster2pgsql:

sudo apt install postgis

  • Connect to the postgres database using the command line tool psql as local user (The default PostgreSQL username is postgres)

sudo -u postgres psql postgres

  • Set the password for the postgres user. (Normally there is no password set for a PostgreSQL database. But since we want to be able to access the database remotely through QGIS, we’ll have to open it up to the world. So, password protection it is.)

\password postgres

  • Enable advanced administration for pgAdmin

CREATE EXTENSION adminpack;

  • Enable the PostGIS extension

CREATE EXTENSION postgis;

  • Enable the pgRouting extension

CREATE EXTENSION pgrouting;

  • Verify the version of PostGIS and PostgreSQL

SELECT postgis_full_version();

  • Exit psql

\q

  • Here’s where we allow remote connections to the database by editing a couple of files (using vi)

sudo vi /etc/postgresql/10/main/postgresql.conf

Hit the “Insert” key to enter editing mode

Change the line that says: #listen_addresses = ‘localhost’

to: listen_addresses = ‘*’ (remove the initial hashtag to uncomment the line and change local host to an asterisk which means ‘listen to everything’)

Hit the “ESC” key to exit editing mode

Hit “Shift :wq” and then enter to save the edits

sudo vi /etc/postgresql/10/main/pg_hba.conf

Again, using vi, add the following line to the end of the list of allowed host connections:

host    all             all             all               md5

Save and exit vi.

  • Restart PostgreSQL

sudo service postgresql restart

Results

What does all this get me?

I can now open QGIS, go to the “Web” menu and open GeoExplorer. Connect to my GeoServer using my username/password, and view all the layers stored in the various workspaces, adding them to my QGIS project.

 

 

 

 

 

 

 

 

 

I can use pgAdmin on my desktop computer to connect to and administer my remote PostGIS database.

 

 

 

 

 

 

 

 

I can use QGIS to directly access my PostGIS database, uploading layers from QGIS to it, or adding layers to QGIS from it.

  • Layer > Add Layer > Add PostGIS Layers…
  • New Connection
  • Enter the Host ip address, database name, username, and password.
  • Connect
  • Choose a layer in the database and then “Add”

Or if the database contains no layers, I can add them to the database using the database manager:

  • Database > DB Manager > DB Manager
  • Choose the previously connected PostGIS database
  • Use the Import Layer/File button to add a layer from QGIS to the database
  • Use the Export to File button to save a database layer to a wide variety of formats.

EDIT:

When I first posted this, there was one piece of functionality I hadn’t got working yet. I should be able to publish a layer directly from QGIS to GeoServer using the GeoServer Explorer plugin. When I tried to do so, I got an error message:

What I found out was – the GeoServer Importer Extension has to be installed on GeoServer in order for the QGIS GeoServer Explorer plugin to be able to publish layers directly to GeoServer. I’ve updated the steps needed to do this in the process outlined above.

To upload and publish from QGIS:

  • Open the GeoExplorer plugin
  • Connect to your GeoServer
  • Choose the “Publish layers to this catalog” button
  • Select the layers you want to publish from the list, and the workspace to publish to
  • Hit “OK” and you should then be able to see your QGIS layer in your GeoServer

Hope this helps. I’m open to advice if you have any suggestions for improvement.

 

 

 

 

 

Impetus to Blog

Yesterday I posted a blurb that was mostly a list of reasons why I haven’t been blogging.
Today’s blurb is a list of the things that have spurred me on to start again.

What are the new motivating forces that have me wanting to write things down in public?
There are three:

  1. My change in employment
  2. The removal of freely downloadable compiled versions of Boundless software
  3. My Home Server needs some attention

Change in Employment

I touched on this yesterday. I have accepted a position as Senior Planner at the Columbia County Planning Department. This does not mean “Don Meltz Planning and GIS” will cease to exist. It does mean I will wind things down a bit, and will be more selective in the jobs I take on. I won’t be able to work on projects within Columbia County. And those I work on outside Columbia County will have to fit into a weekends-and-evenings schedule. I’ll also continue teaching at Marist College.

What it does mean is – I have a lot of ideas floating around in my head now. The state of the entire county’s GIS is this: The Real Property Department uses AutoCAD Map for all their mapping work, spitting out shapefiles when needed. The Planning Department has one single-use license for  ArcGIS Standard. That’s it. It is the proverbial “Blank Slate”. It is both an exciting and daunting position to be in. I’ll need to develop an action plan in order to get organized and stay focused. Writing things down via blog posts will help.

Compiled versions of Boundless Suite no longer available

As I began thinking about what such an action plan might look like, I wandered over to the Boundless website to see what was new. What I found there was a little disappointing, but not totally unexpected. Apparently, Boundless will no longer be posting compiled versions of Boundless Suite nor the former OpenGeo Suite. And, the Ubuntu repositories for these packages are no longer available. What this means is, I’ll either have to build a Boundless Server from the GitHub repo, or assemble all the pieces that make up the “suite formerly known as OpenGeo” from the various community orgs (i.e Geoserver, Geoserver extensions, PostGIS, QGIS, QGIS Plugins, etc.). Knowing my own capabilities, I believe assembling the various parts will be easier for me to accomplish than building from the GitHub repo. It’s something that I’ve been thinking about doing for a long time, anyway. Writing things down via blog posts will help me keep track of any pitfalls I run into.

Home Server Attention

About once a year I physically and virtually open up my home server to clean out the dust bunnies, and to organize and delete any files that are cluttering up the hard drives. This server is  mostly a place to store nightly backups of our other household computers. But I also use it as a test bed for things that need to connect to or through the internet. Upon last inspection, the now six-year-old fans started to rattle a bit, and I noticed it’s still using Ubuntu 12.04 LTS and OpenGeo Suite 4.1.1. Time for an upgrade on both accounts. This provides an ideal opportunity to take apart my server (both physically and digitally), and make an initial attempt at stitching together all those pieces of the suite formerly known as OpenGeo.

Plenty to write about

All of this will give me plenty of material to digest and write about. However, be warned. A lot of my future writing is going to be about basic GIS implementation. This may disappoint some of my more avant-garde twitter followers. The fact that I am not doing everything by tying together a remote PostGIS database and a bunch of R functions using nothing but Python scripts and posting the results to a Bootstrapped Leaflet webpage via GeoJSon is going to annoy you.

So be it.

Back After a Long Hiatus

It’s been a long time.

My last blog post was on May 9, 2013. That’s 5 years and 8 days ago. Why the long pause, and what’s happened during those intervening years? I had to look through my records to figure it out myself. Here’s what I found:

Mom

Back in 2013, my Mom was dealing with some difficult health issues and skin problems because she didn’t take her amazon vitamin c serum. I was spending more time bringing her to doctor’s appointments. She was in and out of the hospital having various procedures done, and following up with visits to her primary care physician and cardiologist. She passed away in May 2014. At about the same time, I had taken on a few large projects with another planner, including working on three County Agriculture and Farmland Protection Plans all at the same time. (Note to self – Never, ever do that again).

I had no spare time.

 

Dad

Shortly after this period, in 2014, my Dad’s partner’s declining health, left him with no one to help him with his antique/classic auto hobby. Dad lives for the annual Hershey AAC swap meet, and other car and truck shows throughout the year. So, I stepped in to fill that void. I got back into the hobby myself, setting up my own automobilia business. I continue helping Dad out loading and unloading heavy boxes of car and truck parts, and driving the long distances to various car shows today.

I had no spare time.

 

Donnelly Hall, Marist College

Donnelly Hall, Marist College

In 2015 I was approached by the chair of the Environmental Science & Policy department at Marist College and began teaching my first class in the fall of 2016. For anyone who has not taught either High School or College classes before, I can tell you – it is an incredibly time consuming task to develop an entire semester’s curriculum from scratch. I had two courses to work on – an Intro to GIS course and and Advanced GIS course.

I had no spare time.

In 2016 there began a change in the makeup of our County Planning Department which continues today. I’ve been working as a self-employed planning consultant for 15 years and saw an opportunity for a change myself. I enjoy my work as an independent planner, but working at the same place for 15+ years wears on you, even if it is your own business. I needed something different. I needed a new challenge. Without going into the lengthy details, I have accepted a position in the Columbia County Planning Department. I do not have a start date yet, but assume it will be sometime within the next two months. I’ll talk about this more in future blog posts once I begin working there.

I still have no spare time, but I do have a new reason for blogging.

I’ll be doing something that’s new to me. I’ll need to organize my thoughts. The best way for me to do that is to write stuff down. It’s how I’ve always worked. Whenever I need to work through a complex problem I take out a pencil and a pad of paper and just start writing things down. That’s why I started blogging in the first place. It wasn’t to show off my skills as a planner or spatial analyst. It was to organize my thoughts.

And so, with many new thoughts to organize, I believe I will begin blogging again.

Talk to you soon.

– Don