Tag Archives: Ubuntu

Resizing my Ubuntu Server AWS Boot Disk

AKA: Building a Bigger GeoSandbox

(Note: This article has been updated to make it clear that expanded EBS volumes will = additional charges from AWS. Something that is not clearly stated in the AWS documentation.)

If you’ve been reading my last few blog posts, you know I’ve been experimenting with various Ubuntu server configurations using Amazon Web Services (AWS) to serve web-maps and spatial data. As my procedures have evolved, the micro-instances I started working with have outgrown their usefulness. Lately, I’ve been testing GeoWebCache, and seeing how that works with GeoServer and the rest of the OpenGeo Suite. As anyone who’s ever delved into the map-tile world knows, tile caches start gobbling up disk space pretty quick once you start seeding layers at larger scales. I had to figure out a way to expand my storage space if I wanted to really test out GeoWebCache’s capabilities without bringing my server to its knees.

The Ubuntu AMIs I’ve been using all start out with an 8GB EBS volume as the root drive with an additional instance-store volume that can be used for “ephemeral” storage. That “ephemeral” storage means, whatever is in there is lost every time the instance is stopped. Supposedly, a reboot will not clear out the ephemeral storage, but a stop and then start, will. There are procedures you can set up to save whatever is in the ephemeral instance-store volume before you stop it, but I was looking for something a bit easier.

A medium instance AMI includes a 400GB instance-store volume, but it still starts out with the same 8GB root drive that a micro instance has. So, what to do? How do I expand that 8GB disk so I can save more data without losing it every time I stop the system?

A little searching led to a couple of articles that described what I wanted to do. As usual, though, I ran into a couple of glitches. So, for my future reference and in case it might be of some help to others, the following paragraphs describe my procedure.

The two articles this post was compiled/aggregated/paraphrased from are:

The standard “Out of the Box” Ubuntu AMI disk configuration

First, connect to the server using WinSCP, SecPanel, or some other means as described in one of my previous posts. Then open a terminal (or PuTTY) window, and enter:

df -h

You should see something like this:

The first line (/dev/xvda1) is the EBS root disk, and it shows 8.0 GB, with about 3.1 GB being used. The last line (/dev/xvdb) is the instance-store “ephemeral” space that’s wiped clean on every stop.

Note: The Ubuntu AMIs use “xvda1” and “xvdb” as device identifiers for the attached disks and storage space, while the AWS console uses “sda1” and “sdb”. In this case, “xvda1” equals “sda1”. Keep this in mind as you’re navigating back and forth between the two.

Step One: Shut It Down

First, look in the AWS console, and make a note of what availability zone your server is running in. You will need to know this later on. The one I’m working on is in “us-east-1d”. Then, using the AWS console stop the EC2 instance (Do not terminate it, or you will wind up rebuilding your server from scratch). Then move to the “Volumes” window, choose the 8GB volume that’s attached to your server, and under the “More…” drop-down button, choose “Detach Volume”. It will take some time for the detach action to complete.

Step Two: Make A Copy

Next, with the same volume chosen, and using the same “More…” button, create a “Snapshot” of the volume. I recommend you give this (and all your volumes) descriptive names so they’re easier to keep track of.

Step Three: Make It Bigger

Once the snapshot is done processing, it will show up in the “Snapshot” window. Again, giving the snapshot a name tag helps tremendously with organization. Choose this snapshot, and then click on the “Create Volume” button.

In the Create Volume dialog, enter the size you want the new root disk to be. Here, I’ve entered 100 GB, but I could enter anything up to the nearly 400GB of storage space I have left in my Medium Instance. Also in this dialog, choose the availability zone to create the volume in. Remember earlier in this post when I said to note the availability zone your server is running in? This is where that little piece of information comes into play. You MUST use the same availability zone for this new, larger volume as your original server volume used. Click the “Yes, Create” button, and a new larger volume will be placed in your list of volumes.

Step Four: We Can Rebuild It

Next, attach the new larger EBS volume to the original Ubuntu server instance. Go back to the Volume window, choose the newly created larger volume, click the “More…” button, and choose “Attach Volume”.

In this dialog box, make sure the correct instance is showing in the “Instance” drop-down. In the “Device” text box, make sure the device is listed as it is shown above. It should be “/dev/sda1”. Note: This will not be the default when the dialog opens. You must change it!

Clicking on the “Yes, Attach” button will begin the attachment process, which will take a minute or two to complete. Once it’s done, you can spin up the server with the new root drive and test it out.

Step Five: Start It Up Again

Choose the server, and under “Instance Actions”, choose “Start”. Once started, connect to the server using your preferred client. Open a terminal or PuTTY window, and once again enter:

df -h

You should now see something like this:

Notice the differences from the first df command. Now the root disk (/dev/xvda1) will show a size of 99GB, or whatever size you might have made your new volume.

More Room To Play

Now I can adjust my root disk size to suit the task at hand. I can store more spatial data in my GeoServer data directory, and seed my map tiles down to ever larger scales. Knowing how to shuffle and adjust these volumes opens up a slew of other possibilities, too. I can imagine setting up a separate volume just to hold spatial data and/or tiles, and using that to copy or move that data back and forth between test servers.

Be mindful though, this extra space is not free. The larger EBS volume does not replace the space on the ephemeral instance-store volume, it is an addition to it. There will be additional charges to your AWS account for the larger EBS volume based on it’s size. This fact is not made clear in the AWS documentation. So, I recommend you increase the size of the EBS root disk as much as you need, but no more.

Oh the possibilities…

Not Just Another MacBook

An account of my recent adventures in repairing and setting up a tri-boot MacBook

Step one – Obtain the MacBook

Dead Mac

I do not recommend obtaining a MacBook the way I did (or wish it upon anyone), but I do believe in making lemonade from lemons whenever possible. Last week I got a call from one of my daughters explaining that her sister, who attends the same college as her, had spilled “something” on her computer, rendering it inoperable. Luckily my girls attend a college that’s not far from home, so the next day was spent bouncing between the Apple store and Best Buy analyzing our options. I’ll leave out the gory details here, but the end result was: I chipped in the cost of a replacement PC, the daughters split the cost of an upgrade to a new MacBook Pro, and I wound up with the old, wet, borked MacBook to use however I wanted.

Here are the two culprits in the dead-Mac fiasco. Daughter on the right is the former dead-Mac owner. Daughter on the left is the Mac-killer.

The Two Culprits

Step Two – Rebuild the MacBook

The tech at the Apple store said there was a significant amount of “liquid” inside the Mac when he opened it up. The estimate for repairs was $750, with no guarantees that it would work properly in the end. He offered us some info for an off-site computer repair person that might be able to fix it for less. But his recommendation was to sell it for parts. With this in mind, I decided to take a chance, open it up, and see what I could do with it. I had nothing to loose.

Upon removing the base plate, it was obvious that the “liquid” spilled was something a bit stronger than water. My guess was some combination of beer and margarita mix, but daughter 1 insisted it was nothing stronger than wine.  I read in a few places that it’s possible to actually wash the logic board in various ways (using water, alcohol, and a few other concoctions) but that seemed a bit extreme to me. Since the “liquid” involved did not contain a lot of extraneous solids and/or sugary substances, I decided a simple drying out was the best route to take.

Mac disassembled

A quick search led me to the fantastic ifixit site that led me through the entire disassembly process. I found the Mac a lot easier to take apart without ruining anything than I have the few PC laptops I’ve worked on. I disassembled the entire bottom half of the Mac, wiped off the “water” droplets, hit everything with compressed air, heated it up with a hair dryer, and put everything back together again. This whole process took about 5 hours. No one was more surprised than I was when I plugged the thing in and it started right up. That felt good.

Step Three – Improve the MacBook

Both my daughters use Macs, so I’ve had a chance to work on them a few times. I’ve never been a fan of the Mac operating system. There’s something about the user interface, with all those expanding and bouncing icons that just turns me off. That and a few other things that I won’t go into here, leave me preferring Windows over OSx. Still, I saw this as a chance to give the Mac another try. One of the best computing decisions I ever made was to turn my work PC into a Win7/Ubuntu dual boot machine. I’ve tried virtual machines in the past, but I think if you’re serious about trying out and learning a new OS, it deserves to be installed on a high quality computer, and run natively on its own partition. Since I did that, I’ve found I enjoy learning and using Ubuntu much more.

So, this time around, I decided to up the ante, and go for three. The big three: Mac OSx, Windows 7, and Ubuntu 11.04.

image_____image_____image

My process follows very closely the steps outlined in the Ubuntu Community Documentation MacBookTripleBoot page. This page was created using a 2008 MacBook, so I’ve pointed out a few specific steps below where I had problems, or deviated from that process.

Bootcamp

First thing to do is get the Mac Bootcamp drivers on a disc or USB drive. I highly recommend getting the Bootcamp drivers assembled before doing anything else. I did not use Bootcamp to setup the partitions, but the drivers are needed after installing Windows in order to use all of the Mac hardware from within Windows. It’s mentioned in a few forum posts that the bootcamp drivers are available from within the bootcamp.app itself. For example:

Solution: Control-click on the Boot Camp Assistant program, choose Show Package Contents, and then navigate into Contents » Resources. Inside there you’ll find DiskImage.dmg. Just burn that disk image, and you’ll have your Windows drivers CD.

However, I was unsuccessful in finding these on my system. You can also download the drivers by running the bootcamp application. However, you must do this BEFORE partitioning your hard drive, or it won’t work. Something I learned the hard way. A third option I did not get to try, is simply using your original Mac OSx disk. Of course the two culprits in this Mac-fiasco could not find said disks, so I was SOL there. I wound up waiting for daughter 2 to come home with her new MacBook Pro, and I downloaded the bootcamp drivers from that, using the burn directly to disk option.

One last note: When you get to the point in the Bootcamp app where it asks you to partition the hard drive, cancel out. You want to use the Mac disc utility app to do that instead.

rEFit

The next step is to download, install, and run rEFit. This is a pretty straight forward process, but I will point out one thing. I suggest you run the rEFit partitioning tool once before actually partitioning the hard drive. This will sync up your partition tables, and fixed some errors I was getting before I did that.

image

Disk Partitions

Use the Mac OSx disk utility to create partitions for the three operating systems. As outlined in the  MacBookTripleBoot page, I used the command line, and entered:

diskutil resizeVolume disk0s2 100G "JHFS+" 4-Linux 60G "MS-DOS FAT32" 4-Windows 0b

…in order to divide my drive up into a 100GB OSx partition, a 60GB Ubuntu partition, and a 90GB Windows partition. (The 0b for the Windows partition assigns whatever’s left to that partition). Then use rEFit to sync the new partition tables again.

Install the OSs

Install Windows on the last partition, as outlined in the MacBookTripleBoot page. I can confirm that using an XP disk and a Win7 upgrade disc does work just fine. No need for a single full install disk.

Install Ubuntu on the middle (third) partition as outlined in the MacBookTripleBoot page. The installation screens appear to be different in the 11.04 edition of Ubuntu. Just make sure you use the correct partition for the root mount point, and install the grub boot loader to that partition’s boot sector, not MBR.

Clean Up

Once you have Windows installed, use the Bootcamp setup program to install the necessary Windows drivers so you can use all the Mac hardware. I had to do this in order to get my wifi, bluetooth, ethernet, and a few other things working.

Ubuntu was able to recognize most of the Mac hardware, but I had to use an ethernet cable to hook up to my router in order to access the internet. Ubuntu then automatically identified the drivers needed for the Mac’s wireless and graphics adapters.

That’s It

Everything seems to be working smooth as silk. If it weren’t for the white case and MacBook logo staring me in the face, I wouldn’t know I was using a Mac. I made one change to the rEFit configuration file. I went into the refit.conf file and changed the timeout setting to 0 in order to disable automatic booting. This loads up the rEFit boot menu, letting me choose whichever OS I feel like using at that time. The only glitches I’ve noticed is during a restart Ubuntu hangs sometimes, requiring a push on the power button. And Windows seems to want to reboot back into Windows occasionally, instead of to the rEFit boot menu. That’s easily fixed by holding down the Option button during the reboot.

IMG_1159IMG_1160IMG_1161

LOGAS–Tweaked

Over the past week I’ve probably generated enough material for a half-dozen blog posts. However, since I have to get some billable hours in and invoices sent out this week, I’ll just post a short update for now on one aspect of project LOGAS. In my prior post I described the steps I took to get an Ubuntu based Apache/GeoServer up and running. Since then I’ve tweaked the process, and with a lot of help, been able to get everything in working order.

One of the roadblocks I faced was how to get Apache/Tomcat and GeoServer working together. Working individually was easy, together was not. As I progressed with my education in JavaScript, I ran into some cross-domain issues due to the fact my website and GeoServer were hosted on different machines. Even when I put them both on the same machine, the problems persisted due to the fact the webserver  and the the map server use different ports (Apache – 80, and GeoServer – 8080). The solution turned out to be setting up a proxy server. Sounds easy. It was not.

After banging my head on the problem to the point of near concussion, a few online cohorts came to my rescue; in particular, @spara. She was generous enough to write up a script that installs Apache and the OpenGeo Suite, and configures Tomcat and the rest in a way that makes them all work together. She’s posted the code on github for anyone to use. Note: My previous post installed a full LAMP system. This one is limited to just Apache and the OpenGeo Suite, with some tweaks to the Tomcat servlet configuration.

I’ve tested her script on a few clean Ubuntu 10.10 setups, but I haven’t been able to get it to work as a single copy/paste yet. For me, it’s still a multi-step process. But, it does work and it is about as painless as it gets. What works consistently for me, is performing the OpenGeo Suite install first, and then running the Apache install/Tomcat configuration separately.

OpenGeo Suite

So far, the only process that works consistently for me is to first sudo to root, then setup the repository and install the OpenGeo suite. Start by typing the following into a terminal window:


sudo su


… hit enter, and enter your password.

Then copy and paste the following into terminal:


wget -qO- http://apt.opengeo.org/gpg.key | apt-key add -

echo “deb http://apt.opengeo.org/ubuntu lucid main” >> /etc/apt/sources.list

apt-get update

apt-cache search opengeo

apt-get install opengeo-suite


After OpenGeo finishes installing, it will ask for a proxy URL (just leave it blank and hit enter unless you know what that means), a user name, and a password. Set these up as you wish. When all that’s done, move on the the next step.

Apache/Tomcat

Here’s the script for the Apache install/Tomcat setup, just copy and paste this into a terminal window:



sudo apt-get install -y apache2
sudo ln -s /etc/apache2/mods-available/proxy.conf /etc/apache2/mods-enabled/proxy.conf
sudo ln -s /etc/apache2/mods-available/proxy.load /etc/apache2/mods-enabled/proxy.load
sudo ln -s /etc/apache2/mods-available/proxy_http.load /etc/apache2/mods-enabled/proxy_http.load
sudo chmod 666 /etc/apache2/sites-available/default
sudo sed -i '$d'  /etc/apache2/sites-available/default
sudo sh -c "echo ' ' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyRequests Off' >> /etc/apache2/sites-available/default"
sudo sh -c "echo '# Remember to turn the next line off if you are proxying to a NameVirtualHost' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPreserveHost On' >> /etc/apache2/sites-available/default"
sudo sh -c "echo ' ' >> /etc/apache2/sites-available/default"
sudo sh -c "echo '<Proxy *>' >> /etc/apache2/sites-available/default"
sudo sh -c "echo '    Order deny,allow' >> /etc/apache2/sites-available/default"
sudo sh -c "echo '    Allow from all' >> /etc/apache2/sites-available/default"
sudo sh -c "echo '</Proxy>' >> /etc/apache2/sites-available/default"
sudo sh -c "echo ' ' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPass /geoserver http://localhost:8080/geoserver' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPassReverse /geoserver http://localhost:8080/geoserver' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPass /geoexplorer http://localhost:8080/geoexplorer' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPassReverse /geoexplorer http://localhost:8080/geoexplorer' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPass /geoeditor http://localhost:8080/geoeditor' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPassReverse /geoeditor http://localhost:8080/geoeditor' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPass /geowebcache http://localhost:8080/geowebcache' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPassReverse /geowebcache http://localhost:8080/geowebcache' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPass /dashboard http://localhost:8080/dashboard' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPassReverse /dashboard http://localhost:8080/dashboard' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPass /recipes http://localhost:8080/recipes' >> /etc/apache2/sites-available/default"
sudo sh -c "echo 'ProxyPassReverse /recipes http://localhost:8080/recipes' >> /etc/apache2/sites-available/default"
sudo sh -c "echo ' ' >> /etc/apache2/sites-available/default"
sudo sh -c "echo '</VirtualHost>' >> /etc/apache2/sites-available/default"
sudo chmod 644 /etc/apache2/sites-available/default


This installs Apache, and configures it and Tomcat so it appears as though your website and all of the OpenGeo Suite apps are using the same port. The last thing to do is test everything as I outlined in my previous post.

Test the Apache2 default website:

on the host computer – http://localhost – It works!
and via a remote computer (substitute your server’s domain here) – http://24.105.210.45/ – It works!

Test the OpenGeo Suite:

Open the dashboard – http://localhost:8080/dashboard/
Launch GeoExplorer
Save the default map and exit GeoExplorer

Test GeoServer by loading a GeoExplorer map:

Open the default map on a remote computer (again, substitute your server’s domain here) – http://24.105.210.45:8080/geoexplorer/viewer#maps/1

If everything is set up correctly, you should see something like this:

OpenGeoExplorerView

There are probably a few more tweaks that can be made to this process, but I don’t know enough about how Ubuntu works yet to make those changes. I do know that this process works most of the time. A restart between processes helps sometimes, and trying them in a different order occasionally works, too. If anyone can come up with a better, more streamlined approach, please feel free to let me know about it. For now, I’ll just have to live with a little complexity.

And in honor of the day, a little more cow bell, too –

LOGAS–Linux OpenGeo Apache Server

Moving the GeoSandbox to Full OpenSource

I’ve run into a couple of roadblocks recently, regarding my experiments with my GeoSandbox. I want to be able to play with some of of the JavaScript libraries available, so I can continue my education on those fronts. Some cross domain issues arose when I started putting JavaScript in my web pages as my website is on a different server than is my GeoServer. So, that means learning some more about setting up web servers and how proxy servers work. Also, my old Dell 600m is starting to feel the effects of the increased demands placed on it as the GeoSandbox becomes more popular, and more complex. That means an eventual move onto a cloud server. Since cloud servers running an open source OS are much less expensive than those running Windows, I felt the need to begin transferring my entire setup over to open source tools.

For now, I’m just going to do a short outline of what it took to get a basic Ubuntu/Apache/OpenGeo Suite operating on my home server. Once I get a better handle on how Ubuntu and Apache work I’ll add some posts about that.

Starting with a clean install of Ubuntu 10.10

Download and install Ubuntu 10.10 Desktop

I downloaded Ubuntuu 10.10 Desktop Edition from their website:

http://www.ubuntu.com/desktop/get-ubuntu/download

Yes, I could have used the Server edition, but Coming from a Windows world, It’s easier for me to work with some kind of GUI than to go 100% command line. After downloading the .iso, I burned it to a CD, installed it on a clean hard drive, and then installed all updates.

Turn Ubuntu Desktop into an Apache Server

Install LAMP with a single command

In case you didn’t know, LAMP stands for “Linux Apache MySQL PHP”. This may or may not be more than I need for my purposes, but a one-line install looked like the easiest way to go, so:

http://nick.onetwenty.org/index.php/2010/12/02/installing-lamp-server-on-ubuntu-10-10-desktop/

Then I opened a terminal window and entered:

sudo apt-get install lamp-server^

Add in the ability to serve maps

Install OpenGeo Suite

This was the most difficult part for me to figure out. Instructions can be found here:

http://projects.opengeo.org/suite/wiki/LinuxPackages

Being somewhat new to Ubuntu, I missed what the first step under “Repository Setup” meant (sudo to root). Once I figured that out, things went smoothly. The steps are:

In terminal, sudo to root:

sudo su

Import the OpenGeo gpg key:

wget -qO- http://apt.opengeo.org/gpg.key | apt-key add -

Add the OpenGeo repository:

echo "deb http://apt.opengeo.org/ubuntu lucid main" >> /etc/apt/sources.list

Update the package list:

apt-get update

Search for OpenGeo packages:

apt-cache search opengeo

Install the OpenGeo Suite package:

apt-get install opengeo-suite

Restart Ubuntu

See if everything works

The last step was to see if everything was working properly, which it was.

Test the Apache2 default website:

on the host computer – http://localhost – It works!
and via a remote computer (substitute your server’s domain here) – http://24.105.210.45/ – It works!

Test the OpenGeo Suite:

Open the dashboard – http://localhost:8080/dashboard/
Launch GeoExplorer
Save the default map and exit Geoexplorer

Test GeoServer by loading a GeoExplorer map:

Open the default map on a remote computer (again, substitute your server’s domain here) – http://24.105.210.45:8080/geoexplorer/viewer#maps/1

In my case, everything worked as expected. After this, I continued trying to set up a custom website to serve my GeoServer maps, but ran into a few problems. I’ll be switching the GeoSandbox back and forth between the Windows and Apache server as I continue my climb up the learning curve, so don’t be surprised if some of the links appear to be broken on occasion.