nTh among all

I’m moving a VM from my own server to EC2. I’ve got $50 worth of credit from a Redhat event, so I would like to make use of that. (That, and my AWS account existed before the free tier was introduced, so I’m not eligible for that.) That said, I want to make that credit last as long as possible. There were two parts to this – resizing the EBS volumes so I don’t get dinged for more than I need.

Amazon says $0.10 per GB-month of provisioned storage – so I’d be paying ~80 cents/month for storage that I’m not using. Admittedly, it isn’t that much, but that’s where I looked first.

I found an official Fedora 17 image, and had started off with that. It comes with 10GB of EBS storage, and I’m not using that much – df -h says 1.1GB used, so I could arguably drop the 10GB EBS volume down to 2GB. It’s just a matter of cloning the filesystem over.

The basic idea behind cloning filesystems doesn’t change in EC2-land. Turn off the instance, connect old drive and new drive to a new instance, copy old to new, shutdown new instance, and spin up old instance, but with the new drive. So I created a new 2GB volume, and turned off the existing micro instance, and spun up a new instance.

The first mistake

At this point, I will point out one thing: I thought the OS didn’t matter, so I just used the default Amazon image. Which is a bad idea – it’s CentOS based, and uses Amazon’s systems for yum. So when I did a yum install parted, I got hit for EC2 regional data transfer. For a temporary instance, I’ll be using an official Fedora AMI by default from now on – Data in is free, and the first GB of data out is free too.

Anyway, I attached the existing 10GB volume as /dev/sdf, and the new 2GB volume as /dev/sdg to the instance using the AWS Volume panel. Then I connected to the EC2 instance, and formatted the 2GB volume (mkfs.ext4 /dev/xvdg). I didn’t want to mess with resizing partitions, filesystems and dd, so I copied over the contents of the 10GB volume to the 2GB volume (cp -a /mnt/* /mnt2).

Now, if you’re following me, don’t do this blindly like me. dracut (Fedora’s initramfs maker) seems to use disk labels. I did not set the label for the new volume. So when I connected the 2GB volume to the original instance, it failed to boot with a dracut error about being unable to find /dev/disk/by-label/\_x2f.

So I shutdown the new instance and reattached the 2GB EBS volume to the temporary AMI. This time, I resized the filesystem on the original 10GB volume to 2GB after running fsck on it – fsck /dev/xvdf && resize2fs /dev/xvdf 2G.

Next I used dd to clone the filesystem – dd if=/dev/xvdf of=/dev/xvdg bs=1M. Oddly enough, dd complained about running out of space on the device. I’m not sure why, but it looked like resizing the fs wasn’t exact – dd said 2049 records in, 2048 records out. (Now, thinking about it, it makes sense because I used dd to clone the drive, not the filesystem. So it’d try to copy the entire 10G block device. So the error makes sense.)

The second mistake

However, one important thing to note is that EC2 also charges on I/O requests. I didn’t think of this when I set the block size for dd to use to 1M. I don’t think it’s a coincidence that the I/O request count ballooned from ~20k to ~180k after cloning the drives twice. So what I should have done was to use a larger block size – maybe bs=128M or 256M. I’m guessing that a larger block size would keep the number of I/O requests down, but, to be honest, it was probably massively blown up by the cp -a, not the dd.

Success

But other than those two mistakes, the resize went fine. I detached the 2GB volume from the temporary instance, and attached it to the original instance. I brought the instance back up, and everything worked as I expected to it. Only strange thing is that the hostname is now “localhost”, even though /etc/hostname exists.

One point to note is that though the volume panel implies that you can only attach volumes as /dev/sdf-sdj, I was able to attach the EBS volume as sda just fine. However, booting from /dev/sda doesn’t appear to work – and I suspect it’s something on Amazon’s side, because if I use /dev/sda for the EBS volume, there’s no corresponding root device appearing in the instance details. However, using /dev/sda1 instead (note the 1!) works fine though.