Archive for category Linux

Terraform import with AWS profiles other than default

I’ll come back and clean this up, but for now:

Undocumented: It will use the default AWS profile – it will pull in your shared credientials, and use the default values if specified.

As per code, use AWS_PROFILE=<name> terraform import aws_db_instance.default <id> to import using a AWS profile that isn’t default.

ELB holds onto subnets that are to be destroyed. Combined with new subnet having the same CIDR, can’t create new subnet because CIDR is in use, can’t update ELB because new subnet isn’t created yet.

No Comments

Using Amazon S3 + CloudFront + Certificate Manager to get seamless static HTTPS support

TL;DR: This post documents the process I took to get S3 to return redirect requests over HTTP + HTTPS to a given domain.

I’m trying to trim down the number of domains and subdomains that I host on my server, since I’m trying a new policy of moving servers every few months in an attempt to make sure I automate everything I can.

One of the things that I’ve done was to start consolidating static files under a single subdomain, and use 301 redirects in nginx to point to the new location. Thanks to Ansible, rolling out the redirect config is a matter of adding a new domain + target pair to a .yml file and running it against a server.

But it’d still be nice if I didn’t have as many moving parts – which meant that I looked at ways to get an external provider to host this for me. I decided to try getting S3’s static website hosting a try to see if it supported everything I wanted it to do. In this case, I want to return either a redirect to a fixed URL, or a redirect to a different domain, but same filename. Essentially, my nginx redirect configs are either return 301 https://kyle.io/$request_uri or return 301 https://kyle.io/fixed-location.

For the purposes of this post, let’s assume I have the domain tw.kyle.io that redirects to my Twitter profile.

Creating the S3 Bucket

I knew that S3 could do static site hosting – but the docs seem to indicate that while redirecting to a different domain is possible, it will still use the same path to the file. So this would work for the different domain, same file name case, but not the fixed URL case.

I found my solution in an example of redirecting on an HTTP 404 error – but it could be adjusted to redirect all the time by removing the Condition elements.

So let’s create the bucket with the AWS CLI: aws s3api create-bucket --bucket tw-kyle-io --create-bucket-configuration LocationConstraint=us-west-2

Then I had to apply the redirection rules. The CLI uses JSON format to specify the redirect rules, so to make things simple, I dumped the config into a file:

{
  "IndexDocument": {
    "Suffix": "redirect.html"
  },
  "RoutingRules": [
    {
      "Redirect": {
        "HostName": "twitter.com",
        "Protocol": "https",
        "ReplaceKeyWith": "lightweavr"
      }
    }
  ]
}

and then called it through the CLI: aws s3api put-bucket-website --bucket tw-kyle-io --website-configuration file://website.json. Note that the use of IndexDocument is misleading: I never actually created a file in S3, but the S3 API requires that something be specified for that key.

People who have created a static website in S3 might be saying that I used a bad bucket name, because now I can’t use CNAMEs with the bucket. Well, I have the useful experience of writing this after discovering that HTTPS requests to the bucket don’t work because S3 doesn’t support HTTPS to the website endpoint. CloudFront doesn’t have restrictions on bucket naming, especially where we’re going to use the website endpoint so we can use redirections. (Hat tip to a StackOverflow question and another one as well.)

Amazon Certificate Manager

So I ended up using CloudFront. But first, I had to create a certificate with ACM. Because I’m creating a subdomain cert, I had to use the CLI – the console only allows you to create a *.domain.com or domain.com cert.

aws acm request-certificate --domain-name tw.kyle.io --domain-validation-options DomainName=tw.kyle.io,ValidationDomain=kyle.io

I had to approve the certificate creation, which required waiting for the email. If you try to create you get an error about the cert not existing: A client error (InvalidViewerCertificate) occurred when calling the CreateDistribution operation: The specified SSL certificate doesn't exist, isn't valid, or doesn't include a valid certificate chain.

CloudFront Magic

Then, it was time to configure the CloudFront distribution. Which isn’t trivial, there’s a bunch of options, and it wouldn’t surprise me if I screwed something up. To get the options below, I created a distribution through the AWS console, then compared that against a generated template (aws cloudfront create-distribution --generate-cli-skeleton).

I put the following in a file named cf.json

{
    "DistributionConfig": {
        "CallerReference": "tw-kyle-io-20160402",
        "Aliases": {
            "Quantity": 1,
            "Items": [
                "tw.kyle.io"
            ]
        },
        "DefaultRootObject": "",
        "Origins": {
            "Quantity": 1,
            "Items": [
                {
                    "DomainName": "tw-kyle-io.s3-website-us-west-2.amazonaws.com",
                    "Id": "tw.kyle.io-redirect",
                    "CustomOriginConfig": {
                        "HTTPPort": 80,
                        "HTTPSPort": 443,
                        "OriginProtocolPolicy": "http-only",
                        "OriginSslProtocols": {
                            "Quantity": 1,
                            "Items": [
                                "TLSv1.2"
                            ]
                        }
                    }
                }
            ]
        },
        "DefaultCacheBehavior": {
            "TargetOriginId": "tw.kyle.io-redirect",
            "ForwardedValues": {
                "QueryString": false,
                "Cookies": {
                    "Forward": "none"
                },
                "Headers": {
                    "Quantity": 0
                }
            },
            "TrustedSigners": {
                "Enabled": false,
                "Quantity": 0
            },
            "ViewerProtocolPolicy": "allow-all",
            "MinTTL": 86400,
            "AllowedMethods": {
                "Quantity": 2,
                "Items": [
                  "HEAD",
                  "GET"
                ],
                "CachedMethods": {
                    "Quantity": 2,
                    "Items": [
                      "HEAD",
                      "GET"
                    ]
                }
            },
            "SmoothStreaming": false,
            "DefaultTTL": 86400,
            "MaxTTL": 31536000,
            "Compress": true
        },
        "Comment": "Redirect for tw.kyle.io",
        "Logging": {
            "Enabled": false,
            "IncludeCookies": false,
            "Bucket": "",
            "Prefix": ""
        },
        "PriceClass": "PriceClass_100",
        "Enabled": true,
        "ViewerCertificate": {
            "ACMCertificateArn": "arn:aws:acm:us-east-1:111122223333:certificate/3f1f4661-f01b-4eef-ae0d-123412341234",
            "SSLSupportMethod": "sni-only",
            "MinimumProtocolVersion": "TLSv1",
            "Certificate": "arn:aws:acm:us-east-1:111122223333:certificate/3f1f4661-f01b-4eef-ae0d-123412341234",
            "CertificateSource": "acm"
        }
    }
}

and then ran the command aws cloudfront create-distribution --cli-input-json file://cf.json.

Testing it

The first thing to test was simply curlling the newly created distribution – after waiting ages for it to actually be created. (I assume CloudFront is just continually working through a queue of distribution changes to each of the 40+ POPs, so it’s actually quite awesome.)

[[email protected] ~]$ curl -v d28h66yisj403x.cloudfront.net
* Rebuilt URL to: d28h66yisj403x.cloudfront.net/
>clipped<
< HTTP/1.1 301 Moved Permanently
< Location: https://twitter.com/lightweavr
< Server: AmazonS3
< X-Cache: Miss from cloudfront
* Connection #0 to host d28h66yisj403x.cloudfront.net left intact
[[email protected] ~]$ curl -v https://d28h66yisj403x.cloudfront.net
* Rebuilt URL to: https://d28h66yisj403x.cloudfront.net/
>clipped<
< HTTP/1.1 301 Moved Permanently
< Location: https://twitter.com/lightweavr
< Server: AmazonS3
< Age: 11
< X-Cache: Hit from cloudfront

Would you look at that, both the HTTP & HTTPS connections worked!

So I changed the CNAME in CloudFlare to point to the new distribution, and tried again, this time with tw.kyle.io.

[[email protected] ~]$ curl -v https://tw.kyle.io
* Rebuilt URL to: https://tw.kyle.io/
>clipped<
< HTTP/1.1 301 Moved Permanently
< Location: https://twitter.com/lightweavr
< Age: 577
< X-Cache: Hit from cloudfront
< Via: 1.1 f360bbb3d1999b5324e1d7ae31da1d7e.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: kl8eZMDzH2BB7T3owENtjFkS2xtfcwqoOsZ4-SNxLY8LMdupbXrp9Q==
< X-Content-Type-Options: nosniff
< Server: cloudflare-nginx
< CF-RAY: 28d9413b3943302a-YYZ

Because I use CloudFlare’s caching layer, parts of the response are overwritten by CloudFlare (notably the Server section, among others). But we can still see that the request ultimately hit the CloudFront frontend. The request was still redirected, so everything looks good.

In Closing

The main thing I didn’t like about this setup? The obtuse documentation. I had far better grasp of everything working through the console, then applying that to the CLI. But there’s still tiny things.

  1. I hit the ancient ‘Conflicting Conditional Operation’ issue about S3 buckets being quick to be deleted from the console, but not actually deleted in the backend. So when I mistakenly thought that --region us-west-2 would be enough to get the S3 bucket to be created in us-west-2, it took ~1 hour to become useable again.

  2. Passing --region us-west-2 to aws s3api create-bucket will create a S3 bucket in the default region, us-east-1. You have to specifically pass --create-bucket-configuration LocationConstraint=us-west-2 to get it created in a specific region.

  3. There’s aws s3 and aws s3api. Why aren’t the s3api operations merged into s3? No other service has an api suffix.

  4. Having to trial and error to find out what parameters are required and what aren’t for different operations. The CloudFront create-distribution command was the worst offender of the commands I ran, just with the sheer number of parameters. I’m hoping the documentation improves before it comes out of beta.

  5. Weird UI bugs/features. Main one I noticed was the ACM certificate selector not being selectable unless a cert exists… and the refresh button is included in that. So the very first time I created a CloudFront distribution, I needed to refresh the page to be able to select the cert, losing my settings.

Now for the important question: Now that I’ve set it up, will I keep on using it? The simple answer is that I’m not sure. It’s nice that it’s offloaded to another provider that keeps it going without my intervention, and that after setting it up it won’t change. But at the same time, it’s another monthly expense. I’ve already put money into a server, and the extra load of a redirection config is negligible thanks to Ansible. There is some overhead with creating & managing Let’s Encrypt certs, particularly with the rate-limiting, and these redirects are entire subdomains, but I just set up an Ansible Let’s Encrypt playbook to run weekly, and the certs will be kept up to date.

It’s going to come down to how much I get billed for this.

No Comments

Upgrading to Fedora 23 on OpenVZ

TL;DR: Run dnf --releasever 23 distro-sync instead of dnf system-upgrade on OpenVZ systems

I run Fedora on my servers almost exclusively. This means I usually fall behind in upgrading to the latest release, leading me to wonder why I don’t just go with the latest version of CentOS.

Then I have lovely cases where CentOS gets horribly outdated, and I remember why I like Fedora with its latest and greatest. (Yes I do like shiny things, thank you very much)

My servers are mostly OpenVZ based, for the simple fact that OpenVZ powered VPSes are rather cheap for what you get, especially where I don’t need high performance. I have just one bad thing about being OpenVZ based: I have no control over the kernel/boot sequence. The vast majority of the time, this isn’t an issue. Sadly, using dnf system-upgrade is one of the times when it is an issue.

Fedora 22 brought in a new way to upgrade your system – dnf system-upgrade. I’ve used it on my laptop, it’s pretty good compared to fedup and past solutions. However, the one thing that rarely failed me in the past was using the yum distro-sync functionality. (The only time I’ve had an issue with it was when the upgrade was stopped midway, but that’s another story.)

Read the rest of this entry »

, ,

No Comments

Let’s not Encrypt on CentOS5

TL;DR – Let’s Encrypt requires a newer version of OpenSSL than CentOS 5 has installed. Unless you want to pass around with compiling OpenSSL yourself, don’t try it.

When your friend will upgrade his CentOS 5 system "someday"

When your friend will upgrade his CentOS 5 system “someday”

Read the rest of this entry »

,

8 Comments

Let’s Encrypt ALL THE THINGS

Got my first domain using a cert from Let’s Encrypt in under ~10 minutes, including setting up Let’s Encrypt itself. Yes, this is rather game changing.

Now to write ansible playbooks around it, and figure out how to get it working for proxied domains automatically.

Read the rest of this entry »

,

No Comments

Building Nginx SRPMS

Companion to my earlier post, this actually has commands

Read the rest of this entry »

,

No Comments

Path to building Nginx Mainline RPMs for Fedora & CentOS

Or: How I spent an afternoon doing a deep dive into the RPM spec and solving a problem for myself

tl;dr – Nginx Mainline packages are being built for Fedora & CentOS at copr.fedoraproject.org/coprs/kyl191/nginx-mainline/


My webserver’s running nginx 1.4.7, a version that hasn’t gotten non-bugfix attention since March 2013, according to the changelog. Oddly enough for a Fedora package, the version in koji is the stable branch – something that makes sense for CentOS/EPEL since that’s a long term support release, but not for Fedora. Doubly annoying was the fact that the packages for Fedora 20 (because Fedora 21 isn’t supported on OpenVZ yet) was one step behind the official stable – 1.4 instead of 1.6.
If I wanted mainline on Fedora, I was going to have to build it from source.
Read the rest of this entry »

, ,

1 Comment

Ansible gotchas

  • Tasks do not like having the remote_user changed mid-playbook if you specify a SSH password
    • Specifically, having an ‘ansible’ user created as the first task, then using that for everything in the rest of the playbook doesn’t work because ansible will always attempt to use the declared password for the newly created user, which promptly fails
    • Solution: Separate runbooks!
    • People debate the usefulness of a separate config account, since it’s effectively root + key-based login. They have a point, since I can get the same security with disabling password-based auth.
  • If your inventory file is +x, ansible will attempt to execute it, even if it is the standard inventory list (.INI-format plain text)
  • I’m liking the look of the Ansible for DevOps book
  • Ansible will automatically import GPG keys during a yum install if the matching GPG key hasn’t been imported yet (Seen after installing EPEL, then installing a package from there)

No Comments

Booting from SD Card on a X230

The SD Card slot is unfortunately on the PCI bus, so it doesn’t show up as a bootable device.

Solution:

Have a /boot partition on an internal drive, point that at the SD card. Reclaimed ~900MB from Lenovo’s system restore partition to make a /boot partition. GRUB was added to the internal drive.

As suggested in ubuntuforums.org/showthread.php?t=986126&page=3&p=11915401#post11915401, edit the initram image to get SD card support added. In Fedora, this was done using dracut’s kernel modules option in /etc/dracut.conf, so any kernel updates would automatically get the appropriate modules added.

Sticking point – Fedora 17 seemed to have the drivers installed natively. Fedora 20 didn’t, systemd would die.

Main issue was naively assuming that grub was using the correct initramfs that was built. I installed Fedora, then did a yum update kernel to generate the initramfs image. But that didn’t work, so I tried various other things, not realising that the bugged initramfs image was still there, and because it was newer, GRUB was automatically using that instead of the initramfs that I was testing.

Other issue was selinux – the context of files no longer matched (oddly enough). Not sure how/why this happened, but it ended up that I couldn’t log in – GDM login screen never came up, and logging into a console just dumped me right back to the blank console login. Had to boot into single user mode (append “single” to the kernel boot line in grub) and found out it was selinux issues when I got avc denial errors about /bin/sh not having permissions to execute. set enforce=0 fixed this.

cd /run/media/liveuser/ba7fca87-462e-4e8e-91bc-494f352f5293/
mount -o bind /dev dev
mount -o bind /sys sys
mount -o bind /proc proc
mount -o bind /tmp tmp
mount -o bind /run/media/liveuser/b887b758-1bed-48f9-9ee9-4c78af34b487 boot (the bootable drive)
chroot
vi /etc/dracut.conf
cd /boot
dracut initramfs-3.11.10-301.fc20.x86_64.img `uname -r` --force

superuser.com/a/368640/35094 appears to suggest that I could boot grub on the internal drive and hand off control to the grub on the SD Card, which means that I can get SD Cards for different distros and not have to worry about keeping copies of the various vmlinuz/initramfses on the hard drive /boot. Which would be the ideal situation.

No Comments

Recovering a broken F18 installation

For some reason (possibly a broken F17 upgrade which moved /lib around?) there were a bunch of empty files in /lib. So F18 refused to boot, with error messages like “/lib/libsystemd.so:0 :File too short”

I discovered yum doesn’t like chroots: Kept on getting a message regarding build_time_vars missing. (Spoiler: This was actually Python!)

I also discovered that yum doesn’t like running from an x86_64 arch when the install arch is i686. Solution was to use setarch as mentioned here: strugglers.net/~andy/blog/2011/08/13/centos-6-in-a-chroot/
Essentially, spawn a new bash shell with arch set to i686, so yum will pick everything up as i686 and (re) install the correct files

Final command (with the borked system drive mounted as /media): yum –installroot=/media –skip-broken –releasever=19 distro-sync

Fingers crossed it works, though it can’t get any worse now…
===
And nope, it didn’t work. –skip-broken seems to have sent it into (circular) dependency hell.

Next thing to try is booting an F18 image and copying over /lib/*. Fingers crossed it’ll fix the lib-systemd missing problem.
===

Yay! No need to boot the F18 image and copy over stuff!

It was a bugged fedup upgrade from F18-F19. yum installed a bunch of packages, left everything in an inconcsistent state – like Python 2.7.5 was ‘installed’. but build_info stuff was never added/created as it would normally be. Manually redownloading python and python-libs for F18 (python-2.7.3), removing Python 2.7.5 (rpm -e python-2.7.5, python-libs-2.7.5) and then installing Python 2.7.3 got yum up and running again.

However, rpm -e $(rpm -qa|grep fc19)’ also removed glibc… which is needed for practically everything. (Should have been rpm -e $(yum list installed|grep fc19|awk {‘print $1’}) ). So I rebooted into the live system and tried downloading glib for fc19 – since everything wanted GLIBC_2.17, and F18 only had glibc-2.16.

Whoops. Ended up with another inconsistent system. Glibc was the only thing that was for F19. AND, because it was a dependency for just about everything, I couldn’t erase it, and yum wouldn’t let me downgrade it either.

So I decided to just do the distro-sync over it. And one distro0sync later, I discovered that I *really* should have mounted /boot (since it lived on a different partition) before installing a new kernel/initrd. So reboot into live cd, copy the files over from the root’s /boot (bloody shadowing!) into the proper /boot, mount the proper /boot into the live cd’s /boot (since doing it in a chroot made grub2-mkconfig complain), and run grub2-mkconfig -o /boot/grub2/grub.cfg.

Other than it mysteriously picking up fedup (I thought that had been disappeared!), it seems to have booted F19… at least, gotten part way there and frozen, but it’s 11:40 at night, I have a flight in 9 hours, and I kind of need to sleep, so I’m off for now…

===

And correcting the root=/dev/mapper/live-rw to the proper path (/dev/mapper/vg_lantea-lv_root) got it to at least boot past the root switch, but now services are failing to start, and I don’t know why… Possibly SELinux trouble, so I’m going to try putting enforcing=0 on the boot command line…

And it kind of worked, in that it gets to the login screen at least, though it won’t let me login…

, ,

No Comments