Posted by Programming on January 13, 2016in
Before I found the
--keep-until-expiring option in the Let’s Encrypt command line client, I was thinking I’d have to parse the cert, extract the expiry date, then check it against the current date before returning True or False.
Thankfully I found the much easier option, but I decided to post the code I wrote to read the date just in case I need something like it in the future. Read the rest of this entry »
dnf --releasever 23 distro-sync instead of
dnf system-upgrade on OpenVZ systems
I run Fedora on my servers almost exclusively. This means I usually fall behind in upgrading to the latest release, leading me to wonder why I don’t just go with the latest version of CentOS.
Then I have lovely cases where CentOS gets horribly outdated, and I remember why I like Fedora with its latest and greatest. (Yes I do like shiny things, thank you very much)
My servers are mostly OpenVZ based, for the simple fact that OpenVZ powered VPSes are rather cheap for what you get, especially where I don’t need high performance. I have just one bad thing about being OpenVZ based: I have no control over the kernel/boot sequence. The vast majority of the time, this isn’t an issue. Sadly, using dnf system-upgrade is one of the times when it is an issue.
Fedora 22 brought in a new way to upgrade your system – dnf system-upgrade. I’ve used it on my laptop, it’s pretty good compared to fedup and past solutions. However, the one thing that rarely failed me in the past was using the yum distro-sync functionality. (The only time I’ve had an issue with it was when the upgrade was stopped midway, but that’s another story.)
TL;DR – Let’s Encrypt requires a newer version of OpenSSL than CentOS 5 has installed. Unless you want to pass around with compiling OpenSSL yourself, don’t try it.
Got my first domain using a cert from Let’s Encrypt in under ~10 minutes, including setting up Let’s Encrypt itself. Yes, this is rather game changing.
Now to write ansible playbooks around it, and figure out how to get it working for proxied domains automatically.
Posted by Personal on November 2, 2015in
I don’t think I’ve published a post on my job searches as part of my time in the SoftEng program at UWaterloo, even though I’ve got a few in my drafts.
Since I’ve had my last interview (and Jobmine has closed & matched), here’s some fun stats without naming companies:
- Favourite interviewer line: “So our alphabet ends with Zee… actually wait, you go to Waterloo, right? So Zed.”
- Favourite interviewer compliment: “If we hired based on domain name, you’d be our first choice!”
- Most common uncommon interview question: “What’s your favourite TF2 class, and why?”
- Most common common interview question: “Tell me about yourself!” (Still don’t have a fixed answer for this)
- Most unexpectedly large number of interviews from a small-ish company: 4
- Most unexpectedly small number of interviews from a large company: 1 (+ code test)
- Weirdest application experience: Rejected on Jobmine, requested on LinkedIn (larger company, left hand doesn’t know what the right is doing)
- Weirdest process experience: “Thanks for your interest in the Winter 2016 Internship position! We’d like to schedule your interview ASAP!” *interview* “Thanks for your interest in the Fall 2016 Internship position! We’d like to schedule your interview ASAP!”
- Most annoying experience: “We’ll definitely consider you for the SRE position since you explicitly requested it!” “Here is the interview schedule for your SWE interviews”
- # of code problems in phone/skype interviews left unfinished: 3
- Best code writing platform: Coderpad
- Worst code writing platform: Google Docs (“Let me just create some indentation groups for you”)
- # of technologies to investigate: 6 (Stemcell, Consul, Elasticsearch, Kibana, Fluentd, Graphana)
Posted by Sysadmin on September 17, 2015in
Because I’ve had to look it up multiple times
Generate the private key
openssl genrsa -out domain.tld.key 2048
then, generate the CSR
openssl req -sha256 -new -key domain.tld.key -out domain.tld.csr
I’m certain there’s a one liner to do this, but didn’t find anything while looking (briefly).
Posted by Programming on August 23, 2015in
TL;DR: Trying to diagnose why my copr builds were abruptly failing, I found an interesting thing: Cloudflare’s Browser Integrity Check apparently doesn’t like Python’s urllib sending requests.
The symptoms in Copr were weird: builds would try importing, and then fail with no log output. To me, trying to diagnose the problem made no sense – I could download the file just fine through Chrome, Firefox and wget, so I thought the issue was with copr. I was all set to file a bug with them, when I decided to look at what other people were building and see if URL importing worked for them.
Surprise surprise, it did. This clearly meant it was isolated to me and my server, so I held off on the bug filing until I could find out more. On my stuff things accessing Jenkins worked fine, so I tried other tools. Hurl.it said there were no issues accessing Jenkins, while RequestBin said… not much at all, I couldn’t get it to work with copr’s URL detection – copr wants the path to end with src.rpm, requestbin uses the path to determine which bin to route stuff to.
Next I looked at Jenkins behind an Nginx proxy. I found an extra Nginx option needed to be set to allow Jenkins CSRF to work properly, which was a plus. (I also found something on serving static files through nginx instead of jenkins, which is now part of my todo list.) I ran through steps on DO’s setting up Jenkins behind nginx tutorial just in case I had missed something, everything looked fine.
Without logs, I couldn’t really trace down what was happening, so I looked at using Github to host the SRPMs, since Github clearly worked. First thing I realized was that keeping 3 branches in an attempt to reduce the number of repos I spawned was a bad idea, and I should fix that. (and maybe follow the git flow strategy…) GitHub has a releases API, so I’ll be spending some time in the near future with it (also, streaming uploads for the 100MB+ pagespeed releases…) Or try the github-release script that someone wrote. Either way, I’d had enough for the night, and stopped.
Today I picked it back up, and took another look. Still thinking it was my server, I tried to isolate which component was failing by serving the src.rpm file on a different domain. Instead of downloading the file and sticking it in a folder, I symlinked a folder to the Jenkins’ workspace directory, and submitted a build. Surprisingly, it worked, so I took a look at my nginx logfile, which revealed that the user agent of what downloaded it was Python-urllib/1.17. That looked like something from the Python stdlib that I could try to replicate the issue with. Started up ipython, googled for python urrlib downfile, and found the urlretrieve function. Tried it against the alternate src.rpm URL, and it worked. Tried it against Jenkins directly, and it failed.
Now I could replicate the symptoms! Next step was to google “Cloudflare blocking urllib”, which suggested that the (lack of) headers was tripping Cloudflare’s Browser Integrity Check. Toggled it off in Cloudflare, and sure enough I could now get the src.rpm file through urllib.
Deciding I now had enough info to file a bug, I went to the Copr codebase to find the relevant line. Downloaded the latest release, unzipped it to grep the code, found the file and line, went to grab the line from the Git repo – and couldn’t find the line. Turns out between last week and this week, the bug was fixed, just not pushed to Copr.
So for now I’ll be regenerating all the missed versions of nginx with the integrity check disabled. Fingers crossed there’s nothing else wrong with Copr in the meantime…
Posted by Sysadmin on March 26, 2015in
Just bundled it into copr, so now there’s a yum repo for fedora 20, 21, 22 and rawhide & CentOS 6+7 – copr.fedoraproject.org/coprs/kyl191/nginx-pagespeed/
Amusingly, Rawhide changed the ABI, which the configure script had problems with. But I found a solution, which is going into ngx_pagespeed – github.com/pagespeed/ngx_pagespeed/issues/942. Documentation fix until the code is brought up to the newer C++ standard.
TodoDone: Fix up the config file – centOS 6 doesn’t like the uncommented pid file set to /run/pid; – default for systemd, doesn’t exist for initfixed by commenting out the pid declaration in the config file
Add pagespeed-by-default config file (copying from developers.google.com/speed/pagespeed/module/configuration)Added config file, automatically gets included by virtue of being in /etc/nginx/conf.d Rewrite spec to conflict with the base nginxObsoletes nginx < 1.7.0, conflicts with nginx >= 1.7.0 Create the pagespeed cache folder as part of the install (specified by FileCachePath, normally /var/ngx_pagespeed_cache)Done as part of .spec file
Also fixed a SELinux boolean needing to be set – ngx_pagespeed needs the httpd_execmem permission
Things that helped:
- Running rpmbuild without the rigid in-root-of-home-directory structure:
rpmbuild -bs --define "_topdir $(pwd)" nginx-pagespeed.spec
- Running mock with
--no-clean, so I could investigate the build directory (located at
/var/lib/mock/<release>/root/build/BUILD, or at least along those lines)
Spec files are now in a git repo as well: github.com/kyl191/nginx-specs
Exhaustive list of libraries that may or may not help nginx: serverfault.com/a/636642/28001
Or: How I spent an afternoon doing a deep dive into the RPM spec and solving a problem for myself
tl;dr – Nginx Mainline packages are being built for Fedora & CentOS at copr.fedoraproject.org/coprs/kyl191/nginx-mainline/
My webserver’s running nginx 1.4.7, a version that hasn’t gotten non-bugfix attention since March 2013, according to the changelog. Oddly enough for a Fedora package, the version in koji is the stable branch – something that makes sense for CentOS/EPEL since that’s a long term support release, but not for Fedora. Doubly annoying was the fact that the packages for Fedora 20 (because Fedora 21 isn’t supported on OpenVZ yet) was one step behind the official stable – 1.4 instead of 1.6.
If I wanted mainline on Fedora, I was going to have to build it from source.
Read the rest of this entry »