Posts Tagged nginx

Cloudflare & Python’s urllib

TL;DR: Trying to diagnose why my copr builds were abruptly failing, I found an interesting thing: Cloudflare’s Browser Integrity Check apparently doesn’t like Python’s urllib sending requests.

The symptoms in Copr were weird: builds would try importing, and then fail with no log output. To me, trying to diagnose the problem made no sense – I could download the file just fine through Chrome, Firefox and wget, so I thought the issue was with copr. I was all set to file a bug with them, when I decided to look at what other people were building and see if URL importing worked for them.

Surprise surprise, it did. This clearly meant it was isolated to me and my server, so I held off on the bug filing until I could find out more. On my stuff things accessing Jenkins worked fine, so I tried other tools. said there were no issues accessing Jenkins, while RequestBin said… not much at all, I couldn’t get it to work with copr’s URL detection – copr wants the path to end with src.rpm, requestbin uses the path to determine which bin to route stuff to.

Next I looked at Jenkins behind an Nginx proxy. I found an extra Nginx option needed to be set to allow Jenkins CSRF to work properly, which was a plus. (I also found something on serving static files through nginx instead of jenkins, which is now part of my todo list.) I ran through steps on DO’s setting up Jenkins behind nginx tutorial just in case I had missed something, everything looked fine.

Without logs, I couldn’t really trace down what was happening, so I looked at using Github to host the SRPMs, since Github clearly worked. First thing I realized was that keeping 3 branches in an attempt to reduce the number of repos I spawned was a bad idea, and I should fix that. (and maybe follow the git flow strategy…) GitHub has a releases API, so I’ll be spending some time in the near future with it (also, streaming uploads for the 100MB+ pagespeed releases…) Or try the github-release script that someone wrote. Either way, I’d had enough for the night, and stopped.

Today I picked it back up, and took another look. Still thinking it was my server, I tried to isolate which component was failing by serving the src.rpm file on a different domain. Instead of downloading the file and sticking it in a folder, I symlinked a folder to the Jenkins’ workspace directory, and submitted a build. Surprisingly, it worked, so I took a look at my nginx logfile, which revealed that the user agent of what downloaded it was Python-urllib/1.17. That looked like something from the Python stdlib that I could try to replicate the issue with. Started up ipython, googled for python urrlib downfile, and found the urlretrieve function. Tried it against the alternate src.rpm URL, and it worked. Tried it against Jenkins directly, and it failed.

Now I could replicate the symptoms! Next step was to google “Cloudflare blocking urllib”, which suggested that the (lack of) headers was tripping Cloudflare’s Browser Integrity Check. Toggled it off in Cloudflare, and sure enough I could now get the src.rpm file through urllib.

Deciding I now had enough info to file a bug, I went to the Copr codebase to find the relevant line. Downloaded the latest release, unzipped it to grep the code, found the file and line, went to grab the line from the Git repo – and couldn’t find the line. Turns out between last week and this week, the bug was fixed, just not pushed to Copr.

So for now I’ll be regenerating all the missed versions of nginx with the integrity check disabled. Fingers crossed there’s nothing else wrong with Copr in the meantime…



, , ,

No Comments


Just bundled it into copr, so now there’s a yum repo for fedora 20, 21, 22 and rawhide & CentOS 6+7 –

Amusingly, Rawhide changed the ABI, which the configure script had problems with. But I found a solution, which is going into ngx_pagespeed – Documentation fix until the code is brought up to the newer C++ standard.


  • Fix up the config file – centOS 6 doesn’t like the uncommented pid file set to /run/pid; – default for systemd, doesn’t exist for init
    fixed by commenting out the pid declaration in the config file
  • Add pagespeed-by-default config file (copying from Added config file, automatically gets included by virtue of being in /etc/nginx/conf.d
  • Rewrite spec to conflict with the base nginx Obsoletes nginx < 1.7.0, conflicts with nginx >= 1.7.0
  • Create the pagespeed cache folder as part of the install (specified by FileCachePath, normally /var/ngx_pagespeed_cache)Done as part of .spec file

Also fixed a SELinux boolean needing to be set – ngx_pagespeed needs the httpd_execmem permission

Things that helped:

  • Running rpmbuild without the rigid in-root-of-home-directory structure: rpmbuild -bs --define "_topdir $(pwd)" nginx-pagespeed.spec
  • Running mock with --no-cleanup-after and --no-clean, so I could investigate the build directory (located at /var/lib/mock/<release>/root/build/BUILD, or at least along those lines)

Spec files are now in a git repo as well:

Exhaustive list of libraries that may or may not help nginx:


No Comments

Building Nginx SRPMS

Companion to my earlier post, this actually has commands

Read the rest of this entry »


No Comments

Getting CodeIgniter/Bonfire running on nginx

So… I’ve been trying CI/Bonfire as a quick PHP dev platform. Unfortunately, it’s pre-packaged for Apache’s .htaccess, so it takes a bit of configuration to get working on nginx. The following assumes PHP-FPM is actually working though.

 Installing CI/Bonfire

For some reason, the install portion has its own index.php. So the default PHP pretty URL rewriting fails – or in this case, causes infinite redirects.

The trick was to add an extra location to nginx’s config file:

# Installing Bonfire/CI requires this - install has its own index.php
     location /install/ {
       try_files $uri $uri/ /install/index.php;

Once the install is done, we can swap it out for URL rewriting:

# URL rewriting generally requires this, PHP specific
location / {
    try_files $uri $uri/ /index.php;
    allow all;

There was one extra thing necessary to do: change in the index_page variable in  bonfire/application/config/config.php. This was probably a result of my testing to try and get things working, but I had “index.php” in it, so all the generated pages had index.php/ prepended to the internal links.

Hat tip to for a good starting point

, , , ,

No Comments

Getting PHP-FPM running on nginx

Getting FastCGI working on nginx

I have two files – a modified version of FastCGI_params, and an extra file that contains my configuration directives for .php files.

The second file is simply named php_fastcgi and is located in the same folder as nginx.conf, and is include php_fastcgi;-d anywhere I need PHP support:

# Process PHP files with FastCGI
location ~* \.php$ {
    fastcgi_pass unix:/var/run/php-fpm/www-pool.socket;
    include /etc/nginx/fastcgi_params;

As for the fastcgi_params file, it’s slight modification of the default file, included here for simplicity:

fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;

fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;
fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;
fastcgi_param  PATH_INFO          $fastcgi_script_name;

fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;

fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;

# PHP only, required if PHP was built with --enable-force-cgi-redirect
fastcgi_param  REDIRECT_STATUS    200;

Getting PHP-FPM sessions working

The default path, /var/lib/php/sessions is – by default – owned by root, and group apache, so the nginx user can’t write to it. So we have to change the folder owner and group:

chgrp nginx /var/lib/php 
mkdir /var/lib/php/session

Alternatively, create a new directory in /tmp, something like /tmp/php-fpm, and make php-fpm the owner of that, and point php-fpm at that directory. (It would also prevent yum ever mucking with permissions on the folder, something which I’m not sure if it was just my imagination, or actually happened…)

, ,


VPS Playtime

So… I found a cheap VPS for me to play around with on LowEndBox after a few weeks of lurking on their site.

$12/year for 256MB ‘guaranteed’ RAM, 15GB of disk space and 300GB of bandwidth. I am pleased.

Except for the fact that it’s an CentOS OpenVZ instance, and I have had bad experiences with OpenVZ.

But other than being unable to run HLDS (runs out of RAM, gets killed by the host), nothing else has had problems. Got nginx, mysql & php-fpm on it following

I was pretty surprised though – First login showed only ~15MB of RAM used. WITH Apache running. (But no PHP or external modules.) But still. 15MB.

Now, as for uses of it… well, that’s yet to be established. I’ll probably be moving my IRC bot over from EC2, because the credit on that is running out end of October.

As for more configs: looks like it has a bunch of tutorials on getting nginx and the like set up

And has stuff on minimal installs.

, ,

No Comments

nginx testing

As part of my web optimization interest, I’ve been looking into alternatives & add-ons for the traditional LAMP stack.

The most obvious thing I’ve found is replacing Apache with Lighttpd or nginx. nginx seems to be more updated (based on what I’m remembering of a cursory web search a while ago), so I’m focusing my efforts on that.

I’m also looking at PHP-FPM instead of proxying PHP requests to Apache. And Varnish, which I’ve been told good things about by Brian. And a PHP accelerator too. APC or memcache/ Something like that.

I’m planning on using an Amazon EC2 instance to play around with nginx and everything else… get some use out of my credit while I have it.

Read the rest of this entry »

, , ,

No Comments