Posts Tagged github
TL;DR: Trying to diagnose why my copr builds were abruptly failing, I found an interesting thing: Cloudflare’s Browser Integrity Check apparently doesn’t like Python’s urllib sending requests.
The symptoms in Copr were weird: builds would try importing, and then fail with no log output. To me, trying to diagnose the problem made no sense – I could download the file just fine through Chrome, Firefox and wget, so I thought the issue was with copr. I was all set to file a bug with them, when I decided to look at what other people were building and see if URL importing worked for them.
Surprise surprise, it did. This clearly meant it was isolated to me and my server, so I held off on the bug filing until I could find out more. On my stuff things accessing Jenkins worked fine, so I tried other tools. Hurl.it said there were no issues accessing Jenkins, while RequestBin said… not much at all, I couldn’t get it to work with copr’s URL detection – copr wants the path to end with src.rpm, requestbin uses the path to determine which bin to route stuff to.
Next I looked at Jenkins behind an Nginx proxy. I found an extra Nginx option needed to be set to allow Jenkins CSRF to work properly, which was a plus. (I also found something on serving static files through nginx instead of jenkins, which is now part of my todo list.) I ran through steps on DO’s setting up Jenkins behind nginx tutorial just in case I had missed something, everything looked fine.
Without logs, I couldn’t really trace down what was happening, so I looked at using Github to host the SRPMs, since Github clearly worked. First thing I realized was that keeping 3 branches in an attempt to reduce the number of repos I spawned was a bad idea, and I should fix that. (and maybe follow the git flow strategy…) GitHub has a releases API, so I’ll be spending some time in the near future with it (also, streaming uploads for the 100MB+ pagespeed releases…) Or try the github-release script that someone wrote. Either way, I’d had enough for the night, and stopped.
Today I picked it back up, and took another look. Still thinking it was my server, I tried to isolate which component was failing by serving the src.rpm file on a different domain. Instead of downloading the file and sticking it in a folder, I symlinked a folder to the Jenkins’ workspace directory, and submitted a build. Surprisingly, it worked, so I took a look at my nginx logfile, which revealed that the user agent of what downloaded it was Python-urllib/1.17. That looked like something from the Python stdlib that I could try to replicate the issue with. Started up ipython, googled for python urrlib downfile, and found the urlretrieve function. Tried it against the alternate src.rpm URL, and it worked. Tried it against Jenkins directly, and it failed.
Now I could replicate the symptoms! Next step was to google “Cloudflare blocking urllib”, which suggested that the (lack of) headers was tripping Cloudflare’s Browser Integrity Check. Toggled it off in Cloudflare, and sure enough I could now get the src.rpm file through urllib.
Deciding I now had enough info to file a bug, I went to the Copr codebase to find the relevant line. Downloaded the latest release, unzipped it to grep the code, found the file and line, went to grab the line from the Git repo – and couldn’t find the line. Turns out between last week and this week, the bug was fixed, just not pushed to Copr.
So for now I’ll be regenerating all the missed versions of nginx with the integrity check disabled. Fingers crossed there’s nothing else wrong with Copr in the meantime…
Because I’ve had to do it a number of times, and had to look it up each time:
- Create the repo on github
- On the desktop/laptop/whatever, in Git Bash, change to the folder with the repo
git remote add origin [email protected]:username/repo.git, then
git branch --set-upstream master origin/master and finally
I’ve been using Git and GitHub for all of my CS137 assignments, in part to keep the files in sync between my desktop and laptop, as well as the UW server that I use for testing. The keeping everything in source control is a bonus, but one that comes in handy, especially branching when I discover that actually my code can be rewritten in a better way.
The primary impetus for this post is that I’ve been creating a branch for each assignment, but I still have yet to get the branching push/pull commands down pat, so I always end up searching Google for the appropriate commands. After 6 branches (and counting), I’ve gotten tired of that, so I shall centralise it in one place:
Pushing a new branch:
git push origin new_branch
Pulling a new branch: or
git pull && git checkout --track origin/new_branch or
git checkout new_branch
I’ve known about ssh-agent for a while, but as I was practically permanently using PuTTY (on Windows), I only bothered with learning about Pageant.
But Git uses ssh to connect to github, and I was getting tired of typing in my password with every push. I got annoyed with InteliJ for making me type in my password in with every push, and this was no different. Because git uses bash as its’ command line on both Windows and Linux, I decided to get started with using ssh-agent.
The first time I ran ssh-agent expecting it to work automagically. Instead it dumped what looked like two environment variables to the screen and quit. Not too helpful, but I manually copied, formatted & pasted the variables, and got it working.
But doing that manually, while awesome when pushing from one system and pulling on another, was also annoying with having to do it every time I logged in. So I looked into automating it. I knew of eval and backticks in bash. So I tried `ssh-agent`.
sh.exe": SSH_AUTH_SOCK=/tmp/ssh-myYvgp1404/agent.1404;: No such file or directory
Hmm. No joy. Ok, let’s try eval ssh-agent. Maybe that’ll make a difference?
$ eval ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-zIQZKN6080/agent.6080; export SSH_AUTH_SOCK;
SSH_AGENT_PID=1092; export SSH_AGENT_PID;
echo Agent pid 1092;
Nope, back to manually editing it. Hmm. Google search time.
Which led me to a uni helpdesk page. Which told me that I had to use both eval and backticks. Huh. How ingenious.
$ eval `ssh-agent`
Agent pid 4288
OH HEY THAT LOOKS GOOD.
So I gleefully ran it on the uni’s servers. And then discovered I had to run it in every bash shell. Hmm. Sounds kind of unoptimal. Fixed my original problem, which was having to type in my password each and every time I pushed or pulled from github. And using eval and backticks meant it was easier than copy & pasting the output. But I didn’t like having to run ssh-add and ssh-agent in every new session, or that ssh-agent wouldn’t auto-terminate when I closed bash. (Which led to a surprise when I ran
ps u and saw at least 8 ssh-agent processes running with my username.)
So I just ran it in a screen session. And just have to use that one window everytime. Fairly straightforward.
The alternative was add to the commands to my .bashrc. But the uni servers seem to do something to bash such that it doesn’t execute a user’s bashrc when they login, only after they run bash. So running it in screen works just as well, since I’d need to run a command anyway.
As an alternative alternative, I also found a script that should work for keeping track of ssh-agent and can be run from .bashrc, so I’ll be looking into that too.