Posts Tagged fedora

Upgrading to Fedora 23 on OpenVZ

TL;DR: Run dnf --releasever 23 distro-sync instead of dnf system-upgrade on OpenVZ systems

I run Fedora on my servers almost exclusively. This means I usually fall behind in upgrading to the latest release, leading me to wonder why I don’t just go with the latest version of CentOS.

Then I have lovely cases where CentOS gets horribly outdated, and I remember why I like Fedora with its latest and greatest. (Yes I do like shiny things, thank you very much)

My servers are mostly OpenVZ based, for the simple fact that OpenVZ powered VPSes are rather cheap for what you get, especially where I don’t need high performance. I have just one bad thing about being OpenVZ based: I have no control over the kernel/boot sequence. The vast majority of the time, this isn’t an issue. Sadly, using dnf system-upgrade is one of the times when it is an issue.

Fedora 22 brought in a new way to upgrade your system – dnf system-upgrade. I’ve used it on my laptop, it’s pretty good compared to fedup and past solutions. However, the one thing that rarely failed me in the past was using the yum distro-sync functionality. (The only time I’ve had an issue with it was when the upgrade was stopped midway, but that’s another story.)

Read the rest of this entry »

, ,

No Comments

Path to building Nginx Mainline RPMs for Fedora & CentOS

Or: How I spent an afternoon doing a deep dive into the RPM spec and solving a problem for myself

tl;dr – Nginx Mainline packages are being built for Fedora & CentOS at copr.fedoraproject.org/coprs/kyl191/nginx-mainline/


My webserver’s running nginx 1.4.7, a version that hasn’t gotten non-bugfix attention since March 2013, according to the changelog. Oddly enough for a Fedora package, the version in koji is the stable branch – something that makes sense for CentOS/EPEL since that’s a long term support release, but not for Fedora. Doubly annoying was the fact that the packages for Fedora 20 (because Fedora 21 isn’t supported on OpenVZ yet) was one step behind the official stable – 1.4 instead of 1.6.
If I wanted mainline on Fedora, I was going to have to build it from source.
Read the rest of this entry »

, ,

1 Comment

Getting OpenVPN to work on an OpenVZ VPS

Note: This is a personal VPN, so I just used static keys. A general guide to getting OpenVPN set up is available on the OpenVPN website, but this guide is targeted at CentOS 5 on an OpenVZ VPS.

This guide should be usable in other RH derivatives without much (any?) modification; and with slight modifications for debian-style distros, especially in installing packages and folder paths.

If you’re not running OpenVZ, I’d recommend following the site where the vast majority of this guide comes from, but I had problems with it – I had to mess around with the config files, and the iptables commands *will* kill your SSH session if you run it. Read the rest of this entry »

, ,

7 Comments

HLDS on EC2

Update Jul 2014 – use the shiny new SteamCMD tool. Also, I started using DigitalOcean instead of EC2 (marginally cheaper, incredibly easier to setup)

Create a Instance – spot works

Install wget, tar, ncompress – depends on your distro, fedora 17 by default didn’t come with this (which, btw, is WTF?!) Also, screen to run disconnected from the server, and vim for text editing

wget http://media.steampowered.com/client/steamcmd_linux.tar.gz && tar -xvzf steamcmd_linux.tar.gz && ./steamcmd.sh

download the actual HLDS server

mkdir tf2 && ./steamcmd.sh +force_install_dir tf2 +login anonymous +app_update  232250 +quit

setup the config files

cd tf2/orangebox/

touch cfg/server.cfg && vim cfg/server.cfg – specify what you want, I pulled mine from forums.srcds.com/viewpost/67692#pid67692

./srcds_run -game tf -autoupdate -maxplayers 32 -console +map mvm_coaltown

Notes:

Pull map names from TF2 wiki, eg wiki.teamfortress.com/wiki/Mann_vs._Machine_%28game_mode%29 has the map filenames listed under “Maps”.

If you’ve got iptables running *in addition to* the aws security groups, disable iptables, or allow exceptions for the HLDS ports (UDP 27000-27015).

A t1.micro instance *isn’t* suitable for a tf2 server. The way Amazon has it set up is that Micro instances will have their CPU stolen by other instances if necessary. While this rarely happens to any great effect, when it does, there’s a large ping spike.

List of console commands, unknown how many map to the server: developer.valvesoftware.com/wiki/List_of_TF2_console_commands_and_variables

Official Valve post on setting up HLDS: support.steampowered.com/kb_article.php?ref=6758-TCMF-2234 (Wasn’t too helpful, also Windows specific.)

On the client, you have to set rcon_password to the server password, *then* issue rcon commands. Doing “rcon password” in the client won’t work.

Some other stuff relating to HLDS here: wiki.teamfortress.com/wiki/Linux_dedicated_server

Potential mods: www.mani-admin-plugin.com/cms/index.php/about and wiki.alliedmods.net/SourceMod

18th Aug update:

For some reason, auto-update failed, had to manually update.

3:07 Updating 'Team Fortress 2 Content' from version 350 to version 352

A symptom was a bunch of messages showing up in the log:

MasterRequestRestart
Your server will be restarted on map change.
Your server will be restarted on map change.

Also, a bunch of config changes coming too.

Aug 19th: Since I’m jumping between EC2 regions spinning up servers as and when I want to play on them, a ‘one’-line command to get the TF2 server up & running is now here

Also, 3 files are edited in the orangebox/tf folder: cfg/server.cfg, mapcycle.txt and motd.txt

A pretty extensive sample server.cfg was posted here: forums.steampowered.com/forums/showpost.php?p=32341843&postcount=1

If the servers were going to be around for a while, register them with Valve as per support.steampowered.com/kb_article.php?ref=2825-AFGJ-3513 for Quickplay support

More plugins: forums.alliedmods.net/showthread.php?t=84801, www.sourcemod.net/newstats.php?mod_id=558&addon_id=0, forums.alliedmods.net/showthread.php?p=592536 and forums.alliedmods.net/showthread.php?p=532620

RCON Guides: forums.srcds.com/viewtopic/13192 and www.uk-tf.co.uk/index.php/rcon-guides#Kicking%20&%20Banning%20players

, , ,

3 Comments

Removing bloat from Fedora 17 on EC2

I’m in the middle of trialing EC2, and I’m using the official Fedora 17 images kindly provided by the community.

They make a great starting point, because I can then install my needed software.

Some of which, though, I consider absolutely crucial. So far, I’ve need to install vim, less, rsync and screen. I can forgive rsync and screen, and to a certain extent less… but vim?

Especially when things like NetworkManager, ModemManager and mobile-broadband-provider-info are taking up space. Then there’s stuff like plymouth – we’re not seeing a graphical boot screen, so I should be able to erase this, right?

I hope that I don’t have to roll my own AMI image though. Just modify this one as needed, and create an AMI out of it. Read the rest of this entry »

, ,

1 Comment

cron screwyness

While trying to diagnose a problem with my VMs (namely, why starting a fedora 16 based vm fails to bring up the network connection), I ran into a strange issue – on my dom0 and some of the domUs, logrotate hasn’t been running, leaving me with insanely long logfiles!

So I set out trying to understand where the problem came from. Both the f16-based dom0 and domU had old logs, but my f14 domU was still working. That pointed to an update failing. And because logrotate never ran, I’ve got the yum logs of what was installed.

logrotate by default is setup to append the date to the filename, so I knew when logrotate was last run – 20111127. So anything between 27th November and 4th December could be the culprit.  To start with, I looked at cron in particular, since that’s what supposed to run logrotate.

When I checked in my dom0, I got this:

[[email protected] log]# grep cron yum.log
 Dec 01 00:23:46 Updated: cronie-anacron-1.4.8-2.fc14.x86_64
 Dec 01 00:23:48 Updated: cronie-1.4.8-2.fc14.x86_64
 Dec 01 01:38:54 Updated: cronie-anacron-1.4.8-2.fc15.x86_64
 Dec 01 01:38:56 Updated: cronie-1.4.8-2.fc15.x86_64
 Dec 01 01:38:58 Updated: crontabs-1.11-2.20101115git.fc15.noarch
 Dec 01 03:05:41 Updated: cronie-anacron-1.4.8-10.fc16.x86_64
 Dec 01 03:05:43 Updated: cronie-1.4.8-10.fc16.x86_64

Which showed me that I upgraded my dom0 to f16 on December 1st. Which is kinda helpful – I did upgrade cronie and cronie-anacron. Also, crontabs was upgraded, but only in F15. Then I looked at my domU. Again, judging from the dates in /var/log, I was looking for something between November 7 – 13. And lo and behold, I found lines that looked exactly like those in my dom0:

[[email protected] log]# grep cron yum.log
Aug 20 15:20:05 Updated: cronie-1.4.8-2.fc14.x86_64
Aug 20 15:20:05 Updated: cronie-anacron-1.4.8-2.fc14.x86_64
Nov 07 18:33:50 Updated: cronie-anacron-1.4.8-2.fc15.x86_64
Nov 07 18:33:54 Updated: cronie-1.4.8-2.fc15.x86_64
Nov 07 18:33:54 Updated: crontabs-1.11-2.20101115git.fc15.noarch
Nov 07 20:50:42 Updated: cronie-anacron-1.4.8-10.fc16.x86_64
Nov 07 20:50:44 Updated: cronie-1.4.8-10.fc16.x86_64

Except the F14 update was done in August, before the problem started. So, hmm, maybe the upgrade to Fedora 15 broke it. It would certainly explain why the F14 domU is still working fine. And I just happened to have an F15 domU that I had yet to upgrade to F16.

But it was strange – when I checked it, logrotate was working fine. /var/log had stuff timestamped 20120429, so the upgrade to F15 couldn’t have been the problem. logrotate kept an old version of yum.log, so I was able to look for cron related entries:

[[email protected] log]# grep cron yum.log*
yum.log:Apr 19 13:16:38 Updated: cronie-anacron-1.4.8-4.fc15.x86_64
yum.log:Apr 19 13:16:40 Updated: cronie-1.4.8-4.fc15.x86_64
yum.log-20120101:Sep 12 18:54:29 Updated: cronie-anacron-1.4.8-2.fc15.x86_64
yum.log-20120101:Sep 12 18:54:30 Updated: cronie-1.4.8-2.fc15.x86_64

Which wasn’t very helpful – I was using virtually the exact same versions that I had upgraded my F16 systems to – 1.4.2. So it wasn’t the F14 -> F15 upgrade. So maybe it was the F15 -> F16 upgrade?

I puzzled over this, then I saw a log named ‘cron’. Which turned out to really be key:

[[email protected] log]# tail -n 5 cron
Dec  1 01:39:01 elemental crond[2011]: (*system*) RELOAD (/etc/cron.d/0hourly)
Dec  1 01:41:01 elemental crond[2011]: (*system*) RELOAD (/etc/cron.d/smolt)
Dec  1 01:43:01 elemental crond[2011]: (*system*) RELOAD (/etc/cron.d/sysstat)
May  1 23:49:29 elemental crontab[9780]: (root) BEGIN EDIT (root)
May  1 23:49:41 elemental crontab[9780]: (root) END EDIT (root)

The crontab made sense – I run crontab -e to see if logrotate was in root’s crontab (which it wasn’t), but the crond entries were interesting – for one, it showed me the exact time it stopped, but also told me that cron has a daemon. And guess what was part of the F15 -> F16 upgrade? Moving stuff to the new systemd.

So I quickly checked crond’s status on a F16 machine:

[[email protected] log]# systemctl status crond.service
crond.service - Command Scheduler
          Loaded: loaded (/lib/systemd/system/crond.service; disabled)
          Active: inactive (dead)
          CGroup: name=systemd:/system/crond.service

Which, oops, that looks bad. But the F15 machine looked a lot better:

[[email protected] log]# systemctl status crond.service
crond.service - Command Scheduler
          Loaded: loaded (/lib/systemd/system/crond.service)
          Active: active (running) since Mon, 30 Apr 2012 14:16:01 +0800; 1 day and 10h ago
        Main PID: 831 (crond)
          CGroup: name=systemd:/system/crond.service
                  â 831 /usr/sbin/crond -n

And finally the F14 machine looked like this:

[[email protected] init.d]# service crond status
crond (pid  1118) is running...

So it seems the fix is to enable crond. I have no clue why crond would be disabled between F15 -> F16 upgrades, but I’m seeing it on two different F16 systems, both of which were upgraded from F15. I might do a clean install of F16 and look, as well as try an F15 -> F16 upgrade, but that’s for later.

For now, I’m doing a systemctl enable crond.service followed by systemctl start crond.service on the three F16 systems and calling it a day.

, ,

No Comments

Because this is worth repeating

I could have sworn I did this a while ago. But I got the error “‘Out of memory’, “xc_dom_boot_mem_init: can’t allocate low memory for domain\n “” when I tried to do a virt-install today.

The problem? I was installing a 64bit domU on a 32bit dom0. Which is a no-no.

Didn’t realise until I found this post on someone else’s blog detailing the error.

Now hopefully I don’t repeat this again…

, ,

No Comments

Why I’m becoming increasingly disillusioned with SELinux

My history with SELinux is a… varied one.

I first remember using it back in Fedora Core 6. I soon gave up on it, the labeling wasn’t consistent and I didn’t have the time nor inclination to relabel everything, especially when a quick one work change in a config file fixed all my problems.

The next time I used it was probably in F10. I ended up disabling it then, for much the same reasons.

Then I got onto F12/13 for my home server, and disabled SELinux again pretty darn quickly, after neither samba nor Apache liked it.

I can’t remember if I ever disabled it on my testing server which was running F15. But when I upgraded the server to F16Beta, I quickly rediscovered SELinux. Mainly because I’d get errors like this one:

avc:  denied  { read } for  pid=1868 comm="dhclient-script" name="ifcfg-br0" dev=dm-4 ino=10894 scontext=unconfined_u:system_r:dhcpc_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file

As far as I can tell, that’s SELinux denying the default network setup script from running. br0 is my custom xen bridge, I’ll admit, but it was working fine in F15, and even F16B until a new update came out sometime on or about Oct 20, breaking the network script.SELinux probably lasted the longest on F16B so far, because, for the most part, it worked perfectlyfine until, well, it just didn’t work.

  1. Sudo -i just  didn’t work. I spent about 30 minutes trying to hunt down what was causing the problems – checking /etc/groups, running visudo, the whole shebang. Everything said “Yes, I should be allowed to use sudo!” But sudo would perpetually tell me “User kyl191 is not in the sudoers file. This incident will be reported.”, and it was. My root terminal would get an extra line reading “You hve new mail” after any command was run.
  2. Starting up VMs just outright didn’t work, SELinux blocked access to the disk, but it was never relayed back to xen – Xen would just die with the exception “Cannot find bootloader”. That was my first inkling that SELinux is still in force.
  3. Last straw, SELinux killed my networking setup when it did…. something after an update that was published on/around Oct 21.

Some will argue that my ifcfg-br0 script is in the wrong context, so of course it would fail. To that my only response is “So…what context should I use then?” I’ve got no clue, and I’m sorry, but I’m not about to spend time looking for a list of contexts, and trying to apply them to each and every file I encounter in such a situation.

I’ve trying to write documentation for the Xen project. On my testing server, I’m not going to bother enabling SELinux because it gives me strange errors. For prod servers, I might use it if there was a VM for each service, otherwise the resulting mish-mash of programs tend to lead to unexpected results.

, ,

No Comments

When a yum update kills itself…

There’s a bunch of clean up to do.

New kernel isn’t working properly – boot from old one.

Try yum update again, get an error message about an existing transaction.

Have yum-utils already installed, so yum-complete-transactions is a command away.

Even after repeating it multiple times, yum still complains about broken packages.

Turns out yum died in the middle of cleanup, so old packages were still in the rpmDB.

Discovery of the day: package-cleanup –cleandupes will fix that error, even though the list of packages to remove looks very scary.

, ,

No Comments