The Life of a SysAdmin

So, I decided it was time for a change.  A new look to an old place.  A lot has changed in the last few months.  I moved a couple times, got a job, got a car, all sorts of madness.  It has been terribly exciting.

So, I’m working at iFactory as a sysadmin.  They don’t call me that but they don’t really know what it means.  You don’t either? Well, I think I can make up something convincing.  And by that I mean Wikipedia can do it for me.

“A system administrator, IT systems administrator, systems administrator, or sysadmin is a person employed to maintain and operate a computer system and/or network. System administrators may be members of an information technology (IT) or Electronics and Communication Engineering department.

The duties of a system administrator are wide-ranging, and vary widely from one organization to another. Sysadmins are usually charged with installing, supporting and maintaining servers or other computer systems, and planning for and responding to service outages and other problems. Other duties may include scripting or light programming, project management for systems-related projects, supervising or training computer operators, and being the consultant for computer problems beyond the knowledge of technical support staff. To perform his or her job well, a system administrator must demonstrate a blend of technical skills and responsibility.”

Yup.  It’s all of that and then some.  I think there is a bigger focus on the programming aspect at iFactory because my predecessors did some programming and I am doing some automation work.  Automation work, you say?  What’s that?

I won’t turn to Wikipedia for that one.  Automation is the process of taking a bunch of tasks and, through scripting or some other means, allowing them to run in a repeatable fashion.  Some of that work is through bash scripts, like the one posted for TF2, and some of is through tool like puppet or cfengine.

I, personally, use puppet.  Just like the reason I fold my towels, it’s what I grew up on.  I’m not a puppet pro but I know it well enough and I’ve seen it used in enough places to have some best practices.  Since starting a iFactory, the time spent on new server setups has decreased from a couple days to a couple hours.

The process isn’t perfect but it’s getting there.  I have a much smaller list of work still to be done than I had 6 months ago.  It’s a good feeling when you can go through a checklist of items and laugh at the estimates beside them.

So there you have it, when the world slips you a Jeffrey, look it up on Wikipedia.

I just want to warn you folks, this one touches on some pretty personal stuff.  I don’t really know how to get it out there any other way.

My grandmother is dying.  She had something up with her gall bladder and went to the hospital.  They did one procedure to get things working again and sent her home.  I knew it wasn’t good but I didn’t completely understand the impact of the situation until my dad told me.  It took me back a little, I didn’t know exactly what to do.

We’re in Maine this weekend and I’m noticing all sorts of changes.  She gets tired very easily and forgets things easily.  She repeats comments after a few minutes.  She sleeps later and seems to be comfortable so it’s not all bad.  When she’s uncomfortable, she will adjust herself or ask for something to be adjusted.  It’s nothing I want to think about too much but it’s also difficult to avoid.

My dad keeps saying it’s important to make sure she’s happy and comfortable.  So we’ll keep plugging along.  As long as she’s happy, we can all be happy, too.

I think I’ll go work on something more exciting now.  Maybe another post coming soon.

Seriously.  It’s like all the stuff you need to make cookies is staring you in the face.  All of a sudden, the eggs break open and slap you in the eye socket.  The milk pours down your front and the flour projects itself on to your favorite shirt.  And just when you think you have them under control, the chocolate chips melt and streak down your pants in an embarrassing manner.

Ok, back to business.  I’m trying to get a letter out to a company to get an interview and a job.  I know I can do all the stuff they ask me to do but I can’t tell them in the right way.  Part of me wants to scream, “oh forget this crap, look at the silly resume.”  Obviously, they want this letter for a reason.

So, the major thing to remember is that they want to make sure you can do the job.  But your resume is about you, not the cover letter.  The cover letter is about THEM.  Make your achievements and experiences tie back to the stuff they outline in the job posting.  I suck at it so I won’t give examples but someone should be able to figure this out.

So I applied for a job.  (yay!)

I interviewed a few times (yay!)

I didn’t get the job (boo!)

I was rather upset yesterday when I heard back.  You know that sinking feeling you get in your stomach when someone says, “unfortunately….”?  Yeah, it was like that.

So, for an hour or so I moped around.  I was in lab when I got the call so I sat there and worked a little more before giving up and leaving.

I sent a text to each of my parents.  My mother, at lunch at the time, was the first to respond.  She was comforting and apologized but I still felt like crap.  So I called my dad.

He was with a client so he said he’d call back.

He called a little while later and we talked about the interview.  I started to feel better.  We talked about random stuff after that.  Eventually, he told me Winnie the Pooh is a Daoist.  I’m not sure where that came from but I hope someone can shed some light on it.

So, I’m looking for a job.  Anyone need a Jr. Systems Administrator with an appetite for knowledge and enough drive to run a small website for months?

#!/bin/bash
# script to bzip and move maps from tf/maps to redirect/maps

# static variables here for easy changing
homedir=/home/tfuser/tf2server/orangebox/tf
wwwdir=/var/www/redirects
homemaplog=/var/log/tf/tfmaps.home.log
wwwmaplog=/var/log/tf/tfmaps.www.log
diffmaplog=/var/log/tf/tfmaps.diff.log
homesoundlog=/var/log/tf/tfsound.home.log
wwwsoundlog=/var/log/tf/tfsounds.www.log
diffsoundlog=/var/log/tf/tfsounds.diff.log

# remove old logs
rm $homemaplog
rm $wwwmaplog
rm $diffmaplog
rm $homesoundlog
rm $wwwsoundlog
rm $diffsoundlog

# write directory listings to file
ls -1 $homedir/maps > $homemaplog
ls -1 $wwwdir/maps > $wwwmaplog
ls -R1 $homedir/sound > $homesoundlog
ls -R1 $wwwdir/sound > $wwwsoundlog

# diff the listings and output to file
# output is used to determine which maps to copy
diff -IbBq $homemaplog $wwwmaplog | grep .bsp >> $diffmaplog
diff -IbBq $homesoundlog $wwwsoundlog | grep .wav >> $diffsoundlog

# remove extra characters from output file
sed -i 's/\(.\{2\}\)//' $diffmaplog
sed -i 's/\(.\{2\}\)//' $diffsoundlog

# to avoid confusion, change to homedir and move maps
cd $homedir/maps
rsync -rlpt $homedir/maps/*.bsp $wwwdir/maps

# change to wwwdir and zip maps
cd $wwwdir/maps
bzip2 -qzk `cat $diffmaplog`

# to avoid confusion, change to homedir and move sounds
cd $homedir/sound
rsync -rlpt $homedir/sound/ $wwwdir/sound

# change the files so they are accessible outside
chmod 775 $wwwdir/maps/*.bz2
chmod -R 775 $wwwdir/sound

ls -1 $homedir/maps |grep .bsp | awk -F"." '{ print $1 }' > /home/tfuser/tf2server/orangebox/tf/addons/sourcemod/configs/adminmenu_maplist.ini
cp /home/tfuser/tf2server/orangebox/tf/addons/sourcemod/configs/adminmenu_maplist.ini /home/tfuser/tf2server/orangebox/tf/mapcycle.txt
cp /home/tfuser/tf2server/orangebox/tf/addons/sourcemod/configs/adminmenu_maplist.ini /home/tfuser/tf2server/orangebox/tf/maplist.txt

So this does a listing of the tf server directory and compares it to the public distribution directory.  Once it has the diff, it copies the remaining ones and bzips the maps so the client can download.  As you can see, it also does sound.  I used that for an old mod that’s now broken.  Fine to leave if you found something that requires sounds to be downloaded.

As a final step, it updates the map listings in sourcemod so the maps all show up for admin menus.  Obviously, if you don’t have sourcemod installed, you can drop those steps.  Or install sourcemod.  I don’t know of a better mod plugin out there.

The script is run as a normal user’s crontab (*/5 * * * *) and will work as long as the folders are user or group read/writable.  One of the steps of my startup/update script is to ensure the tf server home dir is read/writable by group web.  For whatever stupid reason, the source updater changes everything to root:root and must be run by root.

#!/bin/bash
# script to find a running tf2 server and start it, if it is not running

export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

ip=`ifconfig  | grep 'inet addr:'| grep -v '127.0.0.1' | cut -d: -f2 | awk '{ print $1}'`
homedir="/home/tfuser"
rundir="/home/tfuser/tf2server/orangebox"
updatecmd="./steam -game tf -command update -dir tf2server/"
runcmd="./srcds_run -console -game tf -hostport 27015 +maxplayers 12 +map ctf_2fort -ip $ip -debug"
pidfile="/var/run/tf2.pid"

cd $homedir

case "$1" in
 start)
 $updatecmd
 chown -R tfuser:web $homedir
 chmod -R 770 $rundir
 if ! (pgrep srcds_run > $pidfile); then
   echo "tf2 server not running, starting..."
   cd $rundir
   screen -d -m sudo -u tfuser $runcmd
   echo "tf2 server started in screen session"
 else
   echo "tf2 server running in screen session"
 fi
 ;;
 stop)
 # No-op
 echo 'Killing TF2 process from PID file'
 if [ -f $PIDFILE ] ; then
   PID=`cat $pidfile`
   kill -3 $PID
   if kill -9 $PID ; then
     echo "tf2 process stopped"
   else
     echo "tf2 process could not be stopped"
   fi
   rm -f $pidfile
 fi
 ;;
 restart)
 /usr/local/bin/checktf.sh stop
 /usr/local/bin/checktf.sh start
 ;;
 update)
 $updatecmd
 chown -R tfuser:web $homedir
 chmod -R 770 $rundir
 ;;
 noupdate)
 if ! (pgrep srcds_run > $pidfile); then
   echo "tf2 server not running, starting..."
   cd $rundir
   screen -d -m sudo -u tfuser $runcmd
   echo "tf2 server started"
 else
   echo "tf2 server running"
 fi
 ;;
 *)
 echo "Usage: $0 start(implies update)|stop|restart|noupdate|update" >&2
 exit 3
 ;;
esac

A pretty straightforward bash script to start a tf2 server.  It assumes a few things.
1.) You have a separate user for running the server (it’s a security thing)
2.) The server should listen on the eth0 (or whatever is the first in ifconfig) device.

One of the drawbacks of this script, or maybe of the server program in general, is that it runs in a screen.  I may get around to changing that and dumping everything to a log file.  The benefit of that is to have Nagios check the log file for the update notification.  Currently I have a cronjob running a daily restart.  The only benefit of the restart is if the server is dead for a long time and the map doesn’t change, the log files can get bloated and cause severe lag.

Maybe later I’ll post my mapcheck script.  It updates the public directory that serves maps and other custom content to game clients.

And we just have to go along with it.

Man, I’ve been busy.

So, I don’t have a job.  I think that’s why I wanted to sit down and write.

This bothers me.  More than I would prefer and more than I hopefully let on.  It bothers me because I don’t exactly know why I don’t have a job.  Lack of funding?  Poor reason.  Lack of skills?  I hope not but possible.  Lack of trust?  Possible.  I’m very good at making mistakes.  Lack of social acceptance?  Can’t do anything about that but it would also be really disappointing.  Meh.

So, what’s new?

I spent a lot of time the last few months trying to build a network of skills.  I don’t know any new things I’ve learned.  Ruby.  Ok, I learned one thing.  The rest was poking around the Internet for reliable sources for when I get somewhere and have responsibilities for all sorts of things.  I think one of the biggest challenges is staying on top of business and the advancements in the world around you.

Enter social media.  Facebook, Twitter, blogs, IRC, the various flavors of IM.

Oh, remember that post about things to know if you want to be awesome… I mean, a Systems Administrator.  This is part 2.

So, get out there and find people and don’t bother them with silly questions, or at least not too often.

In the last week I have witnessed a few people say really silly things to people who run or contribute often to open-source projects.  That is a huge mistake because everyone laughs at you.  Maybe not to your face but you’ve made yourself seem pretty naive.  That’s completely acceptable because you are probably still young in your career but if you spent some time just following conversations, you could have avoided that.

Anyway, open source projects are cool.  You can pick one up and learn some neat languages and get your name out in the community (resume builder!).  But get to know the people in the project and know their systems before getting too involved.

Programming languages.  Good at them?  Awesome.  I’m not.  I learn logic and then rely on Google.  If I was a full-time programmer, I would probably be worried about this.  The nice part about writing a bunch of different scripts is that I have a reference from those for later works.  Bad side?  I’m not fluent in any particular language.

I understand how a program runs and, usually, why it’s designed a certain way.  If I don’t understand it, I can, again usually, figure it out.  So what happens when I’m standing on my own two feet and have to start these things from scratch or it’s expected that I can weigh in on these things?  I don’t know.  I’ll have to stare at some code and design decisions before then, I guess.

That’s the great part about being on an internship in a place like STM.  There are a bunch of really intelligent people who write clean, well-designed (I hope) code and can explain and backup their decisions.  And when they can’t, they can discuss with others how to go about a certain issue.

So, anyone looking for a Systems Administrator?

Before I start, I will apologize ahead of time for any complete lapses in thought or jumps.  I’m watching the first Red Sox game of the season and it’s a little exciting.  If you don’t understand baseball, I’m sorry.

Today is Easter Sunday.  It is the end of Lent and the day Jesus rose from the dead to ascend into heaven.

The story goes something like this…

Jesus is betrayed on a Thursday night and killed on Friday.  Sunday some people decide to bring some cloth, scents and stuff to his grave site and is shocked to find the body gone.  Mary, arriving first, runs to tell the others and then runs back and finds who she believes to be a gardener and asks where “they” have taken the body.

Today in the sermon, the minister discussed who “they” refers to.  He claims that Mary believes the grave has been robbed.

This idea of grave robbers was actually shocking to me.  I figured these followers of Jesus found an unused tomb and they would most fear government officials or soldiers who came to either take the body away.

If you think about it, this makes sense.  Pontius Pilate and King Herod (??) put Jesus in front of the people asking them who they wanted released, Jesus or Barrabus (a murder, I think).  The people yell out they want Jesus crucified and Barrabus to be released.  Pilate asks the people twice, just to make sure!  He knows that something isn’t right.  After all is said and done, if the officials in the government know the mistake they made, wouldn’t they want to make it right and give Jesus a formal burial?

And on the flipside, if soldiers took the body of Jesus and they treated him anything like they did while he was in captivity, it would be disgrace to the max.  They whipped, beat, and further humiliated Jesus before he was hung on the cross to die.

In either case, I would expect Mary to fear either of those options before grave robbers.  Then again, I haven’t studied the period and I have no real knowledge about grave robbing.

But at the moment, the Red Sox are losing and it’s time to start “cheerin more louda!”  (Thank you Thomas Menino)