Git for System Administrators
Hello and Happy Thanksgiving, this is the belated fourth installment of my blog where I will go over a slightly unconventional, but incredibly powerful use for git the distributed version control system. Git was built by Linus Torvalds and friends to help with Linux kernel development and has since been used by millions of developers to help track changes to source code, but at its core it tracks changes to text files not just source code. This is one way that git has helped me and I thought I would share.
Most System Administrators would like to know when and why configuration changes were made to their servers, and with a little setup git can help here.
Starting out Simply
Let's take a simple example, you just wish to track changes to your local system configuration. I will assume you are running Linux, but if you're running Windows there are some ideas you will be able to walk away with.
So here is the basic idea, we will set up a local git repository to track
changes to the files in our
/etc directory. We could easily use a remote
repository as well and if this was a production system I would definitely
recommend doing so, but for simple illustrative purposes we can skip that
part. My next blog post will probably be about using docker along with a
remote git repository for configuration management, but for now we will
just look at a few of the features git has which will be incredibly useful
in debugging malfunctioning systems and for simply keeping track of who
changed what when.
Setting Up the Local Git Repository
In order to keep things simple we will set up the local repository in our
/etc directory itself. We could set it up somewhere else and had a tool
like rsync update our repo on changes, but unless there's some reason you
would like to keep things separate it will be a lot easier to keep track
of things right in the
NOTE: Since we are working in
/etc permissions will become an issue,
so you can either run all of the commands with
sudo (the easy option) or
change permissions on files paying special attention to security concerns
and what git needs to do it's job. I would recommend using sudo as this will
use the same sets of permissions to keep track of changes as implementing
$ cd /etc $ git init
At this point we have an empty repository, but don't worry it didn't remove or change any of your files it's just not tracking them yet. This is an opportune time to specify any files you would like to ignore. If you ignore them now then they won't be in your repository at all, otherwise it can be a little frustrating to remove all copies of the file from your repository (definitely not impossible or even really all that difficult). But doing it now will save some frustration.
Only you will know what you want to ignore, but the process is fairly simple:
- Find files you would like to ignore
- Create a file in the root of your repository called
.gitignorein our case it will be
- Add glob patterns
matching files you would like to ignore to the
That's it, so for instance if you'd like to ignore anything in
you could issue the following command:
$ echo kernel/ >> .gitignore
or if you wanted to ignore
/etc/bashrc you could issue the following command:
$ echo bashrc >> .gitignore
well you get the idea.
Once you have all the files you wish to ignore set up in the
we can add all other files to our local git repository with the following
commands (please note that everything is meant to be executed in the root
of our repository, or
/etc/ in our case, as git works off of our
present working directory or
$ git add * $ git commit -m "Initial Commit"
The first command will add all changes (Our files are considered changes at this
point because git isn't tracking them yet) to be
staged. Next the
git commit command actually commits the
staged changes to the local
repository. That's it! Now git will be able to tell you if anything changes
from here on out and you can use it to track the whos, whens, whats and whys
of all future changes.
Tracking Some Changes
So, we are going to make some changes to one of our configuration files and track them using git.
First thing to do is check to see if anything has changed and that can be done with the following command:
$ git status
If there are no changes, the output will look like this:
On branch master nothing to commit, working directory clean
otherwise it will list new, modified and deleted files which are not tracked
in their current state. So let's change something and see how this works. On
Fedora 23, the system I'm using, you can disable root login from sshd by
changing the following line in
This is a fairly common change to make for security reasons, so after making
that change (I used
vim, but you can use your favorite text
editor or a carefully crafted
sed command), so let's see how git sees this
$ git status On branch master Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git checkout -- <file>..." to discard changes in working directory) modified: ssh/sshd_config no changes added to commit (use "git add" and/or "git commit -a")
So we have one modified file, let's see what changed:
$ git diff diff --git a/ssh/sshd_config b/ssh/sshd_config index 1861bde..e432c32 100644 --- a/ssh/sshd_config +++ b/ssh/sshd_config @@ -46,7 +46,7 @@ SyslogFacility AUTHPRIV # Authentication: #LoginGraceTime 2m -PermitRootLogin yes +PermitRootLogin no #StrictModes yes #MaxAuthTries 6 #MaxSessions 10
So, it has us removing the one line
PermitRootLogin yes and adding the line
PermitRootLogin no, this is how git sees the changes, line by line not
character by character, this scheme makes reading the reports a little easier.
Next we need to stage and commit the change so it is officially recorded.
We can do that with the following commands:
$ git add ssh/sshd_config $ git commit -m "Disallow root login via ssh"
and now, git knows that this change is officially
Now that we have history of changes (although small) we can review the change history with the following command:
$ git log commit cc46d88f1e193472c748187c629ddb8c6640424e Author: root <firstname.lastname@example.org> Date: Fri Nov 27 11:13:35 2015 -0500 Disallow ssh root login commit 50120a579dcf2ebc16e5ac399e380656ca6e5224 Author: root <email@example.com> Date: Fri Nov 27 10:32:39 2015 -0500 Initial commit
You can note the date and time is recorded along with the author (which is root
since I am performing these commands with
sudo) and the message which we
passed in with the
-m option to
git commit. Very informative commit
messages will go a long way when tracking down a mis-configuration or bug
in one or more of your systems.
We can also look at the differences between commits, simply add the to and from
commit ids to the
git diff command like so:
$ git diff 50120a579dcf2ebc16e5ac399e380656ca6e5224 cc46d88f1e193472c748187c629ddb8c6640424e diff --git a/ssh/sshd_config b/ssh/sshd_config index 1861bde..e432c32 100644 --- a/ssh/sshd_config +++ b/ssh/sshd_config @@ -46,7 +46,7 @@ SyslogFacility AUTHPRIV # Authentication: #LoginGraceTime 2m -PermitRootLogin yes +PermitRootLogin no #StrictModes yes #MaxAuthTries 6 #MaxSessions 10
yeah that's the same differences as before, but that's because we're looking at the same change, this command can be incredibly useful when looking at older changes.
Wrapping it Up
See you soon.