MAKE my day

DISCLAIMER: I’m not good at this yet.

I wrote a lengthy setup setup script in bash that I use to deploy my dotfiles whenever I move them to a new machine.

I keep the whole thing in a git repository so it’s as easy as

git clone git@repo:some-repo

It’s been working for me for a few years but I was always dissatisfied with the way the script handled certain things or needed to be manually massaged when something wasn’t quite right.

So today, I decided to finally try my hand at replacing my script with a MAKEFILE.  Make is available everywhere and is probably already installed on your system. It’s well documented and efficient. And it already does all the things my script was doing. Why re-invent the wheel?

An example: my vimrc

I created a github repo recently for my vimrc and related files.
It’s pretty simple – a .vimrc file, a .vim directory and the makefile itself. I just have to clone the repository, symlink a couple files and update the plugins.

Not really big enough to merit a setup script or makefile – but what the hell – I used it as my first experiment.

This is it:

vimrepo: softlinks 
    @echo "-----------------------------------------------------------"
    @echo "Vim and environment"
    @echo "-----------------------------------------------------------"
    @echo "Open vim and type :PlugInstall to install extension manager"

softlinks: apps 
    @test -L ${HOME}/.vim && unlink ${HOME}/.vim || true
    @(test -d ${HOME}/.vim && mv ${HOME}/.vim ${HOME}/.vim_old) || true
    @ln -vsf ${PWD}/.vim ${HOME}/.vim
    @(test -L ${HOME}/.vimrc && unlink ${HOME}/.vimrc) || true
    @(test -f ${HOME}/.vimrc && mv ${HOME}/.vimrc ${HOME}/.vimrc_old) || true
    @ln -vsf ${PWD}/.vimrc ${HOME}/.vimrc

    @sudo apt install vim

When I type make vimrepo the make utility starts at the vimrepo label which tells it that items under softlinks are dependencies. It then sees that items under appsare dependencies of that! It finally gets to the apps label where it installs vim (or just returns a message that vim is already installed).

In the softlinks label, make checks and removes or renames existing softlinks or files/directories and softlinks the .vimrc file and the .vim directory from the repository to my $HOME directory.

Finally, the vimrepo label just echoes some messages out to the screen telling what happened. In this case – what to do next.

As I said – it’s a simple example, but it works for me so far. In my short time googling for answers I’ve found a lot of options for decision making and portability in the Makefile syntax, none of which I’ve touched here.

Make is a powerful utility used by programmers to automate building of complex applications with multitudes of options. But it’s also a clean, easy-to-understand method of doing smaller tasks more efficiently.

Let me know if my makefile is blatantly wrong. I’m looking forward to replacing my old setup script with makefiles in other more complex projects!

Using DUO on a Raspberry Pi

Duo can be easily added to any Unix system to protect remote (SSH) or local logins with the addition of a simple pam_duo PAM module. It has been tested on Linux (RedHat, Fedora, CentOS, Debian, Ubuntu, Amazon Linux), BSD (FreeBSD, NetBSD, OpenBSD), Solaris, HP-UX, and AIX. The code is open-source and available on GitHub.

We use DUO at work for a variety of reasons and for whatever reason I never really looked into it until I was looking for an authentication method for Shiny Server. Well I still haven’t proven it’ll work with Shiny Server, but I did produce a proof-of-concept on a Raspberry Pi that acts as the jump box for my home network.

Here’s how:

Sign up for a DUO security account

The web page says that there’s a free tier for up to 10 users, but the sign-up page only allows you to sign up for the Free Trial. So I signed up for the free-trial with every intention of using the free tier.

I pretty much followed the guide DUO has here.

You use your smartphone to create the admin account and gain access to the dashboard, where you…

Set up a new application

They have a LOT of pre-made configurations for a number of different services, but I needed to protect SSH on a Raspberry Pi, which isn’t on the list. So I chose a generic “UNIX Application”.

Once on the Application screen you are presented with an Integration key, Secret key, and API hostname. Make note of these – you’ll need them later.

Take this time to change the name of your application to something meaningful. I chose <hostname> SSH/sudo.

Dependencies and Install

Make sure you’ve got the dependencies installed (I used the guide for Debian 7)

sudo apt install libssl-dev libpam-dev

Then install the DUO app from source (no Raspberry Pi repository)

tar zxf duo_unix-latest.tar.gz
cd duo_unix-1.10.1

Build and install duo_unix with PAM support ( pam_duo).

sudo ./configure --with-pam --prefix=/usr
sudo make
sudo make install

Update DUO configs with your account info

Once duo_unix is installed, edit /etc/duo/pam_duo.conf to add the integration key, secret key, and API hostname from your Duo Unix application.

; Duo integration key
; Duo secret key
; Duo API hostname

Update SSH configs

Add or ensure the following lines are present in /etc/ssh/sshd_config

PubkeyAuthentication yes
PasswordAuthentication no
AuthenticationMethods publickey,keyboard-interactive
UsePAM yes
ChallengeResponseAuthentication yes
UseDNS no

If you’re SSH’d into the box you are doing this on, then open another SSH session now as a safeguard in case something goes wrong and you lock yourself out.

Then restart ssh

service ssh restart

Update files in /etc/pam.d

SSH Public Key Authentication – /etc/pam.d/sshd


@include common-auth


#@include common-auth
auth  [success=1 default=ignore]
auth  requisite
auth  required

System-wide Authentication – /etc/pam.d/common-auth


auth  [success=1 default=ignore] nullok_secure
auth  requisite
auth  required


#auth  [success=1 default=ignore] nullok_secure
auth  requisite nullok_secure
auth  [success=1 default=ignore]
auth  requisite
auth  required


At this point you should be able to SSH into the box (try it from localhost first) and be prompted to enroll in DUO Security (you only have to do this once per account/device). Finish the instructions at the web link provided and you should be good to go!

How to install webDAV under Apache on Ubuntu (16.04)

HOW-TO DISCLAIMER: Most of the “how-to” posts are copy/pasted from a personal notes blog I keep for myself and as such sometimes presume familiarity with some of the concepts that might not be obvious. I’d like to make my notes and how-to’s more informative and accessible to a wider audience, so please let me know if something is not clear (or just plain wrong).

Enabling modules

The first thing you must do is enable the necessary modules. Open a terminal window and issue the following commands:

sudo a2enmod dav
sudo a2enmod dav_fs

Restart the Apache server with this command:

sudo systemctl restart apache2

Virtual host configuration

The next step is to create everything necessary for the virtual host. This will require creating specific directories (with specific permissions) and editing an Apache .conf file.

Let’s create the necessary directories. For simplicity, I’ll create a folder called webdav. From the terminal window, issue this command:

sudo mkdir -p /var/www/html/dav

Now we’ll change the owner of that directory to www-data with this command:

sudo chown www-data /var/www/html/dav

The next step is to create a .conf file that will help make apache aware of the virtual host. For this, we’ll create a new .conf file called /etc/apache2/sites-available/webdav.conf. The contents of this file will be:

<VirtualHost *:80>

DocumentRoot /var/www/html/dav/

<Directory /var/www/html/dav/>
    Options Indexes MultiViews
    AllowOverride None
    Order allow,deny
    allow from all
    DAV On
    AuthType Basic
    AuthName "webdav"
    AuthUserFile /var/www/html/dav/passwd.dav
    Require valid-user


Save and close the file.

Now we enable the webdav.conf file with:

sudo a2ensite webdav.conf

Before restarting apache, we need to create the WebDAV password file with this command (USER is a valid username on your system):

sudo htpasswd -c /var/www/html/dav/passwd.dav <user>

NOTE: if you need to update or recreate the users webDAV password, run the same command with out the -c option. (-c creates a new entry)

When prompted enter the password for USER.

Next we must change the permissions of the newly created passwd.dav file so that only root and members of the www-data group have access to it. You’ll do so with the following commands:

sudo chown root:www-data /var/www/html/dav/passwd.dav
sudo chmod 640 /var/www/html/dav/passwd.dav

Restart apache with this command:

sudo systemctl restart apache2

The WebDAV system is ready to test.

Testing your setup

There’s an easy to use tool for testing WebDAV—install the tool with this command:

sudo apt-get install cadaver

Once installed, issue this command (IP_OF_SERVER is the actual IP address of your server):

cadaver http://<server-name>

You should be prompted for a username/password. Enter the USER used when setting up WebDAV and the associated password. If the cadaver command succeeds, you’ll land at the dav:/webdav/> prompt. Type exit to exit cadaver.

Congratulations, WebDAV is working on your Ubuntu server. You can now use whatever tool you need (connect via file managers, web browsers, etc.) to connect to your WebDAV server.


Cherry Music Server on Ubuntu

I have a ton of ripped music. After setting up and getting used to MPD, today I discovered CherryMusic Server

This is what I was looking or originally. I’m sure I’ll use MPD for what it’s good at – but this is a better interface than Plex for listening to streaming music from home (so far).

Get CherryMusic

CherryMusic depends on Python. Although it also runs with Python 2, Python 3 is recommended for best performance and all features.

sudo apt-get install python3

CherryMusic has several optional dependencies, which should be installed for a seamless user experience:

sudo apt-get install mpg123 faad vorbis-tools flac imagemagick lame python3-unidecode

Optionally, you can replace the packages mpg123, faad, vorbis-tools, flac and lame with ffmpeg if you like. The advantage with ffmpeg is that you can also decode WMA files. If you are not running a headless server, consider installing “python3-gi”, which allows you to use CherryMusic’s GTK system tray icon.

Configuration and setup

For security reasons it it highly recommended to run CherryMusic under a dedicated Linux user. First, create the user “cherrymusic”:

sudo adduser cherrymusic

Now, switch to the newly created user:

su cherrymusic

There are two branches of CherryMusic: the stable main release (“master”) and the development version, called “devel”. I highly recommend the development branch, as it often is several steps ahead of the master release and provides all the new features. In this guide I also chose the devel branch. However, if you insist on using the master release, simply replace all occurrences of devel with master.

Now, get CherryMusic:

git clone --branch devel git:// ~/cherrymusic-devel

This command will download the develop branch of CherryMusic and place it in your home directory.

Due to a shortcoming in Debian, the repositories do not provide a recent version of the package cherrypy and the package stagger is not available in the Debian repositories at all. However, they can be fetched locally and simply put into the CherryMusic directory. CherryMusic has a build-in function, that checks if those two packages are available on the operating system and if necessary offers to automatically download and store them locally in the CherryMusic directory — without installing them on your system. This provides a clean way to get CherryMusic running on Debian. Simply change to the CherryMusic directory and start the server application with the –help switch (you will be prompted then):

cd cherrymusic-devel
python3 ./cherrymusic --help

Now, do the initial start-up to generate the configuration and data files in your home directory:

python3 ./cherrymusic

This creates the configuration file ~/.config/cherrymusic/cherrymusic.conf and the directory ~/.local/share/cherrymusic/, where the user data is stored.

Before you head on, edit the configuration file to point to your music library and make any other changes.

CherryMusic uses a database to search and access files in your music collection. Before you can use CherryMusic, you need to do an initial file database update:

python3 ./cherrymusic --update

To reflect changes in your music collection, you need to repeat this step every time you make changes to your music collection. On a standard computer, even very large music collections should not take longer than a few minutes.

Create a systemd service

CherryMusic doesn’t have a daemon, but we can fake it with a systemd script to start the service

Make sure you are logged in as someone with root or sudo rights (exit the cherrymusic user account if you’re still connected) and create the file /etc/systemd/system/cherrymusic@.service with the following contents:

Description=CherryMusic server



Enable and start CherryMusic

Enable the systemd service to start on each boot:

sudo systemctl enable cherrymusic@cherrymusic

Start the service

sudo systemctl start cherrymusic@cherrymusic

Finishing up

Open a web browser on a computer connected to the same LAN the CherryMusic server is in and go to http://<ip>:<port>, where ip is the IP of the server and port the port specified in the CherryMusic configuration file (defaults to “8080”).

Create an admin user and the basic setup is done.


Good news! The above instructions seem to work just fine on Ubuntu’s latest LTS


Kill unresponsive SSH

Let’s say you’re happily typing in your remote shell when all of sudden, the crappy WiFi network you’re using goes down. You end up with a perfectly unusable frozen shell, ugh!

To break out, assuming you’ve lost connectivity, press, in this order:


That is, <Enter key> then tilde (<shift>+`) then period

This will send an escape sequence to your local SSH client and terminate the connection. So this will always work even without a network connection.

Here’s the list of escape sequences:

Supported escape sequences:

~. - terminate connection (and any multiplexed sessions)
~B - send a BREAK to the remote system
~C - open a command line
~R - Request rekey (SSH protocol 2 only)
~^Z - suspend ssh
~# - list forwarded connections
~& - background ssh (when waiting for connections to terminate)
~? - this message
~~ - send the escape character by typing it twice

(Note that escapes are only recognized immediately after newline.)

Do It Out In The Open

I attended my first Ohio Linuxfest this year (2017) and had a great time. I found the lightning talks especially inspirational.

One of the talks I especially like were the lightning talks. Lightning talks are a series of short (5 minute), impromptu talks that anyone can do. They hand you a microphone and you say what you want to say.

There were a lot of Linux advocacy speeches (as you’d expect) and “How Linux has affected me” type talks.

One of the talks that hit home with me was a gentlemen who talked about how when he attended his first Linuxfest, Linux was not his career. Fast-forward a few years and he is back to say that he now has a Linux job that he secured by being active in the community and meeting people and learning on the job.

He stressed that we continue to do what we do, but to do that thing out in the open where you can be seen by others.

Take part in communities or user groups. Attend conventions or meetings. Get yourself known as someone who does your thing. Eventually, you may be asked to do more of that thing. Sometimes, someone will offer to pay you for it!

In that light, I have decided to try to drag my code, documentation and blog posts out into the public. I had moved several of my projects from my self-hosted Gitlab instance to my public Github. I have started this jekyll blog to publish opinion pieces like this and how-to’s I’ve written to help myself do things. I am trying to connect with people on social media. Above all I am sharing the thing I love to do, in the hopes that it may inspire someone else to do the same.

Installing Fiche as a Service

I started playing with fiche as a way to keep a personal pastebin. I like it so much I set it up as a permanent service on my Ubuntu 16.04 box. It’s pretty simple to setup, but there were enough bumps that I wanted to make sure I wrote up a howto in case I want to do this again later.

Get fiche

Get fiche from the Github page

git clone

Build and install fiche

There are instructions on the web site but I had issues. Once I got the repo cloned and changed to it’s directory, the Github page says to build (make) and install (sudo make install).
make went off without a hitch


I, however kept getting an error trying to sudo make install:

[master √] :$ sudo make install
install -m 0755 fiche
install: missing destination file operand after 'fiche'
Try 'install --help' for more information.
Makefile:12: recipe for target 'install' failed
make: *** [install] Error 1

After looking at the Makefile, it seems like all sudo make install does is copy the executable to an unspecified location

After I tried specifying a location, I just gave up and copied the file myself to /usr/local/bin

sudo cp fiche /usr/local/bin/

That seemed to do the trick.

Configuring Apache

I want my fiche snippets to be accessible via the web. Fiche stores each snippet as an index.txt file inside a randomly generated directory name. So I needed to configure fiche to send files to a directory in /var/www/html/ and make Apache serve index.txt files along with the usual suspects (index.htm, index.html…etc)

I decided to make this a subdirectory of my RootDocument rather than a virtual directory, but you can handle it either way.

First of all I made a new directory for my snippets and set the ownership to apache (www-data on Ubuntu, apache on RHEL/CentOS):

sudo mkdir /var/www/html/fiche
sudo chown www-data:www-data /var/www/html/fiche

Then I modified the dir module to accept index.txt as a valid index file:


sudo vim /etc/apache2/mods-available/dir.conf


sudo vim /etc/httpd/conf/httpd.conf

Add index.txt to the DirectoryIndex stanza, so the file looks something like this:

<IfModule mod_dir.c>    DirectoryIndex index.txt index.html index.cgi index.php index.xhtml index.htm</IfModule>

Then restart apache

sudo apachectl graceful

Test fiche

Now to make sure the above worked I started fiche from the command line with the following options:


fiche -s6 -d -o /var/www/html/fiche/ -u www-data:www-data -D


fiche -s6 -d -o /var/www/html/fiche/ -u apache

The -s specifies the slug size (default 4)

The -d specifies the URL from which I want to serve fiche.

The -o specifies the location on the file system to store snippets

The -u specifies the user to run this process as. In this case it runs as the Apache user

The -D daemonizes the process so it runs in the background.

NOTE: add -S to the command line if you want to return an HTTPS URL

Then I sent a test snippet:

echo "This is a test" | nc 9999

Fiche returned a URL which I plugged into my browser and was pleased to see

This is a test

On the screen. Success!

Create a systemd service

Now I want fiche to run all the time without me having to remember to start it after reboots. Since I’m running this on an Ubuntu 16.04 box, I need to create a systemd service.

Create the file /etc/systemd/system/fiche.service

sudo vim /etc/systemd/system/fiche.service

With these contents


ExecStart=/usr/local/bin/fiche -S -d -o /var/www/html/fiche/ -u www-data


Save and quit.

Enable the new service:

sudo systemctl enable fiche

Then start the new service:

sudo systemctl start fiche

Test by sending another snippet

echo "This is a test" | nc 9999

Visit the URL that gets returned and (hopefully) you’re in business!

Cleaning up

If you use fiche a lot you are going to eventually have dozens (or hundreds? thousands?) of randomly named directories on your web server. Common sense says to clean things up after a time. I chose a cron job that purges snips older than 14 days:

# Clean up fiche pastebin at 2am daily
00    02    *    *    *    find /var/www/html/pastebin -type d -mtime +14 -exec rm -rf {} \;


Zoho and Postfix

I’ve recently migrated to zoho to host mail on a domain I own. I also have a couple VPSs that I’d like to send email from. The sensible thing would be to relay through my Zoho account, right? Well, it’s not that easy. This one took a while…


I am configuring this on a Fedora23 server, but the dependencies should be the same on any Linux system.

dnf install postfix postfix-pcre cyrus-sasl cyrus-sasl-lib cyrus-sasl-plain


First of all you need the Zoho email address you want to use when relaying emails through Zoho.

Let’s say that this email address is

It will have a password as well, say apppassword

When configuring postfix, you edit many files. Let’s see them one by one.


The file /etc/postfix/generic maps local users to email addresses.

If email is sent to a local user such root, the address will be replaced with the one you specify.

In my case I have a single line like:


After editing this file remember to hash the file by using the command:

postmap generic


The file /etc/postfix/password contains the passwords postfix has to use to connect to the smtp server.

It’s content will be something like:

You also need to hash this file

postmap password


The file /etc/postfix/tls_policy contains the policies to be used when sending encrypted emails by using the TLS protocol, the one I’m using in this case. Create this file if it doesn’t exist.

The file contains just this line: encrypt

By doing so we force the use of TLS every time we send an email.

And then hash the file

postmap tls_policy


This is the most important file in our case.

The file /etc/postfix/smtp_header_checks contains rules to be used to rewrite the headers of the emails about to be sent. Create this file too, if needed.

It rewrites the sender so that it always matches our Zoho account,

No more ‘Relaying disallowed’ errors!

Put this in the file, replacing your valid email address:

/^From:.*/ REPLACE From: <>

No need for postmap here.

This is the main configuration file postfix uses.

Replace yourhostname with the hostname of your server, the one where postfix is installed on and that is sending emails through Zoho.

# TLS parameters
smtp_tls_policy_maps = hash:/etc/postfix/tls_policy
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_header_checks = pcre:/etc/postfix/smtp_header_checks
myhostname = yourhostname
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mydestination = yourhostname,, localhost
relayhost =
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/password
smtp_sasl_security_options =
smtp_generic_maps = hash:/etc/postfix/generic

In the file /etc/postfix/ I uncommented this line:

smtps     inet  n       -       n       -       -       smtpd

Apply the changes

Reload postfix by typing

postfix reload

Or by restarting the service

systemctl restart postfix


Try sending and email from the command line:

echo "Test" | mail -s "Postfix Zoho Email test"


Hardening SSH

It’s about time I get a standard sshd_config file that doesn’t rely on sane defaults.

Disabling Password-based Authentication

… you should disable password-based authentication altogether. Make sure to only do this once you have verified that you can log in with an account that can escalate to root – or that you have an alternate way of getting back onto your machine. To lock down your server, edit your /etc/sshd/sshd_config and set the following options:

    Protocol 2
    PermitRootLogin without-password
    PubkeyAuthentication yes
    ChallengeResponseAuthentication no
    PasswordAuthentication no
    UsePAM yes

This sets a fairly strict set of defaults that should make most bots give up right after connecting. The options we used were:

Protocol Verify that only protocol version 2 is allowed. There’s no point in supporting the rather dated version 1 and you’re only opening yourself up to ye olde bugs of old. If this reads 1 or 1,2, change it to just 2.

PermitRootLogin The setting without-password is a bit of a misnomer. What that does is it enables root logins, but only if the mechanism to authenticate was not a password – i.e. it enables root logins, but only for public key authentication. This is good. Never set this to yes.

PubkeyAuthentication Make sure this is set to yes, otherwise you won’t be able to log in once you disable passwords.

ChallengeResponseAuthentication Set this to no to disable non-pubkey logins that could otherwise be handled through PAM.

PasswordAuthentication This is what we were here for: set this to no to disable tunneled clear text passwords.

UsePAM If your system has PAM set up, it’ll still be a good idea to keep this enabled even if you disabled password-based authentication. This is because PAM also provides session and account management, so set this to yes. All you need to do now is restart the SSH server, like so:

    /etc/init.d/ssh restart

Note that restarting SSH will not kill your active session, so you should verify that the new settings will actually let you log in before closing your current session – and revert and restart SSH again if they don’t.

For maximum effect, make sure that SSH is the only way to log on to your server. At the very least make really certain that you don’t have telnet enabled! There, now you’re all set and can be very confident that bots won’t be able to access your machines through SSH


Linux doesn’t ask “Are You Sure?”

Are you sure you want to ruin your entire day? (Y/N)

Preface: This happened a couple years back and has completely resolved since. It’s also one of those events that fellow sysadmins might relate to that make you go back and rethink (and rewrite) your backup scripts and policies involving production machines and how to not be stupid, in general.

Today is the day after Easter.

This past Good Friday was not… well… good.

I can blame a cavalier attitude toward my work on a Friday before a holiday weekend or I could blame the bar for keeping me there until midnight the night before or I could blame the webdev people for breaking the DEV web page and requiring me to try to restore it from backup, but in the end I only blame myself.

I have made the joke plenty of times that “Linux doesn’t ask ‘are you sure?’ You tell Linux to delete a file and BAM, that file is GONE! There’s no sissy recycle bin. There’s no ‘are you sure?’ prompt. If you want to shoot yourself in the foot – Linux will shoot you in the most efficient way possible.” I usually follow up that joke with: “That’s also why I have this really fun ulcer and aren’t supposed to take aspirin anymore.”

Well, Friday I wiped out two production databases from the production server.

Intending to delete var/www/html/web-sandbox from /localdepot I typed

rm -rf /var

instead of

rm -rf var

Instead of deleting the var directory tree in the current directory – that preceding slash (which was pure muscle memory) indicated my desire to delete var from the file system root.

The same /var that contains core OS files including ALL of the database instances, their backups, mail and a variety of other important ‘variable data’.

Years of being hyperaware of how long it takes for the cursor to come back after a command helped me catch and kill the command after about 5 seconds. But it was too late to save /var/lib/mysql, /var/empty and /var/spool. (How Linux decides the order in which to delete things still baffles me – it sure isn’t alphabetical)

Immediately afterwards my world turned black. That familiar feeling of adrenaline and bile rushing to my head while my heartbeat almost stops.

I did my best to assess the damage. I scrolled back through my history and wrote down what directories were affected. I briefly thought about how I could cover this up long enough to fix it without anyone knowing. I decided I could not.

I quickly told my boss what happened. Not in detail – but I told him it was my fault and what was affected.

He told his boss it was a ‘hard crash’ on the file system and that I was ‘working on restoring things’. (God, I love my boss)

I set about planning my recovery. I was shaking so bad I kept making typing errors. I double and triple-checked the status line of my screen to ensure I was on the correct server when I typed commands. I started copying things then I canceled them. I checked the backups to see which ones I could recover from and started them copying to the DR server in case I broke the production box beyond repair.

In short – I was a hot mess. And it was only 11:00 am.

By 5:45 pm I had MySQL restored (4 and a half hours of copying and an hour and a half to restore) and sent the email.

In the interim I compared the /var tree to the other RedHat box and started fixing things like my inability to SSH into the server (missing file in /var/empty) and the missing CRON jobs and scripts (also in /var/spool)

I started copying the DEV backup and went home on-time for dinner. By 9:00 pm I had the DEV restored as well.

Hours had gone by and I had heard no new complaints. I started googling ‘deleted /var’ to see what happened to other people when it happened. Most had no recourse but to reinstall. But most had also deleted the entire directory. I only lost a few sub-directories.

I spent this weekend rethinking my backups and re-writing cron scripts. I also took steps to bring the DR server up to date in case I have to roll everything over to it the next time I have to reboot the production box.

This weekend I slept little and thought about how I am the only one that knows and can fix these kinds of problems. (My boss asked if would be worth getting Redhat involved. I told him no, mostly because I KNEW what happened and it was my fuck up)

As of today – all is quiet. There have been no new complaints. Is everything OK?

I won’t know until the next reboot.

But I have a DR plan working and my backups are running. If I have to move this data off this box – I can do it. I also learned where my existing backups are lacking and have addressed those areas.

I hate learning lessons in this manner.

But Linux doesn’t ask ‘are you sure?’

I’ve seen message boards full of threads where people own up to their worst mistakes. I thought this might be a good contribution. I hope it’s seen as a cautionary tale to noobs and a “Thickheaded Thursday” type of story where I explain how I:

  • Screwed up
  • Owned up to my mistake
  • Took ownership of what I did
  • Figured out how to fix it
  • Shared it with others