MAKE my day

DISCLAIMER: I’m not good at this yet.


I wrote a lengthy setup setup script in bash that I use to deploy my dotfiles whenever I move them to a new machine.

I keep the whole thing in a git repository so it’s as easy as

git clone git@repo:some-repo
./setup.sh

It’s been working for me for a few years but I was always dissatisfied with the way the script handled certain things or needed to be manually massaged when something wasn’t quite right.

So today, I decided to finally try my hand at replacing my script with a MAKEFILE.  Make is available everywhere and is probably already installed on your system. It’s well documented and efficient. And it already does all the things my script was doing. Why re-invent the wheel?


An example: my vimrc

I created a github repo recently for my vimrc and related files.
It’s pretty simple – a .vimrc file, a .vim directory and the makefile itself. I just have to clone the repository, symlink a couple files and update the plugins.

Not really big enough to merit a setup script or makefile – but what the hell – I used it as my first experiment.

This is it:

vimrepo: softlinks 
    @echo "-----------------------------------------------------------"
    @echo "Vim and environment"
    @echo "-----------------------------------------------------------"
    @echo "Open vim and type :PlugInstall to install extension manager"

softlinks: apps 
    @test -L ${HOME}/.vim && unlink ${HOME}/.vim || true
    @(test -d ${HOME}/.vim && mv ${HOME}/.vim ${HOME}/.vim_old) || true
    @ln -vsf ${PWD}/.vim ${HOME}/.vim
    @(test -L ${HOME}/.vimrc && unlink ${HOME}/.vimrc) || true
    @(test -f ${HOME}/.vimrc && mv ${HOME}/.vimrc ${HOME}/.vimrc_old) || true
    @ln -vsf ${PWD}/.vimrc ${HOME}/.vimrc

apps:
    @sudo apt install vim

When I type make vimrepo the make utility starts at the vimrepo label which tells it that items under softlinks are dependencies. It then sees that items under appsare dependencies of that! It finally gets to the apps label where it installs vim (or just returns a message that vim is already installed).

In the softlinks label, make checks and removes or renames existing softlinks or files/directories and softlinks the .vimrc file and the .vim directory from the repository to my $HOME directory.

Finally, the vimrepo label just echoes some messages out to the screen telling what happened. In this case – what to do next.


As I said – it’s a simple example, but it works for me so far. In my short time googling for answers I’ve found a lot of options for decision making and portability in the Makefile syntax, none of which I’ve touched here.

Make is a powerful utility used by programmers to automate building of complex applications with multitudes of options. But it’s also a clean, easy-to-understand method of doing smaller tasks more efficiently.

Let me know if my makefile is blatantly wrong. I’m looking forward to replacing my old setup script with makefiles in other more complex projects!

Using DUO on a Raspberry Pi

Duo can be easily added to any Unix system to protect remote (SSH) or local logins with the addition of a simple pam_duo PAM module. It has been tested on Linux (RedHat, Fedora, CentOS, Debian, Ubuntu, Amazon Linux), BSD (FreeBSD, NetBSD, OpenBSD), Solaris, HP-UX, and AIX. The code is open-source and available on GitHub.

We use DUO at work for a variety of reasons and for whatever reason I never really looked into it until I was looking for an authentication method for Shiny Server. Well I still haven’t proven it’ll work with Shiny Server, but I did produce a proof-of-concept on a Raspberry Pi that acts as the jump box for my home network.

Here’s how:

Sign up for a DUO security account

The web page says that there’s a free tier for up to 10 users, but the sign-up page only allows you to sign up for the Free Trial. So I signed up for the free-trial with every intention of using the free tier.

I pretty much followed the guide DUO has here.

You use your smartphone to create the admin account and gain access to the dashboard, where you…

Set up a new application

They have a LOT of pre-made configurations for a number of different services, but I needed to protect SSH on a Raspberry Pi, which isn’t on the list. So I chose a generic “UNIX Application”.

Once on the Application screen you are presented with an Integration key, Secret key, and API hostname. Make note of these – you’ll need them later.

Take this time to change the name of your application to something meaningful. I chose <hostname> SSH/sudo.

Dependencies and Install

Make sure you’ve got the dependencies installed (I used the guide for Debian 7)

sudo apt install libssl-dev libpam-dev

Then install the DUO app from source (no Raspberry Pi repository)

wget https://dl.duosecurity.com/duo_unix-latest.tar.gz
tar zxf duo_unix-latest.tar.gz
cd duo_unix-1.10.1

Build and install duo_unix with PAM support ( pam_duo).

sudo ./configure --with-pam --prefix=/usr
sudo make
sudo make install

Update DUO configs with your account info

Once duo_unix is installed, edit /etc/duo/pam_duo.conf to add the integration key, secret key, and API hostname from your Duo Unix application.

[duo]
; Duo integration key
ikey = INTEGRATION_KEY
; Duo secret key
skey = SECRET_KEY
; Duo API hostname
host = API_HOSTNAME

Update SSH configs

Add or ensure the following lines are present in /etc/ssh/sshd_config

PubkeyAuthentication yes
PasswordAuthentication no
AuthenticationMethods publickey,keyboard-interactive
UsePAM yes
ChallengeResponseAuthentication yes
UseDNS no

If you’re SSH’d into the box you are doing this on, then open another SSH session now as a safeguard in case something goes wrong and you lock yourself out.

Then restart ssh

service ssh restart

Update files in /etc/pam.d

SSH Public Key Authentication – /etc/pam.d/sshd

Before:

@include common-auth

After:

#@include common-auth
auth  [success=1 default=ignore] pam_duo.so
auth  requisite pam_deny.so
auth  required pam_permit.so

System-wide Authentication – /etc/pam.d/common-auth

Before:

auth  [success=1 default=ignore] pam_unix.so nullok_secure
auth  requisite pam_deny.so
auth  required pam_permit.so

After:

#auth  [success=1 default=ignore] pam_unix.so nullok_secure
auth  requisite pam_unix.so nullok_secure
auth  [success=1 default=ignore] pam_duo.so
auth  requisite pam_deny.so
auth  required pam_permit.so

Test

At this point you should be able to SSH into the box (try it from localhost first) and be prompted to enroll in DUO Security (you only have to do this once per account/device). Finish the instructions at the web link provided and you should be good to go!

How to install webDAV under Apache on Ubuntu (16.04)

HOW-TO DISCLAIMER: Most of the “how-to” posts are copy/pasted from a personal notes blog I keep for myself and as such sometimes presume familiarity with some of the concepts that might not be obvious. I’d like to make my notes and how-to’s more informative and accessible to a wider audience, so please let me know if something is not clear (or just plain wrong).

Enabling modules

The first thing you must do is enable the necessary modules. Open a terminal window and issue the following commands:

sudo a2enmod dav
sudo a2enmod dav_fs

Restart the Apache server with this command:

sudo systemctl restart apache2

Virtual host configuration

The next step is to create everything necessary for the virtual host. This will require creating specific directories (with specific permissions) and editing an Apache .conf file.

Let’s create the necessary directories. For simplicity, I’ll create a folder called webdav. From the terminal window, issue this command:

sudo mkdir -p /var/www/html/dav

Now we’ll change the owner of that directory to www-data with this command:

sudo chown www-data /var/www/html/dav

The next step is to create a .conf file that will help make apache aware of the virtual host. For this, we’ll create a new .conf file called /etc/apache2/sites-available/webdav.conf. The contents of this file will be:

<VirtualHost *:80>

ServerAdmin chuck@planethawleywood.com
ServerName webdav.planethawleywood.com
DocumentRoot /var/www/html/dav/

<Directory /var/www/html/dav/>
    Options Indexes MultiViews
    AllowOverride None
    Order allow,deny
    allow from all
    DAV On
    AuthType Basic
    AuthName "webdav"
    AuthUserFile /var/www/html/dav/passwd.dav
    Require valid-user
</Directory>

</VirtualHost>

Save and close the file.

Now we enable the webdav.conf file with:

sudo a2ensite webdav.conf

Before restarting apache, we need to create the WebDAV password file with this command (USER is a valid username on your system):

sudo htpasswd -c /var/www/html/dav/passwd.dav <user>

NOTE: if you need to update or recreate the users webDAV password, run the same command with out the -c option. (-c creates a new entry)

When prompted enter the password for USER.

Next we must change the permissions of the newly created passwd.dav file so that only root and members of the www-data group have access to it. You’ll do so with the following commands:

sudo chown root:www-data /var/www/html/dav/passwd.dav
sudo chmod 640 /var/www/html/dav/passwd.dav

Restart apache with this command:

sudo systemctl restart apache2

The WebDAV system is ready to test.

Testing your setup

There’s an easy to use tool for testing WebDAV—install the tool with this command:

sudo apt-get install cadaver

Once installed, issue this command (IP_OF_SERVER is the actual IP address of your server):

cadaver http://<server-name>

You should be prompted for a username/password. Enter the USER used when setting up WebDAV and the associated password. If the cadaver command succeeds, you’ll land at the dav:/webdav/> prompt. Type exit to exit cadaver.

Congratulations, WebDAV is working on your Ubuntu server. You can now use whatever tool you need (connect via file managers, web browsers, etc.) to connect to your WebDAV server.


Reference

Cherry Music Server on Ubuntu

I have a ton of ripped music. After setting up and getting used to MPD, today I discovered CherryMusic Server

This is what I was looking or originally. I’m sure I’ll use MPD for what it’s good at – but this is a better interface than Plex for listening to streaming music from home (so far).

Get CherryMusic

CherryMusic depends on Python. Although it also runs with Python 2, Python 3 is recommended for best performance and all features.

sudo apt-get install python3

CherryMusic has several optional dependencies, which should be installed for a seamless user experience:

sudo apt-get install mpg123 faad vorbis-tools flac imagemagick lame python3-unidecode

Optionally, you can replace the packages mpg123, faad, vorbis-tools, flac and lame with ffmpeg if you like. The advantage with ffmpeg is that you can also decode WMA files. If you are not running a headless server, consider installing “python3-gi”, which allows you to use CherryMusic’s GTK system tray icon.

Configuration and setup

For security reasons it it highly recommended to run CherryMusic under a dedicated Linux user. First, create the user “cherrymusic”:

sudo adduser cherrymusic

Now, switch to the newly created user:

su cherrymusic

There are two branches of CherryMusic: the stable main release (“master”) and the development version, called “devel”. I highly recommend the development branch, as it often is several steps ahead of the master release and provides all the new features. In this guide I also chose the devel branch. However, if you insist on using the master release, simply replace all occurrences of devel with master.

Now, get CherryMusic:

git clone --branch devel git://github.com/devsnd/cherrymusic.git ~/cherrymusic-devel

This command will download the develop branch of CherryMusic and place it in your home directory.

Due to a shortcoming in Debian, the repositories do not provide a recent version of the package cherrypy and the package stagger is not available in the Debian repositories at all. However, they can be fetched locally and simply put into the CherryMusic directory. CherryMusic has a build-in function, that checks if those two packages are available on the operating system and if necessary offers to automatically download and store them locally in the CherryMusic directory — without installing them on your system. This provides a clean way to get CherryMusic running on Debian. Simply change to the CherryMusic directory and start the server application with the –help switch (you will be prompted then):

cd cherrymusic-devel
python3 ./cherrymusic --help

Now, do the initial start-up to generate the configuration and data files in your home directory:

python3 ./cherrymusic

This creates the configuration file ~/.config/cherrymusic/cherrymusic.conf and the directory ~/.local/share/cherrymusic/, where the user data is stored.

Before you head on, edit the configuration file to point to your music library and make any other changes.

CherryMusic uses a database to search and access files in your music collection. Before you can use CherryMusic, you need to do an initial file database update:

python3 ./cherrymusic --update

To reflect changes in your music collection, you need to repeat this step every time you make changes to your music collection. On a standard computer, even very large music collections should not take longer than a few minutes.

Create a systemd service

CherryMusic doesn’t have a daemon, but we can fake it with a systemd script to start the service

Make sure you are logged in as someone with root or sudo rights (exit the cherrymusic user account if you’re still connected) and create the file /etc/systemd/system/cherrymusic@.service with the following contents:

[Unit]
Description=CherryMusic server
Requires=network.target
After=network.target

[Service]
User=%I
Type=simple
ExecStart=/home/cherrymusic/cherrymusic-devel/cherrymusic
StandardOutput=null
PrivateTmp=true
Restart=always

[Install]
WantedBy=multi-user.target

Enable and start CherryMusic

Enable the systemd service to start on each boot:

sudo systemctl enable cherrymusic@cherrymusic

Start the service

sudo systemctl start cherrymusic@cherrymusic

Finishing up

Open a web browser on a computer connected to the same LAN the CherryMusic server is in and go to http://<ip>:<port>, where ip is the IP of the server and port the port specified in the CherryMusic configuration file (defaults to “8080”).

Create an admin user and the basic setup is done.

Postscript

Good news! The above instructions seem to work just fine on Ubuntu’s latest LTS


Reference

Kill unresponsive SSH

Let’s say you’re happily typing in your remote shell when all of sudden, the crappy WiFi network you’re using goes down. You end up with a perfectly unusable frozen shell, ugh!

To break out, assuming you’ve lost connectivity, press, in this order:

[Enter]
~
.

That is, <Enter key> then tilde (<shift>+`) then period

This will send an escape sequence to your local SSH client and terminate the connection. So this will always work even without a network connection.

Here’s the list of escape sequences:

Supported escape sequences:

~. - terminate connection (and any multiplexed sessions)
~B - send a BREAK to the remote system
~C - open a command line
~R - Request rekey (SSH protocol 2 only)
~^Z - suspend ssh
~# - list forwarded connections
~& - background ssh (when waiting for connections to terminate)
~? - this message
~~ - send the escape character by typing it twice

(Note that escapes are only recognized immediately after newline.)

Installing Fiche as a Service

I started playing with fiche as a way to keep a personal pastebin. I like it so much I set it up as a permanent service on my Ubuntu 16.04 box. It’s pretty simple to setup, but there were enough bumps that I wanted to make sure I wrote up a howto in case I want to do this again later.

Get fiche

Get fiche from the Github page

git clone https://github.com/solusipse/fiche

Build and install fiche

There are instructions on the web site but I had issues. Once I got the repo cloned and changed to it’s directory, the Github page says to build (make) and install (sudo make install).
make went off without a hitch

make

I, however kept getting an error trying to sudo make install:

[master √] :$ sudo make install
install -m 0755 fiche
install: missing destination file operand after 'fiche'
Try 'install --help' for more information.
Makefile:12: recipe for target 'install' failed
make: *** [install] Error 1

After looking at the Makefile, it seems like all sudo make install does is copy the executable to an unspecified location

After I tried specifying a location, I just gave up and copied the file myself to /usr/local/bin

sudo cp fiche /usr/local/bin/

That seemed to do the trick.

Configuring Apache

I want my fiche snippets to be accessible via the web. Fiche stores each snippet as an index.txt file inside a randomly generated directory name. So I needed to configure fiche to send files to a directory in /var/www/html/ and make Apache serve index.txt files along with the usual suspects (index.htm, index.html…etc)

I decided to make this a subdirectory of my RootDocument rather than a virtual directory, but you can handle it either way.

First of all I made a new directory for my snippets and set the ownership to apache (www-data on Ubuntu, apache on RHEL/CentOS):

sudo mkdir /var/www/html/fiche
sudo chown www-data:www-data /var/www/html/fiche

Then I modified the dir module to accept index.txt as a valid index file:

Ubuntu:

sudo vim /etc/apache2/mods-available/dir.conf

RHEL/CentOS:

sudo vim /etc/httpd/conf/httpd.conf

Add index.txt to the DirectoryIndex stanza, so the file looks something like this:

<IfModule mod_dir.c>    DirectoryIndex index.txt index.html index.cgi index.pl index.php index.xhtml index.htm</IfModule>

Then restart apache

sudo apachectl graceful

Test fiche

Now to make sure the above worked I started fiche from the command line with the following options:

Ubuntu:

fiche -s6 -d poe.planethawleywood.com/fiche -o /var/www/html/fiche/ -u www-data:www-data -D

RHEL/CentOS:

fiche -s6 -d poe.planethawleywood.com/fiche -o /var/www/html/fiche/ -u apache

The -s specifies the slug size (default 4)

The -d specifies the URL from which I want to serve fiche.

The -o specifies the location on the file system to store snippets

The -u specifies the user to run this process as. In this case it runs as the Apache user

The -D daemonizes the process so it runs in the background.

NOTE: add -S to the command line if you want to return an HTTPS URL

Then I sent a test snippet:

echo "This is a test" | nc poe.planethawleywood.com 9999

Fiche returned a URL which I plugged into my browser and was pleased to see

This is a test

On the screen. Success!

Create a systemd service

Now I want fiche to run all the time without me having to remember to start it after reboots. Since I’m running this on an Ubuntu 16.04 box, I need to create a systemd service.

Create the file /etc/systemd/system/fiche.service

sudo vim /etc/systemd/system/fiche.service

With these contents

[Unit]
Description=fiche-server

[Service]
ExecStart=/usr/local/bin/fiche -S -d poe.planethawleywood.com/fiche -o /var/www/html/fiche/ -u www-data

[Install]
WantedBy=multi-user.target

Save and quit.

Enable the new service:

sudo systemctl enable fiche

Then start the new service:

sudo systemctl start fiche

Test by sending another snippet

echo "This is a test" | nc poe.planethawleywood.com 9999

Visit the URL that gets returned and (hopefully) you’re in business!

Cleaning up

If you use fiche a lot you are going to eventually have dozens (or hundreds? thousands?) of randomly named directories on your web server. Common sense says to clean things up after a time. I chose a cron job that purges snips older than 14 days:

# Clean up fiche pastebin at 2am daily
00    02    *    *    *    find /var/www/html/pastebin -type d -mtime +14 -exec rm -rf {} \;

References

Zoho and Postfix

I’ve recently migrated to zoho to host mail on a domain I own. I also have a couple VPSs that I’d like to send email from. The sensible thing would be to relay through my Zoho account, right? Well, it’s not that easy. This one took a while…

Pre-requisites

I am configuring this on a Fedora23 server, but the dependencies should be the same on any Linux system.

dnf install postfix postfix-pcre cyrus-sasl cyrus-sasl-lib cyrus-sasl-plain

Configuration

First of all you need the Zoho email address you want to use when relaying emails through Zoho.

Let’s say that this email address is app@planethawleywood.com

It will have a password as well, say apppassword

When configuring postfix, you edit many files. Let’s see them one by one.

Generic

The file /etc/postfix/generic maps local users to email addresses.

If email is sent to a local user such root, the address will be replaced with the one you specify.

In my case I have a single line like:

root app@planethawleywood.com

After editing this file remember to hash the file by using the command:

postmap generic

Password

The file /etc/postfix/password contains the passwords postfix has to use to connect to the smtp server.

It’s content will be something like:

smtp.zoho.com:587 app@planethawleywood.com:apppassword

You also need to hash this file

postmap password

tls_policy

The file /etc/postfix/tls_policy contains the policies to be used when sending encrypted emails by using the TLS protocol, the one I’m using in this case. Create this file if it doesn’t exist.

The file contains just this line:

smtp.zoho.com:587 encrypt

By doing so we force the use of TLS every time we send an email.

And then hash the file

postmap tls_policy

smtp_header_checks

This is the most important file in our case.

The file /etc/postfix/smtp_header_checks contains rules to be used to rewrite the headers of the emails about to be sent. Create this file too, if needed.

It rewrites the sender so that it always matches our Zoho account, app@planethawleywood.com

No more ‘Relaying disallowed’ errors!

Put this in the file, replacing your valid email address:

/^From:.*/ REPLACE From: <app@planethawleywood.com>

No need for postmap here.

main.cf

This is the main configuration file postfix uses.

Replace yourhostname with the hostname of your server, the one where postfix is installed on and that is sending emails through Zoho.

# TLS parameters
smtp_tls_policy_maps = hash:/etc/postfix/tls_policy
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_header_checks = pcre:/etc/postfix/smtp_header_checks
 
myhostname = yourhostname
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mydestination = yourhostname, localhost.com, localhost
relayhost = smtp.zoho.com:587
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/password
smtp_sasl_security_options =
smtp_generic_maps = hash:/etc/postfix/generic

master.cf

In the file /etc/postfix/master.cf I uncommented this line:

smtps     inet  n       -       n       -       -       smtpd

Apply the changes

Reload postfix by typing

postfix reload

Or by restarting the service

systemctl restart postfix

Test

Try sending and email from the command line:

echo "Test" | mail -s "Postfix Zoho Email test" email@domain.com

References:

Hardening SSH

It’s about time I get a standard sshd_config file that doesn’t rely on sane defaults.


Disabling Password-based Authentication

… you should disable password-based authentication altogether. Make sure to only do this once you have verified that you can log in with an account that can escalate to root – or that you have an alternate way of getting back onto your machine. To lock down your server, edit your /etc/sshd/sshd_config and set the following options:

    Protocol 2
    PermitRootLogin without-password
    PubkeyAuthentication yes
    ChallengeResponseAuthentication no
    PasswordAuthentication no
    UsePAM yes

This sets a fairly strict set of defaults that should make most bots give up right after connecting. The options we used were:

Protocol Verify that only protocol version 2 is allowed. There’s no point in supporting the rather dated version 1 and you’re only opening yourself up to ye olde bugs of old. If this reads 1 or 1,2, change it to just 2.

PermitRootLogin The setting without-password is a bit of a misnomer. What that does is it enables root logins, but only if the mechanism to authenticate was not a password – i.e. it enables root logins, but only for public key authentication. This is good. Never set this to yes.

PubkeyAuthentication Make sure this is set to yes, otherwise you won’t be able to log in once you disable passwords.

ChallengeResponseAuthentication Set this to no to disable non-pubkey logins that could otherwise be handled through PAM.

PasswordAuthentication This is what we were here for: set this to no to disable tunneled clear text passwords.

UsePAM If your system has PAM set up, it’ll still be a good idea to keep this enabled even if you disabled password-based authentication. This is because PAM also provides session and account management, so set this to yes. All you need to do now is restart the SSH server, like so:

    /etc/init.d/ssh restart

Note that restarting SSH will not kill your active session, so you should verify that the new settings will actually let you log in before closing your current session – and revert and restart SSH again if they don’t.

For maximum effect, make sure that SSH is the only way to log on to your server. At the very least make really certain that you don’t have telnet enabled! There, now you’re all set and can be very confident that bots won’t be able to access your machines through SSH


Reference(s):