Let’s Encrypt intermediate certificate change

I checked one of the sites I use Let’s Encrypt free SSL certificates on SSL Labs’ SSL Test recently and found to my dismay that they only got a “B” grade while I was pretty sure I got an “A” grade before. Looking at the details it showed the intermediate certificate chain to be incomplete and the dreaded “extra download” warning.

When I set up Let’s Encrypt I used the simple acme-tiny tool and naturally also configured Apache to to use the generated certificate properly, including sending the intermediate certificate. Some searching suggested that it might be because the relevant directive was deprecated in Apache 2.4.8. But the affected sites (incl. this blog) run on a VPS with Ubuntu 14.04 which includes Apache 2.4.7, so that change was not relevant to me (it will be when you upgrade Apache to a later version, like when upgrading to Ubuntu 16.04!).

So after some experimenting I finally figured it out: when the Let’s Encrypt beta went live all the certificates were signed by the “Let’s Encrypt Authority X1”. However sometime in March they changed this and now all certificates are signed by “Let’s Encrypt Authority X3”. My server was sending the wrong intermediate certificate (X1 instead of X3). It’s one of those problems that once you find out what is wrong, is very easy to fix, just understanding what was wrong took some effort.

Links for Let’s Encrypt intermediate certificates can be found here. Acme-Tiny has updated their readme as well.

Enabling http/2 on your sites with Ubuntu and nginx

HTTP is the most important protocol on the web, but hasn’t had a major update in a long time (HTTP 1.1 was introduced in 1997). In May 2015 however the HTTP 2.0 (HTTP/2) standard was published. The new standard promises increased speed.

As of right now http/2 will only work on HTTPS sites. Although the HTTPS requirement isn’t in the standard (although it nearly did), the most popular browser implementations (Chrome, Firefox, etc) don’t support http/2 with anything but HTTPS.

Now to enable http/2 on your Ubuntu server, just run this command (as root, prefix with sudo when necessary):

add-apt-repository ppa:nginx/development

Then run apt-get update and apt-get install nginx as usual. This will install the latest mainline version of nginx (http/2 support is included as of version 1.9.5). Despite the name of the PPA repository this is actually the recommended version: “We recommend that in general you deploy the NGINX mainline branch at all times”.

Now there is a slight problem: if you use this method on the current (as of writing) version of Ubuntu LTS (14.04) then an important part of http/2 will not work: ALPN. This TLS extension is used to enable http/2 during the connection phase. Without it another roundtrip to the server is needed for browser and server to agree on using http/2.

To enable ALPN you need a version of nginx that is built with OpenSSL 1.0.2. This version of OpenSSL is included from Ubuntu version 15.10 onwards (including 16.04 LTS). When installing on Ubuntu 14.04 the nginx PPA repository assumes you have the default version of OpenSSL installed (v1.0.1), hence no ALPN support.

That leaves three options: upgrade to 15.10 (not an LTS release, so not preferable on servers), build the latest version of nginx and OpenSSL yourself (which creates an upgrade burden) or live with having slightly crappy http/2 support until Ubuntu 16.04 LTS is available.

Once enabled you can test http/2 here: https://tools.keycdn.com/http2-test

(http/2 is not currently available on this blog, but is enabled on The Emulator Zone Forum)

Getting snmpd to work through IPv6

I have Cacti installed so that I have pretty graphs (CPU/load/memory/disk/network) for all my servers (VPS) in one place. I just enabled full IPv6 on one of them and Cacti stopped working. The problem wasn’t Cacti though, it was the server it was now trying to contact through IPv6.

As I only need SNMP for Cacti, the snmpd.conf file on all my servers is very simple, just:

rocommunity public

However for IPv6 that isn’t enough. On Ubuntu 14.04 (what the server is running) snmpd will only work on IPv4 this way. I quickly found out you can use an agentAddress directive in the snmpd.conf file to enable the daemon to listen on IPv6, but more changes are required if you want to actually return data through IPv6. As it turns out the “rocommunity” directive is apparently also IPv4 only.

The final solution was this:

agentAddress udp:161,udp6:161
rocommunity public
rocommunity6 public

Warning: above configuration will cause snmpd to listen on all network interfaces. Be sure to lock down access to UDP port 161 in your firewall to only authorized hosts. And make sure your firewall supports IPv6.

Let’s Encrypt in Public Beta

Let’s Encrypt is now in public beta phase. It allows you to create free unlimited trusted SSL certificates that work in all browsers (all relevant browsers anyway). There are now no excuses for adding SSL encryption to your website(s).

That isn’t to say that Let’s Encrypt isn’t without flaws: the certificates you can get are only valid for a maximum of 90 days. While you can of course renew them, if you don’t have some kind of automated way of renewing your certificates it can quickly become a pain to keep them up-to-date.

The other issue (at the moment) is that Let’s Encrypt own monolithic tool for requesting certificates requires to be run as root on your server, which no doubt some people will have issues with. Fortunately the protocol (ACME) used is public and alternative clients are available. I personally used letsencrypt-nosudo to issue my first certificate (this tool allows you to register an account with an email address with Let’s Encrypt, which might be useful) and intent to issue further certificates (and renewals) with the very simple acme-tiny.

If you haven’t set up SSL on your server before, these resources might also be useful:

I intend to use Let’s Encrypt for all my less important domains (which includes this site) until its reliability is proven and CheapSSLSecurity (Comodo PositiveSSL 3 years for $4.99/year) for the rest in the mean time. No need to pay any more.

Mandrill support for vBulletin

Update February 2016: Mandrill has announced changes that if I understand them correctly would mean for my sending volume I’d go from paying $65 in the past two years to $30/month in the future. Needless to say, I’ll be moving away from them a.s.a.p. and do not recommend them anymore to anyone! Of course I am not the only one who is not happy about that. While I have yet to properly evaluate alternatives, Mailgun looks promising. I’ll update my plugins when I’ve moved to another service.

If you have a server that needs to send mail, I highly recommend using a specialized service such as SendGrid or my personal favorite Mandrill (up to 12k sends per month are free). While integrating Mandrill in certain web software is easy (like their wpMandrill plugin for WordPress), vBulletin is a bit more tricky.

While it is easy to configure vBulletin to send mail through Mandrill (just create an API key and configure mail in the vBulletin settings) a problem is that the vBulletin Contact Us form uses the user’s email address in the “From:” field and that pollutes your Mandrill account very quickly.

As solution I’ve written a little plugin that rewrites the “From:” field into a “Reply-To:” field and then adds your webmaster email address as “From:” field, which solves this issue.

Now on to one of the neat things you can do with Mandrill: automated handling of bounced mail. Mandrill supports webhooks where you can have certain events be reported to automatically, such as “hard-bounce” or “reject” email events. Such a webhook can then take action on this information.

I’ve written a Mandrill webhook for vBulletin that will automatically move a user to the “Users Awaiting Email Confirmation” usergroup if such an event occurs, so that no more mail is sent to them (until they update and reconfirm their email address), which will improve your email delivery score.

The plugin and webhook plus further info can be found here: https://bitbucket.org/ghdpro/vbulletin-mandrill

Tip: for greater security, limit your Mandrill API key access (in Mandrill settings) to your server’s IP address and for vBulletin (or any service that only uses SMTP access) you only need to allow the “send-raw” API call, nothing else.