Securing Hatchbox servers with Cloudflare

I recently moved my Jammed Rails app to Hatchbox, and wanted to secure the server with Cloudflare. Hatchbox is excellent for Rails, and has native support for Wildcard SSL certs issued from the Cloudflare API, so it’s an exceedingly happy union. What Hatchbox does not setup for you for Cloudflare however is the stronger security settings, so I thought I’d write up what I did to get it working.

Cloudflare security settings

I pay for a Pro account on Cloudflare, so I have access to the WAF and other security settings. I wanted to enable the following:

• Always use HTTPS
• HTTP Strict Transport Security (HSTS)
• Opportunistic Encryption
• Web Application Firewall (WAF)

The first three are easy to setup, and are available on all plans. The WAF is only available on the Pro plan and above, and is a bit more involved to setup. Before moving behind Cloudflare, I used to see many automated requests to my server for Wordpress and other CMS admin pages, which I assume were bots trying to find a way in. I wanted to block these requests, and the WAF is a good way to do this.

SSH keys

You want to make sure your server is as secure as possible from SSH attacks, and one way to do this is to disable password authentication and only allow SSH access via SSH keys. By default Hatchbox manages the SSH keys on the server cluster and disables password authentication. This means that only you can access the server via SSH, and all other requests are blocked.

ed25519 keys

I use the more mordern ed25519 keys for SSH access, as they are more secure than RSA keys. You can generate an ed25519 keys on the Terminal, or if you use 1Password, you can generate them from there. I use 1Password to store my secrets, and then add the public keys to the server via the Hatchbox UI.

## Firewall rules

The single most important thing to secure my origin servers was to block all traffic to the server, except for when it’s through the Cloudflare network. This means that only valid and authenticated requests from Cloudflare can access the server, and all other traffic is blocked. You do this by adding firewall rules, which blocks all traffic except from Cloudflare IP addresses, your own IP address, and any other IP addresses you want to allow.

You can find a list of Cloudflare IP addresses here, I haven’t listed them here as they might become outdated in the future. I added these as accepts to the server firewall, and then added a rule to block all other traffic. This means that only Cloudflare can access the server, and all other traffic is blocked.

Allow these IPs via HTTPS only, to prevent block all other traffic.

Permitting SSH access

Alongside the Cloudflare IP addresses, Hatchbox needs to be able to access the server via SSH. You can find the IP addresses for Hatchbox here. I added these to the firewall rules, and allowed them to access the server via SSH. I also added my own home IP address to the firewall rules, so I can access the server via SSH. This means that only Cloudflare, Hatchbox and my own IP address can access the server via SSH.

Bastion server

A bastion server is a server that acts as a gateway to other servers within the same network behind the firewall.

I have a bastion server setup on Digital Ocean, which I use to access my other servers, whcih means that all SSH traffic can be routed through this single node, and SSH access does not need to be given to all the other servers. I have a single SSH key on the bastion server, which I use to access all my other servers.

Hatchbox also allows you to deploy via a bastion server, which means that you can deploy to your servers without giving them SSH access.

## Cloudflare WAF

Within Jammed, there are superadmin and special pages that should only be accessed by me. I wanted to block all other requests to these pages, and the WAF is a good way to do this. I setup a WAF rule to block all requests to these pages, and then added an exception for my own IP address. This means that only I can access these pages, and all other requests are blocked.

IP rotation

After you’ve set this all up, assuming that you used to have IP address listed on the domain DNS, you’ll want to rotate the public IP addresses. The old IP address will still be accessible (even if they are behind a firewall), and DNS records are cached and can take a while to update. Within Digital Ocean, you can generate a new floating IP address, and then assign this to the server. This means that the old IP address is no longer accessible, and all traffic is routed through the new IP address. Then you can update the DNS records to point to the new IP address.

Because of how Cloudflare works, your origin server IP address is not exposed, and all traffic is routed through the Cloudflare network.

After the IP rotation, these new IP addresses have never been public, and are only accessible via Cloudflare, so you can be sure that only Cloudflare can access your server or even knows the IP address.

## Conclusion

I hope this helps anyone else who is looking to secure their Hatchbox servers with Cloudflare. I’m really happy with the setup, and it’s given me peace of mind that my servers are as secure as they can be from the outside world.