Let's Encrypt, Ruby on Rails 5.2, Content Security Policy and Nginx

The story of how I spent more hours fighting with Content Security Policy configs than migrating Rails application to Digital Ocean.

TL;DR: Check your /etc/letsencrypt/options-ssl-nginx.conf file and make sure you don’t have CSP and other security headers conflicting with Rails headers in there.

Some time ago I was working on the Rails project running on AWS. The architecture of that application was pretty simple:

  • The monolith encapsulates all required business logic and served by Puma.
  • The deployment process is organized around AWS Elastic Beanstalk and its config files.
  • Amazon Elastic Load Balancer (ELB) sits in front of the application instances.
  • Any application instance1 includes Nginx that acts as a reverse proxy for Puma.
  • Amazon RDS for PostgreSQL as a primary database.
  • A tiny Redis cluster on Amazon ElastiCache and Sidekiq for processing background jobs.

Choosing the AWS platform, the team was hoping to scale infrastructure with minimum effort. And it went well, especially for such a small startup. What didn’t work out is a business model and the company eventually failed. Management took a decision to keep the product’s website running and migrate the infrastructure on the cheapest cloud – Digital Ocean.

The migration process wasn’t hard:

  • Create a droplet and do the initial setup (permissions, firewall, packages etc.)
  • Create a database and restore the dump.
  • Setup Redis for Sidekiq jobs.
  • Move all static files on the droplet.
  • Choose the deployment tool (Capistrano in my case) and deploy the application. Make sure Puma processes are running.
  • Setup Nginx and configure it as a reverse proxy for Puma.
  • Install Certbot and obtain an SSL certificate.
  • Restart Nginx.
  • Update DNS records.

And that’s it! It took me only a few hours to spin up a new environment. I’ve checked the website. At first glance, it was working well – I could sign up, sign in and do some stuff with the product. I thought, my work is done here. But all of a sudden, I’ve noticed missing fonts and some other static assets. That looked strange to me. I’ve opened Chrome DevTools and found tens of similar errors:

Content Security Policy (CSP) errors

🤔Hmmm…Then I’ve checked the response headers: Strict-Transport-Security, Content-Security-Policy, Referrer-Policy, X-Frame-Options, X-Content-Type-Options, X-XSS-Protection were duplicated all over the place with strictest rules which I didn’t set up. And that’s a reason why I’ve got all these CSP errors. What the hack?!

I knew that Rails 5.2 has introduced inbuilt DSL for configuring Content Security Policy with default presets. But we were using Rails 5.2 for a long time and didn’t have such issue while running on AWS. Just for experiment, I removed ours config/initializers/content_security_policy.rb initializer and deployed the app one more time. Almost nothing changed. I’ve spent an hour trying to figure out what’s wrong with the Rails application but didn’t find any good reason.

After taking a short break I’ve decided to look at the problem from a different angle. What DevOps / SysAdmin guy would be doing first in such situation? Probably checking configs of third-party packages. Obviously, the right candidate for that is Nginx. I’ve got an idea that Nginx somehow adds problematic headers to all HTTP requests. Seems to be a good start 💡

Reading /etc/nginx/nginx.conf and other general configs haven’t helped much. All stuff looked good to me. I’ve stuck. My last hope was the /etc/nginx/sites-enabled/mysite.conf config file for the running website:

# ...

ssl_certificate /etc/letsencrypt/live/mysite-0001/fullchain.pem;
ssl_trusted_certificate /etc/letsencrypt/live/mysite/chain.pem;
ssl_certificate_key /etc/letsencrypt/live/mysite-0001/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

# ...

Finally, I’ve found one line staring on me:

include /etc/letsencrypt/options-ssl-nginx.conf;

I’ve opened the file and, voilà!

# ...

add_header Strict-Transport-Security "max-age=15768000; includeSubdomains; preload;";
add_header Content-Security-Policy "default-src 'none'; frame-ancestors 'none'; script-src 'self'; img-src 'self'; style-src 'self'; base-uri 'self'; form-action 'self';";
add_header Referrer-Policy "no-referrer, strict-origin-when-cross-origin";
add_header X-Frame-Options SAMEORIGIN;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";

I’ve commented out the security headers, rolled back the Rails initializer and after restarting Nginx, my issue has gone.

P.S. It’s very nice that software, frameworks, and libraries attempt to protect developers from security breaches. But sometimes that could lead to unpredictable and undocumented behavior. And I recently got this experience. The experience when the software silently fighting with each other for our safety.

  1. Orchestrated by Elastic Beanstalk [return]