You are here: Home page > Computers > Blocking hotlinks
Advertisement

Questionmarks in the clouds.

Stopping hotlinks to your Cloudfront files

by Chris Woodford. Last updated: September 25, 2013.

You've been bitten by the cloud bug! Excited by the possibility of supercharging your website, you've shifted your static content (your images, your CSS, and whatever else doesn't change that often) onto a content delivery network (CDN), such as Amazon Cloudfront or Rackspace Cloud Files. Everything goes well for the first few months and then, a little while later, a dreadful realization slaps you round the face: your monthly bill is increasing more quickly than your monthly traffic! You're the person "paying-as-you-go" for every single hit on your CDN—but you're not the only person generating those hits. Other people are happily hotlinking your files, duplicating your content on search engines, and running up your bill. The absolute, ultimate, nightmare scenario? Someone embeds or links to one of your really big files in a viral link; days or weeks later, you discover millions of downloads and an unexpectedly massive credit-card bill. Blocking hotlinks on a standalone server is easy enough—a two-minute edit of your Apache htaccess file—but most CDNs don't offer the same kind of HTTP referrer checking. Of course, you could simply rename the files people are hotlinking, but if there are hundreds or thousands of them (I have at least 3000) that's not an option. So what to do? Here's a quick and easy solution. It's not a permanent fix, but it certainly solves for the problem for a time.

Photo: Cloud computing questionmarks: Moving files to the "cloud" has advantages but it can bring security and management problems too; having people hotlink to your Cloudfront files is just one of them.

Using a CDN with a CNAME

For the sake of illustration, let's suppose I'm using Amazon Cloudfront as my CDN. The same method also works fine with Rackspace Cloud Files and with any other CDN that lets you refer to your files using a CNAME.

When you set up a Cloudfront distribution, you can either reference your files directly from your distribution's URL (typically something like d1234567890.cloudfront.net) or, more sensibly, set up a DNS CNAME (essentially a kind of alias) based on your own domain name. So if I wanted to serve files from Cloudfront based on this website's domain name (explainthatstuff.com) I could set up a CNAME something like fred.explainthatstuff.com and point that to my Cloudfront distribution, d1234567890.cloudfront.net, then simply make any links to files on Cloudfront by linking to fred.explainthatstuff.com. Only the terminally curious—DNS anoraks who go to sleep reading the output from "Live HTTP headers"—will ever discover that my files are actually coming from Cloudfront.

A CNAME looks much more professional, but the other great advantage is that you can change it very easily: you can point it to another CDN, point it back to your original server, or delete it entirely—and that's exactly how we can tackle Cloudfront Hotlinking. If you're not currently using a CNAME (if you're linking directly to your Cloudfront distribution with raw URLs like d1234567890.cloudfront.net), you're not going to be able to take advantage of the trick I'm about to share unless you set up a new distribution, copy your files across, and link to them with a CNAME.

Changing your CNAME

If you've reached the end of your tether with the hotlinkers, it's time to change your CNAME! And it's really easy to do, with (hopefully) no downtime to your website and complete transparency to search engines. Here's what you do.

1. Set up a new DNS CNAME

It doesn't matter what name you choose. If my first CNAME was fred.explainthatstuff.com, I could set up a new CNAME fred2.explainthatstuff.com and point it to the same Cloudfront distribution. Remember to give it the standard 48 hours to propagate over the Internet, though if you're lucky it'll be live much more quickly.

2. Go into Cloudfront and add the new CNAME to your existing distribution

You can use up to 10 CNAMEs to point to a single Cloudfront distribution so it's no problem to have two CNAMEs linked to the same one. How do you do it? If you're using the AWS console, go into the Cloudfront tab, click to edit your distribution and where it says CNAME, add the new CNAME either after the existing one (with a comma) or on a new line. Cloudfront will take a little while to propagate that information through its edge servers and then switch back to "Deployed". Don't sit there waiting for it; there's something else you can be doing in the meantime...

3. Now change your website files

I have maybe 10,000 or more IMG tags in about 500 HTML files pointing to maybe 2500 images on my CDN, so changing my files by hand isn't an option! Fortunately, I set them up a while ago so all the IMG tags are explicitly referencing full URLs on my CDN, such as fred.explainthatstuff.com/myimage.jpg.

Using perl, it's a cinch to change all those URLs so they point to fred2. With all your HTML files in the same directory, you need a quick line of perl something like this:

perl -pi -e 's%fred.explainthatstuff.com%fred2.explainthatstuff.com%' *.html

That takes no more than a second.

4. Test that your new CNAME is working

Once Amazon says your Cloudfront distribution is deployed, try referencing some of your files using the new CNAME to make sure it's working. If it is, you can safely upload your HTML files. Once you've done that, you've effectively switched over to your new CNAME. Use that from now on.

5. Switch off the old CNAME

So now you're no longer using the old CNAME, you have two options: you can either just "switch it off" or you can 301 redirect it for a time to help search engines find where your files have gone to, and then switch it off later. If you don't care about the search engines, simply go into Cloudfront and edit your distribution again, deleting the CNAME you no longer require (fred.explainthatstuff.com) and leaving the new CNAME (fred2.explainthatstuff.com) as the one and only active CNAME. You'll find Cloudfront quite rapidly denies any links to fred.explainthatstuff.com. Next, edit your DNS settings and delete your old, unwanted CNAME (fred.explainthatstuff.com). In 48 hours or less, fred.explainthatstuff.com will be history: any files that still hotlink to it will be rendered useless by a DNS name that doesn't resolve. It's a great (and instantly satisfying solution): that unwanted traffic will be stopped dead in its tracks before it goes anywhere near your server, slows it down, or costs you a dime.

6. Redirect the old CNAME

Instantly killing the old CNAME is a bit crude and drastic—because you're potentially also going to lose any useful links to your images on search engines and from other sites. Fortunately, there's a solution to that too. Once you've deleted the old CNAME, you can set up a new A record with exactly the same domain name and point it to the same place as your ordinary server (www.explainthatstuff.com). Now all attempts to reference fred.explainthatstuff.com (formerly at Cloudfront) will go to your main server instead and you can handle them, however you want to, with rewrites and redirects from an htaccess file (if you're using cPanel, use its built-in redirects—which simply edit your htaccess file behind the scenes). You can create a wildcard, 301 redirect to point any legitimate links from search engines to files like fred.explainthatstuff.com/myimage.jpg so they go to fred2.explainthatstuff.com/myimage.jpg and neatly return the 301: Moved permanently code. You'll need something like this:

RewriteCond %{HTTP_HOST} ^fred.explainthatstuff.com$
RewriteRule ^(.*)$ "http\:\/\/fred2\.explainthatstuff\.com\/$1" [R=301,L]

That's purely for the benefit of the search engines and traffic you want to retain. At the same time, you can block links from anywhere else so they return a custom "Hotlinks blocked" image—much as you'd do if you were blocking hotlinks on an ordinary server in the absence of a CDN (I'm not going to go into how you do that here—there are plenty of other pages telling you how to block hotlinks with htaccess). Once you're confident the search engines have got the message, you can delete the redirect code and the old CNAME from your DNS. (I found Google had reindexed all my images at the new CNAME within a few days—though you might want to wait longer. In my experience, Bing takes longer to catch up.)

7. Monitor the redirected traffic

If you're redirecting this way, it's very wise to monitor incoming traffic at the place you're redirecting to (for a few days at least). Using a stats package (such as Awstats), or even simply inspecting raw access logs, you'll get a list of people who were previously hotlinking your Cloudfront files. Some of them will be innocent links (maybe text links in an HTML page rather than actual image hotlinks); most will be exactly the kind of hotlinks you're trying to knock out; and a few will be scraped copies of your entire web pages (text and images) that you can pursue with polite emails or takedown notices, as you wish. But some may also be links from your own website (or other websites you operate) that you'll need to change over to avoid your own web pages getting one of those "Hotlinks blocked" messages. It's also worth watching out for blocked links coming from search engines that you haven't correctly exempted in your htaccess file. For example, I was using some old code and discovered that I was allowing links from live.com but blocking its new incarnation bing.com, which I quickly addressed.

Inspecting your logs will also give you a sense of how many people were hotlinking, how much bandwidth you were losing, and how much it was costing you; it can be quite hard to establish that from the raw data Cloudfront provides (not least because logging is not enabled by default and, even when it is, you don't automatically get meaningful statistics). In my case, after about 9–10 months of Cloudfront use, I was getting about 500 hotlinked image hits a day, making my total bandwidth loss maybe 300–500MB per month or so—nothing too serious compared to my own monthly bandwidth use of about 50GB, but still not trivial. If I'd left it another 9–10 months, I could easily have been draining 1GB a month.

A copied, hotlinked webpage from explainthatstuff.com. The same page with its hotlinks blocked and replaced by dummy images.
Photo: Goodbye hotlinks—at least for now. Left: Before: One of the many web pages I've found ripping off my words and photos and hotlinking to my Cloudfront files; Right: After: Moments later, with my CNAME switched over, this page is now trying to get its images from my ordinary server and being redirected to blocked image files. I could leave that there indefinitely to advertise my site but after a month or two, once the search engines have indexed my images again, I'll probably delete the DNS entry I was previously using for my CDN (the original CNAME and now an A record) so those linking images break completely.

8. Until next time

Of course this is only a temporary solution to the Cloudfront hotlinking problem—but it is a solution. Most of your hotlinkers probably won't even spot that their links are broken, but there are always new ones waiting to take their place. There's nothing to stop you changing your CNAME regularly if hotlinking is driving you crazy or costing you money. Maybe the CDN providers will eventually implement some sort of decent http referrer checking on incoming requests—but don't hold your breath. I've seen this question asked numerous times on Amazon Cloudfront forums and the answer from AWS is always a variation of "referrer checking is not a feature we support at this time". If you know different, please let me know!

Find out more

On this website

On other sites

Sponsored links

Please do NOT copy our articles onto blogs and other websites

Text copyright © Chris Woodford 2011. All rights reserved. Full copyright notice and terms of use.

Amazon Web Services, AWS, Amazon Cloudfront, and Cloudfront are trademarks or registered trademarks of Amazon Web Services LLC in the United States and/or other countries.

Follow us

Rate this page

Please rate or give feedback on this page and I will make a donation to WaterAid.

Share this page

Press CTRL + D to bookmark this page for later or tell your friends about it with:

Cite this page

Woodford, Chris. (2011) Blocking Cloudfront hotlinks. Retrieved from http://www.explainthatstuff.com/blocking-cloudfront-hotlinks.html. [Accessed (Insert date here)]

More to explore on our website...

Back to top