In this article I’ll describe the attack, what we’re doing about it, how you can ensure that your site will continue to work, and how you can protect your users who have not upgraded their browsers yet.

The attack

In a traditional cross-site scripting (XSS) attack, the attacker finds a way to inject JavaScript code fragments into a web page that they can’t read. When a legitimate user of the targeted site loads the page, the attacker’s code executes. It might send information back to the attacker’s servers, or it might forge commands to the targeted site. CSS data theft also involves injecting strings into a page that the attacker can’t read, but this time, the strings are fragments of a style sheet.

The diagram above shows the course of a CSS data theft attack, from the perspective of a network hub that can see all the traffic. Before the attack begins, a victim user (the laptop in the middle) logs into their favorite website, Clockworks (mockup icon, on the right). Clockworks sends down a session cookie.

Some time later, while the victim is still logged into Clockworks, they click on an ad for dancing hamsters, and get sent to the attacker’s website (Badenov, on the left). The attacker’s website sends down an innocent-looking webpage that contains a <link> tag whose URL points to the victim’s private-messages page at Clockworks.

The victim’s browser duly requests the private-messages page from Clockworks; since the victim is still logged in, it sends the session cookie, so the reply will include information meant only for the victim. The query string, chosen by the attacker, causes Clockworks’ server to inject strings into the HTML on either side of an interesting piece of secret information.

Because the attacker’s website is being rendered in quirks mode, the victim’s browser ignores the Content-Type header and feeds HTML to the CSS parser. Of course, the very first HTML tag in the file causes a CSS syntax error, but CSS has predictable, lenient rules for recovering from syntax errors. The attacker’s injected strings make the CSS parser ignore most of the target page, and capture the secret as the value of the CSS background property.

Finally, since the background property applies to the body tag, the browser needs to download the image it specifies in order to render the attacker’s website. The image URL has been wrapped around the secret information that the attacker wants. So the browser sends that secret to the attacker’s server as a query string.

This attack has been known for some time. The earliest public description I have found was by GreyMagic Corporation in 2002. It has been rediscovered at least twice since then: by Matan Gillon in 2005, and by pseudonymous blogger ‘ofk’ in 2008 (article in Japanese). There are many variations, some of which no longer work, and some of which only work in IE. The variation I’ve described works everywhere that hasn’t deployed a defense; security researchers at CMU were able to use this attack to steal the contents of private messages from a bulletin board and two different webmail providers, with victims using unpatched versions of all the popular browsers.

Since the attack relies on the CSS parser’s error recovery behavior, sites may be immune because of accidental properties of their page structure. For instance, most browsers do not allow newlines in url() literals. If there had been a newline in the middle of the secret information in the diagram, just because that’s the way Clockworks generates its HTML, the attack would only work against victims using IE.

Browser-side defense

This attack works because a webpage in quirks mode can load anything as a style sheet, even if it’s really a HTML page coming from someone else’s server. If the attacker’s page were in standards mode, the browser would pay attention to the HTTP Content-Type header on the target page, declaring it not to be CSS, and refuse to load it as a style sheet.

The attacker, of course, controls whether their page is in quirks mode. But the attacker’s page is on a different server than the target page, which means the attack can be blocked by an extension of the same-origin policy. Even if a page is in quirks mode, it’s not allowed to load a style sheet with a Content-Type header declaring it to be something other than CSS, unless that sheet comes from the same origin. Firefox 4 and IE 9 will implement this rule.

Unfortunately, there are a few websites out there that are rendered in quirks mode, and load their style sheets from a different origin, and put a Content-Type header on those style sheets that says they’re not CSS. These sites aren’t common—the aforementioned CMU security researchers found 62 in the Alexa top 100,000, and most of those have been fixed already—but Firefox 4 and IE 9 will break them.

To give folks more time to fix their sites, while blocking the attack as soon as possible, we implemented a more lenient rule in Firefox ≥3.5.11 and ≥3.6.7. If a page is in quirks mode, loads a style sheet cross-origin, and that sheet has the wrong Content-Type, we’ll start parsing it as CSS anyway … but we’ll stop and throw the sheet away if we encounter a syntax error before the first complete rule has been parsed. HTML tags cause CSS syntax errors, so unless the attacker can inject text at the very beginning of a page, they won’t be able to make the attack work. Safari, Google Chrome, and Opera have also adopted this rule.

It’s possible that this rule could break sites too. For instance, if a style sheet begins with an @-rule that Firefox 3.5 does not understand, that will count as a syntax error, and the sheet will be discarded.

UPDATE: August 28: The HTML5 spec for <link rel="stylesheet"> now requires the strict rule adopted by Firefox 4 and IE 9 (see HTML5 defect report 9834).

Fixing your website

You only have to worry about your site being broken by the defense if you load your style sheets from a different server than the HTML and you use quirks mode. If your site works with a Firefox 4 beta, you’re fine. Current versions of Firefox 3.5 and 3.6 will warn you in the error console when they see a site that will break in Firefox 4, so you can also test that way. (Unfortunately, due to limitations of our translation process, part of this warning will always be in English.)

If your site breaks, all you have to do to fix it is make sure that your style sheets are being served with Content-Type: text/css in the HTTP headers. Please also consider switching to standards-mode rendering. If you cannot fix your website, we want to hear from you.

Protecting your users

If a browser tries to load a style sheet, and the HTTP response it gets has no Content-Type header, it will just assume that it has been sent some CSS, even if it’s a cross-origin load. Therefore, your users are not fully safe from the attack, even if they all have browsers with the defense, unless your servers put Content-Type headers on all content requiring authentication. Check your web services as well as human-readable content.

You should also make sure that those headers are correct. Most importantly, ensure that if your server can’t figure out what Content-Type to put on a response, it falls back to application/​octet-stream or text/​plain. Certain other possibilities (for instance, */* and application/​x-unknown-content-type) may be treated the same as if you hadn’t sent a Content-Type at all.

It is also vital to provide an accurate charset= option in your Content-Type headers for all textual data. If you don’t, an attacker can bypass your XSS filters by encoding injected strings in UTF-7. Declaring the charset in a meta tag or <?xml...?> instruction is not enough to defend against a CSS data theft attack encoded in UTF-7; the CSS parser doesn’t pay any attention to them.

To protect users that are still using browsers that have no defense against CSS data theft, you should block this attack in your filters for user-submitted content. All you have to do is add {, }, and @ to the set of characters that get replaced with equivalent HTML entities (&#123, &#125, and @, respectively). If you can’t be sure that you are always producing Content-Type headers with the correct charset= option, you should also entity-encode + to +.

More information

Chris Evans rediscovered this exploit in late 2009 and has been instrumental in getting it fixed. He has two other blog posts that go into more detail. Collin Jackson and his team at CMU have also been very helpful in understanding the full scope of the attack and ensuring all major browsers fixed it. Their paper will appear at the ACM Computer and Communications Security conference in October.

For technical details of the fixes, see Mozilla bug 524223, Chromium bug 9877, and Webkit bug 29820.

My SO and I are moving. We have a whole lot of ceramic objects that we made. We would like to send them to people rather than find a new home for a giant stack of pottery in our new apartment. Please let us know what you want and we’ll send it to you!

Hers: http://www.flickr.com/photos/pamgriffith/sets/72157624185501258/
Mine: http://www.flickr.com/photos/zackw/sets/72157624186453266/

(Please post requests on the Flickr pages for the objects you want, if at all possible.)

[EDIT 23 Jun 2010: This offer is no longer open. We have donated what hadn’t already been claimed to the San Jose chapter of Empty Bowls.]

David Bolton wanted to know why some of the error screens asked the user to visit other sites manually, rather than doing checks behind the scenes. The main reason, honestly, is that that made a good example thing the user could do next. In practice we probably would want to do at least some checks in the background. Right now, another reason would be that error pages do not have “chrome” privileges so they can’t do anything of the sort (this is part of why the certificate error screen pops up a separate dialog box if you say you want to add an exception) but we may be able to get around that in a real implementation.

John Barton, in email, points out that SSL errors often come up in practice because of server-side configuration changes that ought to have been transparent to users, but a sysadmin goofed. I’ve been using the Certificate Patrol extension, which brings up warnings when a site’s cert changes in any way; this reveals that cert handling mistakes happen even on very popular and well-staffed sites (recently, for instance, mail.google.com flipped back and forth between its own cert and the generic *.google.com cert several times in one day). Of course that would have been invisible to most people, but it’s not much harder to make mistakes that do trigger warnings in a stock browser.

My general feeling on that is, yes, it is way too hard to administer an SSL-encrypted web site, and I would wholeheartedly support an initiative to make it easier, especially for sites that carry information of only moderate sensitivity (e.g. the plethora of Bugzilla instances with self-signed certs out there in the wild). I don’t think that should stop us from raising the visibility of SSL administration mistakes, as long as we improve the presentation and advice on those mistakes so we are not just training people to click through the errors.

John also points out that most people won’t have any idea what “Herdict” is or why they are trustworthy. The explicit mention of Herdict was mainly because I was riffing off Boriss’ earlier proposal to use Herdict information to improve page not found errors. Indeed, we should probably put it more like “Other people who try to visit this website get (something) which (is/isn’t) what you got.” We should credit whatever service we use for that information, but it doesn’t have to be as prominent as I made it.

Someone else (whose name I have lost; sorry, whoever you were!) pointed me at the Perspectives extension, which is said to do more or less exactly what I proposed, as far as comparing certificates seen by the user with those seen by “notaries” at other network locations. I like the use of the term “notary” and the proof of concept; unfortunately, Perspectives seems not to be actively maintained at the moment, and doesn’t work with Firefox 3.6. Also, for privacy, we want to make the queries to the notaries as uninformative as possible to an adversary that can observe network traffic. Reusing the same system that is used for “is this site down?” requests would help there. (Ideally, the notaries would also be unable to tell which users are asking what about which sites, but that might not be tractable.)

I apologize in advance if this causes the RSS feed to spew old posts all over Planet Mozilla or your feed reader. I hope it won’t, but you never know with RSS.

I don’t follow a lot of people on Twitter, but I still sometimes have trouble deciding whether the accounts I’m following are worth it. Folks with much longer follow lists presumably have even harder going.

Enter The Twit Cleaner, a service that scans your follow list and automatically categorizes the behavior of everyone on it. They have some straightforward heuristics for deciding whether someone is worth following, mostly documented in their FAQ:

Q. How are the (potential) bad guys broken down?

A. The possible categories are:
Dodgy – spam phrases, @ spamming, duplicate links etc
Absent – No updates in a month, or fewer than 10 tweets.
Repetitive – High numbers of duplicate tweets or links
Flooding – So high volume you can’t see anyone else
Non-Responsive – No interaction & those that follow back < 10%
Little New Content – Retweeting lots or just posting quotes

This is generally a good scheme, but its focus on conversational use of Twitter means that it misidentifies a few types of legitimate account as unsavory. I think a few special case categories would go a long way to making the service’s advice more useful.

Announcement channels

These are the Twitter equivalent of a news ticker—they broadcast announcements related to something, but they don’t converse with people (as a general rule). The Cleaner dings them as “dodgy behavior: tweeting the same links all the time” and/or “not interactional: hardly follow anyone.” Examples include @NBCOlympics, @CDCemergency, @asym, @Astro_Soichi, and (ironically) @TwitCleaner itself (the problem here appears to be public “@somebody, your report is ready at ” directed tweets when direct messages fail).

These can probably be machine-identified as extreme outliers in follower-to-followed ratio. @asym and @Astro_Soichi don’t follow anyone; @NBCOlympics and @CDCemergency follow less than 0.1% of their follower numbers. @TwitCleaner likes to follow users of the service, though; maybe they should just whitelist themselves? Also, if Twitter-verified users are not already whitelisted (I wasn’t able to tell from my own report), perhaps they should be.

Lurkers

Lurkers are the opposite of announcement channels: they just read Twitter, they never post anything. Lurking is a time-honored tradition on the Internet and people shouldn’t be penalized for it. I have several lurkers on my follow list just on the off chance that they might start posting in the future.

Accounts that have never posted at all should be distinguished from accounts that post rarely. (The latter are often spammers. Lately Twitter itself has gotten a lot better about finding and banning spammers, but they still turn up now and then.)

Fictional character accounts

There are any number of fictional characters who regularly use Twitter—that is, their authors write and post tweets under their names, usually to provide a bonus story line, or to implement the fourth wall mail slot. Examples include @Othar of Girl Genius and the entire cast (caution: mildly NSFW; @pintsize0101 consistently links to egregiously NSFW images of the “where’s my brain bleach” variety) of Questionable Content. Fictional characters may absent themselves for long periods because the bonus story line is on hold (Othar recently didn’t post anything for four months but is now back) and might not follow anyone but other characters from the same fictional world (the QC cast does this); both things get them unfairly dinged by the Cleaner.

It probably isn’t possible to identify fictional accounts in a mechanical way. However, you could pick out cliques in the follow graph, sets of accounts that are followed by many but that follow no one but each other, as deserving human attention. If Twitter implemented some sort of account-labeling scheme that would let the people behind the curtain mark accounts as fictional characters, that would be awesome.

This game is worth playing just for the chance to drive the protagonist’s hot rod around and see all the epic scenery. The art department had fun with this game. So did the character modelers. They licensed about a hundred classic metal tracks for the background music, which means it’s thematically appropriate, and never gets repetitive enough to earworm you. (The magical guitar solos, on the other hand, I got a bit tired of.) The gameplay itself is a little spotty, but I think that’s been well covered elsewhere. My main beef was with poor integration of the side quests into the story line—you don’t benefit much from doing them, even though they could have added quite a bit of interest and strategic ramification. The up side of that, though, is that I never felt like I was being forced to level-grind. There was one infuriating point where me and Pam spent three hours losing one stage battle over and over again, but that was because we were doing it wrong.

So that’s all good, but now I want to complain, at length, about the storyline.

I am about to spoil the ENTIRE PLOT. You have been warned.

Problem numero uno is, of course, that the entire Drowning Doom arc could have been avoided if Eddie hadn’t picked up the idiot ball upon overhearing Doviculus in the ruins of Lionwhyte’s castle. Eddie is genre-savvy enough that he should have assumed Doviculus knew they were there and was deliberately trying to deceive them. (I think Doviculus doesn’t ever lie in the technical sense, but I think we all know just how little that means.) Eddie is level-headed enough to be head roadie for a boyband whose members are all too dumb to live. Eddie is perfectly aware that Lita is wound way too tight, has issues with Ophelia, and has just had to watch her brother get killed. Eddie isn’t the kind of asshole who claims to trust someone when nobody else does, in order to get them into bed. Ditching Ophelia was way the hell out of character. Yeah, she’s clearly not telling him something important, but I see his in-character reaction as something like “Now is not the time for this, but when we get somewhere safe, you and me and Lita are going to sit down and have a long talk. I still trust you, but I need you to trust me with the whole story.” This does take away Doviculus’ big reveal at the end, but screw him, why should the antagonist always get the big reveal?

It’s not like there was no other way to arrange for the “Metal versus Goth” battles, either. The most obvious fix would be for the protagonists to decide they have to have the power of the Sea of Black Tears to stand a chance against the Tainted Coil. It’s Ophelia who has the best odds of being able to control it, but at first she can’t, and they have to fight her a bit. Or else it’s Lita who loses it and throws herself into the Sea. Either way, this probably couldn’t have been dragged out as far as the as-written storyline did, but the Tainted Coil deserved a bit more screen time as the immediate enemy, anyway. And would it not have been even more awesome if Ironheade and the Drowning Doom could have teamed up?

I was also disappointed in the, um, complete lack of closure at the end of the game. Okay, Drowned Ophelia was a doppelganger created by the Sea of Black Tears, and now we have the real one back, but it was the real Ophelia who got ditched in the snow outside the ruined, demon-infested castle. I’d expect her to be more than a little angry and hurt, still. The antagonist leaders are all dead but all four armies still have squads running around the map attacking each other, with no explanation given. In the closing sequence Eddie drives off into the sunset, without Ophelia (wtf?), we see her shed one black tear (I have to assume this is a sequel hook because otherwise it makes no sense), and after that all the protagonists are off in their own corners of the map and you can have a little wordless scene with each one, but that’s it. Yeah, they’re giving you a chance to drive around and finish up side-quests, ok. Still not satisfying, and I cannot be bothered to get to 100% completion just to see if there’s another cutscene. (Might be different if someone assured me there was another cutscene, although maybe I should just go find it on Youtube.)

Even if these small-scale plot problems were resolved, it doesn’t seem to me that the story that the game tells is the story it should have told. Is the ultimate metal epic really just about how generic bondage demons manipulate power metal musicians into fighting first their hair metal counterparts and then some goths? I don’t think so. It’s not about subgenres of rock fighting each other, it’s not about rock and roll versus the blues, it’s not both of those versus country and western, it’s not even about electric versus acoustic. (Although it certainly could appear to be about each of these in turn, as we peel back layer upon layer of manipulative level bosses.) The ultimate metal epic ought to be about nothing less than good music versus bad music; to be precise, Music With Rocks In versus Extruded Music Product. Eddie, Ophelia, Lars, and Lita travel the land assembling a motley crew (pun intended) of every kind of musician who ever played, because no lesser force can stand against the ultimate evil. And what could that be but … the record industry? (It occurs to me that this is the plot of We Will Rock You, and that’s just about right.)

I’m not sure there’s a place for the Tainted Coil in a game that’s telling that story, but you know, I’m okay with that. They weren’t that interesting. I’d rather have had more shout-outs to the great heroes of music, anyway. Where were Freddie Mercury and Janis Joplin and John Lennon and Bob Marley? Buddy Holly? B.B. King? Heck, where was Leo Fender? (Okay, Leo Fender was probably one of the Titans. BUT STILL.)

Small conference rooms (memes)

• All your base
• Bike shed (used as a storage area for a while)
• Chuck Norris
• Dancing Baby
• End of the World
• Fail
• Get to da choppa
• Hampster Dance
• Icanhascheezburger
• Jibjab
• Keyboard Cat
• Leeroy Jenkins
• Mullets Galore
• Numa Numa
• Orly
• Peanut butter jelly
• QQ
• Rickroll
• Strong Bad
• Tron Guy
• Ultimate
• Very good, very mighty
• Win
• Xiao Xiao
• Yatta
• Zombocom

Large conference rooms (Star Trek)

• 10 Forward (break room)
• Holodeck
• The Bridge
• Warp core

The management hopes this is not a horrible inconvenience for the two or three people still reading this site.

This is a big block of jargon which doesn’t do anything to tell the user how big the risk actually is, or help them distinguish a minor problem from a major one. If you click on “technical details” you get a little bit more information about what went wrong, but it still doesn’t make any effort to give advice.

The Firefox UI team has been talking about using Herdict or a similar service to improve network error screens, especially the site not found screen. I think we could get a lot of mileage out of that for SSL errors as well. We should also make use of the user’s history with the site, and pay attention to what exactly is wrong with the credential. Here are some examples.

The only problem with self-signed certificates is they haven’t been signed by a trusted third party. The connection is secure, but you might not be talking to who you think you are. In the first section, we emphasize that the concern here is with identity, and we use Herdict information to deduce that this is probably not a hijacked site, because lots of people get the same credential. (“The same credential” means exactly the same, not just some self-signed cert, but we needn’t bother people with that unless they want to see the details.)

In the “What should I do?” section, we give some examples of things that might be unsafe to trust this site with, but we go ahead and let them visit the site, automatically storing the self-signed cert and marking it valid for this site only. We implement bug 251407, so we can promise to notify the user if the site’s credentials change in the future.

I’ve front-loaded the information that used to be in the “technical details” section, so it has been replaced with “Inspect the Credentials”. If you open that area up, it shows the certificate, but in a more user-friendly way than the existing certificate dialog box does. Especially important here is to reveal the interesting parts immediately, highlight suspicious things, and deemphasize the jargon and the long hexadecimal numbers.

“I understand the risks” is still there, but in this case, it’s for people who didn’t read the rest of the page. It’s meant to make people stop, slow down, and reread. If you click on it you get another link to the page.

There are exploits in the wild that take over your WiFi hub, or your cable modem. Once they’ve done that, they are in a position to tamper with all your Internet traffic. I ran into one of these for reals last week; I was in a café and getting certificate errors on every secure site I tried to visit, including Mozilla’s mail server. (The theory is that you’ll just click through the error messages because you want to get your email, or whatever; one of the staff at the café did just that when I complained.) Here’s where Herdict could really come in handy: if you are getting certificate errors but nobody else is, we can deduce a problem near your computer.

Again, the first section tries to be clear and specific about the problem: we suspect that someone is tampering with your Internet connection, and here is why. The second section underlines how big a deal this is: “Do not log into any site or buy anything online.” It then suggests a test: visit another secure website and see if the problem persists. This scenario should put the whole browser into a paranoid mode, where it will not load saved passwords and continues to try to work out whether there’s something wrong with the local router. Ultimately, we should advise people in this boat to factory-reset their WiFi hub and/or contact their ISP for help, but we should take care to be certain in our diagnosis first.

In this scenario, the “I understand the risks” section gives you access to the certificate-exception dialogs, as it does now.

Finally, here’s what it looks like in the comparatively rare scenario that SSL certificates were originally intended to defend against: the server has been hijacked (but the attackers do not have access to the cert). We can tell from browser history that the cert has changed, and we can tell from Herdict that it has changed for everyone. We tell the user not to visit this website, and again, suggest trying another secure site. (We need to take care to distinguish this case from an expired or legitimately changed cert.)

This mug was designed by Steven Frank and printed by Zazzle. The top part of the design was much darker six months ago. Zazzle’s process appears to involve shrink-wrapping a layer of plastic over the mug and then printing on that; you can’t see it in the photo, but the plastic has started to peel off near the top of the handle. I have another such mug, printed using a different process in 2003 for the Stanford Film Society’s “Film Our Way” festival; it didn’t fade nearly as fast, and there wasn’t any plastic to peel off, but after seven years of use the design is almost gone.

The problem with these mugs is, the design is printed on top of the glaze. Truly permanent decorations on ceramic are either done with the glaze itself, or are inked directly on the unglazed piece and then covered by transparent glaze. Either way, the decoration happens before the glaze firing. Unfortunately, glaze kilns are typically designed to process hundreds of pieces per batch, and take several days to go through a complete cycle. That’s not practical for a print-on-demand outfit.

I think you could design a much smaller kiln, with space for just a few mugs, though. It’d be lined with fiberglass instead of firebrick, to reduce the thermal mass; since there’s no need for a reduction phase with clear glazes, it could use electric heat. It’s not possible to do a stoneware firing in less than about 24 hours start to finish, because the clay will crack if you heat or cool it too fast (this is why raku-glaze pieces are often fragile) but there would be no need for several days’ worth of cooling time as is typical for large batch kilns.