Why ordering HTTP headers is important


RFC 2616 in decay

If you code against Akamai hosted sites, you could be rejected because your HTTP library sends request headers in the wrong order. In fact, most libraries use undefined order, as the IETF specification says it doesn’t matter.

In casu:

$ URL=http://www.bulgari.com
$ UA="User-Agent: Mozilla/5.0 My API Client"
$ ACCEPT="Accept: */*"

$ curl -v -H "$UA" -H "$ACCEPT" $URL |& grep '< HTTP'
< HTTP/1.1 403 Forbidden

$ curl -v -H "$ACCEPT" -H "$UA" $URL |& grep '< HTTP'
< HTTP/1.1 302 Moved Temporarily

My guess: they identified that major browsers send HTTP headers in a specific order, and they implemented this trick to fend off spammers.

Update After some more experimenting, it appears that this behaviour depends on order and the Accept header:

$ ACCEPT="Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"

$ curl -v -H "$UA" -H "$ACCEPT" $URL |& grep '< HTTP'
< HTTP/1.1 302 Moved Temporarily

$ curl -v -H "$ACCEPT" -H "$UA" $URL |& grep '< HTTP'
< HTTP/1.1 302 Moved Temporarily

Also, no block without Mozilla/5.0 in the User-Agent.

Conclusion: they will block your request if:

Update 2 Other sites at Akamai don’t expose this behaviour, so it could be a single site issue and/or a configurable setting.

(picture by Olli Homann)

Hacked store? Hire me! And get updated on new posts via Twitter: