2007-10-12

HTTP response signing abstract model

I've argued that there's a need for an HTTP-specific mechanism for signing HTTP responses. So let's try and design one.  (Usually at this point, I would start coding, but with security-related stuff, I think it's better to have more discussion up front.)

Before drilling down into syntax, I would like to work out the design at a more abstract level.  Let's suppose that the mechanism will take the form of a new Signature header.  Here is my current thinking as to the steps involved in constructing a Signature header:

  1. Choose which security token to use and create one or more identifiers for it.
  2. Choose the suite of cryptographic algorithms to use.
  3. Choose which request and response headers to sign.
  4. Compute the cryptographic hashes of the request and response entity bodies if they are present.  Base64-encode those hashes.
  5. Create a  Signature header template containing the information from steps 1 to 4; this differs from the final Signature header only in that it has a blank string at the point where the final Signature header will have the base64-encoded signatures value.
  6. Canonicalize the request headers.
  7. Combine the response headers with the Signature header template and canonicalize them.
  8. Create a string that encodes the HTTP method, the request URI, the canonicalized request headers from step 6, the response status, and the canonicalized response headers from step 7.
  9. Compute the cryptographic hash of the string created in the previous step.
  10. Sign the cryptographic hash created in the previous step.  Base64-encode this to create the signature value.
  11. Create the final Signature header by inserting the base64-encoded signature value from the previous step into the Signature header template from step 5.

Let's flesh this out with a bit of q&a.

  • What kinds of security token can be used?
    At least X.509 certificates should be supported.  But there should be the potential to support other kinds of token.
  • How are security tokens identified?
    It depends on the type.  For X.509, it would make sense to have a URI that allowed the client to fetch the certificate.  It would also be desirable to have an identifier that uniquely identifies the certificate, so that the client can tell whether it already has the certificate without having to go fetch it.  As far as I can tell, in the X.509 case, people mostly use the SHA-1 hash of the DER encoding of the certificate for this.
  • How does the server know what kind of signature (if any) the client wants?
    The client can provide a Want-Signature header in the request.  (It makes more sense to me to call this Want-Signature than Accept-Signature, just as RFC 3230 uses Want-Digest, since any client should be able to accept any signature.)
  • What if the response uses a transfer encoding?
    It's the entity body that's hashed in step 4. Thus, the sender computes the hash of the entity body before applying the transfer encoding.
  • What if the response uses a content encoding?
    It's the entity body that's hashed. Thus, the sender computes the hash of the entity body after applying the content encoding. From 7.2 of RFC 2616:
      entity-body := Content-Encoding( Content-Type( data ) )
  • What if the response contains a partial entity body (and a Content-Range header)?
    The hash covers the full entity-body (that is. the hash is computed by the recipient after all the partial entity bodies have been combined into the full entity-body).  In the terminology of RFC 3230, it covers the instance (where there is a relevant instance), rather than the entity. Thus the hash identifies the same thing as a strong entity tag (which in my view makes the terminology of RFC 3230 rather unfortunate).
  • Why and how are headers canonicalized?
    As I understand the HTTP spec, proxies are allowed to change various semantically insignificant syntactic details of the headers before they pass them on.  For example, section 2.2 of RFC 2616 says any linear white space may be replaced with a single SP. Section 4.2 says explicitly proxies must not change the relative order of field values with the same header field name, but it seems to imply that proxies are allowed to change syntactic details of the header fields that are not syntactically significant (such as the relative order of headers with different field names).  It is not clear to me to what extent section 13.5.2 overrides this. Perhaps some HTTP wizard can help me out here. In any case, there needs to be just enough canonicalization to ensure that the canonicalization of the headers sent by the origin server will be the same as the canonicalization of the headers as seen by the client, no matter what HTTP/1.1 conforming proxies (or commonly-deployed non-conforming proxies) might be on the path between the origin server and the client. I would note that the Amazon REST Authentication scheme does quite extensive canonicalization.
  • Can there be multiple signatures?
    Yes. In the normal HTTP style, the Signature header should support a comma-separated list of signatures. The order of this list would be significant.  There should be a way for each signature in the list to specify which of the previous signatures in the list are included in what it signs.  There's a semantic difference between two independent signatures, and a later signature endorsing an earlier signature.
  • How about streaming?
    Tricky.  The fundamental problem is that HTTP 1.1 isn't very good at enabling the interleaved delivery of data and metadata.  This is one of the things that Roy Fielding's Waka is supposed to fix.  I don't think it's a good idea to put a lot of complexity into the design of a specific header to fix a generic HTTP problem.  The fact that the hash of the entity body is computed in a separate, independent preliminary step makes it a bit easier for the server to precompute the hash when the content is static. Also the signature header could be put in a trailer field when using a chunked transfer-coding.  However, although this helps the server, it screws the client's ability to stream, because the client needs to know the hash algorithm up front.  And, of course, it also requires all the proxies between the client and the server to support trailer fields properly.

13 comments:

Anonymous said...

I can't see the utility nor feasibility of including headers in the data being signed. It's just not going to work in general because of all things intermediaries can change/add/remove.

If you want message integrity with HTTP you need to do it at transport layer, e.g. with SSL and NULL ciphers as someone commented on your previous entry.

If you want entity integrity don't get distracted by trying to sign header values.

Anonymous said...

Anon said it first. While I think that signing (and encrypting) the payload would be very valuable, including transport details (headers) in that seems to add unnecessary complexity. I would say that leaving transport security to the TLS layer is a better choice, and focus on the payload.

Anonymous said...

There's good motivation for including the request headers, but it does mean that only the requester (or those s/he chooses) can verify the signature; is that your intent? Perhaps allow for a variant that doesn't include the request. That also might caching more effective, too -- I can return the same document for conditional-get, e.g.

Anonymous and Gustavsson are wrong. :)

Anonymous said...

Overall this seems reasonable and fairly simple to implement. I agree with Rich in that signing the request headers may prove counterproductive as far as caching is concerned. Hop-by-hop headers need to be omitted from the signature; and we would need a way of listing the specific headers that were signed and the ordering of their values.

Anonymous said...

Rich, out of curiosity, which request and response headers would you find necessary to sign (where TLS would not be appropriate)?

I think that signing data rather than at the transport level is of interest if we plan on keeping the data and the signature around beyond the HTTP request/response (like storing it for some future processing or sending it to a third party). But then, the HTTP headers from the initial request/response would no longer be of any interest.

But then again, my use cases might not be your use cases :-)

Also, I fail to see how this would have impact of conditional-gets, but I'm likely missing something. Feel free to educate me.

/niklas

Anonymous said...

Ok, so I've got the basic model you describe implemented and it's fairly straight forward and easy to do. I would recommend against signing the request AND response entities and the request AND response headers. Instead, let's focus on signing individual HTTP messages, e.g. signing the request or signing the response. The signature would be cover the entity body and selected headers in the message. If it's a response, the status code is also included.

Here's the rough output my quick impl is generating:

sigalg="SHA1withDSA";hashalg="MD5";token="1157395547";headers="Bar,Foo";content="jXd/OF09/siBXSD3SWAm3A==";sig="MCwCFHaT1nsPAYMyB0ZkKYa6Q5wtkpCyAhRXfoYoDtHkW/3LDGW57/MQtvXQaQ=="

token is the serial number of the X.509 Cert; we would obviously need to provide a link to the cert and a better way of identifying it. The headers field lists the headers whose values were signed in the order they were signed. The canonicalization I applied was to simply order the headers alphabetically by name, and strip leading and trailing whitespace from the value. If multiple values are specified for the header, the values are concatenated to generate the sig.

Other than that, there are no major showstoppers that I can see.

Anonymous said...

Your comment page cut off the full output in the display. Here is a newline delimited alternate version

sigalg="SHA1withDSA";
hashalg="MD5";
token="1157395547";
headers="Bar,Foo";
content="jXd/OF09/siBXSD3SWAm3A==";
sig="MCwCFHaT1nsPAYMyB0ZkKYa6Q5wtkpCyAhRXfoYoDtHkW/3LDGW57/MQtvXQaQ=="

btw, the content field is the base64 encoded hash of the entity payload.

Anonymous said...

Signing the headers ties the request to the response. For example, including the content-negotiation means that I can "prove" you said a low-res image was acceptable.

As for if-modified-since; with a signed reqeust/response, intermediate proxies would need to see this and know which fields aren't covered by the signature and therefore can be ignored in terms of sending out a proxied response.

Anonymous said...

"For example, including the content-negotiation means that I can "prove" you said a low-res image was acceptable."

Hmmm... I'm not convinced that nonrepudiation of the request is really all that useful. Got a use case?

Unknown said...

Having just gone through all this with the OAuth spec, I wonder if signing the response could be proposed as an extension to OAuth? The signature algorithm would be basically the same, except with the request body as part of the signing string (maybe instead of the request parameters).

Davanum Srinivas (dims) said...

Just ran into this spec on James Snell's blog comments:
http://www.httpsec.org/1.0/

Anonymous said...

Niklas, I don't know which request headers to sign; I haven't thought about it much.

James, the intent of including the request in the signature is to tie the two together. For example, an auction site returning a "winning bid" message should tie that response to the winning bid.

As with everything else in security, it's a trade-off.

Anonymous said...

Amazon's S3 uses something similar and proprietary for signing the HTTP headers and messge. But of course it's Amazon specific. I ran into problems finding software that would treat S3 as a ftp site because of this problem, very few custom clients.

It would be great if there were a standardized solution and then there would be all the different sites, clients and servers to support it.

I also wonder if any of this is in scope for the HTTPbis WG?