Bootstrapping WebFinger decentralized discovery with WebFist

The goal of the WebFinger protocol is to make it easy for people to interact with you on the web.

We commonly identify each other online with email addresses (e.g., myname@example.com) not web URLs. But email addresses have a fundamental problem: The only useful thing you can do with an email address is send it email. With just an email address you can't answer simple questions like, "What is the profile photo of the person with this email address?"

There are many things I want to share with someone who has my email address. A quick list:

My business address (as part of a hCard)
My public key (for private communication)
My personal webpage address and RSS feed
My photo hosting provider (for sharing images, like Flickr)
My web drive hosting provider (for sharing documents, like DropBox)
My preferred way of logging into websites (Facebook, Twitter, OpenID)

WebFinger translates an email address into something that provides this information. It is general and decentralized. It is not controlled by a single company. There is no single point of failure. The flow for developers that want to use WebFinger is really simple:

Start with an email address: pithy.example@gmail.com
Translate it into a URL: http://gmail.com/.well-known/webfinger?resource=acct:pithy.example@gmail.com
Fetch that URL and get back JSON

The resulting service document looks like this:

{
  "subject": "acct:pithy.example@gmail.com",
  "links":
  [
    {
      "rel": "avatar",
      "type": "image/jpeg",
      "href": "http://www.example.com/~myname/profile.jpg"
    },
    {
      "rel": "homepage",
      "type": "text/html",
      "href": "http://www.example.com/~myname/"
    },
    {
      "rel": "public-key",
      "type": "text/plain",
      "href": "http://www.example.com/~myname/public.key",
      properties: {
        "fingerprint": "c6:1e:fb:a8:1a:b8:80:66:e5:22:94:e6:18:09:98:96",
        "algorithm": "rsa"
      }
    }
  ]
}

The link types (e.g., "avatar") aren't part of the specification. WebFinger only provides the standard mechanism for finding this service document. It enables systems to offer better service to you and the people you communicate with.

The problem with WebFinger is it requires the domain name that owns your email address to participate. When you own your domain name this is trivial. But if you use Gmail and they don't implement WebFinger handlers you are out of luck. We in the open web community hoped everyone would adopt WebFinger in time. That hasn't happened.

Bootstrapping

Our solution to the adoption problem is WebFist. WebFist is a fallback when providers don't support WebFinger natively. It lets you do WebFinger lookups for Gmail, Facebook, and other email addresses even if the owner of the domain name isn't playing along. WebFist works because of a judo move on an existing infrastructure: DomainKeys Identified Mail (DKIM).

The goal of DKIM is to stop email spam and scams. Every major email provider out there uses DKIM to cryptographically sign the email messages they send on your behalf. That enables people who receive email from you to verify that your account actually sent it.

WebFist uses DKIM-signed email to prove that you, the user, want to participate in WebFinger, regardless of what your provider says. By sending a single email you can delegate your WebFinger profile to your own website host or anything that can serve the service document over HTTP (e.g., Google Drive). This is ridiculously easy for users. You can even set up WebFist via a mailto link on a webpage.

Here's an example email that does this:

MIME-Version: 1.0
Date: Mon, 24 Jun 2013 22:53:15 -0700
Message-ID: <CAK3M4CXHHrvLArWPCcdnPh0i5JGZkQw8L_RW-YsB8eKhrivm9g@mail.gmail.com>
Subject: 
From: Example Account <pithy.example@gmail.com>
To: verify@webfist.org
Content-Type: multipart/alternative; boundary=047d7b15b1172d343004dff425dd

--047d7b15b1172d343004dff425dd
Content-Type: text/plain; charset=ISO-8859-1

webfist = http://example.com/my-delegation-here.json

--047d7b15b1172d343004dff425dd
Content-Type: text/html; charset=ISO-8859-1

<div dir="ltr">webfist = <a href="http://example.com/my-delegation-here.json">http://example.com/my-delegation-here.json</a><div><br></div></div>

--047d7b15b1172d343004dff425dd--

Once a WebFist node receives an email like this it will answer WebFinger queries on behalf of this user. Fetching a URL like http://webfist.org/.well-known/webfinger?resource=acct:pithy.example@gmail.com will produce a response like this:

{
  "subject": "pithy.example@gmail.com",
  "links": [
    {
      "rel": "http://webfist.org/spec/rel",
      "href": "http://example.com/my-delegation-here.json",
      "properties": {
        "http://webfist.org/spec/proof":
          "http://webfist.org/webfist/proof/08e01fb3123de74555528daaeb2d33b513f50f88-
           c255b91b02617c067df89a3809f0e17197b52413?decrypt=pithy.example%40gmail.com"
      }
    }
  ]
}

The special WebFist link relation tells clients where the real service document lives. WebFinger implementors can use a flow like this to fallback to WebFist:

Start with an email address: pithy.example@gmail.com
Translate it into a URL: http://gmail.com/.well-known/webfinger?resource=acct:pithy.example@gmail.com
Fetch that URL and receive a bad response
Formulate a WebFist URL: http://webfist.org/.well-known/webfinger?resource=acct:pithy.example@gmail.com
Fetch the WebFist URL and get back JSON
Follow the delegation path and fetch the real service document

The important thing here is the WebFist look-up server could be any number of peer servers that participate in the fist-bump network. This makes WebFist decentralized, redudant, and load-balanced.

Fist bump

To accomplish decentralization, WebFist servers take delegation emails, encrypt them into blobs, and distribute the blobs safely across a pool of peer servers. These servers synchronize with a "fist bump", transferring just encrypted blobs without secret keys. This makes it near impossible to enumerate every email address in WebFist.

WebFist nodes participate like this:

Receive a verification email, verify its DKIM signature
Use the email address to very slowly generate an AES encryption key with scrypt; the email address is the password
AES128 encrypt the email using the generated key
Save the email to storage, identified by "[scrypt(password)]-[hash(encrypted-email)]"
Publish the list of recently encrypted blobs
Find and share encrypted blobs with peer servers

The only password that will decrypt an encrypted blob is the email address it contains. Brute-forcing an encrypted blob is hard because scrypt is computationally difficult and would take a long time for each attempt. Every WebFist node should have every encrypted blob, eventually. At ~2k per address max, and 10 billion active email addresses, it'll take a node less than 20TB of storage to have the entire state of the world.

As a bonus, WebFist responses include a link to "proof" that a user opted in to WebFist. The proof link is the original encrypted blob that established the WebFist delegation. You can download the blob and see what the rest of the fist-bump network sees (garbage bits). Or you can supply the decrypting password, the email address you were discovering, and see the original email that established the connection.

What's next

We built a prototype of WebFist at IndieWebCamp this year. Thanks to the attendees for all their ideas and brainstorming. There is still a lot left to do, and we'd like your help. Here's a quick list:

Update client libraries to fallback to the WebFist network when WebFinger returns no result
Build adapter tools from your favorite silo that happens to send DKIM-signed emails
Make fist-bumps real-time with PubSubHubbub
Save/cache DNS/DKIM records for each email so future re-verification is possible
Add scrypt caching at all layers of the WebFist server
Add rate-limiting for clients, especially look-up failures
Run your own WebFist lookup node that participates in the first-bump network
JavaScript client support (CORS, etc)

Find us on the WebFinger mailing list.

One Big Fluke

25 June 2013

Bootstrapping WebFinger decentralized discovery with WebFist