True value comes from understanding what information means, not just what it says.
Individual bits of data are relevant to specific purposes, but far deeper meaning comes from examining the relationships between them.
For example, a bank account number is essential information for every financial transaction, but alone it's pretty boring.
However, when it's related to a payee's name and an amount, it becomes much more interesting. And when that transaction is then related to the content of a purchase (say, an eBook on a given topic), it becomes more interesting still.
Put together enough information like that and you can build quite a detailed picture of someone.
That's something we know the government and large corporations do, much to their benefit. The question we asked is, "Can we give Merlin users that ability, but privately"?
In other words, can we design an analysis engine that gives a Merlin user the same sort of deep understanding of their personal information by discovering previously hidden relationships within their own data? What would it take to give people a super-private, personal analytic engine?
We experimented with algorithms and tested data thresholds to determine what kind and how much raw data it takes before previously undiscovered relationships between the people, places and things you know might yield non-trivial observations...
We found that simply by matching basic information such as names, physical and electronic addresses, and simple topics, we could discover surprising relationships across a remarkably modest amount of data. It turns out that having just a few hundred emails and text messages is often enough, let alone the many thousands most of us have in a year.
Here's a simple case in which Merlin lets you know that seemingly disconnected people actually know each other.
Let's say that your Merlin contacts include family, friends, neighbors, business associates and customers.
You use Merlin to communicate with them all, mostly by regular email, but also by secure voice/text/email with those who also have Merlin. Emails and messages flow in and out every day. Some of them are direct, and some are notes cc'ed to groups.
As Merlin watches it all for you (and only you!) it notices that deep in the cc list of an email you got from a relative there was a name and email of someone Merlin hadn't seen before.
You didn't notice it, why should you? You don't even know that other person, and besides, there were 17 people cc'd. This happens all the time. But Merlin notices and remembers, it just doesn't do anything... yet.
Sometime later, a customer with whom your having a secure message chat mentions that same name. It means nothing to you, but it means something to Merlin, which knows that 16 months ago your relative cc'd an email to someone with that name.
Suddenly Merlin can advise you that a relative knows someone who in turn might know your customer. So Merlin notifies you by highlighting the name in chat and showing you the email address it collected back then. Then it suggests you gently probe the person with whom you're chatting for the email address of the person they just mentioned. If it's the same, Merlin will alert you.
Perhaps this discovery is meaningless. Or perhaps it's a useful opportunity to solidify your business relationship. Maybe it's even potentially harmful. There's no way for Merlin to tell, but at least now you know there's a new way to connect with your customer.
This is a pretty simple example; Merlin is doing much more sophisticated stuff by constantly looking for connections in everything you do. That's one of the reasons it can file messages and emails so intelligently, an amazingly useful ability we talk more about in the Born of frustration blog.
What makes this possible is not so much the algorithms we use, although they are pretty cool.
Rather, the real magic lies in the simplest fact of all: Merlin's security features1 make it safe to consolidate all your information in one place. And that creates a landscape broad enough for it to discover previously undetected relationships.
We've coined a couple of new words to describe this...
The more you use Merlin the more amazing its discoveries become.
1. Multi-factor login, multiple personas, onion-like layers of encryption, and plausible deniability.