When we first started thinking about the concepts that became the Empathy Project, we looked at the potential of Twitter and how we could mine the incredibly rich source of data to get a true real-time picture of what people were talking about in any instant of time, and what topics might trend in the future. This immediately presents a fundamental problem, not only is Twitter huge (over 200m tweets every day) but it doesn't work in a structured way, it is chaotic and has no start or end points. This is a problem for analysis: how do you analyze a chaotic system which doesn't have a beginning, a middle or an end?
The nature of the problem
One of the first things they teach in maths is that when you want to solve a problem, first you have to understand the nature of the problem: is it linear or is it distributed? Linear problems need linear solutions, distributed problems need distributed solutions. Trying to solve a distributed problem with a linear solution is like trying to understand the climate by locating the main cloud in the sky, analyzing that, and then moving on to the next one. You'll spend a long time learning not very much.Twitter isn't like that, Twitter isn't linear, Twitter is distributed.
Trends propagate without any central control or linear direction – it's like a flock of birds, or stock market shares, each actor is independent, influencing the others, but control is distributed throughout the group. There is no boss bird telling the flock which way to go or no big central share guiding the market up and down. Twitter trends are the same, they cannot be traced back along a coherent line to a single source that controls the content; after the first tweet, the thing takes on a life of its own, propagating exponentially.
the Twitter Revolution
The Empathy Project
Predicting the future
We use this understanding to provide a distributed approach to Twitter. We were fortunate enough to work with the Basque government to develop this idea. We built a tool to see a real-time view of what was being said in the Basque Country and what terms were likely to trend before they actually did so. The Software Factory team built on this success with La Voz de Galicia newspaper to develop a tool to track local elections in and listen for mentions of the political parties and personalities, providing a notoriety ranking for each candidate which was converted into a prediction of the final results with remarkable accuracy. The key to this success was understanding that semantics are not the key and that Twitter is a distributed system that needs to be tackled in the same way that we predict the weather: not by looking at one cloud at a time, but having a system-wide understanding of the distributed whole.
Predicting customer demand
The demand curve for many products is fairly easy to predict. There is a fairly constant demand across the year and a seasonal variation: popular authors like Stephen King and groups like U2 sell a fairly constant number of products across the year. Same would be true for a standard umbrella design, a frozen pizza, or a block of A4 paper for example. There will be a constant demand which doesn't change in the medium-term. The seasonal variation is also well understood, we know that people buy more books at Christmas and again in the summer to have something to read on the beach. Ice-cream sales jump in the summer, woolly hats sell more in the winter. So far so predictable, we can anticipate this element of demand and be proactive in ensuring that we have sufficient supply to maximize sales. Stock will be kept roughly equal to the constant demand plus or minus the known seasonal variation.
However the last element is the unpredictable variable - and this is where the Empathy Project comes in. With umbrellas we know what the stimulus is, it's the rain. We don't know when it's going to rain, but we know it will rain and across a timescale of weeks or months, we can predict the variable umbrella demand with reasonable accuracy - we don't need Twitter to tell us it's raining.
However demand for U2 will not change by that sort of stimulus, it will increase when the band tour, or appear on TV, just as Stephen King's sales will jump if he's interviewed or if he turns up on the news ... anything that increases his notoriety. If Julia Roberts walks out in the latest Hunter Boots, demand for those Hunter Boots will go up. This sort of variable cannot be consistently predicted because it is dependent on the notoriety of the item at any given moment in time: this is notoriety demand, highly specific, highly volatile, and a huge opportunity for eCommerce.
This is where Twitter comes in. Twitter is a thermometer for reading what's hot: for reading notoriety. For example, if Julia Roberts and Hunter Boots shoot up the trending topics, clever retailers should be configuring their search engines to hoover up traffic searching using these terms. If Stephen King wins the Nobel Prize for literature his notoriety increases, when someone types "king" into the bookstore search engine, Stephen King should come out ahead of an old out-of-print biography of King George III.
This is how Colbenson Software Factory is innovating and developing SearchBroker, using Twitter as a tool for providing extra notoriety information to search engines. This means that SearchBroker can rank results according to notoriety or provide quick links for high notoriety options, methods which can dramatically increase conversion rates on eCommerce sites. It can even cleverly capture external search engine traffic by making the correct relationship between Julia Roberts and Hunter boots and have a results landing page ready to receive that traffic.
The Empathy Project is an ongoing innovation, continually looking for new ways to make using search engines into a more effective and more personal experience for users, and a tool which directly contributes to organizational objectives.