I recently came across this O’Reilly post about CrunchBase, the open database of information on startup companies, asking whether CrunchBase will remain free in the long term.  While the post itself is interesting, the part that puzzles me is why there is so little business-oriented information freely available out there.  Data as a service has been generating a lot of excitement recently, and I think it’s well warranted.  However, the only prominent sources of open business information are the SEC’s Edgar database, LinkedIn, and CrunchBase.  After that, the field gets very thin.  Considering how much effort companies put into business intelligence and competitive intelligence, it seems like there should be a great profit motive for someone to provide a deeper business information layer.  So I don’t really understand why we’re in this situation, but it does seem like a big opportunity.

These days, most companies are still creating expensive in-house data sets for purposes like market analysis and competitive intelligence. Data providers like Thomson Reuters and CapitalIQ do the same thing and charge through the nose for it.  As a result, big companies end up paying a lot for often not very impressive information. Meanwhile, the rest of the market makes do with sources like Google that just aren’t intended for business research.

The open source analogy

The current landscape reminds me of the days when every big company did custom development for its own IT applications, even though each company’s payroll or data processing system did basically the same thing. That’s good for consultants and developers, but it makes process improvement very expensive for corporations. Similar walled gardens still dominate the business information market.  We need to move to a mindset closer to the open source world. In the last ten years, open source software has become a very competitive option compared to commercial software.  While the core movement is still based on developers writing code based on their personal interests, an important catalyst for the movement has been the rise of companies that add a layer of support and commercialization on top of the core technology. Open source is now a lively market, not just a movement. What’s interesting is that users of open source, from Google and Facebook to NASA, probably contribute more to it now than the companies that sprung up to commercialize it, like Red Hat.  The crowd is contributing to tools that would often be prohibitively time-consuming to build internally or too expensive to license from a vendor.  All the users benefit as a result.

The Wikipedia of business?

Wikipedia and other crowdsourced sites like Wikinvest (a wiki for public company information for investors) are a bit analogous to the open source movement. People make contributions for fun or or to burnish their reputations, not for monetary incentives. The results have been amazing – the English language Wikipedia alone has over 3 million articles. But there are also limits to the phenomenon. Growth in articles is estimated to have peaked in 2006, and you can tell that sometimes articles of marginal interest to the community languish or are even deleted. And as the base of articles becomes more static, it will be interesting to see whether the relatively mundane task of updating all those articles still motivates people.

Wikinvest is another amazing tool, but also one with limits. Interesting companies are updated frequently, but articles on, say, pulp and paper companies tend to be out of date. (As an aside, I get the impression that contributors to Wikinvest are overwhelmingly undergraduate business majors. I wonder what the incentive is. Do corporate recruiters look at the site?)

So there are limits to the open source model for data, particularly commercial data like company information. Somehow writing up a stock analysis doesn’t have the same craft appeal as working on a piece of software.

The commercial data layer

But I do think there’s potential for a real commercial data layer, crowdsourced but with profit and cost savings motives as a driver. There are myriads of players who need information on topics like companies and industries (or who want to sell it), and right now they’re all creating expensive, in-house data sets.  Moreover, much of the data is inaccurate or rapidly goes out of date because maintaining that volume of information requires massive scale. It’s a lot like the software world before first off the shelf and then open source became prevalent.  There are huge economies of scale to be realized with a more consolidated data layer.

Facebook owns the social graph. FourSquare or Yelp will likely do the same with local business information (although Groupon could be well positioned to horn in on it). Right now, no one has claimed pole position in creating this data layer for commercial purposes. There are also lots of aspirants to that position: Thomson Reuters, Capital IQ, Bloomberg, and on and on. I just don’t think any of them are taking the right approach. We need a platform, not a silo.

Once commodity information on companies becomes more widely available, it will be interesting to see what kinds of applications people can build on that information.  Tools like Google Finance, which is quite amazing for a free service, are just the tip of the iceberg. Imagine if you never had to do another bubble chart yourself again! Freeing people from basic research will make it much easier to do deep, thoughtful business analysis instead of spending your time trolling for information online. Basic business information needs to become an open commodity.

• U.S. tries open-source model for health data systems
  • The government and a variety of big companies are working to quickly develop a standard for sharing healthcare information among different providers.  The early version basically appears to be encrypted email.  It sounds ridiculously simple, but maybe getting a basic standard in place and seeing what healthcare users need to help them do their jobs is a better approach than spending years in standard-setting only to realize the specification doesn’t really meet people’s needs.

• A new challenge looks for a smarter algorithm to improve healthcare
  • The Heritage Provider Network, a big primary care network in Southern California, is offering a Netflix-style prize of $3 million to whoever can best predict which patients will end up in the hospital based on anonymous medical data.  The idea is that statistical analysis might catch risk trends that physicians miss, allowing riskier patients to receive better care and thus mitigating the chance of costly hospitalization.
• Can we lower medical costs by giving the neediest patients better care?
  • In New Jersey, Boston, and Atlantic City, a few doctors are taking a much more intensive approach to the sickest patients, trying to minimize extremely expensive chronic hospitalizations.  Even more interestingly, the article mentions companies like Verisk that are attempting to use data analysis to minimize healthcare costs in much more sophisticated ways than usual.
• Massive Health uses big data, mobile phones to fight chronic disease
  • Details are a bit slim, but Massive Health recently got a couple of million in funding from top-tier venture investors like Andreessen Horowitz to help patients manage their chronic diseases with the help of mobile phone data.  The idea is that real-time feedback on their behavior will encourage them to change it, the same way real-time mileage information changes driving patterns.

Of course, these initiatives are all different.  And there are very legitimate reasons to be skeptical of whether they can all be successful, or more importantly whether initial, localized success can be extended to the overall healthcare landscape.

But these initiatives already seem more likely to have an impact than efforts like Google Health and Microsoft Health Vault, both of which seem like platforms looking for a problem to solve.  Most patients simply aren’t that interested in the abstract idea of online health records.  To make inroads with health data, companies need to either focus on helping healthcare providers or create consumer applications that focus on self-improvement rather than healthcare per se.   The paradox, as always, is that the patients who need to be the most vigilant about health issues are usually the hardest to motivate.  That challenge won’t simply wash away in a wave of new technology.  Innovation in medical business models and practices will also be necessary, and progress will probably be slow.

From a technology perspective, we can only hope that the insoluble problems of healthcare decision-making become more tractable over time.  Speech recognition, computer vision, and the tricorder all once seemed like pie in the sky but are now either a reality or just around the corner.  Now that healthcare information is finally becoming digital, perhaps similar advances will finally be possible with the treatment of chronic disease.

First of all, Blekko is a clear counter-example. It’s a viable search option and has a much lower burn rate. DuckDuckGo is also an interesting product, although it doesn’t quite qualify as a contradiction of Thiel’s thesis since it runs partially on APIs like Yahoo’s.

Ignore the Existing Cost Base

More generally, taking the existing cost base as a given in technology markets is pretty foolish, from a couple of perspectives.  Consider the 80/20 rule.  In most areas, Google and Bing have already exceeded user needs and are working on features that provide very incremental benefits.  It might be nice getting to a result a couple of seconds faster with Google Instant, and Google’s spelling correction (which occupies an entire development team) is pretty cool.  However, a new competitor can get to 80% of Google’s spelling correction pretty quickly. No one’s going to miss the remaining 20% if that product provides some truly novel benefit.  Whether DuckDuckGo’s privacy or Blekko’s slashes qualify as something many people want remains to be seen.  But Thiel is ignoring the entire concept of disruptive versus sustaining innovation.

Would Thiel make the same argument in the car industry?  If we looked at Toyota and General Motors, would we assume that starting a car company requires billions in fixed costs?  Clearly not; just look at Tesla.

Fixed Costs Aren’t Really Fixed

We all know that fixed technology go down over time, from hardware to bandwidth to (open source) software.  New upstarts in search will be able to do more with less, especially if they choose to focus on providing value along a specific dimension of customer needs rather than trying to do everything at once. Think about what that $5-10 billion in fixed costs really represents: video search, image search, ad serving, real-time search, and a million other things.  Do you really think every search engine needs to offer all of those features?  Clearly not!

The search field is disintegrating into more specialized applications.  Although it’s in Google and Bing’s interest to keep all of those applications under the hood of one monolithic search engine, that’s clearly not the only model.  I will bet you that Hipmunk is going to beat the pants off Google and Bing for flight search (at least until they get bought), and with much less investment. There’s incredible value in not being tied to an established mindset and infrastructure.  That’s the whole point of startups.

What’s Really Disruptive?

The key for search startups in avoiding the fixed cost trap is that their innovations must be truly disruptive.  They have to satisfy a need that current search engines do not, that users care about, and that is not too tightly coupled to the entire search infrastructure that incumbents have in place.  I think this the main problem for Powerset, the semantic search startup Thiel invested in.  They were targeting the mass market, and thus their strategy was tightly coupled to all the features that mass market users have come to expect.  In addition, generalist users probably didn’t care enough about semantic search to give up other features.  If customers aren’t willing to give up other benefits for your innovation, then it’s not really disruptive.  Powerset might have done better if they had focused on a niche that absolutely had to have semantic features (maybe patent search?) and then expanded over time into other segments.

I’m Biased

Since I’m working on a vertical search engine, I’m hopelessly biased. And I actually think Thiel’s plans to invest in underlying infrastructure technology make a lot of sense. I just wish he hadn’t described the economics of search in such a superficial way.

With that in mind, here’s an only slightly tongue-in-cheek segmentation of Twitter users based on my experience with the service so far.

The Broadcaster

• Broadcasters are on Twitter to promote content, whether their own or someone else’s.  Consciously or unconsciously, many broadcasters see Twitter as being a medium like an email newsletter; one more channel to shoot their message into.  They’ve heard that interacting with other people on Twitter is valuable somehow, but they just haven’t gotten their heads around how or why.  A slightly different type of Broadcaster is looking to reinforce personal status, and their tweets tend to highlight their intelligence, sophistication, and wealth.  If they retweet, it’s typically to show off their good taste or high-status friends.
• 15% of users

The Consumer

• This user comes to Twitter from the newsreader paradigm.  Twitter is a source of interesting information, and the Consumer is here to find that information, not to talk.  In extreme cases, these users are so quiet that it can be difficult to distinguish them from inactive accounts.  Interestingly, information consumers are estimated to make up about 90% of most user-generated online communities.  On Twitter, the number feels much, much smaller, perhaps because the 140 character bar is low enough that more people feel comfortable expressing themselves.
• 10% of users

The Chatter

• This crowd comes to Twitter from the IM / MySpace / Facebook / SMS world.  They send enormous numbers of stream of consciousness tweets to each other.  Check out the Twitter trending topics to get a sense of this world.  I read somewhere that the average teenage girl sends 100 text messages per day.  Twitter is either supplanting or complementing that firehose of communication.
• 50% of users

The Conversationalist

• These people are the uber-marketers on Twitter, in a good way.  They promote their point of view and content like Broadcasters do, but they manage to mix that with a genuine dialogue with their followers.  I’m thinking of people like Tom Peters and Gary Vaynerchuk here.  They somehow overcome the inherent pain of using Twitter as a communication medium, in part by being more savvy about Twitter clients.  They’re amazing to watch, although following them can be exhausting because of the sheer volume of tweets they generate.
• 5% of users

The Faker

• The Faker is the evil twin of the Conversationalist.  They go through the same motions at first glance, but you can tell that the Faker is just trying to get you to buy something.  Repeated links to the same products or sites and a lack of original thought are what give them away, although at the margins the line between Fakers and Conversationalists is a thin one.  Outright spammers probably fall in this bucket as well.
• 20% of users

Now the questions is, how much has Twitter the company thought through these segments?  What do they mean for the business model and the evolution of the medium overall?  I’ll write about that another day, but I’d be interested in your thoughts.

Update: As it turns out, the percentages in this post are way off.  I still think they provide an interesting qualitative view into how prevalent different types of users seem on Twitter, but they don’t really match up to some of the public numbers.  See here for a more updated analysis on Twitter, which is at least slightly more data-driven.