Information overload is in the eye of the beholder and I’d agree that it is largely an anxiety issue and it can drive you crazy, if you grant it that permission.  So much for mental health, but what about the rapid expansion of knowledge in medicine and its positive versus negative effects on physicians, healthcare workers and patients?  Doctors are now faced with so much information that diagnoses take longer to arrive at, and in some cases no diagnosis at all, as more and more specialist tests are called for.  Add to this the under-load of information in the form of electronic medical records where 90% of data capture is still done manually.  No wonder medical liability insurance can cost some physician $500,000 a year or more.  So what I think we see here is disequilibrium of information loads both over and under.

Tropical medicine is a good case to point to here.  The CDC and the London School of Hygiene & Tropical Medicine estimate that the state of our knowledge in this field stands at 3% to 4%.  Globalization and climate change are speeding up the rate of transfer of tropical diseases to Western economies and until very recently the pharmaceutical industry comfortably ignored the problem as belonging to the developing world and inherently unprofitable from an R&D perspective.  Research and eradication has been left to philanthropists like Jimmy Carter and the Bill & Melinda Gates Foundation to begin to address the world’s ten most deadly diseases.  Along with entrepreneurs like Victoria Hale of the Institute of One World Health: http://www.oneworldhealth.org/ which is now licensing patents back to Big Pharma to deploy in Western markets.

Information overload can be a wonderful excuse to turn our eyes away from the real suffering in the world.  Information under load in the form of the low priority placed on geography and foreign languages in the American education system, for example, allows politicians to play to protectionist and xenophobic tendencies in a population ill equipped to address the real issues involved in, say the Dubai Ports deal.  Meanwhile the mainstream media [MSM] is unlikely to embark upon investigative journalism into the Carlyle Group, Halliburton and their ilk, simply because they have bought into the same establishment and become part of the problem.  MSM is very wary of looking for solutions that may jeopardize their license to operate, or as Jon Stewart said at the Time person of the year awards last year “enjoy your scrod.”  MSM will not be overloading us with information any time soon, so onward and upward to all your efforts at Future Monitor.

Amidst all the technofetishism of the ETECH alpha geeks this year, there has been a very encouraging theme of humanising technology and understanding the need for attenuation as a way of protecting the scarce resource of our attention from the onslaught of information overload.

Humanising the Enterprise through Ambient Social Knowledge

Josh, I wish the survey respondents were identified by age—we might know more about the anxiety component. Discussion with two eighteen year-olds yielded one point (in addition to their “all the people you asked were too old for computers”): if you don’t feel responsible for anything, whatever information you you have is enough to do your job.  But people in companies or families feel that the information they know they lack diminishes their personal competence or corporate competitiveness—so “overload” is one name for that concern.

But here’s another idea, related to Josh’s  signal-to-noise discussion, and the notions of density and underload discussed by John and albing [hi Alph]: that the frequency with which new information impinges on us through news tickers, CNN bulletins, RSS feeds, etc.—creates an impression of news where none exists. Nassim Taleb, in Fooled By Randomness, discusses the effect of reading a data stream more or less frequently. He imagines a talented trader who will predictably make a 15% annual return, plus or minus 10%--in other words, in 95% of years, he will earn between -5% and +35%. Taleb points out that if this trader checks his portfolio annually, in 93% of the years he will see a positive return.  But if he checks daily, only 54% of his results will be positive. Checking every second works out to a 50.02% probability of good news. Half the trader’s life would be made miserable.

How much of our information unease comes from the availability of new information at a frequency that conveys more noise than signal? Psychology experiments have shown in various venues that as creatures lose control over their environment, they suffer from anxiety, hypertension, hypertrophied adrenal cortices, etc.  So as we are bombarded by information that seems to tell us that we are not making good decisions, but actually means nothing because it reflects primarily randomness, we may suffer similarly.

What does this suggest for behavior? Separate our lives into the parts that are intended to deal with randomness—traders reacting instantly to market movements are under no illusion that they have an understanding of underlying value—from those dealing with longer term signal, which we should check only infrequently. The application to investing is obvious, but how many times daily should we weigh ourselves if we’re trying to lose weight?  How about our children’s grades?  How about our company’s sales results?

While sales reflects both randomness and many kinds of actual information about market conditions and company performance, few companies use the available statistical tools to separate these.  Economists routinely perform seasonal adjustment to remove a predictable effects like the annual cycle of car buying, apparently buoyed by estival animal spirits, but most companies use much rougher tools.  In 1975 I worked on econometric methods for banks to separate the influence of industry performance on the growth of their loan portfolios, but such analysis is rare.  And simple smoothing and moving averages help damp out randomness, as in our weight.

Of course, technology relies on very fast feedback to achieve real time control—if we cut out that information we lose even more control.  So it would seem that we might be more conscious of the type of information we are receiving. Should we switch from The Wall Street Journal to the Economist? If we could get rid of the obligation we feel to be up to the second regarding information that carries no signal, we might be able to relax.