Quickly: automation is outstripping human common sense.

Content: Two cases are relevant. 1) Google’s dissemination of old United Airlines news resulting in a precipitous UAL stock slide, and 2) exotic, computer-generated financial instruments leading Wall Street into fatal waters.

What happened? In the first case, a system unchecked by human beings made a mistake resulting in massive damage to an already fragile public company. But that’s nothing compared to the wreckage caused by the inscrutable mortgage securities.

My father was a part-time banker who loved to extol the virtues of what he called “small town banking.” It was all pretty simple: his bank took deposits and loaned that money to people whom the bank officer had gone to high school with. This was important, because bankers are not the highest IQ animals on the savanna — they needed good doses of trust and information to sleep at night.

Step forward forty years to sub-prime mortgages. Using complex algorithms, Goldman, Lehman and others created securities that blended mortgages of divergent risk. These securities were supposedly safe, backed by another exotic instrument — credit default swaps. The trust and information of the old days was replaced by computer models and arcane mathematics. The evil great-great computer grandchildren of Hal were in charge.

An east coast, mid-sized commercial bank has avoided this mess. How? In 2006, when the computer-concocted financial instruments were raging, his VPs said, “We have to get into this market.” He said that he would consider, but first he demanded a two hour presentation on how the securities worked. He listened for two hours, then sat up in his chair and said, “We’re not going to do this — because I can’t understand it.”  That wasn’t the case with Wachovia, Washington Mutual, Lehman and others — they had their oxygen supplies cut by Hal (rent 2001: A Space Odyssey if you’re under 40).

Automation and computers do much good in the world. But we have stepped over the line into a world where, without the check of human common sense and understanding, they can do great harm.

What it means one: Apply a simple rule: “If it doesn’t make sense, it doesn’t make sense.” Don’t believe that computer-driven models can reside beyond the realm of human understanding. Either grasp their import, or pull their plug.

What it means two: Risk assessment and management programs (perhaps within Sarbanes) should be placed on alert to identify the Hals in your company. Pinpoint the danger.

What it means three: If you are the leader, never be afraid to say Andy Grove’s favorite business word: “No.”

Let me know if you have any other examples of computing out of control…I’d love to get your thoughts.