There are moments in the history of software, when a product created by a tiny team has invented or transformed entire markets.  I was struck by evidence for a similar model in evolutionary biology: the theory of punctuated equilibrium, where there are long periods of relatively little change followed by short and dramatic periods of upheaval.

This kind of rapid change, brought on by an insightful act of creation, has happened repeatedly.  Afterwards, the market leaders usually settle into a period of stability and incremental refinement, backed by large teams.  For example:

UNIX: Ken Thompson and Dennis Ritchie were working at Bell Labs, and became disenchanted with a large-scale project to design a novel operating system, called Multics.  In reaction to its complexity, they created a much simpler alternative and impishly called it “Unics” (later renamed to Unix).  Thompson described Multics as “overdesigned and overbuilt and over everything” (a common complain about systems built by armies!).  Unix went on to become one of the two dominant server operating systems; it is now enhanced and maintained by an army of its own – thousands of developers around the world.

Visicalc: while a student at Harvard Business School, Dan Bricklin got tired of doing spreadsheets on paper.  He thought that he could use a personal computer to help, so he wrote Visicalc for the Apple II and transformed the role of the personal computer in business.  The spreadsheet has long since moved to the other model – Microsoft Excel has dominated the market for many years.

Electronic Arts: The company was originally created by Trip Hawkins in 1982 with a vision: find the brilliant programmers who could create amazing games for small computers, and celebrate those artists and their creations just as we do with great musicians.  One of my friends from junior high school wrote an early EA game (called Axis Assassin).  It was a very impressive homage to his favorite arcade game, Tempest, and he was the sole programmer who built it.  I still have the packaging with his picture and bio in it (at the ripe age of 18).  But that notion of the individual artist was abandoned by EA many years ago.  Now they make games like Madden NFL, which is built by 30 developers and has more than 10 million lines of source code.  The big PC and console games involve massive multi-year investments.  The artisans have moved to mobile, a disruptive domain where very popular games with millions of users are still being created by tiny teams of people.

American culture, I think, has always been caught between a celebration of the small (the lone inventor in a workshop building a better mousetrap, the small farmer setting forth in a Conestoga wagon on the Oregon trail, the “small is beautiful” philosophy of the 70s) and a celebration of the large (the great skyscrapers – “cathedrals of the machine age”, the massive industrial output that powered our economy and made us victorious in the world wars, the space program that landed astronauts on the moon).  Like the technology industry, our culture is tugged back and forth between these two extremes.

The Model

Across all of these examples, I find that there is a somewhat consistent pattern.  Typically, a new opportunity opens up due to some key technology transformation (like the advent of the PC or the Internet), which is not yet dominated by large established organizations.  Some ground-breaking programmers dream up a new kind of product and create it, often as an act of passion rather than of calculation. The new idea gathers popularity, and over time it either grows a large company around it or (occasionally) is taken over by some existing player that wakes up quickly enough and buys or builds their way into a dominant position.

Once a kind of software has become well-established, it usually stops being the realm of the artisan and becomes dominated by large, well-financed teams.  Division of labor was one of the foundational ideas of the machine age, allowing our society to generate massively more output than it ever could have from the labors of loosely organized artisans in their guilds.  The artisans in most industries were washed to the sea by mega-giants.

But division of labor has a great flaw in times of turbulence – it is extremely hard to rapidly re-architect large products or large teams.  At scale, nobody knows the full story of how either one functions.  In software, the products may be millions of lines of code.  The working relationships among hundreds of specialized experts on an engineering team is a tremendously complex system of its own – redesigning it can be even harder than redesigning the software.  Clayton Christenson and others have eloquently written about how hard it is for established players to reinvent themselves.

It’s Not Just Software

This alternation between artisan and army is not restricted to software.  I was fascinated to learn what happened in the 1800’s in manufacturing, elegantly described in the book The Tycoons (which I highly recommend).

During the Industrial Revolution, British industry was transformed by machine-based manufacturing.  Individual artisans were unable to compete and were largely supplanted.  That part I knew.  What I didn’t know, is that there were some key inventions in manufacturing that had the side-effect of making the roles within a factory require far less individual skill and judgment.  The British were slow to adopt them, and most British plant workers were artisans – in the 1890s, three-quarters of them were highly trained crafts-people with a lifetime of expertise.  By contrast, American plants had virtually none – somebody could be trained for almost any role in the plant relatively quickly.

For this and other reasons, American industry production sky-rocketed past the British.  In 1860, US output was one-third of Britain’s.  Fifty years later, it was 2.3 times larger (!).  Manufacturing has remained dominated by mass scale ever since, though there are interesting early signs of another major shift.  The rise of 3D printing technology makes it cost effective to do very small runs – we may see a new renaissance for the artisan manufacturer.

Another example is movie-making.  United Artists was founded in 1919 by four directors and actors, including Charlie Chaplin and Mary Pickford (two of the most popular actors of the day).  The vision, embodied in its name, was to create a place where the artists dominated, not what today we might call the “suits”.  The company struggled because the industry was moving to longer movies with high production values that required large teams and big investments (sound familiar?).

The company, and the broader industry, seesawed back and forth.  In the 1960s, United Artists created the Bond blockbuster franchise.  In the 1970s, they were involved in the shift towards small “auteur” movies that represented the singular vision of a one person (like Midnight Cowboy).  Then back to blockbusters in the 1980s.  Lately, “indie” pictures are all the rage.  The dynamics of the movie industry have interesting similarities to software, shifting back and forth across the spectrum between creation by artisan or by army.

We’re In a Time of Radical Change

As I’ve written elsewhere, because of the cloud and devices, technology is in a time of radical transformation – a lot of equilibriums are being punctured right now.  It’s a hard time to be an established leader – a threat can develop out of nowhere from a couple of passionate developers with a new idea, and it can grow to massive size in the relative blink of an eye.

But it’s a magical time for the artisans – they can challenge the dominions of the giants, tweaking the noses of the biggest companies in the industry.  And if they are right and they have a winning idea, they can have a tremendous impact.  You don’t need an army to change the shape of an industry – you can build a program and, with no capital investment at all, make it available to a billion people in an hour.  If you crack the code and build something that has real demand, there are ready accelerants poised to support you.  Investment capital is abundantly available to companies that have gotten traction, and people with every kind of specialized skill are ready to jump on the train once it has begun gathering speed.  I call it “scale fast or fail fast”.

I believe that during the next several years, many domains of human endeavor will be radically reshaped by small teams of scrappy challengers.  They will seize this period of transformation to forge and pursue new visions that will change the dynamics of whole industries.  It will be fascinating to see what they come up with.