Heartbleed highlights a contradiction in the web

The Heartbleed bug that made news last week drew attention to one of the least understood elements of the Internet: Much of the invisible backbone of websites from Google to Amazon to the FBI was built by volunteer programmers in what is known as the open-source community.

Heartbleed originated in this community, in which these volunteers, connected over the Internet, work together to build free software, to maintain and improve it and to look for bugs. Ideally, they check one another's work in a peer review system similar to that found in science, or at least on the nonprofit Wikipedia, where motivated volunteers regularly add new information and fix others' mistakes.

This process, advocates say, ensures trustworthy computer code.

But since the Heartbleed flaw got through, causing fears - as yet unproved - of widespread damage, members of that world are questioning whether the system is working the way it should.

"This bug was introduced two years ago, and yet nobody took the time to notice it," said Steven M. Bellovin, a computer science professor at Columbia University. "Everybody's job is not anybody's job."

Once Heartbleed was revealed, nearly two weeks ago, companies raced to put patches in place to fix it. But security researchers say more than 1 million web servers could still be vulnerable to attack. Mandiant, a cyberattack response firm, said Friday that it had found evidence that attackers used Heartbleed to breach a major corporation's computer system, although it was still assessing whether damage was done.

What makes Heartbleed so dangerous, security experts say, is the so-called OpenSSL code it compromised. That code is just one of many maintained by the open-source community. But it plays a critical role in making our computers and mobile devices safe to use.

OpenSSL code was developed by the OpenSSL Project, which has its roots in efforts in the 1990s to make the Internet safe from eavesdropping. "SSL" refers to "secure sockets layer," a kind of encryption. Those who use this code do not have to pay for it as long as they credit the OpenSSL Project.

Over time, OpenSSL code has been picked up by companies like Amazon, Facebook, Netflix and Yahoo and used to secure the websites of government agencies like the FBI and Canada's tax agency. It is baked into Pentagon weapons systems, devices like Android smartphones, Cisco desktop phones and home Wi-Fi routers.

Companies and government agencies could have used proprietary schemes to secure their systems, but OpenSSL gave them a free and, at least in theory, more secure option.

Unlike proprietary software, which is built and maintained by only a few employees, open-source code like OpenSSL can be vetted by programmers the world over, advocates say.

"Given enough eyeballs, all bugs are shallow" is how Eric S. Raymond, one of the elders of the open-source movement, put it in his 1997 book, "The Cathedral & the Bazaar," a kind of manifesto for open-source philosophy.

In the case of Heartbleed, though, "there weren't any eyeballs," Raymond said in an interview this week.

Although any programmer may work on OpenSSL code, only a few regularly do, said Ben Laurie, a Google engineer based in Britain who donates time to OpenSSL on nights and weekends. This is a problem, he said, adding that the companies and government agencies that use OpenSSL code have benefited from it but give back little in return.

"OpenSSL is completely unfunded," Laurie said. "It's used by companies who make a lot of money, but almost none of the companies who use it contribute anything at all."

According to the project's website, OpenSSL has one full-time developer - Stephen N. Henson, a British programmer - and three so-called core volunteer programmers, including Laurie, in Europe.

Open-source coders hardly blame Henson, considering that the OpenSSL project has operated on a shoestring annual budget of $2,000 in donations - most from individuals - which is just enough for volunteers to cover their electric bills.

Five years ago, Steve Marquess, then a technology consultant for the Defense Department, was struck by the contradiction that OpenSSL was "ubiquitous," yet no one working on the code was making any money. When he met Henson, Marquess said, Henson was working on OpenSSL code full time and "starving."

So Marquess started the OpenSSL Software Foundation to help programmers like Henson make money by consulting for government agencies and companies that were using the code. It also takes in some minimal donations, he said.

Over the past five years, the foundation has never made more than $1 million in commercial contracting revenue a year. This does not go very far in paying for the programmers' work, Marquess said.

Most corporate OpenSSL users do not contribute money to the group, Marquess said. Google and Cisco say they contribute by encouraging their own engineers to look for bugs in the code while they are on the clock. The OpenSSL website shows that a Cisco engineer and several Google engineers have discovered bugs and created fixes over the years.

A Google engineer, Neel Mehta, discovered the Heartbleed bug earlier this month, and two other Google engineers came up with the fix.

Likewise, Microsoft and Facebook created the Internet Bug Bounty initiative, which pays engineers who responsibly disclose bugs in widely used systems like OpenSSL. The group paid Mehta $15,000 for his discovery - a windfall he donated to the Freedom of the Press Foundation.

But open-source advocates say organizations that rely on the code should do more to help.

"Open source is not magic fairy dust," said Tim O'Reilly, an early advocate of open source and the founder of O'Reilly Media. "It happens because people work at it."

© 2014, The New York Times News Service