The Programming Elite, Programmers Who Read

About half an hour into the StackOverflow podcast, I started wondering whether I should stop listening to all the navel gazing about the soon to be launched Stack Overflow. Then it got a little more interesting when the discussion veered to the kind of readers/listeners that the site might get after the launch. Joel Spolsky makes an interesting comment (around 29:17):

"The audience of people that read Coding Horror and the audience of people that read Joel on Software are already fairly elite in the programmers, because they are the kind of people who read things in order to better themselves as programmers. And that's already, you know, 5-10% of practising programmers. It's not the vast masses of Java monkeys who were formerly VB monkeys who were formerly COBOL monkeys who are just doing, you know, large swathes of extremely boring stuff internally somewhere. Ahh, Who have I not offended ?"

Joel also adds a little later, "Don't bother writing in, I will just commit suicide."

Seriously though, Joel's comment struck a chord with me. The programmers who read online, especially technical stuff unrelated to their work are a minority. The ones who read books are an even smaller group. Tom DeMarco and Timothy Lister's classic "Peopleware" has the following to say about reading habits of programmers

"The statistics about reading are particularly discouraging: The average software developer, for example, doesn't own a single book on the subject of his or her work, and hasn't ever read one. That fact is horrifying for anyone concerned about the quality of work is the field; for folks like us who write books, it's positively tragic."
- From Peopleware, Productive Projects and Teams by Tom De Marco and Timothy Lister (2nd Ed, page 12)

Paul Graham On Why Some Popular Web Sites for Nerds Went Downhill

I don't think it's as hard to keep a site from sliding as one might think from the examples of previous sites where things went downhill as they got more popular. Slashdot, Digg, and Reddit were all companies. They wanted to grow. Whereas News.YC is a side project. We don't care about growth. It's much easier to do things to keep up the quality when you're willing to sacrifice growth.
-Paul Graham in a comment on Hacker News

Steve Jobs On Recruiting

"Recruiting is hard. It's just finding the needles in the haystack. We do it ourselves and we spend a lot of time at it. I've participated in the hiring of maybe 5,000-plus people in my life. So I take it very seriously. You can't know enough in a one-hour interview. So, in the end, it's ultimately based on your gut. How do I feel about this person? What are they like when they're challenged? Why are they here? I ask everybody that: 'Why are you here?' The answers themselves are not what you're looking for. It's the meta-data."

- From an interview with Fortune

Professional Stereotypes and High Performers

In response to Paul Graham's essay A Fund Raising Survival Guide, Gordon Mohr has posted a fascinating insight in the comments:

The very last sentence of the last footnote caught my eye:

Oddly enough, the best VCs tend to be the least VC-like.

I suspect that rather than being odd this is nearly tautological for any profession -- "the best X tend to be the least X-like". Professional stereotypes are set by the multitudes in the middle, not the highest-performing outliers.

Further, atypical behavior can be both a cause and effect of excellence. Being 'different' helps them be 'better', but also by being 'better' they gain freedom and confidence to deviate from norms.

(Of course, "the worst X tend to be not very X-like" is also true. But they're more likely to at least try to emulate the average X.)

- (via Hacker News)

Joshua Schachter on the Yahoo del.icio.us Rewrite

The old, perl codebase was a pain -- I should know, I wrote it.

...

of course, this was just integration with an off the shelf search solution that was available internally. but engineering was focused on the rewrite so nobody had time to replace the old search engine (xapian, which did an admirable job, but not at that scale)

...

The writer is accidentally correct - we were told that it had to be in PHP to get ops support. Curiously the PHP part didn't take that much time - the majority of the "business logic" is in C++ which took forever and ever to write. I think the open question now is whether the remaining team will be able to innovate or be stuck in complicated codebase hell.

...

Right. The PHP part was not the time sink. On the other hand, the general rewrite took forever.

...

They had the technology, but the relevant staff had higher priorities.

(via reddit)

While it seems to be par for the course for big companies to screw interesting start-ups after a take over, these comments are pretty revealing all the same. Rewrite software to make it easier for ops ? As more and more software moves to the data center, ops is becoming one of the potential pain points and political power centers. I have been ranting about this to a couple of friends and don't seem to be getting my point across. I think one way out of this mess is a new breed of programmer administrators and automation tools like Puppet.

IBM Executive To Linux Desktop Developers: 'Stop Copying Windows'

IBM, whose decision to back Linux years ago was a driving force in its adoption by business, called on developers of the open-source operating system to make it more "green" and to stop copying Windows, if they want to see Linux on the desktop.

Bob Sutor, VP of open source and standards at IBM, told attendees of the LinuxWorld Conference in San Francisco, that what the open source community needs to make Linux popular as a desktop OS used by consumers and businesses are "some really good graphic designers."

"Stop copying 2001 Windows. That's not where the usability action is," Sutor said during his afternoon keynote.

- Information Week

Wow ! I suspect most reactions are going to be predictable. The real problem though, might be that "some really good graphic designers" are not the only factor that could result in a great Linux desktop. Linux development is now heavily influenced by corporations that are not known for any great user facing design efforts. Great UI work does not seem to lend itself to the kind of distributed development style that has made many open source projects successful. Successful open source projects which are usable by the ordinary user are few and far between. Good UI requires a different mindset as John Gruber explained once in an excellent post:

UI development is the hard part. And it’s not the last step, it’s the first step. In my estimation, the difference between:

• software that performs function X; and
• software that performs function X, with an intuitive well-designed user interface

isn’t just a little bit of extra work. It’s not even twice the work. It’s an entire order of magnitude more work. Developing software with a good UI requires both aptitude and a lot of hard work.

...

It’s not something every programmer can learn. Most programmers don’t have any aptitude for UI design whatsoever. It’s an art, and like any art, it requires innate ability. You can learn to be a better writer. You can learn to be a better illustrator. But most people can’t write and can’t draw, and no amount of practice or education is going to make them good at it. Improved, yes; good, no.

Conversely, some people who are good UI designers aren’t programmers. But the rock stars are the guys who can do both, and they are few and far between.

If there’s a glib, nutshell synopsis for why Linux desktop software tends to suck, it’s this: Raymond and his ilk have no respect for anyone but themselves.

They have no respect for the fact that UI design is a special talent.

They have no respect for the fact that good UI design requires a tremendous amount of time and effort.

And, most importantly, they have no respect at all for real users. The idea that GUI software needs to be designed for “dumb users” — which is Raymond’s own term, and an indication of what he really means when he refers to dear old A.T. — is completely wrong.

Great software developers don’t design for morons. They design for smart, perceptive people — people just like themselves. They have profound respect for their users.

...

Movies are collaborative art, but require strong direction. So it is with end user software.

Gruber's post is so good, that I run the risk of reproducing it in its entirety.

Funny Perl Quote

My only argument against perl would be it looks like a chicken ran across your keyboard.

- Michael Whalen's comment on a blog post titled In defense of Perl

Linux Kernel Development Stats from Greg Kroah Hartman

Linux kernel hacker Greg Kroah Hartman's June 5, 2008 talk at Google titled "The Linux Kernel" was chock-full of details about kernel development. I noted down some of the the things he said. Please note that the talk was delivered on June 5th, 2008 and all stats mentioned by GKH are relative to that date.

• Code changes per day in 2007-2008 so far
  
  • 4,300 lines added
  • 1,800 lines removed
  • 1,500 lines modified
    
  These changes don't include moving files around.It works out to about 3.69 changes per hour, 24x7 . It's not just the drivers that are changing at this rate, it's the the entire kernel.
  
• The kernel itself is about 9.2 million lines, and has been increasing in size by about 10% every year since 2.6.0 (when GKH started tracking it). The drivers make up about about 55% of the code, while architecture specific code is second in terms of LOC. The core of the kernel is about 5%.
• Supports more processors and devices than any other OS in history.
• One of the consequences of the scorching pace of the kernel development is that, the kernel is a better place for patches , than for it to be maintained separately. GKH pointed out that it is difficult to keep pace with Linux kernel development and gave the example of Xen; The Xen folks apparently never played nice with the kernel developers and are now trying to get their patches into the kernel. From his tone it sounded as thought it was a tough climb uphill for the Xen guys, and the KVM is already in the kernel.
• The Kernel development hierarchy (it has one !) has developers feeding patches to the maintainers, who in turn feed them to subsystem maintainers (pci, security, usb, etc).Sub-system maintainers maintain their own trees. Steve Rothwell (IBM, Austalia) pulls changes into the "Next" tree every night and does daily builds, while Andrew Morton pulls changes into his tree once a week or so. When Linus says the merge tree window is open, all the sub-system maintainers hit Linus. There is also the stable release tree, which is not maintained indefinitely.
• He likened the patch submission process to a lossy network routing algorithm - it can handle people dropping out . There is no other way to develop at such high speed.If you own a file or subsystem, you have to accept the fact that other people are going to be changing it. Maintainers can always revert changes if they don't like it.
  
• There is no good way to test the kernel except to run it. The hundreds of permutations of devices and interactions makes it impossible to test it comprehensively. The only way out is for the developers to test the rc releases.
• There are no stable and unstable releases anymore. For the last four years,they have been replaced by releases every 2 and 3/4 months.
  
• There have been 2399 independent contributors to the Linux kernel in the last year. 50% of the contributors submitted only one patch; half of the half contributed two patches. the top of the curve is getting flatter. Top 30% do only 30% of the work. The number of individual contributors is going up.
  
• Top developers by quantity (for the last one and half years)
  
  1. Adrian Bunk 754
  2. Al Viro 698
  3. Thomas Gleixner 656
  4. David S.Miller 655
  5. Bart Zolnierkiewicz 637
  6. Paul Mundt 610
  7. Ralf Baechle 604
  8. Ingo Molnar 596
  9. Patrick McHardy 554
  10. Tejun Heo 530
• Top contributors by sign-offs (shows who the gatekeepers are). Note that Linus doesn't sign off on patches from subsystem maintainers.
  
  1. Andrew morton 9086
  2. Linus Torvalds 8960
  3. David S.Miller 4926
  4. Jeff Garzik 2960
  5. Ingo Molnar 2489
  6. Greg Kroah-Hartman 2098
  7. Thomas Gleixner 1098
  8. Mauro Carvalho Chehab 1822
  9. Paul Mackerras 1675
  10. John Linville 1461
• Who's funding Linux kernel development ?
  
  1. Amateurs 18.5%
  2. Red Hat 11.6%
  3. IBM 7.5%
  4. Novell 6.6%
  5. Unknown individuals 5.5%
  6. Intel 4.1%
  7. Oracle 2.2%
  8. Consultants 2.2%
  9. Academia 1.5%
  10. Renesas Technology 1.5%
  Google is at number 13 with 1.4% contribution. Without Andrew Morton's contributions Google's would be at the fortieth spot.
• Canonical had about 6 changes in the past 5 years; they are in the 300th
  position. GKH was very emphatic that 'Canonical does not give back to the community'.
• 75% of Linux kernel work is paid for.
• Linus jokes that the kernel is not intelligent design, it is evolution. GKH also added that "We react to stimuli that's happening in the world" and "We don't over plan things".
• As GKH put it, 'We broke all software development rules and we are continuing to break it'.

The Importance of Apple

Recently at work I couldn't help chuckling and shaking my head as I went through a list of requirements for a well known MNC - buried in the middle of was 'Oracle support' - there, that's an enterprise customer, I thought. It's one of those 'enterprise' requirements that clients who work in multi-storeyed glass and concrete buildings with large cube farms feel concerned about regardless of whether it makes any real difference to their work. This was not the first time that I had heard users asking for Oracle support. It's all about making the right noises, which is what enterprise software is all about. I was more annoyed with the business development folks for not having nipped this in the bud than with the customer. A couple of days later I was reading an interview with the author of Cocoa Programming for Mac OS X when I came across this heart warming section:

Scott: The Mac appears to be making real inroads in the mainstream consumer computing market, and certainly the iPhone is doing the same. Do you expect to see this carry over to the corporate world?

Aaron: Apple seems to doing its best to keep Macs out of the corporate world. Most dialogues between Apple and a corporation go something like this:

Corporation: Hey, does Cocoa include something to help us write apps that talk to our Oracle database?

Apple: Um, nope. But isn't GarageBand cool?

Corporation: Hmm.

Indian IT Companies and Internet Access at Work

Shyam points out the hypocrisy of a TCS executive advocating net access at work while denying the same to his own employees. A couple of years back one of my ex-colleagues along with one of the sales guys went to TCS to make a sales pitch. Using our product meant all your data was hosted off-site; the data could be accessed using a browser based interface, as well as command line tools and GUIs. They spent a couple of hours explaining the details of this product, and answering all kinds of questions about edge cases and close to impossible usage scenarios. They were feeling a little exasperated about the oddness of the questions, especially, since they were informed that this was an audience of senior managers. Finally they learnt that it would be difficult for the TCS staff to use the product, since they did not have Internet access. My ex-colleagues were more than a little peeved about their time being wasted.

IT work in recent years has become very dependent on access to search engines, online forums, mailing lists and IRC groups. Why try to fix something from first principles, when you can paste the error message into a search engine and get an easy answer ? Makes sense in most cases, especially crappy software, which means most software. It makes little sense to attempt to understand the workings of twisted minds that develop such software. Add to this the amount of open source software in use at these days. The very culture of open source means you will never really be able to use it meaningfully without internet access.

Management concerns about internet access cannot be wished away either. It seems to revolve around a couple of key concerns.

• Downloading and installation of various kinds of malware by employees and subsequent IT resources and time spent on cleaning up the network.
  
• Browsing sites unconnected to work and chat. I have to admit that, I find the amount of time that some recent graduates spend on social networking sites and chat borders on the pathological. And it's not always for lack of work to do. It's a huge time sink.
  
• Legal obligations of employers - what if the employees are surfing porn sites ? That could be construed as sexual harassment by their female colleagues. Isn't the employer liable in this case ?
• Data security has become a huge concern with management - nobody seems to have a good solution to this problem. There seem to be a few back office type operations that apart from blocking net access disable access to USB drives and such. There have been quite a few scams that seem to have made managements paranoid.
  

Employer trust is not something that is always reciprocated by employees. A liberal employer is often misunderstood to be a weak personality, who can be taken advantage of. Misuse of office facilities is a reality that cannot be wished away.

There are no hard answers to a complex techno-social problem like this. An evolutionary approach is needed that takes into account the concerns of both employers and employees. Employers need to realise that, increasingly, internet access is not a priviliege that they 'allow' employees to enjoy; In many cases such as software development it helps to work effectively. Allowing employees to finish domestic chores such as paying bills, check their bank accounts, or keep in touch with their friends and relatives (in moderation) contribute to productivity at work.

I once heard from a consultant in Bangalore, that his client not only had him setup a proxy server for net access, but also had him setup an intranet site where everyone's browsing records were publicly available. This seems like an interesting approach that could work in some cases. In any case I don't have a problem with employers monitoring my internet activities. Employers should make their policy on net connectivity very clear; what's acceptable and what's not should be explained; new employees should have it clearly explained to them that net access is not a 'given' and that their activity is being monitored. Most companies do not have a policy on this matter and their only solution is a blanket ban on any kind of access, with a patronising common 'net PC' thrown in.

OLPC - Trying to make Water flow Downhill

I don't understand why we need a program to spend millions of dollars to make laptops cheaper, when laptops get cheaper everywhere. I mean, that would be like setting up a world bank program to make water flow downhill.
- Dr. Joel Selanikio co-founder of DataDyne in an interview with Jon Udell

That must be one of the funniest and most insightful comments on the OLPC project. I suspect it's actually one of the most nagging questions that bothers folks who are not evil.

Virtualization, The Jamie Zawinski Remix

Some people, when confronted with a problem, think "I know, I'll use virtualization." Now they have two problems.

(Jamie Zawinski - Wikiquote)

Automated Testing and Programming Skills

Joel Spolsky's lecture at Yale university makes some interesting points about the dangers of automated testing. Joel is good at latching on to the unintended consequences of latest trends, especially the kind that nerds with Asperger's typically overlook. He speculates that 'automatic testing religion' was what did Vista in. There is an interesting bit about the fate of testers without programming skills at Microsoft:

Microsoft has had a long term policy of eliminating all software testers who don’t know how to write code, replacing them with what they call SDETs, Software Development Engineers in Test, programmers who write automated testing scripts.

I can sympathise with this, having interacted with testers without programming skills myself. If Microsoft's policy were to be adopted in India, there would be quite a few unemployed testers. Joel's article is a helpful reminder of the dangers of overdoing automated testing. But I think the situation in India represents the other extreme - we have lots of testers without any real skills just trying to fill up bug trackers and warming chairs in head-count based projects. Automation is the last thing that concerns either testers or software companies in India. After the dot-com bust, testers have become an overvalued commodity in Indian software companies with clueless managements.

John Gilmore's test of a good programmer

John Gilmore suggested to me a test of a good programmer: one who has written a piece of software that at least 1000 people have downloaded from the Internet.

- Arun Mehta

OLPC Laptop Application Virtualization

I was listening to a fascinating presentation (video)by Ivan Krstić, the director of security architecture of the One Laptop Per Child (OLPC) project. What struck me most was the extent to which virtualization is being used. Unless you have been living under a rock, you know that virtualization is a 'hot' area now. But this must be one of the few attempts to use it to provide desktop security (Incidentally Anti-virus tools routinely use virtual machine containers) The Bitfrost security model involves running each application in its own virtual machine container. As the Wikipedia entry explains,

Every program, when first installed, requests certain bundles of rights, for instance "accessing the camera", or "accessing the Internet". The system keeps track of these rights, and the program is later executed in an environment which makes only the requested resources available. This is implemented by a fully-fledged, container-based virtual machine.

By default, the system denies certain combinations of rights; for instance, a program would not be granted both the right to access the camera and to access the internet. Anybody can write and distribute programs that request allowable right combinations. Programs that require normally unapproved right combinations need a cryptographic signature by some authority. The laptop's user can use the built-in security panel to grant additional rights to any application.

Running each application inside a virtual machine is a scary idea, especially given the hardware configuration of the OLPC laptop. Krstic explained that the Linux kernel VServer patch had been extended to do OLPC specific tasks. Perhaps, the most fascinating piece of information that I learnt from the whole talk was the low overhead associated with such Linux VServer virtual machines. Quote from Krstic's presentation:

"The interesting thing about this, by the way is you know people are terrified of how are you going to do virtualization with 466MHZ CPU - turns out with Linux Vserver that the overhead you pay is 32 K per task struct, but there is 0% measurable CPU overhead with up to 65,000 Virtual Machines running ."

I have mixed feelings about this approach. Having done virtualization related work for the last two years, I find this approach fascinating;but, it could also start a disturbing trend of people opting for this approach instead of trying to solve security problems at a more fundamental level. But I got to admit that the clever Copy-on-Write approach of the VServer just blew me away. As Krstic said after explaining the low overhead associated with the virtual machines,

"I will let that sink in for about two seconds"

It's not as bad as the Sendmail configuration file

But the Apache web server configuration file is pretty close. I once heard one of the founders of the Apache project confess that, the only configuration file worser than it was the one used by Sendmail; I mean, that's not saying much, is it ? I wasted the whole afternoon yesterday wrestling with the VirtualHost directive. Finally, I threw in the towel and got it working in five minutes with twisted web. It's httpd.conf that's twisted. All I wanted was a couple of web servers on my laptop to test my program. The worst part - it makes you feel so stupid for no fault of your own.

The Glorious History of HR in Software Companies

Mitch Kapor's wife Freada was in charge of HR at Lotus in the early years. (As he is at pains to point out, they did not become romantically involved till afterward.) At one point they worried Lotus was losing its startup edge and turning into a big company. So as an experiment she sent their recruiters the resumes of the first 40 employees, with identifying details changed. These were the people who had made Lotus into the star it was. Not one got an interview.
- Paul Graham in a footnote to his essay News from the front

Extreme Pair Programming - Guy Steele and Richard Stallman

"We sat down one morning," recalls Steele. "I was at the keyboard, and he was at my elbow," says Steele. "He was perfectly willing to let me type, but he was also telling me what to type.

The programming session lasted 10 hours. Throughout that entire time, Steele says, neither he nor Stallman took a break or made any small talk. By the end of the session, they had managed to hack the pretty print source code to just under 100 lines. "My fingers were on the keyboard the whole time," Steele recalls, "but it felt like both of our ideas were flowing onto the screen. He told me what to type, and I typed it."

The length of the session revealed itself when Steele finally left the AI Lab. Standing outside the building at 545 Tech Square, he was surprised to find himself surrounded by nighttime darkness. As a programmer, Steele was used to marathon coding sessions. Still, something about this session was different. Working with Stallman had forced Steele to block out all external stimuli and focus his entire mental energies on the task at hand. Looking back, Steele says he found the Stallman mind-meld both exhilarating and scary at the same time. "My first thought afterward was: it was a great experience, very intense, and that I never wanted to do it again in my life."
- Guy Steele on a hacking session with RMS in the 1970s.

Beautiful Code - The Paris Hilton Connection

The odds of finding truly beautiful code in most production systems seem to be on par with the odds of finding a well-read copy of IEEE Transactions on Software Engineering in Paris Hilton’s apartment. Furthermore, enterprise software, which represents the bulk of code in existence, deals mostly with forms, reports, etc., which – one might argue – seldom require much thinking or cleverness.

- Albert Savoia in a post on O'Reilly's Beautiful Code blog

It's not just enterprise software - you find bad code in lots of startups, where there is an exclusive focus is on getting things done rather than doing it correctly. In fact, even lots of open source code bases are full spaghetti code. A great hacker even recommends FreeBSD code over Linux code.

Sadly, a lot of Linux and Gnome open source is poorly written.

- Trevor Blackwell in response to 'How can I become a better programmer ?'

Startups are not Democracies

If you are not one of the founders of a startup, you really need to aware of this. I wish someone had reminded me about this :-( .

Startups are not a democracy. Want a democracy? Go run for class president, Bueller.
- From Mark Fletcher's 15 Startup commandments