One Reason Why Software Is Expensive

Joel Spolsky recently spoke at Y Combinator about selling software to corporate customers. He said that in most companies software costing up to about $1000 could be bought by individual managers without any additional approvals. Above that threshold, software purchases generally had to be approved by a committee. But babysitting this process was so expensive for software vendors that it didn't make sense to charge less than $50,000. Which means if you're making something you might otherwise have charged $5000 for, you have to sell it for $50,000 instead.

The purpose of the committee is presumably to ensure that the company doesn't waste money. And yet the result is that the company pays 10 times as much.

- Paul Graham in 'The Other Half of "Artists Ship"'

How Fortran Was Developed

While going through a presentation (pdf) by John Anderson of the University of Edinburgh at LISA'08, I came across a fascinating quote:

As far as we were aware, we simply made up the language as we went along.We did not regard language design as a difficult problem, merely a simple prelude to the real problem: designing a compiler which could produce efficient programs.
- John Backus, Developer of Fortran and inventor of BNF

Googling a bit led to Backus' paper (pdf) from which the quote had been taken.

From The Horse's Mouth: The Free Desktop Has Been In 'Catchup' Mode

The free desktop has been in “catchup” mode: catching up to first Windows and now nipping at the heels of the Mac. Our path has been obvious to date. In some areas, our technology and applications lead; in others we still lag. From here on, progress becomes much less clear, though I’ll bet on the moving herd and natural selection of free software over directed closed commercial development any day.

How now to move from such a reactive strategy to true leadership in all areas? How do you set strategy, when our very culture is that of serendipity, discovery, sharing of ideas, and creation? where a single vision cannot rule?

- From Jim Gettys' post 'Time To Lead ...'

In case you are a newbie or one of those irrational defenders of crappy open source desktops, please check out the Wikipedia entry for Jim Gettys. No wait, I will save you that click with this one line from that Wiki entry:

He is one of the original developers of the X Window System at MIT and worked on it again with X.Org, ...

Andy Bechtolsheim On a Common Start-Up Mistake

Lean staffing also helps Arista keep its costs down. The Menlo Park, Calif., company has fewer than 50 employees and started shipping systems a few months ago even though it had no formal chief executive.

“One mistake a lot of start-ups make with the encouragement of venture capitalists is to hire the whole management team upfront,” said Mr. Bechtolsheim. “You have a lot of people twiddling their thumbs and spending money.”

- via NYT article Sun Loses Co-Founder to Start-Up

A Marketing Guy Nails the Problem with Software Companies

Watching Steve Johnson's short six and half minute presentation titled 'Software:Business or Hobby' (video) at the Business of Software Conference (2007), I was struck by something he said (around 04:13):

One of the challenges, I think that we all face is so many people in the organization are making decisions about that they are not qualified to make decisions about. Nothing seems hard to the people, who don't know what they are talking about.

- Steve Johnson of pragmatic marketing

You can watch the whole presentation here.

Mercurial vs Git: The Biggest Non-Technical Difference

The biggest non-technical difference between git and mercurial is the rabid culture surrounding git. mercurial users fairly happily and quietly use their tool, while I've had to send two separate door-to-door git missionaries away today alone.
- Dustin Sallings

Software Development - Fire Fighters vs Real Heroes

In a Dilbert cartoon, the pointy-haired boss, apparently frustrated by the company's sub-par products, announces that he'll reward each bug fix with a $10 bill. Wally says: "Hooray! I'm gonna code me a minivan!"

Unfortunately the heroes, those who seem to save the organization in a great flurry of activity, are often reacting dramatically to the problems they created. Like Wally, they're rewarded for the successes while no one notices that furious activity is no substitute for doing things carefully.

Solving problems is a high-visibility process; preventing them is much better, but earns few rewards. This is illustrated by an old parable:

In ancient China, there was a family of healers, one of whom was known throughout the land and employed as a physician to a great lord. The physician was asked which of his family was the most skillful healer. He replied, "I tend to the sick and dying with drastic and dramatic treatments, and on occasion someone is cured and my name gets out among the lords.

"My elder brother cures sickness when it just begins to take root, and his skills are known among the local peasants and neighbors.

"My eldest brother is able to sense the spirit of sickness and eradicate it before it takes form. His name is unknown outside our home."

Unfortunately, sometimes the very best developers get the least acknowledgement, even from their own teams.
- Jack Ganssle, Embedded development expert

The Programming Elite, Programmers Who Read

About half an hour into the StackOverflow podcast, I started wondering whether I should stop listening to all the navel gazing about the soon to be launched Stack Overflow. Then it got a little more interesting when the discussion veered to the kind of readers/listeners that the site might get after the launch. Joel Spolsky makes an interesting comment (around 29:17):

"The audience of people that read Coding Horror and the audience of people that read Joel on Software are already fairly elite in the programmers, because they are the kind of people who read things in order to better themselves as programmers. And that's already, you know, 5-10% of practising programmers. It's not the vast masses of Java monkeys who were formerly VB monkeys who were formerly COBOL monkeys who are just doing, you know, large swathes of extremely boring stuff internally somewhere. Ahh, Who have I not offended ?"

Joel also adds a little later, "Don't bother writing in, I will just commit suicide."

Seriously though, Joel's comment struck a chord with me. The programmers who read online, especially technical stuff unrelated to their work are a minority. The ones who read books are an even smaller group. Tom DeMarco and Timothy Lister's classic "Peopleware" has the following to say about reading habits of programmers

"The statistics about reading are particularly discouraging: The average software developer, for example, doesn't own a single book on the subject of his or her work, and hasn't ever read one. That fact is horrifying for anyone concerned about the quality of work is the field; for folks like us who write books, it's positively tragic."
- From Peopleware, Productive Projects and Teams by Tom De Marco and Timothy Lister (2nd Ed, page 12)

Paul Graham On Why Some Popular Web Sites for Nerds Went Downhill

I don't think it's as hard to keep a site from sliding as one might think from the examples of previous sites where things went downhill as they got more popular. Slashdot, Digg, and Reddit were all companies. They wanted to grow. Whereas News.YC is a side project. We don't care about growth. It's much easier to do things to keep up the quality when you're willing to sacrifice growth.
-Paul Graham in a comment on Hacker News

Steve Jobs On Recruiting

"Recruiting is hard. It's just finding the needles in the haystack. We do it ourselves and we spend a lot of time at it. I've participated in the hiring of maybe 5,000-plus people in my life. So I take it very seriously. You can't know enough in a one-hour interview. So, in the end, it's ultimately based on your gut. How do I feel about this person? What are they like when they're challenged? Why are they here? I ask everybody that: 'Why are you here?' The answers themselves are not what you're looking for. It's the meta-data."

- From an interview with Fortune

Professional Stereotypes and High Performers

In response to Paul Graham's essay A Fund Raising Survival Guide, Gordon Mohr has posted a fascinating insight in the comments:

The very last sentence of the last footnote caught my eye:

Oddly enough, the best VCs tend to be the least VC-like.

I suspect that rather than being odd this is nearly tautological for any profession -- "the best X tend to be the least X-like". Professional stereotypes are set by the multitudes in the middle, not the highest-performing outliers.

Further, atypical behavior can be both a cause and effect of excellence. Being 'different' helps them be 'better', but also by being 'better' they gain freedom and confidence to deviate from norms.

(Of course, "the worst X tend to be not very X-like" is also true. But they're more likely to at least try to emulate the average X.)

- (via Hacker News)

Joshua Schachter on the Yahoo del.icio.us Rewrite

The old, perl codebase was a pain -- I should know, I wrote it.

...

of course, this was just integration with an off the shelf search solution that was available internally. but engineering was focused on the rewrite so nobody had time to replace the old search engine (xapian, which did an admirable job, but not at that scale)

...

The writer is accidentally correct - we were told that it had to be in PHP to get ops support. Curiously the PHP part didn't take that much time - the majority of the "business logic" is in C++ which took forever and ever to write. I think the open question now is whether the remaining team will be able to innovate or be stuck in complicated codebase hell.

...

Right. The PHP part was not the time sink. On the other hand, the general rewrite took forever.

...

They had the technology, but the relevant staff had higher priorities.

(via reddit)

While it seems to be par for the course for big companies to screw interesting start-ups after a take over, these comments are pretty revealing all the same. Rewrite software to make it easier for ops ? As more and more software moves to the data center, ops is becoming one of the potential pain points and political power centers. I have been ranting about this to a couple of friends and don't seem to be getting my point across. I think one way out of this mess is a new breed of programmer administrators and automation tools like Puppet.

IBM Executive To Linux Desktop Developers: 'Stop Copying Windows'

IBM, whose decision to back Linux years ago was a driving force in its adoption by business, called on developers of the open-source operating system to make it more "green" and to stop copying Windows, if they want to see Linux on the desktop.

Bob Sutor, VP of open source and standards at IBM, told attendees of the LinuxWorld Conference in San Francisco, that what the open source community needs to make Linux popular as a desktop OS used by consumers and businesses are "some really good graphic designers."

"Stop copying 2001 Windows. That's not where the usability action is," Sutor said during his afternoon keynote.

- Information Week

Wow ! I suspect most reactions are going to be predictable. The real problem though, might be that "some really good graphic designers" are not the only factor that could result in a great Linux desktop. Linux development is now heavily influenced by corporations that are not known for any great user facing design efforts. Great UI work does not seem to lend itself to the kind of distributed development style that has made many open source projects successful. Successful open source projects which are usable by the ordinary user are few and far between. Good UI requires a different mindset as John Gruber explained once in an excellent post:

UI development is the hard part. And it’s not the last step, it’s the first step. In my estimation, the difference between:

• software that performs function X; and
• software that performs function X, with an intuitive well-designed user interface

isn’t just a little bit of extra work. It’s not even twice the work. It’s an entire order of magnitude more work. Developing software with a good UI requires both aptitude and a lot of hard work.

...

It’s not something every programmer can learn. Most programmers don’t have any aptitude for UI design whatsoever. It’s an art, and like any art, it requires innate ability. You can learn to be a better writer. You can learn to be a better illustrator. But most people can’t write and can’t draw, and no amount of practice or education is going to make them good at it. Improved, yes; good, no.

Conversely, some people who are good UI designers aren’t programmers. But the rock stars are the guys who can do both, and they are few and far between.

If there’s a glib, nutshell synopsis for why Linux desktop software tends to suck, it’s this: Raymond and his ilk have no respect for anyone but themselves.

They have no respect for the fact that UI design is a special talent.

They have no respect for the fact that good UI design requires a tremendous amount of time and effort.

And, most importantly, they have no respect at all for real users. The idea that GUI software needs to be designed for “dumb users” — which is Raymond’s own term, and an indication of what he really means when he refers to dear old A.T. — is completely wrong.

Great software developers don’t design for morons. They design for smart, perceptive people — people just like themselves. They have profound respect for their users.

...

Movies are collaborative art, but require strong direction. So it is with end user software.

Gruber's post is so good, that I run the risk of reproducing it in its entirety.

Funny Perl Quote

My only argument against perl would be it looks like a chicken ran across your keyboard.

- Michael Whalen's comment on a blog post titled In defense of Perl

Linux Kernel Development Stats from Greg Kroah Hartman

Linux kernel hacker Greg Kroah Hartman's June 5, 2008 talk at Google titled "The Linux Kernel" was chock-full of details about kernel development. I noted down some of the the things he said. Please note that the talk was delivered on June 5th, 2008 and all stats mentioned by GKH are relative to that date.

• Code changes per day in 2007-2008 so far
  
  • 4,300 lines added
  • 1,800 lines removed
  • 1,500 lines modified
    
  These changes don't include moving files around.It works out to about 3.69 changes per hour, 24x7 . It's not just the drivers that are changing at this rate, it's the the entire kernel.
  
• The kernel itself is about 9.2 million lines, and has been increasing in size by about 10% every year since 2.6.0 (when GKH started tracking it). The drivers make up about about 55% of the code, while architecture specific code is second in terms of LOC. The core of the kernel is about 5%.
• Supports more processors and devices than any other OS in history.
• One of the consequences of the scorching pace of the kernel development is that, the kernel is a better place for patches , than for it to be maintained separately. GKH pointed out that it is difficult to keep pace with Linux kernel development and gave the example of Xen; The Xen folks apparently never played nice with the kernel developers and are now trying to get their patches into the kernel. From his tone it sounded as thought it was a tough climb uphill for the Xen guys, and the KVM is already in the kernel.
• The Kernel development hierarchy (it has one !) has developers feeding patches to the maintainers, who in turn feed them to subsystem maintainers (pci, security, usb, etc).Sub-system maintainers maintain their own trees. Steve Rothwell (IBM, Austalia) pulls changes into the "Next" tree every night and does daily builds, while Andrew Morton pulls changes into his tree once a week or so. When Linus says the merge tree window is open, all the sub-system maintainers hit Linus. There is also the stable release tree, which is not maintained indefinitely.
• He likened the patch submission process to a lossy network routing algorithm - it can handle people dropping out . There is no other way to develop at such high speed.If you own a file or subsystem, you have to accept the fact that other people are going to be changing it. Maintainers can always revert changes if they don't like it.
  
• There is no good way to test the kernel except to run it. The hundreds of permutations of devices and interactions makes it impossible to test it comprehensively. The only way out is for the developers to test the rc releases.
• There are no stable and unstable releases anymore. For the last four years,they have been replaced by releases every 2 and 3/4 months.
  
• There have been 2399 independent contributors to the Linux kernel in the last year. 50% of the contributors submitted only one patch; half of the half contributed two patches. the top of the curve is getting flatter. Top 30% do only 30% of the work. The number of individual contributors is going up.
  
• Top developers by quantity (for the last one and half years)
  
  1. Adrian Bunk 754
  2. Al Viro 698
  3. Thomas Gleixner 656
  4. David S.Miller 655
  5. Bart Zolnierkiewicz 637
  6. Paul Mundt 610
  7. Ralf Baechle 604
  8. Ingo Molnar 596
  9. Patrick McHardy 554
  10. Tejun Heo 530
• Top contributors by sign-offs (shows who the gatekeepers are). Note that Linus doesn't sign off on patches from subsystem maintainers.
  
  1. Andrew morton 9086
  2. Linus Torvalds 8960
  3. David S.Miller 4926
  4. Jeff Garzik 2960
  5. Ingo Molnar 2489
  6. Greg Kroah-Hartman 2098
  7. Thomas Gleixner 1098
  8. Mauro Carvalho Chehab 1822
  9. Paul Mackerras 1675
  10. John Linville 1461
• Who's funding Linux kernel development ?
  
  1. Amateurs 18.5%
  2. Red Hat 11.6%
  3. IBM 7.5%
  4. Novell 6.6%
  5. Unknown individuals 5.5%
  6. Intel 4.1%
  7. Oracle 2.2%
  8. Consultants 2.2%
  9. Academia 1.5%
  10. Renesas Technology 1.5%
  Google is at number 13 with 1.4% contribution. Without Andrew Morton's contributions Google's would be at the fortieth spot.
• Canonical had about 6 changes in the past 5 years; they are in the 300th
  position. GKH was very emphatic that 'Canonical does not give back to the community'.
• 75% of Linux kernel work is paid for.
• Linus jokes that the kernel is not intelligent design, it is evolution. GKH also added that "We react to stimuli that's happening in the world" and "We don't over plan things".
• As GKH put it, 'We broke all software development rules and we are continuing to break it'.

The Importance of Apple

Recently at work I couldn't help chuckling and shaking my head as I went through a list of requirements for a well known MNC - buried in the middle of was 'Oracle support' - there, that's an enterprise customer, I thought. It's one of those 'enterprise' requirements that clients who work in multi-storeyed glass and concrete buildings with large cube farms feel concerned about regardless of whether it makes any real difference to their work. This was not the first time that I had heard users asking for Oracle support. It's all about making the right noises, which is what enterprise software is all about. I was more annoyed with the business development folks for not having nipped this in the bud than with the customer. A couple of days later I was reading an interview with the author of Cocoa Programming for Mac OS X when I came across this heart warming section:

Scott: The Mac appears to be making real inroads in the mainstream consumer computing market, and certainly the iPhone is doing the same. Do you expect to see this carry over to the corporate world?

Aaron: Apple seems to doing its best to keep Macs out of the corporate world. Most dialogues between Apple and a corporation go something like this:

Corporation: Hey, does Cocoa include something to help us write apps that talk to our Oracle database?

Apple: Um, nope. But isn't GarageBand cool?

Corporation: Hmm.

Indian IT Companies and Internet Access at Work

Shyam points out the hypocrisy of a TCS executive advocating net access at work while denying the same to his own employees. A couple of years back one of my ex-colleagues along with one of the sales guys went to TCS to make a sales pitch. Using our product meant all your data was hosted off-site; the data could be accessed using a browser based interface, as well as command line tools and GUIs. They spent a couple of hours explaining the details of this product, and answering all kinds of questions about edge cases and close to impossible usage scenarios. They were feeling a little exasperated about the oddness of the questions, especially, since they were informed that this was an audience of senior managers. Finally they learnt that it would be difficult for the TCS staff to use the product, since they did not have Internet access. My ex-colleagues were more than a little peeved about their time being wasted.

IT work in recent years has become very dependent on access to search engines, online forums, mailing lists and IRC groups. Why try to fix something from first principles, when you can paste the error message into a search engine and get an easy answer ? Makes sense in most cases, especially crappy software, which means most software. It makes little sense to attempt to understand the workings of twisted minds that develop such software. Add to this the amount of open source software in use at these days. The very culture of open source means you will never really be able to use it meaningfully without internet access.

Management concerns about internet access cannot be wished away either. It seems to revolve around a couple of key concerns.

• Downloading and installation of various kinds of malware by employees and subsequent IT resources and time spent on cleaning up the network.
  
• Browsing sites unconnected to work and chat. I have to admit that, I find the amount of time that some recent graduates spend on social networking sites and chat borders on the pathological. And it's not always for lack of work to do. It's a huge time sink.
  
• Legal obligations of employers - what if the employees are surfing porn sites ? That could be construed as sexual harassment by their female colleagues. Isn't the employer liable in this case ?
• Data security has become a huge concern with management - nobody seems to have a good solution to this problem. There seem to be a few back office type operations that apart from blocking net access disable access to USB drives and such. There have been quite a few scams that seem to have made managements paranoid.
  

Employer trust is not something that is always reciprocated by employees. A liberal employer is often misunderstood to be a weak personality, who can be taken advantage of. Misuse of office facilities is a reality that cannot be wished away.

There are no hard answers to a complex techno-social problem like this. An evolutionary approach is needed that takes into account the concerns of both employers and employees. Employers need to realise that, increasingly, internet access is not a priviliege that they 'allow' employees to enjoy; In many cases such as software development it helps to work effectively. Allowing employees to finish domestic chores such as paying bills, check their bank accounts, or keep in touch with their friends and relatives (in moderation) contribute to productivity at work.

I once heard from a consultant in Bangalore, that his client not only had him setup a proxy server for net access, but also had him setup an intranet site where everyone's browsing records were publicly available. This seems like an interesting approach that could work in some cases. In any case I don't have a problem with employers monitoring my internet activities. Employers should make their policy on net connectivity very clear; what's acceptable and what's not should be explained; new employees should have it clearly explained to them that net access is not a 'given' and that their activity is being monitored. Most companies do not have a policy on this matter and their only solution is a blanket ban on any kind of access, with a patronising common 'net PC' thrown in.

OLPC - Trying to make Water flow Downhill

I don't understand why we need a program to spend millions of dollars to make laptops cheaper, when laptops get cheaper everywhere. I mean, that would be like setting up a world bank program to make water flow downhill.
- Dr. Joel Selanikio co-founder of DataDyne in an interview with Jon Udell

That must be one of the funniest and most insightful comments on the OLPC project. I suspect it's actually one of the most nagging questions that bothers folks who are not evil.

Virtualization, The Jamie Zawinski Remix

Some people, when confronted with a problem, think "I know, I'll use virtualization." Now they have two problems.

(Jamie Zawinski - Wikiquote)