Sunday, May 27, 2007

Programmer Interviews: Two Warning Signs

I just need to get this off my chest. There are a lot of interviewers who will ask you trivial syntax questions, like, what does ^ mean in Ruby? Some good blog posts have made it common knowledge among better companies that these are basically stupid questions that tell you nothing about what a programmer can actually do, and don't represent vital knowledge anyway, since the answers can be found immediately on Google.

Unfortunately, a lot of companies know that these questions are trivial questions, and they know they're supposed to abandon the practice, but they don't know what they're supposed to do instead, so they just ratchet it up a notch. Instead of asking you questions which are trivial because any idiot could answer them by going on Google, they ask you questions which are trivial because any idiot could answer them by pulling Knuth off the shelf. Questions like, "what is a tree sort? What is it good for?" You can give them credit for having higher standards, because they make the hypothetical idiot go to a more sophisticated source, but they're still asking questions which any idiot could answer with access to the right reference materials. But the ability to locate reference materials isn't an important criterion in hiring programmers; it's an important criterion in hiring librarians.

Algorithm questions are becoming the huge warning sign, to me, that syntax questions were in the past. If you take a job with a company that asks you questions any idiot could answer, sooner or later they're going to put you on a project with some idiot who answered them. It's something to watch out for.

The best way to hire programmers is to read their blogs and look at their code. If you don't read their blogs and look at their code before the interview, you've gotten it backwards. The phone screen should come after the Google screen.

Even though I sometimes post embarassing things on my blog, I'm always happier in an interview when I find out the person I'm talking to has read my blog beforehand. I never talk to anybody without researching their company first, and if you googled the company but they didn't google you, that means that the first thing you find out about a prospective employer is that they're less diligent than you are. That's something to watch out for too.


  1. I can't remember who wrote it, but somebody had an article whose hypothesis was that google asked those questions not because they were relevant, but instead that they were intended to weed out bad developers.

    That is to say - they might wipe out some good developers too, but that might be acceptable if the thesis "people who can answer difficult algorithmic questions off the top of their head tend to be good programmers" was true.

  2. Which questions? The NY Times article in the link is about Google abandoning a quiz which included details of the HTTP spec (iirc).

  3. I have yet to find an employer who actually read my blog, despite it's prominance on my resume. In one interview I wasted 20 minutes on some dumb logic problem about pirates and gold. For interviews I say, read the blog, look at code, talk with former coworkers (not managers). You can't fake that.

  4. I agree that knowing exactly how arcane tree-sorting algorithms work is not a very useful interview question.

    What is important is that a candidate has heard of Tree/Hashtable structures and knows (roughly) their characteristics and when to use them.

    There's a class of interview/test questions where a correct answer is impressive but an incorrect (or 'I don't know') answer tells you precisely nothing about the candidate.

    "How is quicksort implemented?" is an example of this sort of question. The fact that someone doesn't know this tells you nothing.

  5. I find it slightly amusing that researching stuff off a book has become more sophisticated than googling for it.

    Are books becoming deprecated? That gives me an eerie-science-fiction-kind-of feeling :P

  6. Name some employers who will read a blog if you have one and look over open source code you've released.

  7. i tend to disagree . As important it is to ask questions relevant to the work the person might be getting hired for , its also important to test algorithm and data structure knowledge. Reading blogs and pet projects done by developers is a good idea . But you know how busy ppl are n how HR ppl work ..

  8. @anonymous - I'm not going to name any such employers because I already linked to them in the exact sentence you're responding to.

    @godfoca - come on. it's knowing what Google is vs. knowing who Donald Knuth is. the delivery method is irrelevant.

    @andrewr & karanjude - it's true these things matter, but a lot of people are just using it as a crutch, the same crutch syntax questions used to be.

  9. You said

    "But the ability to locate reference materials isn't an important criterion in hiring programmers; it's an important criterion in hiring librarians."

    Au contraire :) I feel that the ability to locate the right kind of reference materials in the right situation is something that is sadly lacking in even some otherwise wonderful coders. In fact, one of my interview questions in the late 90s was "What do you read?"

    Even today, it seems that many coders think themselves too busy to actually read up on the right ways to deal with certain challenges, preferring instead to use their 'initiative' and fudge around the problem rather than seeking out prior art.

  10. So I guess those of us who don't share the compulsion to divulge the last of our innermost thoughts to the world the second we have them, or whose meaningful code is locked up under NDA, should be discriminated against in the workplace? And what happened to "past performance is no guarantee of future potential"?

    No. Pretty much the only way to evaluate someone is to evaluate them. Let them do the job for a while, with careful but not suffocating supervision, and see whether or not they can get up again when they fall on their arse. Hiring is, and should be, a risk, just like any other investment; but a decent trial period (I've tended to suggest three months) really is the only fair way to see whether someone's up to snuff. Anything else is basically a more or less sophisticated form of fortune telling.

  11. Asking somebody who claims to be a Ruby programmer what the "^" character means in Ruby code seems perfectly reasonable to me. I'd ask a Java programmer what the difference between "==" and ".equals(x)" is too. In both cases, the goal is to determine whether the person has actually written much code. Surely no Ruby programmer does a Google search between every line of code they type.

    Similarly, asking about familiarity with basic computational theory is a good way to tell the difference between a hack programmer and somebody who really does know the difference between a good sorting algorithm and a bad one.

    Now, I definitely agree that questions like "what's on page 147 of Knuth volume 1" are silly, as are questions about arcana buried in RFC's and protocol specs. The real qualifier for "good" questions should be, "is knowing the answer to the question an important qualifier for filling the job description? Can somebody who doesn't know the answer really be expected to get the work done the way we expect it to be done?"

    I don't know much about router configuration, but I know what it means. In a pinch, I could probably do Google searches and read manuals and manage to reconfigure a router. I would hope, however, that nobody would consider me therefore a qualified network admin.

  12. I'm writing this post as a public service for anyone considering hiring me in a Ruby environment.

    I don't know what the ^ operator does in Ruby. I haven't used it, probably because I don't know what it does.

    I kind of remember Giles writing that it's an exclusive or operator. And if I had tried to guess before reading that post, I would have guessed exponentiation.

    Here's an interesting follow-up: is it a method or an operator?

    In other words, can you define it for one of your classes, or is it one of those warty things in Ruby that look like the other things but have their own special implementation in the interpreter?

    If you want short-cut evaluation in Ruby, you need to define it in the interpreter. Compare and contrast this implementation to how Lisp and Scheme handle things with macros or how Smalltalk makes everything a black. Both of those languages let you define your own operators with short-cut semantics. Ruby does not.

  13. I'm an unusual programmer. I've worked for and consulted at over 30 companies. I often get offers to go full-time and often turn them down.

    unlike a lot of programmers, I didn't graduate from college, and I didn't study computer science. I studied art, music, Latin, Ancient Greek, and acting. and I've mentored and even taught programmers with master's degrees and doctorates.

    in my 20s I did office temping so I could spend the majority of my energy on writing fiction and making music. so if you count that office temping, the number of companies I've worked for is probably more like 50 or 60.

    If there's one thing I know through and through, it's how to spot warning signs in a corporate culture very, very quickly.

    I think most of the people posting on this thread are well-informed and well-intentioned, but if you're a young programmer trying to decide which job to take, I want you to understand, this post is for you, not them, and the only important thing I have to say is that these warning signs are absolutely warning signs. they are not absolute deal-breakers, but they are definite warning signs. no company is without warning signs. there are worse warning signs. but these are still things to watch out for.

    that being said:

    @gwen - you need to read Clay Shirky's essay on micropayments. the essay's economics apply to code samples and effort just as readily as they apply to blog posts and cash. less and less people are going to go to the effort of requesting code samples as it becomes easier and easier to obtain somebody's code samples by googling them. if your code falls under NDA, you will appear to be a programmer who does not produce code. I know because it happened to me.

    @reg - I assumed ^ was exponentiation when I first saw it too. in fact it often serves double-duty as an XOR, in Perl, Python, C, and many other languages, according to somebody who commented on my post about it. this is pretty relevant; there are lots of corners to any programming language, and some people will know particular idiosyncrasies inside-out. a lot of people don't ever even use XOR, but obviously some people use it often enough that they want it as an operator.

    the problem with asking ANY question with a predetermined correct answer is that an expert on Ruby idiosyncrasies can be totally missed by that screening process. if the XOR programmer doesn't remember the correct terminology for a tree sort, oops, too bad. screening out anybody who doesn't know one particular corner of computer science can cost you a potential hire who was an expert in some other corner. this means you should only screen for relevant corners.

    this is the other reason companies should do a Google screen before a phone screen. you shouldn't have to waste time on the phone on the question of whether or not somebody knows the language. that's a silly question. it's silly because a good candidate will in almost every case have proof of their knowledge somewhere on the Web, either through blogging, open source, or questions in forums. if you don't have that proof on the Web, you should put it on the Web, and if you can't put it on the Web, you should include it in the e-mail with your resume.

    one way or another, that question should be answered before you ever get on the phone. it's a waste of cellphone minutes and it makes the conversation depressing. nobody enjoys trying to spot a liar; nobody enjoys talking to someone who's trying to guess if they're telling the truth or not. that poisons the process, and this type of situation happens every time a technology gets hot. it's an ongoing problem in this industry, tied directly to its cycles. the solution is to use the technologies that the industry created. Google people, read their blogs, find code they've written - and then decide if you want to bring them in for an interview. likewise, if you're too busy to do that, have the people in human resources do it. they don't need Ruby programming skills just to google somebody's name. have an office temp do it for $15 per hour. the money you save will be worth it.

  14. Giles:

    It's quite possible that many good candidates will not have Google-able trails. And as an interviewer, it's my job to find out whether they're a good candidate even if Google turns up zilch.


    I get three CVs, one is from someone whose work has been under NDA, one is from a blogger who has exhibited intelligence and an ability to communicate in writing, and one is from someone who has a Ruby Gem available for download and inspection.

    What do you think happens next?

  15. This comment has been removed by the author.

  16. "if you don't have that proof on the Web, you should put it on the Web, and if you can't put it on the Web, you should include it in the e-mail with your resume."

    That's your opinion, and a minority one. You are part of a collection of people that wants everyone to know things about yourself that most (as in, most people aren't blogging, don't contribute to newsgroups, don't want to post pictures of themselves with a dildo on their head, etc) people want to keep private. The privacy issue is completely orthogonal to how good a prospective client is.

  17. I'm going to have to go with "Blogs aren't always about code" on this one, Alex. Sure, you can blog about the idiosynchrosies of your particular pet language every day, but one minor slip up and a mention of your political views on this or that, and suddenly, your blog has become a libability and not an asset. I'd greatly frown on anyone using google to find out background information on someone - they might find code, or they might find you espousing your support for Barak Obama(and they happen to be a Clinton fan), thus, subconsiously, you're off the list.

  18. Anonymous: The simple solution to not posting pictures of a dildo on your head is to not post pictures of a dildo on your head.

    Maintaining a squeaky-clean, unobjectionable-to-all image may be hard. (But then, as this post establishes, no image at all is objectionable to some. This battle you can not win.) Maintaining an image of a human who can be hired is not so hard. There is no 100% solution, and if there were, "no online identity at all" is clearly not it.

  19. @reg - that's exactly my point. I'm not saying it's fair. I'm saying it's the way it is.

    @anonymous - the line you quoted was about proof of being able to code. you're making a huge mental leap there. if you don't have any code you can put on the Web, write something and put it on the Web. you don't have to be an exhibitonist to put together a small site with code samples.

    @austin - I totally agree with you. I learned this the hard way when I used a metaphor from comparative religion under the assumption that since this was a programming blog people would take it all with a grain of salt and remain calm-headed. this was a very, very wrong assumption and I ended up killing the post to avoid getting people upset over a metaphor.

    but again, I'm not saying it's perfect. I'm just pointing out that it's the way it is. there's a great article about how this is affecting kids, I'll see if I can find it. the reality is, as David Brin predicted in The Transparent Society, anonymity is becoming a thing of the past.

    the first time a company hires some ax murderer and some stockholder finds his ax murdering in the first page of search results for the guy's name, googling potential employees is going to become an official best practice. there's no point fighting it.

    @jerf - There is no 100% solution, and if there were, "no online identity at all" is clearly not it.

    Exactly! I did something embarassing in 2006 and got hired by two companies in a row that knew about it. I didn't even find out they knew until I'd been working for them for a while, in each case. They googled me, found out about my flaws as well as my advantages, and hired me. I think this will become more and more normal in future. And let's face it, if you ask somebody in an interview what their flaws are, is that really useful information? Google them, find out for yourself what their worst flaws are, and then decide if it's a dealbreaker or not. The dude wearing a dildo on his head might still win if he's a better programmer than the upstanding citizen in the suit at church. Nobody hires people to go to church, they hire you to write programs. What you do on your own time is your own business.

    Personally I sometimes think posting a picture of myself with a dildo on my head was a good business move. Maybe a company which couldn't handle that wouldn't be a good fit. Maybe I'm doing myself a favor by scaring away conservative companies. Maybe I'm just screening bad customers here. Sometimes these things are simple.

  20. "The best way to hire programmers is to read their blogs and look at their code. "

    Geez, that's what I was hoping for but I'm having a tough time with that. I put out at least a dozen applications recently for programming gigs and not a single one has asked for a code sample or even looked at my blog. Not one. Instead I'm getting cookbook-style hiring questions like "we need someone with exactly 2 years experience with Rails that has 3 years with that you?" And at that point I hangup the phone and hide under the bed. I'm getting the feeling that I should have spent the last nine months studying Erlang.

    Awesome post as usual.

  21. Heh. Thank you. Actually that's the other reason I recommend this philosophy. Companies that ask you how many years don't really know what it is you should know. Companies that read your blog and look at your code expect to learn something in the process. You can't do that unless you know what to look for.

    Nine months studying Erlang would be a good idea, by the way. The Erlang market is much smaller than the C# market, and has much more interesting people in it. Chad Fowler's "My Job Went To India" has some really solid and maybe surprising perspectives on commodity programming, both from economics and marketing vantage points. Long story short, specializing in something you really dig is a really good job-hunting move.

  22. Glad to read articles like this. Thanks to author!

  23. Excellent website. Good work. Very useful. I will bookmark!


Note: Only a member of this blog may post a comment.