Whiteboard coding in software developer interviews.

[gkm]

Isn't 5, 8, or 10 years experience enough of a sign? Though, there are many areas to coding, and stuff that get used daily, others once in 2 years unless you do the same stuff everyday. If you develop window apps every day you're not going to be great at your web page stuff if you've done a web page 3 times a year.

Not always. I have interviewed people with that much experience and while they know all the buzzwords, only during the coding session one discovers that what they actually did was upload some files very day or something like that, and did not do much actual coding.

 

[gkm]

Coming back to the example of Max Howell, he had developed not only 1 of the most popular package managers but also a very popular Promises & Futures Framework; both are complex topics and challenging to get right. Binary tree applications can be considered quite topic specific, e.g. search engines, AI decision trees, parser combinators, path finding, etc... Hence it would seem strange to want to interview someone who had not yet demonstrated expertise in fields you would normally associate with binary tree structures + if you were recruiting ability, then why would you disqualify someone based on something that could so easily be mastered.

That for me is just short-sighted and is very indicative of what's wrong with WB interviews.

Companies interview for more than coding skills. I very much doubt he failed the interview battery because of coding, more likely there were other reasons.

 

[semaphore]

I had to write a lock free queue mechanism where I am now. Hamster just laughed in the interview

 

[Hamster]

I had to write a lock free queue mechanism where I am now. Hamster just laughed in the interview

The bank asked me "how is the concurrent dictionary implemented in .NET 3.5 and is it a good solution". Just shrugged my shoulders and said I don't know, I don't read through the source code on MSDN. Interview was just a formality since they had to let us in to come fix their source control and release issues

Eventually, after a couple of months here I found their own implementation of it. Did a couple of benchmark tests against it, .NET 3.5, 4 and 4.5. Found all the .NET implementations faster and since we were switching to 4.5 the memory issue was solved. Hit the delete button, replaced with default and off she went to prod.

 

[cguy]

Also think about ability vs. knowledge for a second; these are not some tyoe of locked in generic traits. So disqualification of someone who is unfamiliar with binary trees assumes the persons is incapable of acquiring this knowledge.

This was not a knowledge question. This was a very simple "can you do basic algorithms" question. That he got it wrong would have been a huge red flag, but almost certainly not why he was disqualified.

Coming back to the example of Max Howell, he had developed not only 1 of the most popular package managers but also a very popular Promises & Futures Framework; both are complex topics and challenging to get right. Binary tree applications can be considered quite topic specific, e.g. search engines, AI decision trees, parser combinators, path finding, etc... Hence it would seem strange to want to interview someone who had not yet demonstrated expertise in fields you would normally associate with binary tree structures + if you were recruiting ability, then why would you disqualify someone based on something that could so easily be mastered.

That for me is just short-sighted and is very indicative of what's wrong with WB interviews.

"Binary Tree Applications" are irrelevant - this was an algorithm question, that binary trees where used instead of queues, or B-Trees or Hash Tables or array processing (sorting, medians, uniqueness, special stucture, etc.) was likely entirely arbitrary. You don't need special expertise to solve these, just basic computer science.

It is possible to write something (like a very popular package manager), and over time become an expert in all the code required to implement and the nuances of working with the various OS's that it supports, etc., which is difficult and challenging, and it is definitely a strong achievement, however, if Google isn't hiring you to work on their internal package manager, this knowledge is largely useless, and if you've let the skills relevant to Google atrophy (or if they weren't there in the first place), and especially if you get bent out of shape about the question, it makes sense for them to not hire you.

 

[cguy]

Isn't 5, 8, or 10 years experience enough of a sign? Though, there are many areas to coding, and stuff that get used daily, others once in 2 years unless you do the same stuff everyday. If you develop window apps every day you're not going to be great at your web page stuff if you've done a web page 3 times a year.

No - the odds of past coding being specifically relevant to what they use/do at Google is likely small. What is relevant is the skill set, not the job/company specific knowledge you may have gained. So the questions will focus on general computer science, algorithms, problem solving, maths (if relevant), current research in your field of expertise, "culture-fit", etc.

 

[Ancalagon]

This was not a knowledge question. This was a very simple "can you do basic algorithms" question. That he got it wrong would have been a huge red flag, but almost certainly not why he was disqualified.



"Binary Tree Applications" are irrelevant - this was an algorithm question, that binary trees where used instead of queues, or B-Trees or Hash Tables or array processing (sorting, medians, uniqueness, special stucture, etc.) was likely entirely arbitrary. You don't need special expertise to solve these, just basic computer science.

It is possible to write something (like a very popular package manager), and over time become an expert in all the code required to implement and the nuances of working with the various OS's that it supports, etc., which is difficult and challenging, and it is definitely a strong achievement, however, if Google isn't hiring you to work on their internal package manager, this knowledge is largely useless, and if you've let the skills relevant to Google atrophy (or if they weren't there in the first place), and especially if you get bent out of shape about the question, it makes sense for them to not hire you.

Knowledge about algorithms such as the general principles or O(n) notation is general enough to be used in all programming jobs.

But knowledge about particular and specific algorithms is based upon whether you have used or encountered that algorithm. It isn't that useful as a test. Let's say you have written merge sort and quick sort algorithms, and one day get an interview question on the bubble sort. You fail the test even though you have knowledge of superior sorting algorithms.

Does that make any sense? Should you learn particular algorithms off by heart just to pass a test?

Heck I implemented the A* pathfinding algorithm years ago, but I can't remember much about it now. Does that mean I'm a poor programmer?

 

[cguy]

I think the problem mentioned in the article is not actually the whiteboard - it is the specific knowledge requested. In other words, whether they asked the candidate to implement a reversal of a binary tree using Notepad++, Visual Studio, paper or a whiteboard, the point is that reversing a binary tree is a highly specific piece of knowledge. If you happen to have studied it or worked with it, then great. If not, you're in trouble. Knowing how to reverse a binary tree does not mean you are a good programmer - neither does not knowing how to reverse one imply you are a bad programmer.

You see the same thing in South Africa when you have interviews in which you are asked very specific questions about technology. Such as, what is the viewstate, or what does the internal modifier mean? All stuff you could discover on Google in 2 seconds flat.

Really???

 

[cguy]

Knowledge about algorithms such as the general principles or O(n) notation is general enough to be used in all programming jobs.

But knowledge about particular and specific algorithms is based upon whether you have used or encountered that algorithm. It isn't that useful as a test. Let's say you have written merge sort and quick sort algorithms, and one day get an interview question on the bubble sort. You fail the test even though you have knowledge of superior sorting algorithms.

Does that make any sense? Should you learn particular algorithms off by heart just to pass a test?

Heck I implemented the A* pathfinding algorithm years ago, but I can't remember much about it now. Does that mean I'm a poor programmer?

Nobody asked him specific knowledge about a particular algorithm - they asked him for any algorithm to reverse a binary tree, which is trivial (edit: I.e. To figure out/create on the spot). If I couldn't give any algorithm to sort, I would be a crap computer scientist. If I can't give you a breakdown between the differences of a Bitonic Sort and and a Merge Sort, that's a different story (and a bad question).

I had never heard of "inverting a binary tree", but after seeing the problem statement, I took me all of 30 seconds to see how I would code a solution.

 

[[)roi(]]

This was not a knowledge question. This was a very simple "can you do basic algorithms" question. That he got it wrong would have been a huge red flag, but almost certainly not why he was disqualified.

"Binary Tree Applications" are irrelevant - this was an algorithm question, that binary trees where used instead of queues, or B-Trees or Hash Tables or array processing (sorting, medians, uniqueness, special stucture, etc.) was likely entirely arbitrary. You don't need special expertise to solve these, just basic computer science.

It is possible to write something (like a very popular package manager), and over time become an expert in all the code required to implement and the nuances of working with the various OS's that it supports, etc., which is difficult and challenging, and it is definitely a strong achievement, however, if Google isn't hiring you to work on their internal package manager, this knowledge is largely useless, and if you've let the skills relevant to Google atrophy (or if they weren't there in the first place), and especially if you get bent out of shape about the question, it makes sense for them to not hire you.

Silly analogy, because what is knowledge but "can you do xxx" or "have you done xxx"?
The application of particular algorithms or constructs are about as irrelevant as categorising engineers by skill-set: compiler, front-end, ... because not everyone will be familiar with writing parsers, or building fluid UIs or even building a dependency package manager; IMO binary tree algorithms are far common domain for the compiler engineer than the latter two. The only glaring exception is someone fresh out of varsity, but that's largely down to the fact that they haven't specialised, hence their general knowledge is possibly all they have, making WB far more advantageous.

In the case of the article; clearly both of the engineers had specialised in a particular field, which for me at least, implies that their general knowledge wouldn't be as fresh as someone just out of varsity, or more specifically someone who had recently studied for exams.

But then let's also not discount a possibility of poor shortlisting of candidates i.e. the recruitment agency that Google used didn't do a proper match up of the candidates capabilities against the job profile, for example:

• You would expect a person skilled in building package managers and Promises / Future frameworks to be a poor match for a job requiring a search speciality.

Either way it doesn't discount the need for relevance.

 

[[)roi(]]

Nobody asked him specific knowledge about a particular algorithm - they asked him for any algorithm to reverse a binary tree, which is trivial (edit: I.e. To figure out/create on the spot). If I couldn't give any algorithm to sort, I would be a crap computer scientist. If I can't give you a breakdown between the differences of a Bitonic Sort and and a Merge Sort, that's a different story (and a bad question).

I had never heard of "inverting a binary tree", but after seeing the problem statement, I took me all of 30 seconds to see how I would code a solution.

There's a lot of assumptions in your response; one for example: that he too had never heard of "inverting a binary tree", and that's according to his twitter posts is where the interview terminated.
For Compiler, Search engineers this would naturally be easy peasy... i.e. there would be no need for clarification. Hence I mention it could have been a poor match up by a recruitment agency.

As to inverting binary trees; yeah sure it's easy (once you understand what is being asked), and knowing Max he certainly wouldn't have any issues quickly coding a solution.
Imagine for a moment a different question; On the WB: draft the underlying structure of parser combinator. For a compiler engineer this again would be easy peasy, but not so for someone who had specialised on the front-end.

 

[biometrics]

I'd think showing a prospective employer what you've actually developed is of much more value. All this theory stuff is BS imo. That's not how we work.

 

[[)roi(]]

I'd think showing a prospective employer what you've actually developed is of much more value. All this theory stuff is BS imo. That's not how we work.

It really depends on the job, for most corporate jobs; algorithms are probably less relevant than design patterns, but yeah you're right re what you!ve already achieved.

 

[cguy]

Silly analogy, because what is knowledge but "can you do xxx" or "have you done xxx"?

In the context of an interview, a "knowledge based question" is one that requires specific or uncommon knowledge to answer. The interview question here did not require more than a basic understanding of how a tree data-structure works. What was being tested, was the ability to conceive of and implement a simple algorithm, in the most basic of settings.

The application of particular algorithms or constructs are about as irrelevant as categorising engineers by skill-set: compiler, front-end, ... because not everyone will be familiar with writing parsers, or building fluid UIs or even building a dependency package manager;

I don't see how this is relevant - he wasn't asked to apply a particular algorithm, he was asked to create an algorithm to solve the problem. Anybody who claims basic competency as a computer scientist knows what trees are.

IMO binary tree algorithms are far common domain for the compiler engineer than the latter two. The only glaring exception is someone fresh out of varsity, but that's largely down to the fact that they haven't specialised, hence their general knowledge is possibly all they have, making WB far more advantageous.

In the case of the article; clearly both of the engineers had specialised in a particular field, which for me at least, implies that their general knowledge wouldn't be as fresh as someone just out of varsity, or more specifically someone who had recently studied for exams.

But then let's also not discount a possibility of poor shortlisting of candidates i.e. the recruitment agency that Google used didn't do a proper match up of the candidates capabilities against the job profile, for example:

• You would expect a person skilled in building package managers and Promises / Future frameworks to be a poor match for a job requiring a search speciality.

Either way it doesn't discount the need for relevance.

Google doesn't use recruitment agencies as far as I know - I have always been contacted by their HR department directly, as has everyone I know of. Also, they don't try interview you for position X when you join, they interview you to see if you have the pre-requisite skills, and then decide what to do with you after the hiring decision has been made. In special cases (where someone has expert/world-class experience in something they're interested in), some indication of the groups wanting to hire them is given, although interviewers are still selected from different groups, and the candidate would be moved to a different department if they felt that the candidate would be a better fit somewhere else.

Specializing in a particular field, and letting your general abilities decay is fine if you're going to remain strictly in that field, however, losing basic skills like being able to create simple algorithms, will never be acceptable by a company that hires computer scientists, who are expected to be experts at developing algorithms. Relevance is important, but it's relevance to your new job that matters, not your old one.

 

[cguy]

There's a lot of assumptions in your response; one for example: that he too had never heard of "inverting a binary tree"

Where did I make that assumption? I have no idea whether he had heard of it or not, nor do I think it would make a difference either way.

What are my other assumptions?

, and that's according to his twitter posts is where the interview terminated.
For Compiler, Search engineers this would naturally be easy peasy...

For a good high school student, this would be "easy peasy".

It is possible that they terminated his interview here, because struggling with this would be pretty damn scary - Personally, I wouldn't terminate an interview on just this one question, but I would ask another algorithm question to see if this part of the candidates skill set could possibly be so broken - given that, it is entirely possible that they had already "worked down" to this question due to his inability to answer any of the other more complex questions.

It is also likely that they terminated it here because he got bent out of shape by the question... or perhaps they terminated it here because it it was the last interview question at the end of the day.

Whatever the reason, given how he reacted, I think that Google made the right decision.

i.e. there would be no need for clarification. Hence I mention it could have been a poor match up by a recruitment agency.

As to inverting binary trees; yeah sure it's easy (once you understand what is being asked), and knowing Max he certainly wouldn't have any issues quickly coding a solution.

Yet he did have issues coding a solution... Do you think that the issue was that he didn't understand what was being asked? Given how simple the problem is, that would be an even bigger red flag.

Imagine for a moment a different question; On the WB: draft the underlying structure of parser combinator. For a compiler engineer this again would be easy peasy, but not so for someone who had specialised on the front-end.

That's why they asked a question that should be "easy peasy" to anyone with basic computer science skills.

 

[cguy]

I'd think showing a prospective employer what you've actually developed is of much more value. All this theory stuff is BS imo. That's not how we work.

For a company like Google, work that has already been done is more than likely irrelevant. When the work is irrelevant it is difficult to assess (because the interviewer may not be familiar with the work/industry) how much work the candidate did themselves, how difficult their work was, how much help they had, how much innovation they actually contributed, what part they really played in their team, etc. (Funny story: I once interviewed 3 people from the same team, who all claimed to be the "lead" developers). In the case it is not irrelevant, they will get asked about the work they have done in detail, and that will play a big part in how the candidate is rated.

None of the above really tells you how well you will work with that person on the type of things you usually work on, so you throw a few algorithm questions in, and a few theory questions in, since these are things that are used daily, and form a common ground dialogue. If the basic theory is missing, it is cause for concern, if the interaction with the person is off (and I've seen everything from being sworn at, to a candidate chasing the interviewer around the table!), then it's a no - the person's previous work doesn't tell you this.

It really depends on the job, for most corporate jobs; algorithms are probably less relevant than design patterns, but yeah you're right re what you!ve already achieved.

I agree with this (depends on type of company) - people tend to think that the most important thing is "can they do the technical work?", and for many companies this is the most important thing. At Google, and similar companies, there is usually no specific job to hire for, so this is hard to test for. Engineers move as needed, and can move into a group that is almost as different as if they had moved companies, so the ability to learn new infrastructure, ideas, algorithms, and begin to innovate and contribute in a new environment needs to somehow be assessed.

Similarly, one of the biggest issues faced, are personality issues that cause intra-team politics - you just don't want to hire a bad fit - I know this happens everywhere, but at companies like Google, it can cost many millions of dollars (partly because of scale, and partly because collaboration and communication are so important at such a firm, that there are very few "siloed" positions that problem personalities can be contained in) - apart from their direct costs and perhaps negatively impacting what they work on, managing them, managing employees that hit bumps due to dealing with them, the man-hours lost by other employees just "talking" about them, the loss of morale, and all the management politics and HR/Legal work involved is an immense cost. The best way to achieve this is to actually "do" what employees at companies like Google "actually" do on a daily basis when interacting, which is go into a meeting room with the person, and hash through some ideas for solving some artificial (or sometimes real) problems on a White Board.

 

[freddster]

I'd think showing a prospective employer what you've actually developed is of much more value. All this theory stuff is BS imo. That's not how we work.

What is a class?

 

[[)roi(]]

Where did I make that assumption? I have no idea whether he had heard of it or not, nor do I think it would make a difference either way.

What are my other assumptions?



For a good high school student, this would be "easy peasy".

It is possible that they terminated his interview here, because struggling with this would be pretty damn scary - Personally, I wouldn't terminate an interview on just this one question, but I would ask another algorithm question to see if this part of the candidates skill set could possibly be so broken - given that, it is entirely possible that they had already "worked down" to this question due to his inability to answer any of the other more complex questions.

It is also likely that they terminated it here because he got bent out of shape by the question... or perhaps they terminated it here because it it was the last interview question at the end of the day.

Whatever the reason, given how he reacted, I think that Google made the right decision.



Yet he did have issues coding a solution... Do you think that the issue was that he didn't understand what was being asked? Given how simple the problem is, that would be an even bigger red flag.



That's why they asked a question that should be "easy peasy" to anyone with basic computer science skills.

Go back and read your post prior to my response, assumptions are clear; unless of course you were part of these interviews.
You make generalized statements about individuals classifying their skill as if you're the authority on these matters, maybe you should take time to research their work before making crazy statements, especially when these statements leave no possibility of fault on the part of Google, ... do I really need to remind you that Max was employed by Apple, and his work since joining Apple is visible here:

• https://github.com/apple/swift-package-manager

Does this really look like the work of some fool that is incapable of reversing a binary tree, especially considering that you said: "I had never heard of 'inverting a binary tree'"; but still you can't imagine a scenario where you are in an interview asked the same question, and you simply don't know. if he really was such a lost case as you imply, then why did Apple employ him and choose to task him with a vital part of Swift's architecture; oh wait let me guess the answer on this... because Apple got it wrong... and Google doesn't.

FFS who really studies algorithms every day; for most developers its a situation of review the required topics and write the code. I could probably draw a bit of blank with many of these, except of course if you ask me to verbatim rewrite the stuff I recently coded, for example: parser combinator and applicative functors, .. But heck ask me to write a quicksort and expect a blank stare, similarly before I wrote the parser combinator & related applicative token functors, I had no clear way to tackle it; Google and a book helped frame that.

 

[[)roi(]]

Nobody asked him specific knowledge about a particular algorithm - they asked him for any algorithm to reverse a binary tree, which is trivial (edit: I.e. To figure out/create on the spot). If I couldn't give any algorithm to sort, I would be a crap computer scientist. If I can't give you a breakdown between the differences of a Bitonic Sort and and a Merge Sort, that's a different story (and a bad question).

I had never heard of "inverting a binary tree", but after seeing the problem statement, I took me all of 30 seconds to see how I would code a solution.

While I'm at it let me jump in on another bugbear, the use of the term Computer Scientist or Computer Science, when nothing we do justifies that classification (unless of course you're advancing in the field of computer science e.g. Quantum Computing, Lambda Calculus, ...), but for most of us it's a very big misnomer. I like to think of what we do as more of art than science; science is like baking; precision in ingredients and process is vital, but programming like art comes down to the skill of the artisan in using the tools at hand.

• http://letscallitbob.blogspot.co.za/2006/10/why-computer-science-is-misnomer.html
• http://www.whilenotdeadlearn.com/blog/2007/11/i-am-not-computer-scientist
• http://c2.com/cgi/wiki?IsComputerScience

Donald E Knuth captured it well with his bookset Title: "The Art of Computer Programming" or even how Steve Jobs would often refer to the sector: "The intersection between art and science". Artists (artisans) like programmers are continually learning new techniques, incorporating new tools, and whilst these skills can be learnt, some artists just have a natural advantage over their peers.

 

[Hamster]

While I'm at it let me jump in on another bugbear, the use of the term Computer Scientist or Computer Science, when nothing we do justifies that classification (unless of course you're advancing in the field of computer science e.g. Quantum Computing, Lambda Calculus, ...), but for most of us it's a very big misnomer. I like to think of what we do as more of art than science; science is like baking; precision in ingredients and process is vital, but programming like art comes down to the skill of the artisan in using the tools at hand.

• http://letscallitbob.blogspot.co.za/2006/10/why-computer-science-is-misnomer.html
• http://www.whilenotdeadlearn.com/blog/2007/11/i-am-not-computer-scientist
• http://c2.com/cgi/wiki?IsComputerScience

Mr. Simpson, B.Art