In early 2015, we decided to revamp the Ettus Research R&D hiring process. We had a lot of candidates coming through the pipeline, all the way through an on-site interview, that we would ultimately decide weren't a good fit. Additionally, it wasn't uncommon for interviewers to finish interviews with ambivalent opinions about a candidate's skills. We were spending a lot of time on hiring and didn't have commensurate results to show for it. So, we organized a focused effort to improve our hiring procedures. We considered aspects of other interview methods (some of which I summarized in my previous post, "Hiring is Hard"), and analyzed the results of our own interviews (i.e., who got offers and who didn't).

With some effort, we landed on a process that seems to be working fairly well for us. Hiring is still very time-intensive, but we waste much less time now than we did then, and when we give an offer to someone we are confident in it. So, after seeing this question asked on, I thought I would share what is working for us.

Technical Questions

First, we addressed our "ambivalence" problem. Up to that point, each time we had an interview the team would get together beforehand and plan out the interview questions. The questions were always relevant to the work they were doing (and, indeed, were often inspired by something recent) - but they changed almost every time. This made it really difficult to equitably weigh candidates' responses.

As I wrote about in my previous post, whiteboard interviews generally miss the mark. For candidates who were not active in the same open-source communities in which we participate, though, we couldn't think of a better option - so we instead tried to normalize the results.

We got some engineers together (software, firmware, hardware, etc.,), and designed questions together. The questions were open-ended, but focused on topics that are actually relevant to what the candidate would be expected to do, if hired. We created a software design question, a question about debugging FPGA logic based on an instrumented signal, a question about translating a DSP equation into software, and so forth. We then assigned each question to a handful of people so that the same question would be asked by approximately the same people every time.

This, on its own, was a tremendous improvement. With the same people asking the same questions, it became much easier to assess responses. In retrospect, this sounds obvious, but in talking to colleagues this structure is surprisingly rare. Given that interviews are really just a form of informed estimation, simple changes like this, which improve the estimator performance, can be really significant.

Technical Presentation

A 30-minute technical presentation has always been part of the Ettus Research interview process. When we sat down to analyze our hiring results, though, we realized something interesting: the technical interview was highly correlated to whether or not the candidate would receive an offer. Candidates with bad presentations would usually end up not receiving an offer, and the converse was true for good presentations. This seemed to be the case regardless of when in the day the presentation was scheduled.

Relative to the time consumed by a typical on-site interview, a technical presentation is a low investment. Even better, it can be done remotely, which meant we could reduce the occurrence of asking someone to travel out to California only to then not give them an offer; this is expensive for both parties in terms of time, dollars, and opportunity cost.

Hiring Process

The two primary goals of every company's hiring process are:

  1. Source and hire the best people possible for your organization.
  2. Limit the investment made by both the company and by candidates on hires that won't work out.

Below is our hiring process, after making the improvements discussed above. Each step represents progressively more investment by more people; it is designed to limit wasted expense by both the candidate and the company by trying to catch bad matches as soon as possible, while giving good matches progressively more opportunities to interact with the team.

  1. We source candidates through contributions to open-source projects we care about (e.g., code, activity on mailing lists, discussions on bug trackers, etc.,), networking (especially with professors), and submitted resumes.
  2. If the hiring manager decides you might be a potential fit, they set up a 20-minute phone call to make sure you and the position are aligned at a high level. We discuss details of the position like job responsibilities, expectations regarding things like title & salary, and physical office location, then briefly talk through your resume.
  3. We then go to a 30-minute technical phone screen, administered by one or two engineers in a similar role. In some cases, this involves writing code in a shared document.
  4. Next up is the technical presentation. If you are local, you have the option to come in and do it in-person, but we always offer the remote option to make it easier to schedule. The presentation usually takes 30-45 minutes, including questions and discussion.
  5. If the presentation goes well, we bring the candidate on-site for a half-day interview, including a group lunch with the team. The on-site interview is conducted by a series of teams, each asking a question they have practiced and honed in prior interviews.

If a candidate comes through all five steps and we still feel good about them, we can generally be confident that they are a good match.

I'll note a major exception to the above process, which is interns. Like most technical companies, Ettus Research often makes full-time offers to interns. Joel Spolsky (CEO of Stake Overflow, among other things) makes a really interesting statement about interns: for top talent, an internship might be the last time that person will be freely available on the job market for a very long time.

Making the Best Hires Possible

As many people have pointed out, claiming that you "only hire the best and brightest" is both inaccurate and meaningless. Even with a narrowed scope of just "the best people possible", though, hiring well is still a huge challenge.

One of the critical parts of the interview process I described, above, is evaluating culture fit, which is at least as important as technical talent. That said, this is a tricky thing to assess. How can you ascertain that someone will engage in toxic office politics or sabotage your workplace in an interview?

How can you tell that someone is not only a talented engineer, but a mature engineer - the sort of people that are keystones of success - by your hiring process? You can certainly gather evidence, but it's extremely difficult to be sure.

Like all of our other procedures, we are always trying to improve our hiring process with what we learn. The one constant, though, is that hiring is hard.