2. Do not ask basic software literacy questions. First of all, this was completely stupid even before LLMs. Secondly, its easy to cheat. If you absolutely have to do this then do it terms of measures. Most people in software are entirely incapable of measuring anything and LLMs cannot fix their personality deficiency.
3. Ask all questions where the expected answer is a not some factoid nonsense but a decision they must make. Evaluate their answer on the grounds of risk, coverage, delivery, and performance. For example if you are interviewing a AI/ML guy ask them about how they overcome bias in the algorithms and how they weigh the consequences of different design outcomes. If they are a QA ask them about how they will take ownership of quality analysis for work already in production or how they will coach developers when communicating steps to reproduce a defect.
4. As an interviewer you should know, by now, how to listen to people. That is so much more than just audible parsing of words. If their words say one thing, but their body language says something different then they are full of shit. Its okay that they aren't experts in everything. Their honesty and humility is far more important. They can get every question wrong, but if their honesty is on and they can make solid decisions then they are at least in the top half of consideration.
5. Finally, after evaluating their decision making ability and risk analysis then ask them for a story where they have encountered such a problem in the past and had to learn from failure.
This question comes up at least once a month so this answer is copy/paste from a prior comment.
Some software literacy questions are fine. My favorites are what is grep, what's functional programming, what is the difference between TCP and UDP, what does hashing do.
Even if they were cheating, you could tell because the definition would be too definition-like. I expect people to stumble over what is functional programming, but they should be able to bring up things like immutability or the map() thing lol.
Another favorite is "what is TDD". The wrong answer is being fanatical/emotional about it either way.
It could be worth reviewing Ogletree v. Cleveland State University before doing this. A court ruled that a room scan violated a person’s 4th amendment rights.
There is also the risk that the person could have something on their wall which could indicate they are in a protected class, and if they don’t get the job, they could claim you used this information against them.
While I’m sure cameras for interviews are likely commonplace, it does open up some risk. Some may see it as an acceptable amount of risk, others may not.
That is entirely the candidate's liability. If they don't want certain identifying materials or decorations made visible during the interview then don't make them visible to the camera. There is no law that says as a hiring manager I must wear a blindfold or otherwise hide from the candidate.
As a fully remote hiring manager I am interviewing a person and I need to see that person to know if or when they are full of shit when they talk. If that isn't acceptable they can interview somewhere else.
> Almost all candidates nowadays seem to have some form of external help or LLM-based assistance setup during remote interviews.
I haven't experienced this, but isn't it super obvious when someone is using some kind of text/audio interface, waiting for a reply, regurgitating an answer they don't understand, and not talking in a conversational way?
Doesn't this fall apart if you ask for quick clarifications about what they're saying because they won't be able to get the answer fast enough to reply naturally?
Ask about the candidates’ previous experiences, setbacks, and lessons learned from them. LLMs can certainly make up all of that, but it is much harder for candidates to hide inaccuracies in their backgrounds, particularly over time.
Probably best to ask them detailed questions about their particular resume. In theory you could cram all that into an LLM but the quality will certainly deteriorate.
I haven't interviewed people at all in a year or so though.
I'll ask them to sketch out a systems architecture in real time for me. Like put the camera on the paper and tell me how the data should flow, what tools are being used etc
Not a complete answer by any means, but it may be worthwhile plugging your questions into popular LLMs. Recently I interviewed a very suspiciously behaving candidate. The algorithm they approached the problem with, which I had not seen anyone use in years of asking that question, was also the exact same that ChatGPT suggested for the problem.
One suggestion I saw recently in a thread was asking deliberately incorrect questions, such as how to implement a particular solution using an irrelevant technology. LLMs are so 'eager to please' they seem to just BS some nonsense in response. Not sure how I feel about that approach however.
Nothing changes. I ask them to describe the project that they are most proud of or they found the most challenging and ask them to describe the business and technical implementation and dig into their thinking process.
Unpopular opinion, if your company is like 90% of the companies, you’re doing boring framework CRUD development. Your candidates don’t need to reverse a b tree on the white board while riding a unicycle on a tightrope.
The complexity in most development is managing business complexity and a large code base. You’re not going to be able to suss that out by coding interview.
System design based on their actual experience and behavioral questions are better anyway.
Besides if your coding interview can be passed by using an LLM and your day to day coding can’t, by definition your interview isn’t an accurate assessment of whether they can do the job.
The assumption that every programming round is a b tree reversal is not right, and I think programming rounds tell a lot about the candidates thought process, how they structure the code, how they handle edge cases and a lot of other things.
Both system design / verbal discussions and programming rounds are critical to the interview process.
What’s the percentage of the engineers at your company who could pass your current interview? If that number is not 100%, your interview process is rather useless.
What’s wrong with the usage of tools? Before LLMs I was searching for “behavioral interview questions” all over the net. I was searching github for previous home take assignments, I was asking ex employees about how company is like, and what kind of questions they ask in the interview. It took time. Nowadays if I can save some time using LLMs, again, what’s wrong with that?
People don’t work in isolation. We use tools to enhance our productivity. If you ask me to write code in Notepad, I won’t be as effective as I could be if i were using Jetbrains IDEs. If I need to do some calculations, I’ll rely on a calculator rather than doing maths in my head. I’ll use grammarly (well, not anymore) to polish my non-native english writing… LLMs are just tools man.
If any, you want to hire people who are good with tools, rather than “geniuses”
1. Interview candidates with cameras on.
2. Do not ask basic software literacy questions. First of all, this was completely stupid even before LLMs. Secondly, its easy to cheat. If you absolutely have to do this then do it terms of measures. Most people in software are entirely incapable of measuring anything and LLMs cannot fix their personality deficiency.
3. Ask all questions where the expected answer is a not some factoid nonsense but a decision they must make. Evaluate their answer on the grounds of risk, coverage, delivery, and performance. For example if you are interviewing a AI/ML guy ask them about how they overcome bias in the algorithms and how they weigh the consequences of different design outcomes. If they are a QA ask them about how they will take ownership of quality analysis for work already in production or how they will coach developers when communicating steps to reproduce a defect.
4. As an interviewer you should know, by now, how to listen to people. That is so much more than just audible parsing of words. If their words say one thing, but their body language says something different then they are full of shit. Its okay that they aren't experts in everything. Their honesty and humility is far more important. They can get every question wrong, but if their honesty is on and they can make solid decisions then they are at least in the top half of consideration. 5. Finally, after evaluating their decision making ability and risk analysis then ask them for a story where they have encountered such a problem in the past and had to learn from failure.
This question comes up at least once a month so this answer is copy/paste from a prior comment.
Some software literacy questions are fine. My favorites are what is grep, what's functional programming, what is the difference between TCP and UDP, what does hashing do.
Even if they were cheating, you could tell because the definition would be too definition-like. I expect people to stumble over what is functional programming, but they should be able to bring up things like immutability or the map() thing lol.
Another favorite is "what is TDD". The wrong answer is being fanatical/emotional about it either way.
> 1. Interview candidates with cameras on.
It could be worth reviewing Ogletree v. Cleveland State University before doing this. A court ruled that a room scan violated a person’s 4th amendment rights.
There is also the risk that the person could have something on their wall which could indicate they are in a protected class, and if they don’t get the job, they could claim you used this information against them.
While I’m sure cameras for interviews are likely commonplace, it does open up some risk. Some may see it as an acceptable amount of risk, others may not.
That is entirely the candidate's liability. If they don't want certain identifying materials or decorations made visible during the interview then don't make them visible to the camera. There is no law that says as a hiring manager I must wear a blindfold or otherwise hide from the candidate.
As a fully remote hiring manager I am interviewing a person and I need to see that person to know if or when they are full of shit when they talk. If that isn't acceptable they can interview somewhere else.
> Almost all candidates nowadays seem to have some form of external help or LLM-based assistance setup during remote interviews.
I haven't experienced this, but isn't it super obvious when someone is using some kind of text/audio interface, waiting for a reply, regurgitating an answer they don't understand, and not talking in a conversational way?
Doesn't this fall apart if you ask for quick clarifications about what they're saying because they won't be able to get the answer fast enough to reply naturally?
Well; anything that’s added here is immediately training food for the next model..:..:.
Just ask them how photons lose energy due to inflation and where that energy goes.
Ask about the candidates’ previous experiences, setbacks, and lessons learned from them. LLMs can certainly make up all of that, but it is much harder for candidates to hide inaccuracies in their backgrounds, particularly over time.
Probably best to ask them detailed questions about their particular resume. In theory you could cram all that into an LLM but the quality will certainly deteriorate.
I haven't interviewed people at all in a year or so though.
I'll ask them to sketch out a systems architecture in real time for me. Like put the camera on the paper and tell me how the data should flow, what tools are being used etc
Not a complete answer by any means, but it may be worthwhile plugging your questions into popular LLMs. Recently I interviewed a very suspiciously behaving candidate. The algorithm they approached the problem with, which I had not seen anyone use in years of asking that question, was also the exact same that ChatGPT suggested for the problem.
One suggestion I saw recently in a thread was asking deliberately incorrect questions, such as how to implement a particular solution using an irrelevant technology. LLMs are so 'eager to please' they seem to just BS some nonsense in response. Not sure how I feel about that approach however.
Nothing changes. I ask them to describe the project that they are most proud of or they found the most challenging and ask them to describe the business and technical implementation and dig into their thinking process.
I have a standard set of behavioral questions.
This definitely helps, but what about the programming or implementation rounds ?
Unpopular opinion, if your company is like 90% of the companies, you’re doing boring framework CRUD development. Your candidates don’t need to reverse a b tree on the white board while riding a unicycle on a tightrope.
The complexity in most development is managing business complexity and a large code base. You’re not going to be able to suss that out by coding interview.
System design based on their actual experience and behavioral questions are better anyway.
Besides if your coding interview can be passed by using an LLM and your day to day coding can’t, by definition your interview isn’t an accurate assessment of whether they can do the job.
The assumption that every programming round is a b tree reversal is not right, and I think programming rounds tell a lot about the candidates thought process, how they structure the code, how they handle edge cases and a lot of other things.
Both system design / verbal discussions and programming rounds are critical to the interview process.
What’s the percentage of the engineers at your company who could pass your current interview? If that number is not 100%, your interview process is rather useless.
What’s wrong with the usage of tools? Before LLMs I was searching for “behavioral interview questions” all over the net. I was searching github for previous home take assignments, I was asking ex employees about how company is like, and what kind of questions they ask in the interview. It took time. Nowadays if I can save some time using LLMs, again, what’s wrong with that?
People don’t work in isolation. We use tools to enhance our productivity. If you ask me to write code in Notepad, I won’t be as effective as I could be if i were using Jetbrains IDEs. If I need to do some calculations, I’ll rely on a calculator rather than doing maths in my head. I’ll use grammarly (well, not anymore) to polish my non-native english writing… LLMs are just tools man.
If any, you want to hire people who are good with tools, rather than “geniuses”
Seem like you are confused about what an interview is.
Interview is there to judge if you have the requisite knowledge of programming based on your position and experience.
If interviewer wants to test your usage of tools, it will be structured accordingly.