Docs›Best Practices

Best Practices for AI-Era Hiring

Expert guidance for creating effective assessments, evaluating candidates, and building successful AI-literate teams.

Assignment Design Best Practices

The REAL Framework

Design assignments that are Realistic, Engaging, AI-dependent, and Learnable.

Realistic

Mirror actual work scenarios the candidate would face in the role.

Engaging

Interesting problems that motivate candidates to do their best work.

AI-dependent

Require AI collaboration to complete effectively within time limits.

Learnable

Provide clear success criteria and helpful examples.

✅ Assignment Do's

• Include specific deliverables and success criteria
• Provide enough context without overwhelming
• Set realistic time limits (60-90 minutes typically)
• Add constraints that require strategic thinking
• Include examples of good and bad AI interactions
• Test assignments with your team first
• Use clear, role-appropriate language

❌ Assignment Don'ts

• Make tasks solvable without AI assistance
• Use vague or overly broad problem statements
• Set unrealistic time constraints
• Include industry jargon without explanation
• Design single "correct answer" scenarios
• Create overly complex multi-part tasks
• Forget to include AI usage examples

Evaluation Excellence

The CHAT Evaluation Method

Evaluate candidates across four key dimensions of AI collaboration effectiveness.

Communication Quality

How candidate communicate with AI?

Look for: Specific prompts, clear context, appropriate scope, building conversations

Human Judgment

Does the candidate validate and improve AI output?

Look for: Critical evaluation, fact-checking, customization, quality control

Adaptive Thinking

How well do they iterate and refine their approach?

Look for: Multiple refinement cycles, learning from responses, progressive improvement

Task Execution

Strategically break down and solve the problem?

Look for: Problem structuring, systematic approach, goal achievement

Pro Evaluation Tips

• Read the entire chat log, not just the AI assessment
• Look for patterns across multiple interactions
• Consider the assignment difficulty and time constraints
• Value strategic thinking over perfect execution
• Get multiple team perspectives on borderline cases
• Document specific examples to support decisions

Building Effective Review Teams

Team Composition Strategy

AI Expert

Evaluates prompt quality and AI utilization effectiveness

Domain Expert

Assesses technical knowledge and industry understanding

Team Lead

Evaluates fit with team culture and collaboration style

Collaborative Review Process

Independent Review: Each team member evaluates separately
Comment Discussion: Review team feedback together
Consensus Building: Discuss disagreements and align on standards
Decision Documentation: Record rationale for future reference

Managing Disagreements

• Specific Examples: Point to chat interactions that support your view
• Role Context: Consider what matters most for the specific position
• Follow-up Interviews: Use disagreements to generate interview questions
• Calibration: Regularly review decisions as a team to improve consistency

Real Chat Examples: Strong AI Collaboration

Case Study: Debugging Login Issue

This example demonstrates excellent AI collaboration skills - systematic problem-solving, specific prompts, iterative refinement, and strategic thinking.

user

Bug report: 'Login sometimes fails on Android'. What specific information should I collect first to understand the scope and cause?

assistant

Let's gather systematic debugging information: Immediate Information Needed:

Device specifics - Android OS version, device model, app version
Error details - Exact error messages, screenshots, reproduction steps
Frequency - How often it occurs (1/10 attempts, daily?)
Network conditions - WiFi vs cellular, specific carriers
User patterns - Time of day, specific user segments affected Backend Investigation:

Authentication service logs around failure times
Database connection status and query performance
API response times and error rates Start with user-facing data collection, then correlate with backend metrics.

user

The report mentions "session expired" error specifically. Help me form hypotheses about root causes and suggest tests.

assistant

Based on "session expired" error, here are likely hypotheses: Hypothesis 1: Token refresh mechanism failing

Theory: JWT tokens expiring but refresh calls failing silently
Test: Monitor token lifecycle → capture refresh API calls → check 401/403 responses Hypothesis 2: Network timeout during auth requests
Theory: Slow networks causing auth timeouts, showing as "expired"
Test: Simulate slow network → measure request timing → check timeout handling Hypothesis 3: Race condition in session management
Theory: Concurrent requests invalidating active sessions
Test: Automated parallel API calls with same session → check premature invalidation I'd start with Hypothesis 1 - the specific "session expired" message suggests users see actual session state rather than network failures.

user

Let's focus on the token refresh hypothesis. Outline a reproduction plan to confirm this root cause.

assistant

Token Refresh Reproduction Plan: Setup:

Android device with network monitoring (Charles Proxy)
Enable auth service debug logging
Identify token expiry time (check JWT payload or session duration) Reproduction Steps:
Fresh login → capture initial tokens (access + refresh)
Wait near expiry → use app normally until ~30 seconds before expiry
Trigger refresh → make API call requiring auth → monitor refresh attempt
Capture results → log refresh response, new tokens, error states What to Monitor:

Refresh token API calls (endpoint, payload, response codes)
Token expiry times vs actual refresh timing
Client-side retry behavior on 401 responses
Server logs showing refresh token validation failures Expected Findings:
Refresh requests returning 400/401 with expired refresh tokens
Client not handling refresh failures gracefully
Missing retry logic after failed refresh This should confirm if the token mechanism is the root cause vs other factors.

✅ Why This Is Excellent

• Structured approach: Systematic information gathering → hypothesis formation → testing
• Specific prompts: Asks for concrete deliverables (questionnaire, reproduction plan, metrics)
• Strategic thinking: Considers rollback plans, success criteria, and business impact
• Iterative refinement: Builds on each AI response to go deeper into the problem
• Professional output: Ends with a structured bug ticket ready for implementation

Advanced Evaluation Techniques

Conversation Flow Analysis

Look beyond individual prompts to understand the candidate's strategic thinking process.

Strong Flow Patterns

• Exploration → Planning → Execution → Refinement
• Building complexity over time
• Connecting ideas across interactions
• Adapting strategy based on AI responses

Weak Flow Patterns

• Random, unconnected requests
• No clear progression or strategy
• Repetitive questions without learning
• Jumping between unrelated topics

Red Flag Detection

Identify concerning patterns that might indicate poor AI collaboration skills.

Over-Dependency

Asking AI to do everything without adding human insight or validation.

Prompt Copying

Using generic prompts found online without customization for the specific task.

No Validation

Accepting AI output without checking accuracy or appropriateness.

Scaling Your AI Hiring Process

Building Assignment Libraries

• Create role-specific assignment templates
• Document what works for different positions
• Share successful assignments across teams
• Regularly update based on hiring outcomes
• Version control assignment improvements

Team Training & Calibration

• Regular review calibration sessions
• Share examples of strong vs. weak submissions
• Document evaluation decisions and rationale
• Train new team members on AI assessment
• Continuously improve evaluation standards