{
  "eval_id": "tool-selection-2026-07-11",
  "generated_at": "2026-07-11",
  "source": "Internal eval records (cases, per-model results, transcripts; not themselves published) -- every case id, prompt, arg, and score below is copied from those records, run against the shipped tool descriptions.",
  "method": "Anthropic tool-eval rubric, three axes, graded by three fresh-context subagents. Picker sees ONLY the tool names + descriptions and a realistic user ask, and must name the right tool with no args. First-call-success sees names + descriptions + params and must emit the exact first call (tool + JSON args), validated against the frozen zod schemas. Response-usability sees a real production response envelope and must answer the user's question from it and name its next action. No agent saw the expected answers. All results reflect the shipped tool descriptions as of the run date.",
  "models": {
    "picker": "Haiku (the smallest model the descriptions must survive)",
    "first_call": "Sonnet",
    "response_usability": "Sonnet"
  },
  "limitations": "This eval measures tool-selection and first-call / response-envelope usability only: whether a model picks the right tool for a realistic ask, emits a valid first call, and can read and act on a real response envelope. It does NOT measure data quality, coverage, or freshness, and it does NOT measure API availability or latency. It covers the LinkedIn toolset (five LinkedIn tools plus get_usage); the Reddit and X tools shipped after this run and have not yet been through this eval.",
  "picker": {
    "description": "Model sees only the tool names + descriptions; must name the right tool for a realistic ask.",
    "score": "13/13",
    "cases": [
      {
        "id": "P1",
        "ask": "Find the LinkedIn profile of the CTO of Datadog.",
        "expected": [
          "linkedin_search_people"
        ],
        "selected": "linkedin_search_people",
        "pass": true
      },
      {
        "id": "P2",
        "ask": "What has Anthropic been posting on LinkedIn recently?",
        "expected": [
          "linkedin_get_posts"
        ],
        "selected": "linkedin_get_posts",
        "pass": true
      },
      {
        "id": "P3",
        "ask": "Pull the full work history for linkedin.com/in/satyanadella.",
        "expected": [
          "linkedin_get_profile"
        ],
        "selected": "linkedin_get_profile",
        "pass": true
      },
      {
        "id": "P4",
        "ask": "How many employees does Stripe have on LinkedIn?",
        "expected": [
          "linkedin_get_company"
        ],
        "selected": "linkedin_get_company",
        "pass": true
      },
      {
        "id": "P5",
        "ask": "A user pasted this link, what is it? https://www.linkedin.com/company/openai/?utm_source=chatgpt",
        "expected": [
          "linkedin_resolve_url",
          "linkedin_get_company"
        ],
        "selected": "linkedin_resolve_url",
        "pass": true
      },
      {
        "id": "P7",
        "ask": "How many credits do I have left?",
        "expected": [
          "get_usage"
        ],
        "selected": "get_usage",
        "pass": true
      },
      {
        "id": "P8",
        "ask": "Find software engineers at Shopify who used to work at Amazon.",
        "expected": [
          "linkedin_search_people"
        ],
        "selected": "linkedin_search_people",
        "pass": true
      },
      {
        "id": "P9",
        "ask": "Get the education history of williamhgates.",
        "expected": [
          "linkedin_get_profile"
        ],
        "selected": "linkedin_get_profile",
        "pass": true
      },
      {
        "id": "P10",
        "ask": "I only have the website vercel.com. Get their LinkedIn company info.",
        "expected": [
          "linkedin_get_company"
        ],
        "selected": "linkedin_get_company",
        "pass": true
      },
      {
        "id": "P11",
        "ask": "Did Bill Gates post anything about malaria this month?",
        "expected": [
          "linkedin_get_posts"
        ],
        "selected": "linkedin_get_posts",
        "pass": true
      },
      {
        "id": "P12",
        "ask": "Look up the profile of the person named 'Jordan Rivera' who works at Datadog.",
        "expected": [
          "linkedin_search_people"
        ],
        "selected": "linkedin_search_people",
        "pass": true
      },
      {
        "id": "P13",
        "ask": "Check whether my key is still valid and what plan I'm on.",
        "expected": [
          "get_usage"
        ],
        "selected": "get_usage",
        "pass": true
      },
      {
        "id": "P14",
        "ask": "Read this specific post for me: https://www.linkedin.com/feed/update/urn:li:activity:7123456789/",
        "expected": [
          "linkedin_resolve_url",
          "none"
        ],
        "selected": "none",
        "pass": true
      }
    ]
  },
  "first_call_success": {
    "description": "Model sees names + descriptions + params; must emit the exact first call (tool + JSON args), validated against the frozen zod schemas.",
    "score": "8/8",
    "cases": [
      {
        "id": "F1",
        "ask": "Pull the work history for linkedin.com/in/satyanadella.",
        "tool": "linkedin_get_profile",
        "args": {
          "identifier": "linkedin.com/in/satyanadella",
          "sections": [
            "experience"
          ]
        },
        "pass": true
      },
      {
        "id": "F2",
        "ask": "Find software engineers at Shopify who used to work at Amazon.",
        "tool": "linkedin_search_people",
        "args": {
          "title": "Software Engineer",
          "current_company": "Shopify",
          "past_company": "Amazon"
        },
        "pass": true
      },
      {
        "id": "F3",
        "ask": "I only have the website vercel.com. Get their LinkedIn company info.",
        "tool": "linkedin_get_company",
        "args": {
          "identifier": "vercel.com"
        },
        "note": "Website domains are first-class identifiers: the domain reaches the company in one call, zero extra turns.",
        "pass": true
      },
      {
        "id": "F4",
        "ask": "What has Anthropic been posting on LinkedIn recently?",
        "tool": "linkedin_get_posts",
        "args": {
          "identifier": "anthropic"
        },
        "pass": true
      },
      {
        "id": "F5",
        "ask": "Get the education history of williamhgates.",
        "tool": "linkedin_get_profile",
        "args": {
          "identifier": "williamhgates",
          "sections": [
            "education"
          ]
        },
        "pass": true
      },
      {
        "id": "F6",
        "ask": "Search up to 30 results for CTOs at Anthropic, but do not spend more than 15 credits on the call.",
        "tool": "linkedin_search_people",
        "args": {
          "keywords": "CTO Anthropic",
          "title": "CTO",
          "limit": 30,
          "max_credits": 15
        },
        "note": "The hard case: a result cap plus a hard credit ceiling. The model followed the description's budget rule (quote 12 <= max_credits 15) and avoided the company-name-filter-plus-3-pages combination the planner rejects; the server's teaching error remains the safety net for agents that miss it.",
        "pass": true
      },
      {
        "id": "F7",
        "ask": "Get LinkedIn company info for microsoft.com with a hard 6-credit ceiling.",
        "tool": "linkedin_get_company",
        "args": {
          "identifier": "microsoft.com",
          "max_credits": 6
        },
        "note": "Correctly predicted from the published prices that domain resolution quotes 8 credits, exceeding max_credits=6, and expected the call to reject uncharged.",
        "pass": true
      },
      {
        "id": "F8",
        "ask": "(a LinkedIn company URL passed as the identifier)",
        "tool": "linkedin_get_company",
        "args": {},
        "note": "Passed the LinkedIn URL straight through as identifier, without misrouting it through linkedin_resolve_url first.",
        "pass": true
      }
    ]
  },
  "response_usability": {
    "description": "Model sees a real production response envelope; must answer the user's question from it and name its next action.",
    "score": "3/3",
    "cases": [
      {
        "id": "U1",
        "ask": "How many employees does Microsoft have on LinkedIn, and what industry is it in?",
        "envelope_source": "production linkedin_get_company identifier=microsoft",
        "answer": "Microsoft has about 233,334 employees on LinkedIn (listed in the 10001+ range), and its industry is Software Development.",
        "grading_note": "Both facts extracted correctly from the envelope's employee_count / employee_count_range / industry fields.",
        "pass": true
      },
      {
        "id": "U2",
        "ask": "How many credits do I have left and what did my last few calls cost?",
        "envelope_source": "production get_usage",
        "answer": "You have 486 credits remaining. Your last two calls: linkedin_search_people cost 10 credits, and linkedin_get_company cost 4 credits (both on 2026-07-10T19:57:07Z).",
        "grading_note": "Balance and per-call costs with receipt dates extracted correctly from balance_remaining and recent_receipts.",
        "pass": true
      },
      {
        "id": "U3",
        "ask": "Find CTOs at Anthropic; I will want their profiles next.",
        "envelope_source": "production linkedin_search_people keywords=CTO current_company=1035 limit=10",
        "answer": "These results don't look like Anthropic CTOs. The search's current_company=1035 is Microsoft's company id (matching the linkedin_get_company result for Microsoft), not Anthropic's, so the search appears to have run against the wrong company filter. Of the 5 results, three are Microsoft/Xbox-affiliated and two are CTOs of unrelated small startups; none show an Anthropic affiliation. Next action: re-run linkedin_search_people with the correct Anthropic company id, not 1035, before spending further credits on profile fetches.",
        "grading_note": "The eval prompt itself mislabeled the search (the envelope was actually a Microsoft-id (1035) search mislabeled 'Anthropic'). The agent caught the mismatch from the envelope's own echoed current_company field, refused to spend credits fetching wrong-company profiles, and named the correct next action.",
        "pass": true
      }
    ]
  },
  "response_envelopes_used": {
    "note": "Response-usability cases were graded against real production response envelopes captured 2026-07-10. Full envelope bytes are not republished here: they carry a live account's receipt ids and claim token from that snapshot. Envelope shape is documented separately at /openapi.json.",
    "U1": "production linkedin_get_company identifier=microsoft",
    "U2": "production get_usage",
    "U3": "production linkedin_search_people keywords=CTO current_company=1035 limit=10"
  },
  "availability_and_latency": {
    "status": "pending",
    "note": "30-day availability and p50/p95 latency are synthetics-derived and require an accumulation window; no data exists yet. This section will be filled in once synthetic monitoring has accumulated a 30-day window."
  },
  "verdict": "Descriptions pass the picker axis on a small fast model (Haiku) and the first-call axis on a mid-size model (Sonnet). The hardest cases (a result cap combined with a hard credit ceiling; a website domain as the only identifier) pass on the first call with the shipped descriptions."
}