Benchmark · tool searchInjection defense →
How often search finds the right tool.
95.4%recall@5 for Toolrails
recall@5 means the right tool is in the top five results. Measured on 1,840 labeled queries over 4,213 tools, re-run on every merge.
- Toolrails95.4%
- StackOne (reported)~94%
- Zero-shot classifier82%
- Composio71%
- Keyword search (BM25)40%
StackOne's figure is self-reported accuracy from a public talk, not re-run on this set. Last run 22 Sep 2026 · commit 3f9a2c1
Try a query
Same query. Two searches.
Keyword search (BM25)1 of 3 found
github.list_reposmissstripe.list_chargesmissslack.list_channelsmisszendesk.list_ticketsmatchshopify.list_ordersmiss
Toolrails3 of 3 found
linear.list_issuesmatchzendesk.list_ticketsmatchjira.search_issuesmatch
Toolrails knows "tickets" also means issues and cases, and drops unrelated list tools.
By query type
Scroll sideways to see every system →
| Query type | Toolrails | StackOne | Composio | Arcade | BM25 |
|---|---|---|---|---|---|
| Synonyms (tickets = issues) | 96% | n/a | 68% | 64% | 31% |
| Steps that must run first | 93% | n/a | 22% | 18% | 12% |
| Broad requests | 94% | n/a | 70% | 66% | 44% |
| Exact action names | 99% | n/a | 97% | 96% | 95% |
| Average rank (MRR) | 0.91 | n/a | 0.62 | 0.58 | 0.34 |
How search works
Keyword and meaning-based search run side by side and their results are merged. A reranker only steps in when the top results are too close to call.
Fewer, better results
Weak matches are dropped instead of padding the list. Median results per query:
2.4.Run it yourself
The test set and harness are open source.
npx @toolrails/tool-search bench