Robocurve

| Aspect | Robocurve / Inspect Robots | BenchBot | RobotPerf | |--------|----------------------------|----------|----------| | Starting Price | Open-source (MIT) for Inspect Robots; services pricing not published | Open-source research stack (paper + GitHub); some naming collisions exist in public listings | Open-source benchmark suite (pricing not applicable) | | Primary use case | Real-robot evaluation harness with reproducible logs and visualization | Benchmarking semantic scene understanding tasks across sim and real robots | Standardized ROS 2 compute performance benchmarking across hardware platforms | | Strengths | General harness for multiple models/embodiments and an emphasis on trace logging and visualization (Rerun) | Established academic suite for a specific family of tasks; sim+real focus | Clear focus on compute performance with ROS 2 baselines; hardware-agnostic measurement orientation | | Limitations | Evaluation rigor depends on experimental controls; adapter implementation burden; limited independent review data for services/pricing | Narrower scope (semantic understanding tasks); may not focus on LLM/VLM-driven control | Not an end-to-end manipulation/behavior benchmark; targets compute-layer metrics rather than task success | Narrative analysis - Direct competitors and differentiators - Robocurve’s distinguishing characteristics include a public evaluation posture and an open-source harness oriented to modern LLM/VLM-driven robot control with detailed trace logging and visualization. - BenchBot is academically oriented with a defined task family and a multi-repo stack; it is less focused on LLM/VLM agent policies but comparable as an evaluation framework. - RobotPerf competes for standardization in the compute-performance domain and measures different layers of the stack (compute graphs and performance metrics) rather than embodied task capability. - Pricing comparison: value proposition vs alternatives - The core frameworks are open-source; engineering effort and hardware costs drive real customer expenditure. Any services offered by a vendor would add procurement and coordination costs. - Market share and adoption trends - Robocurve appears to be a recent entrant. Frequent release activity for Inspect Robots was visible on public package indices recently, indicating active development, but public evidence of broad production adoption was not found. - When customers choose competitors instead (and why) - RobotPerf is a better match for compute profiling (latency/throughput/power) of ROS 2 systems. - BenchBot is appropriate for semantic scene understanding benchmarks in its target task family. - Organizations may choose in-house frameworks when they require tight safety integration, proprietary hardware support, or strictly controlled experimental protocols.