remotelabor.ai

What It Is

Remote Labor Index (RLI) is a research benchmark developed by authors associated with Center for AI Safety, Scale AI, and others. Its goal is to measure the end-to-end automation capability of AI Agents on real remote-work projects. It is not an AI app or SaaS tool in the usual sense, but a benchmark framework for evaluating whether AI systems can complete economically valuable work, accompanied by a paper, GitHub code, and a results dashboard.

Core Capabilities and Evaluation Dimensions

RLI’s key feature is that its tasks come from real remote labor scenarios, covering multiple industries including game development, product design, architecture, data analysis, video animation, and more. According to the main text, these projects represent more than 6,000 hours of real work and over $140,000 in value; some individual projects cost more than $10,000 and took over 100 hours to complete. Both project costs and working hours are based on actual human professionals who completed the work, making the benchmark closer to real economic activity than many knowledge QA or reasoning benchmarks.

Result Quality and Limitations

According to the page, current frontier AI Agents perform at “near-floor level” on RLI, and even the best models still have a very low automation rate. This means that when tasks must meet the acceptable quality standards of real commissioned work, existing AI systems still struggle to complete the vast majority of complex remote projects. RLI’s value lies in providing a shared metric that can be tracked over time, but the page does not show specific model rankings or full scores, only noting that the latest results can be viewed at dashboard.safe.ai.

Pricing, Integrations, and Support

The main text does not disclose commercial pricing, free quotas, trial policies, payment methods, or API integration details. The page provides a paper, GitHub code, and citation format, making it look more like an open research project than a commercial product. Service support, data privacy, Chinese-language interface, and support for Chinese-language tasks are also not explained in the captured text.

Pros, Cons, and Who It’s For

Its strengths are realistic tasks, broad coverage, and clearly defined economic value, making it suitable for AI researchers, Agent development teams, enterprise automation strategy teams, and policy researchers. Its drawbacks are that it is not a direct productivity tool for ordinary users, the barrier to use may be research-oriented, and it lacks information on commercial deployment, Chinese-language support, and privacy compliance.

Access from China

The page does not provide information about access from mainland China, network stability, or payment-related matters, so this remains unknown. For similar capability evaluations, users may also follow Agent and complex-task benchmarks such as SWE-bench, GAIA, OSWorld, and WebArena.