About EvalCore
What is EvalCore?
EvalCore is a powerful tool designed to record and replay the behavior of AI models, ensuring that any changes made to the models can be accurately tracked and verified. It operates as a single binary that supports any programming language, eliminating the need for flaky tests and complex setups. With EvalCore, users can quickly get started by installing the open-source software, which is available for various platforms including macOS and Linux.
Key features of EvalCore include:
Deterministic evaluations: Every model call is recorded to a local SQLite cassette, allowing for precise replay without the need for network access or API keys.
Compatibility: Works seamlessly with OpenAI, vLLM, Ollama, and any OpenAI-compatible gateways, as well as custom REST APIs.
Easy integration: Users can ship their first evaluation in minutes with minimal dependencies and straightforward configuration.
CI-friendly: EvalCore supports continuous integration by replaying recorded evaluations, ensuring consistent results across different environments.
EvalCore Features
EvalCore is a powerful tool designed to record and replay AI model evaluations, ensuring consistent behavior across different environments. It operates as a single binary that supports any programming language, making it versatile and easy to integrate into existing workflows. With EvalCore, users can run evaluations offline, replay trials in continuous integration (CI) environments, and compare model performance without the need for network access or API keys.
Key features and capabilities of EvalCore include:
Recording every model call to a local SQLite cassette for deterministic replay.
Support for various AI models and APIs, including OpenAI and custom REST APIs.
Ability to run evaluations with minimal configuration using YAML files.
Offline operation, ensuring that evaluations can be conducted without internet connectivity.
Integration with CI systems to automatically replay evaluations and check for regressions.
Why EvalCore?
EvalCore offers significant advantages for developers and teams working with AI models by providing a reliable way to record and replay model evaluations. This ensures that any changes made to the AI are thoroughly tested against previous behaviors, allowing for consistent and deterministic results without the need for complex setups or flaky tests.
Some key benefits of using EvalCore include:
Offline replay capabilities, which eliminate the need for network access and API keys during testing.
Deterministic evaluations, ensuring that identical inputs yield identical outputs every time.
Easy integration with various AI models and APIs, including OpenAI and custom REST APIs.
Minimal setup requirements, with a single dependency-free binary that can be installed quickly.
How to Use EvalCore
To get started with EvalCore, you can easily install it as a dependency-free binary that works across various platforms including macOS, Linux, and CI runners. Simply run the commandcargo install evalcoreto install it. If you prefer not to set up a Rust toolchain, prebuilt binaries are available for your convenience.
Once installed, you can begin using EvalCore by running your evaluations with the command$ evalcore run evals.yaml. This allows you to record and replay trials, compare models, and analyze agent traces without the need for complex setups or flaky tests.
Record every model call to a local SQLite cassette for deterministic evaluations.
Replay evaluations offline without any network dependencies.
Compatible with various AI models and APIs, including OpenAI and custom REST APIs.
Ready to see what EvalCore can do for you?and experience the benefits firsthand.
Key Features
How to Use
Visit the Website
Navigate to the tool's official website.