testpilot test

Test the scenarios in the provided pilot plans

testpilot test [path/to/plan.pilot.yaml | path/to/directory/with/pilot/plans] [flags]

Examples​

# Run all tests in the current directory
testpilot test

# Run tests from a specific pilot plan file
testpilot test ./my-test.pilot.yaml

# Run tests from a specific directory
testpilot test ./tests/

# Run only tests with names containing "login"
testpilot test --filter login

# Run tests and save reports to a custom directory
testpilot test --outdir ./test-results

# Run tests in shard 1 of 3 total shards
testpilot test --shard 1/3

Options​

Option	Description
-f, --filter string	only run tests with names containing the given string
-h, --help	help for test
--concurrency int	number of tests to run concurrently (default 10)
--driver string	driver to use for the test (playwright or appium) (default “playwright”)
—launch-vars stringToString	Environment variables passed when launching the application under test. Currently only applies to iOS apps and tests (default [])
--os string	operating system to run tests on (local, linux, macos, windows) (default “local”)
--os-version string	operating system version to run tests on (e.g., “13”, “18.3”)
--outdir string	Directory in which to save reports (default “testpilot-out”)
--runtime string	runtime environment to execute the test in (chromium, firefox, webkit, native) (default “chromium”)
--shard value	Split tests into n shards and only run tests in shard x
--udid value	specify the UDID of the iOS device you want to test. Note: this cannot be used along with --os-version

Advanced Options​

The following options are available but not commonly used:

Option	Description
--show-browser	show the browser
--viewport ints	screen dimensions to use for your web browser (playwright driver only)

Environment Variables​

The following environment variables are required or affect the behavior of the test command:

Variable	Required	Description
OPENAI_API_KEY	Yes	OpenAI API key for LLM interactions
ANTHROPIC_API_KEY	No	Anthropic API key (optional)
OPENAI_BASE_URL	No	Custom OpenAI API base URL

SEE ALSO​

• testpilot - Testpilot CLI