Shortest: an AI automated testing tool that uses natural language for end-to-end testing

General Introduction

Shortest is an AI-powered natural language end-to-end testing framework developed by the Anti-Work team. It is built on Playwright and supports GitHub integration and two-factor authentication (2FA).The main feature of Shortest is to write test cases through natural language and utilize Anthropic Claude API for AI-driven test execution. The framework is designed to streamline the testing process, improve testing efficiency, and enable developers to write and maintain test cases more easily.

Function List

• Natural Language Test Writing: Writing test cases in natural language lowers the threshold for writing tests.
• AI-driven test execution: AI-driven test execution utilizing the Anthropic Claude API with a high degree of automation.
• Based on Playwright: Built on top of Playwright to provide powerful browser automation.
• GitHub Integration: Seamless integration with GitHub is supported, including two-factor authentication (2FA).
• email verification: Email validation via Mailosaur ensures comprehensive testing.
• callback function: Supports the use of callback functions to add additional assertions and logic.
• lifecycle hook: Provides lifecycle hooks that allow custom logic to be executed before and after test execution.

Using Help

Installation process

1. Install Shortest using npm, pnpm or yarn:

   npm install -D @antiwork/shortest
# 或者
pnpm add -D @antiwork/shortest

2. commander-in-chief (military).shortest/add to.gitignorein order to store screenshots and caches of each test run:

   echo ".shortest/" >> .gitignore

3. Identify the test portal and add the Anthropic API key to the configuration file:

   // shortest.config.ts
import type { ShortestConfig } from "@antiwork/shortest";
export default {
headless: false,
baseUrl: "http://localhost:3000",
testPattern: "**/*.test.ts",
anthropicKey: process.env.ANTHROPIC_API_KEY,
} satisfies ShortestConfig;

Usage Process

1. Write test cases:

   // app/__tests__/login.test.ts
import { shortest } from "@antiwork/shortest";
shortest("Login to the app using email and password", {
username: process.env.GITHUB_USERNAME,
password: process.env.GITHUB_PASSWORD,
});

2. Use callback functions to add additional assertions and logic:

   import { shortest } from "@antiwork/shortest";
import { db } from "@/lib/db/drizzle";
import { users } from "@/lib/db/schema";
import { eq } from "drizzle-orm";
shortest("Login to the app using username and password", {
username: process.env.USERNAME,
password: process.env.PASSWORD,
}).after(async ({ page }) => {
const clerkId = await page.evaluate(() => {
return window.localStorage.getItem("clerk-user");
});
if (!clerkId) {
throw new Error("User not found in database");
}
const [user] = await db
.select()
.from(users)
.where(eq(users.clerkId, clerkId))
.limit(1);
expect(user).toBeDefined();
});

operational test

1. Run tests using npx, pnpm, or yarn:

   npx shortest
# 或者
pnpm shortest
# 或者
yarn shortest

Detailed Functions

• Natural Language Test Writing: Shortest allows users to write test cases in natural language, such as "Log in to the app and use your email and password". This approach lowers the technical barrier to writing tests and allows non-technical people to participate in test writing.
• AI-driven test execution: Shortest utilizes the Anthropic Claude API for AI-driven test execution that intelligently understands and executes test cases to improve test automation and accuracy.
• Based on Playwright: Shortest is built on Playwright and provides powerful browser automation with support for testing on multiple browsers and devices.
• GitHub Integration: Shortest supports seamless integration with GitHub, including two-factor authentication (2FA), ensuring the security and reliability of the testing process.
• email verification: Email validation via Mailosaur ensures comprehensive and accurate testing.
• callback function: Shortest supports the use of callback functions to add additional assertions and logic, allowing users to perform further validation and manipulation after test execution.
• lifecycle hook: Shortest provides lifecycle hooks that allow users to execute customized logic before and after test execution, such as setting up the test environment before the test and cleaning up the test data after the test.