forked from kevadesu/forgejo
Kd/ci playwright go test (#20123)
* Add initial playwright config * Simplify Makefile * Simplify Makefile * Use correct config files * Update playwright settings * Fix package-lock file * Don't use test logger for e2e tests * fix frontend lint * Allow passing TEST_LOGGER variable * Init postgres database * use standard gitea env variables * Update playwright * update drone * Move empty env var to commands * Cleanup * Move integrations to subfolder * tests integrations to tests integraton * Run e2e tests with go test * Fix linting * install CI deps * Add files to ESlint * Fix drone typo * Don't log to console in CI * Use go test http server * Add build step before tests * Move shared init function to common package * fix drone * Clean up tests * Fix linting * Better mocking for page + version string * Cleanup test generation * Remove dependency on gitea binary * Fix linting * add initial support for running specific tests * Add ACCEPT_VISUAL variable * don't require git-lfs * Add initial documentation * Review feedback * Add logged in session test * Attempt fixing drone race * Cleanup and bump version * Bump deps * Review feedback * simplify installation * Fix ci * Update install docs
This commit is contained in:
Kyle D
GitHub
parent
5710ff343c
commit
c8ded77680
644 changed files with 1857 additions and 1027 deletions
93
tests/e2e/README.md
Normal file
93
tests/e2e/README.md
Normal file
|
|
@ -0,0 +1,93 @@
|
|||
# End to end tests
|
||||
|
||||
E2e tests largely follow the same syntax as [integration tests](tests/e2e/README.md).
|
||||
Whereas integration tests are intended to mock and stress the back-end, server-side code, e2e tests the interface between front-end and back-end, as well as visual regressions with both assertions and visual comparisons.
|
||||
They can be run with make commands for the appropriate backends, namely:
|
||||
```shell
|
||||
make test-sqlite
|
||||
make test-pgsql
|
||||
make test-mysql
|
||||
make test-mysql8
|
||||
make test-mssql
|
||||
```
|
||||
|
||||
Make sure to perform a clean front-end build before running tests:
|
||||
```
|
||||
make clean frontend
|
||||
```
|
||||
|
||||
## Install playwright system dependencies
|
||||
```
|
||||
npx playwright install-deps
|
||||
```
|
||||
|
||||
|
||||
## Run all tests via local drone
|
||||
```
|
||||
drone exec --local --build-event "pull_request"
|
||||
```
|
||||
|
||||
## Run sqlite e2e tests
|
||||
Start tests
|
||||
```
|
||||
make test-e2e-sqlite
|
||||
```
|
||||
|
||||
## Run MySQL e2e tests
|
||||
Setup a MySQL database inside docker
|
||||
```
|
||||
docker run -e "MYSQL_DATABASE=test" -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" -p 3306:3306 --rm --name mysql mysql:latest #(just ctrl-c to stop db and clean the container)
|
||||
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --rm --name elasticsearch elasticsearch:7.6.0 #(in a second terminal, just ctrl-c to stop db and clean the container)
|
||||
```
|
||||
Start tests based on the database container
|
||||
```
|
||||
TEST_MYSQL_HOST=localhost:3306 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=root TEST_MYSQL_PASSWORD='' make test-e2e-mysql
|
||||
```
|
||||
|
||||
## Run pgsql e2e tests
|
||||
Setup a pgsql database inside docker
|
||||
```
|
||||
docker run -e "POSTGRES_DB=test" -p 5432:5432 --rm --name pgsql postgres:latest #(just ctrl-c to stop db and clean the container)
|
||||
```
|
||||
Start tests based on the database container
|
||||
```
|
||||
TEST_PGSQL_HOST=localhost:5432 TEST_PGSQL_DBNAME=test TEST_PGSQL_USERNAME=postgres TEST_PGSQL_PASSWORD=postgres make test-e2e-pgsql
|
||||
```
|
||||
|
||||
## Run mssql e2e tests
|
||||
Setup a mssql database inside docker
|
||||
```
|
||||
docker run -e "ACCEPT_EULA=Y" -e "MSSQL_PID=Standard" -e "SA_PASSWORD=MwantsaSecurePassword1" -p 1433:1433 --rm --name mssql microsoft/mssql-server-linux:latest #(just ctrl-c to stop db and clean the container)
|
||||
```
|
||||
Start tests based on the database container
|
||||
```
|
||||
TEST_MSSQL_HOST=localhost:1433 TEST_MSSQL_DBNAME=gitea_test TEST_MSSQL_USERNAME=sa TEST_MSSQL_PASSWORD=MwantsaSecurePassword1 make test-e2e-mssql
|
||||
```
|
||||
|
||||
## Running individual tests
|
||||
|
||||
Example command to run `example.test.e2e.js` test file:
|
||||
|
||||
_Note: unlike integration tests, this filtering is at the file level, not function_
|
||||
|
||||
For SQLite:
|
||||
|
||||
```
|
||||
make test-e2e-sqlite#example
|
||||
```
|
||||
|
||||
For other databases(replace `mssql` to `mysql`, `mysql8` or `pgsql`):
|
||||
|
||||
```
|
||||
TEST_MSSQL_HOST=localhost:1433 TEST_MSSQL_DBNAME=test TEST_MSSQL_USERNAME=sa TEST_MSSQL_PASSWORD=MwantsaSecurePassword1 make test-e2e-mssql#example
|
||||
```
|
||||
|
||||
## Visual testing
|
||||
|
||||
Although the main goal of e2e is assertion testing, we have added a framework for visual regress testing. If you are working on front-end features, please use the following:
|
||||
- Check out `main`, `make clean frontend`, and run e2e tests with `VISUAL_TEST=1` to generate outputs. This will initially fail, as no screenshots exist. You can run the e2e tests again to assert it passes.
|
||||
- Check out your branch, `make clean frontend`, and run e2e tests with `VISUAL_TEST=1`. You should be able to assert you front-end changes don't break any other tests unintentionally.
|
||||
|
||||
VISUAL_TEST=1 will create screenshots in tests/e2e/test-snapshots. The test will fail the first time this is enabled (until we get visual test image persistence figured out), because it will be testing against an empty screenshot folder.
|
||||
|
||||
ACCEPT_VISUAL=1 will overwrite the snapshot images with new images.
|
||||
120
tests/e2e/e2e_test.go
Normal file
120
tests/e2e/e2e_test.go
Normal file
|
|
@ -0,0 +1,120 @@
|
|||
// Copyright 2022 The Gitea Authors. All rights reserved.
|
||||
// Use of this source code is governed by a MIT-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
// This is primarily coped from /tests/integration/integration_test.go
|
||||
// TODO: Move common functions to shared file
|
||||
|
||||
package e2e
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"code.gitea.io/gitea/models/unittest"
|
||||
"code.gitea.io/gitea/modules/graceful"
|
||||
"code.gitea.io/gitea/modules/log"
|
||||
"code.gitea.io/gitea/modules/setting"
|
||||
"code.gitea.io/gitea/modules/util"
|
||||
"code.gitea.io/gitea/modules/web"
|
||||
"code.gitea.io/gitea/routers"
|
||||
"code.gitea.io/gitea/tests"
|
||||
)
|
||||
|
||||
var c *web.Route
|
||||
|
||||
func TestMain(m *testing.M) {
|
||||
defer log.Close()
|
||||
|
||||
managerCtx, cancel := context.WithCancel(context.Background())
|
||||
graceful.InitManager(managerCtx)
|
||||
defer cancel()
|
||||
|
||||
tests.InitTest(false)
|
||||
c = routers.NormalRoutes(context.TODO())
|
||||
|
||||
os.Unsetenv("GIT_AUTHOR_NAME")
|
||||
os.Unsetenv("GIT_AUTHOR_EMAIL")
|
||||
os.Unsetenv("GIT_AUTHOR_DATE")
|
||||
os.Unsetenv("GIT_COMMITTER_NAME")
|
||||
os.Unsetenv("GIT_COMMITTER_EMAIL")
|
||||
os.Unsetenv("GIT_COMMITTER_DATE")
|
||||
|
||||
err := unittest.InitFixtures(
|
||||
unittest.FixturesOptions{
|
||||
Dir: filepath.Join(filepath.Dir(setting.AppPath), "models/fixtures/"),
|
||||
},
|
||||
)
|
||||
if err != nil {
|
||||
fmt.Printf("Error initializing test database: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
exitVal := m.Run()
|
||||
|
||||
tests.WriterCloser.Reset()
|
||||
|
||||
if err = util.RemoveAll(setting.Indexer.IssuePath); err != nil {
|
||||
fmt.Printf("util.RemoveAll: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if err = util.RemoveAll(setting.Indexer.RepoPath); err != nil {
|
||||
fmt.Printf("Unable to remove repo indexer: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
os.Exit(exitVal)
|
||||
}
|
||||
|
||||
// This should be the only test e2e necessary. It will collect all "*.test.e2e.js"
|
||||
// files in this directory and build a test for each.
|
||||
func TestE2e(t *testing.T) {
|
||||
// Find the paths of all e2e test files in test test directory.
|
||||
searchGlob := filepath.Join(filepath.Dir(setting.AppPath), "tests", "e2e", "*.test.e2e.js")
|
||||
paths, err := filepath.Glob(searchGlob)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
} else if len(paths) == 0 {
|
||||
t.Fatal(fmt.Errorf("No e2e tests found in %s", searchGlob))
|
||||
}
|
||||
|
||||
runArgs := []string{"npx", "playwright", "test"}
|
||||
|
||||
// To update snapshot outputs
|
||||
if _, set := os.LookupEnv("ACCEPT_VISUAL"); set {
|
||||
runArgs = append(runArgs, "--update-snapshots")
|
||||
}
|
||||
|
||||
// Create new test for each input file
|
||||
for _, path := range paths {
|
||||
_, filename := filepath.Split(path)
|
||||
testname := filename[:len(filename)-len(filepath.Ext(path))]
|
||||
|
||||
t.Run(testname, func(t *testing.T) {
|
||||
// Default 2 minute timeout
|
||||
onGiteaRun(t, func(*testing.T, *url.URL) {
|
||||
cmd := exec.Command(runArgs[0], runArgs...)
|
||||
cmd.Env = os.Environ()
|
||||
cmd.Env = append(cmd.Env, fmt.Sprintf("GITEA_URL=%s", setting.AppURL))
|
||||
var stdout, stderr bytes.Buffer
|
||||
cmd.Stdout = &stdout
|
||||
cmd.Stderr = &stderr
|
||||
err := cmd.Run()
|
||||
if err != nil {
|
||||
// Currently colored output is conflicting. Using Printf until that is resolved.
|
||||
fmt.Printf("%v", stdout.String())
|
||||
fmt.Printf("%v", stderr.String())
|
||||
log.Fatal("Playwright Failed: %s", err)
|
||||
} else {
|
||||
fmt.Printf("%v", stdout.String())
|
||||
}
|
||||
})
|
||||
})
|
||||
}
|
||||
}
|
||||
57
tests/e2e/example.test.e2e.js
Normal file
57
tests/e2e/example.test.e2e.js
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
// @ts-check
|
||||
import {test, expect} from '@playwright/test';
|
||||
import {login_user, save_visual, load_logged_in_context} from './utils_e2e.js';
|
||||
|
||||
test.beforeAll(async ({browser}, workerInfo) => {
|
||||
await login_user(browser, workerInfo, 'user2');
|
||||
});
|
||||
|
||||
test('Load Homepage', async ({page}) => {
|
||||
const response = await page.goto('/');
|
||||
await expect(response?.status()).toBe(200); // Status OK
|
||||
await expect(page).toHaveTitle(/^Gitea: Git with a cup of tea\s*$/);
|
||||
await expect(page.locator('.logo')).toHaveAttribute('src', '/assets/img/logo.svg');
|
||||
});
|
||||
|
||||
test('Test Register Form', async ({page}, workerInfo) => {
|
||||
const response = await page.goto('/user/sign_up');
|
||||
await expect(response?.status()).toBe(200); // Status OK
|
||||
await page.type('input[name=user_name]', `e2e-test-${workerInfo.workerIndex}`);
|
||||
await page.type('input[name=email]', `e2e-test-${workerInfo.workerIndex}@test.com`);
|
||||
await page.type('input[name=password]', 'test123');
|
||||
await page.type('input[name=retype]', 'test123');
|
||||
await page.click('form button.ui.green.button:visible');
|
||||
// Make sure we routed to the home page. Else login failed.
|
||||
await expect(page.url()).toBe(`${workerInfo.project.use.baseURL}/`);
|
||||
await expect(page.locator('.dashboard-navbar span>img.ui.avatar.image')).toBeVisible();
|
||||
await expect(page.locator('.ui.positive.message.flash-success')).toHaveText('Account was successfully created.');
|
||||
|
||||
save_visual(page);
|
||||
});
|
||||
|
||||
test('Test Login Form', async ({page}, workerInfo) => {
|
||||
const response = await page.goto('/user/login');
|
||||
await expect(response?.status()).toBe(200); // Status OK
|
||||
|
||||
await page.type('input[name=user_name]', `user2`);
|
||||
await page.type('input[name=password]', `password`);
|
||||
await page.click('form button.ui.green.button:visible');
|
||||
|
||||
await page.waitForLoadState('networkidle');
|
||||
|
||||
await expect(page.url()).toBe(`${workerInfo.project.use.baseURL}/`);
|
||||
|
||||
save_visual(page);
|
||||
});
|
||||
|
||||
test('Test Logged In User', async ({browser}, workerInfo) => {
|
||||
const context = await load_logged_in_context(browser, workerInfo, 'user2');
|
||||
const page = await context.newPage();
|
||||
|
||||
await page.goto('/');
|
||||
|
||||
// Make sure we routed to the home page. Else login failed.
|
||||
await expect(page.url()).toBe(`${workerInfo.project.use.baseURL}/`);
|
||||
|
||||
save_visual(page);
|
||||
});
|
||||
60
tests/e2e/utils_e2e.js
Normal file
60
tests/e2e/utils_e2e.js
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
import {expect} from '@playwright/test';
|
||||
|
||||
const ARTIFACTS_PATH = `tests/e2e/test-artifacts`;
|
||||
const LOGIN_PASSWORD = 'password';
|
||||
|
||||
// log in user and store session info. This should generally be
|
||||
// run in test.beforeAll(), then the session can be loaded in tests.
|
||||
export async function login_user(browser, workerInfo, user) {
|
||||
// Set up a new context
|
||||
const context = await browser.newContext();
|
||||
const page = await context.newPage();
|
||||
|
||||
// Route to login page
|
||||
// Note: this could probably be done more quickly with a POST
|
||||
const response = await page.goto('/user/login');
|
||||
await expect(response?.status()).toBe(200); // Status OK
|
||||
|
||||
// Fill out form
|
||||
await page.type('input[name=user_name]', user);
|
||||
await page.type('input[name=password]', LOGIN_PASSWORD);
|
||||
await page.click('form button.ui.green.button:visible');
|
||||
|
||||
await page.waitForLoadState('networkidle');
|
||||
|
||||
await expect(page.url(), {message: `Failed to login user ${user}`}).toBe(`${workerInfo.project.use.baseURL}/`);
|
||||
|
||||
// Save state
|
||||
await context.storageState({path: `${ARTIFACTS_PATH}/state-${user}-${workerInfo.workerIndex}.json`});
|
||||
|
||||
return context;
|
||||
}
|
||||
|
||||
export async function load_logged_in_context(browser, workerInfo, user) {
|
||||
let context;
|
||||
try {
|
||||
context = await browser.newContext({storageState: `${ARTIFACTS_PATH}/state-${user}-${workerInfo.workerIndex}.json`});
|
||||
} catch (err) {
|
||||
if (err.code === 'ENOENT') {
|
||||
throw new Error(`Could not find state for '${user}'. Did you call login_user(browser, workerInfo, '${user}') in test.beforeAll()?`);
|
||||
}
|
||||
}
|
||||
return context;
|
||||
}
|
||||
|
||||
export async function save_visual(page) {
|
||||
// Optionally include visual testing
|
||||
if (process.env.VISUAL_TEST) {
|
||||
await page.waitForLoadState('networkidle');
|
||||
// Mock page/version string
|
||||
await page.locator('footer div.ui.left').evaluate((node) => node.innerHTML = 'MOCK');
|
||||
await expect(page).toHaveScreenshot({
|
||||
fullPage: true,
|
||||
timeout: 20000,
|
||||
mask: [
|
||||
page.locator('.dashboard-navbar span>img.ui.avatar.image'),
|
||||
page.locator('.ui.dropdown.jump.item.tooltip span>img.ui.avatar.image'),
|
||||
],
|
||||
});
|
||||
}
|
||||
}
|
||||
57
tests/e2e/utils_e2e_test.go
Normal file
57
tests/e2e/utils_e2e_test.go
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
// Copyright 2019 The Gitea Authors. All rights reserved.
|
||||
// Use of this source code is governed by a MIT-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package e2e
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"code.gitea.io/gitea/modules/setting"
|
||||
"code.gitea.io/gitea/tests"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
func onGiteaRunTB(t testing.TB, callback func(testing.TB, *url.URL), prepare ...bool) {
|
||||
if len(prepare) == 0 || prepare[0] {
|
||||
defer tests.PrepareTestEnv(t, 1)()
|
||||
}
|
||||
s := http.Server{
|
||||
Handler: c,
|
||||
}
|
||||
|
||||
u, err := url.Parse(setting.AppURL)
|
||||
assert.NoError(t, err)
|
||||
listener, err := net.Listen("tcp", u.Host)
|
||||
i := 0
|
||||
for err != nil && i <= 10 {
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
listener, err = net.Listen("tcp", u.Host)
|
||||
i++
|
||||
}
|
||||
assert.NoError(t, err)
|
||||
u.Host = listener.Addr().String()
|
||||
|
||||
defer func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||
s.Shutdown(ctx)
|
||||
cancel()
|
||||
}()
|
||||
|
||||
go s.Serve(listener)
|
||||
// Started by config go ssh.Listen(setting.SSH.ListenHost, setting.SSH.ListenPort, setting.SSH.ServerCiphers, setting.SSH.ServerKeyExchanges, setting.SSH.ServerMACs)
|
||||
|
||||
callback(t, u)
|
||||
}
|
||||
|
||||
func onGiteaRun(t *testing.T, callback func(*testing.T, *url.URL), prepare ...bool) {
|
||||
onGiteaRunTB(t, func(t testing.TB, u *url.URL) {
|
||||
callback(t.(*testing.T), u)
|
||||
}, prepare...)
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue