Gua

Gua is a runtime UI automation protocol for games.

It exposes a semantic UI tree from a running game, so test runners and AI agents can inspect, query, click, and verify in-game UI without relying on fragile image recognition or coordinate-based input.

NuGet Packages

Gua.Core:
P/Invoke bindings for using the Gua C ABI runtime from .NET, including the Windows x64 native runtime.
Gua.Testing:
Adds Gua locators, waits, assertions, and adapter test loops to regular .NET tests.
Gua.Testing.Godot:
Starts a Godot process and provides helpers for controlling and verifying a running scene through the Gua bridge.
Gua.Runtime: shared managed wrapper over the gua_runtime C ABI for engine adapters. Targets net10.0 and netstandard2.1.
Gua.Testing.Unity: builds and starts Unity Editor Play Mode or Mono Windows players and exposes the shared remote Gua context.
Gua.Testing.Visual:
Adds opt-in PNG baseline comparison and machine-readable visual failure artifacts.
Gua.Testing.Recording:
Records semantic UI operations and replays them with correlated host completion.

MCP and Inspector

gui-mcp:
A thin MCP server that exposes Gua runtime actions to AI agents through the same WebSocket bridge used by the Inspector.
Gua Inspector:
A browser and Windows desktop UI for inspecting the semantic UI tree, node state, screenshots, and logs, and for sending runtime commands.

await game.getByRole("button", { name: "Start Game" }).click()
await expect(game.getById("loading")).toBeVisible()

The current implementation exposes that shape for C++ and C# over the C ABI, adds Inspector and MCP consumers, and includes Godot 4.7 adapters and samples:

gua::testing::get_by_role(ctx, "button", "Start Game").click();
gua::testing::wait_for_text(ctx, "Loading...").to_be_visible();

GuaAssertions.GetByRole(ui, "button", "Start Game").Click();
GuaAssertions.WaitForText(ui, "Loading...").ToBeVisible();

Click() intentionally enqueues a click request; it does not directly mutate the game. A game adapter, such as the ImGui or Godot adapter, must consume the request on a later frame and emit the observed click event. Tests that do not run a real engine adapter can use GuaTestHost for the same loop:

using var ui = new GuaContext();
var host = new GuaTestHost(ui);
var loading = false;

host.Frame("title", frame =>
{
    frame.Button("start", "Start Game", new GuaBounds(100, 100, 240, 64));
});

GuaAssertions.GetByRole(ui, "button", "Start Game").Click();

host.Frame("title", frame =>
{
    frame.Button("start", "Start Game", new GuaBounds(100, 100, 240, 64));
});

host.DrainClickEvents(id => loading = id == "start");

host.Frame("loading", frame =>
{
    if (loading)
    {
        frame.Text("loading", "Loading...", new GuaBounds(100, 180, 240, 24));
    }
});

GuaAssertions.WaitForText(ui, "Loading...").ToBeVisible();

Real .NET tests should use an existing test runner such as NUnit. Gua.Testing does not try to become a Vitest-style runner; it provides Gua-specific locators, waits, assertions, and adapter test loops inside normal NUnit tests. One C# file can contain multiple [Test] methods:

using Gua.Core;
using Gua.Testing;
using NUnit.Framework;

[TestFixture]
public sealed class TitleScreenTests
{
    [Test]
    public void StartClickShowsLoadingText()
    {
        using var _ = GuaAssertionScope.UseNUnit(Assert.Fail);
        using var ui = new GuaContext();
        var host = new GuaTestHost(ui);

        host.Frame("title", frame =>
        {
            frame.Button("start", "Start Game", new GuaBounds(100, 100, 240, 64));
        });

        GuaAssertions.GetByRole(ui, "button", "Start Game").Click();
    }
}

The repository includes a runnable NUnit sample in examples/dotnet-nunit.

Godot scene tests can use the Gua.Testing.Godot package to start a Godot process and assert against the live Gua bridge:

using var host = GodotSceneTestHost.Load("game/scenes/title_screen.tscn");

GuaAssertions.GetByRole(host.Context, "button", "開始").ToBeVisible();
host.Click("CenterPanel/Content/ButtonBox/StartButton", nextScene: "game/scenes/village_list.tscn");
GuaAssertions.GetByRole(host.Context, "button", "Create").ToBeVisible();

For v0.2.0 and later, repository-local GuaLiveAssertions helpers can migrate directly to WaitForId / WaitForText / WaitForVisible / WaitForEnabled, locator WaitForCount*, and correlated node actions such as ClickAsync. Wait-returned expectations retain the successful completed-frame snapshot; use Refresh() or WaitUntil* when a later frame is required. Repository GodotHostFactory wrappers should keep only game-specific process and fixture policy while GodotSceneTestHost owns executable/project discovery, automatic loopback ports, rendered/headless startup, bridge diagnostics, and reset policy. All APIs are additive over the existing C ABI and WebSocket protocol.

Failure diagnostics can be coordinated through the framework-independent GuaDiagnosticsSession. Assertions and completed actions use the same artifact layout, return typed absolute paths/media types/secondary capture errors, and can attach through a callback. Godot sessions can include live process metadata, stdout/stderr, runtime version, and an opt-in on-demand screenshot; successful tests do not create artifacts unless capture is explicitly requested.

Want to try that in GitHub Actions without wiring every setup step by hand? link1345/gua-tester provides reusable Actions for Godot GDScript projects. It downloads Godot on the runner, links the released Gua Godot addon into your project, sets GODOT_EXECUTABLE, and runs your .NET tests that use Gua.Testing.Godot.

For a typical consumer repository, the workflow can be as small as:

- uses: link1345/gua-tester@v1
  with:
    project-path: game
    test-project: tests/GuaTester.Tests.csproj
    godot-version: "4.7"
    godot-status: stable

If you have a Godot project that starts the Gua bridge, give it a spin in CI and see whether the semantic UI tree catches regressions before they land.

context.log(gua::LogLevel::info, "title screen opened");
context.set_screenshot("data:image/png;base64,...", 1280, 720);

std::cout << context.ui_tree_json() << '\n';
std::cout << context.logs_json() << '\n';
std::cout << context.screenshot_json() << '\n';

while (ui.TryPollEvent(out var e))
{
    if (e.Type == GuaEventType.Click && e.NodeId == "start")
    {
        ShowLoading();
    }
}

Gua is not a game engine.
Gua is not an editor MCP.
Gua is not an image-recognition QA bot.

Gua is a small bridge between a game runtime and automation tools.

Scope

The first implementation focuses on a small, stable core:

Protocol specification and JSON schemas
C ABI runtime core
Thin C++ wrapper
ImGui adapter
C++ and C# testing helpers
.NET P/Invoke binding over the C ABI
Inspector for tree, node detail, screenshot, logs, and runtime commands
MCP server for AI-agent access to the runtime bridge
Experimental Godot 4.7 C# sample for basic tree reflection and button clicks over the shared native runtime bridge
Recommended Godot 4.7 GDScript addon using the same runtime bridge through GDExtension, including the full standard-Control adapter

Engine-specific integrations such as Unity, Unreal Engine, Godot, and MonoGame are expected to be adapters built on top of the protocol, not the center of the project.

Native Toolchain

Runtime compatibility can be inspected through the additive C ABI gua_copy_version_json, the WebSocket get_version command, or .NET GetVersion(). The response follows protocol/schema/version.schema.json and lists stable capability IDs. Non-Godot runtimes report godotPluginVersion as null; release workflows inject the release version and commit build ID.

Windows native development uses MSVC as the primary toolchain:

cmake --preset windows-msvc-debug
cmake --build --preset windows-msvc-debug

The portable boundary remains the C ABI. macOS and iOS should use Apple Clang, and Android should use Android NDK Clang when those targets are added.

.NET Testing

The .NET packages are published on NuGet and can also be packed locally:

dotnet pack bindings/dotnet/src/Gua.Core/Gua.Core.csproj --configuration Release
dotnet pack bindings/dotnet/src/Gua.Testing/Gua.Testing.csproj --configuration Release
dotnet pack bindings/dotnet/src/Gua.Runtime/Gua.Runtime.csproj --configuration Release
dotnet pack bindings/dotnet/src/Gua.Testing.Unity/Gua.Testing.Unity.csproj --configuration Release
dotnet pack bindings/dotnet/src/Gua.Testing.Godot/Gua.Testing.Godot.csproj --configuration Release
dotnet pack bindings/dotnet/src/Gua.Testing.Visual/Gua.Testing.Visual.csproj --configuration Release
dotnet pack bindings/dotnet/src/Gua.Testing.Recording/Gua.Testing.Recording.csproj --configuration Release

The packages are written to artifacts/packages. Gua.Testing declares a NuGet dependency on the matching Gua.Core package, so a test project only needs the testing package:

<PackageReference Include="Gua.Testing" Version="0.5.0-preview.3" />

Gua.Core is also delivered as a NuGet package and Gua.Testing depends on the matching version. The Windows x64 native runtime is included in Gua.Core under runtimes/win-x64/native/gua.dll, so a normal package restore/build copies it to the consuming app or test output. GUA_NATIVE_DIR remains as an override for locally built native runtimes. A missing or wrong architecture native library is reported with the exact checked paths.

Run the standalone testing sample:

cmake --preset windows-msvc-debug
cmake --build --preset windows-msvc-debug --target gua
$env:GUA_NATIVE_DIR = "$PWD\build\windows-msvc-debug\native\gua-core\Debug"
dotnet run --project examples/dotnet-testing/GuaDotNetTestingSample.csproj

Run the NUnit testing sample:

cmake --preset windows-msvc-release
cmake --build --preset windows-msvc-release --target gua
dotnet pack bindings/dotnet/src/Gua.Core/Gua.Core.csproj --configuration Release
dotnet pack bindings/dotnet/src/Gua.Testing/Gua.Testing.csproj --configuration Release
dotnet test examples/dotnet-nunit/GuaDotNetNUnitSample.csproj

Unity 6 Windows x64

Gua.Core, Gua.Testing, Gua.Testing.Visual, and Gua.Testing.Recording target both net10.0 and netstandard2.1. Unity 6 projects using the default .NET Standard 2.1 API Compatibility Level can load the managed assemblies without changing the native C ABI. The first verified native configuration is Windows Editor x64: place the managed assemblies and their NuGet dependency closure under Assets/Plugins/Gua/Managed, and place gua.dll under Assets/Plugins/x86_64. Unity's Plugin Import Settings must enable the DLL for the Windows Editor and Windows Standalone x86_64 targets.

Build the precompiled Unity Package Manager artifact and .tgz archive with:

.\scripts\build-unity-package.ps1

Each automated Gua GitHub Release also attaches com.link1345.gua-<version>.tgz. Download that file and install it from Unity Package Manager with Add package from tarball. Release builds require the UNITY_LICENSE, UNITY_EMAIL, and UNITY_PASSWORD secrets in the GitHub release environment so the precompiled Unity assemblies are produced by the same release workflow as the Windows native plugins.

The package starts automatically and reflects UI Toolkit, uGUI, and TextMeshPro runtime UI. GuaId and GuaScreen are optional overrides; game code does not register semantic nodes manually. Gua.Testing.Unity provides Editor Play Mode and Mono Windows Player hosts. See examples/unity-smoke for the verified Unity 6000.5.3f1 fixture. Windows x64, Unity 6000.0+, and Mono are supported in this initial release; IL2CPP, other operating systems, IMGUI, and EditorWindow UI automation are not.

Inspector

The Inspector is a React application that consumes Gua protocol snapshots. It is not tied to MCP. The UI talks to a GuaInspectorClient abstraction so transport implementations can be added for mock data, WebSocket bridges, HTTP bridges, MCP, saved files, or the native runtime bridge.

Run the browser version:

bun run --filter @gua/inspector dev

Run the sample WebSocket bridge in another terminal:

bun run bridge:ws

Or build and run the native C++ bridge that serves a real gua::Context:

cmake --preset windows-msvc-debug
cmake --build --preset windows-msvc-debug --target gua-native-bridge-example
.\build\windows-msvc-debug\examples\native-bridge\Debug\gua-native-bridge-example.exe

The ImGui example also hosts the same WebSocket bridge while the UI is running:

cmake --preset windows-msvc-debug
cmake --build --preset windows-msvc-debug --target gua-cpp-imgui-example
.\build\windows-msvc-debug\examples\cpp-imgui\Debug\gua-cpp-imgui-example.exe

Then connect the Inspector to:

ws://127.0.0.1:8765

The ImGui bridge pushes snapshot notifications while the UI is running, so the Inspector updates without polling. The Poll toggle in the Inspector remains as a fallback for bridges that only implement request/response.

The Inspector Automation panel can record semantic actions issued from the UI, download or import a recording.schema.json document, replay every semantic action, and supply sensitive replay values from an in-memory JSON map. Its Visual comparison controls accept the current screenshot or a selected image as a baseline, compare in the browser, and download Actual/Expected/Diff images plus a machine-readable manifest. Browser Inspector files are explicit downloads; it does not silently write arbitrary local paths. Coordinate fallback recordings are accepted as schema v1 documents but are not executed by the Inspector; replay remains semantic-target-only by default.

The bridge speaks the same JSON command shape used by Gua runtime adapters:

{ "id": 1, "type": "get_ui_tree" }
{ "id": 2, "type": "click_node", "nodeId": "start" }

Build the static Inspector:

bun run --filter @gua/inspector build

Run the Tauri desktop shell during development:

bun run --filter @gua/inspector tauri:dev

Tauri requires a Rust toolchain in addition to the JavaScript dependencies.

MCP

The MCP server is a thin protocol consumer over the same bridge used by the Inspector. Start a runtime bridge first:

bun run bridge:ws

Then run the MCP server over stdio:

bun run mcp

The MCP server is published to npm as gui-mcp. MCP clients can start it with:

bunx gui-mcp@latest mcp

By default it connects to ws://127.0.0.1:8765. Override that for another runtime adapter:

$env:GUA_BRIDGE_URL = "ws://127.0.0.1:8765"
bunx gui-mcp@latest mcp

The MCP tool surface is:

get_ui_tree
click_node
focus_node
set_value
set_checked
select
scroll
press_key
wait_for_node
get_screenshot
get_logs
start_recording
stop_recording
save_recording
replay_recording
compare_screenshot
get_visual_artifacts
run_test

Recording, baseline, and visual failure files default to .gua. Set GUA_ARTIFACT_DIR to choose a different root. Names supplied to MCP tools cannot escape that root. Semantic action tools wait for request-ID-correlated host completion when the connected bridge supports it.

Release automation

GitHub Actions publishes runtime artifacts from main when a related protocol consumer changes. The Inspector, Godot plugin, and ImGui plugin are built together and attached to one versioned gua-v* GitHub Release. MCP and .NET packages continue to use their independent npm and NuGet publishing workflows.

The MCP workflow uses npm trusted publishing through GitHub Actions OIDC. The gui-mcp package must be configured on npm with this repository, the .github/workflows/mcp-publish.yml workflow, and the release environment.

Godot 4.7 C# Sample

Experimental — basic functionality only. This sample demonstrates the shared runtime bridge, basic semantic tree reflection, screenshots, and button clicks. It is not feature-equivalent to the GDScript adapter. For new Godot integrations, including .NET-enabled projects, use the GDScript addon below; Godot projects can use it alongside C# game scripts.

The v0.5 C# runtime sample lives in examples/dotnet-godot. It is a minimal Godot 4.7 project using Godot.NET.Sdk/4.7.0, net10.0, project references to Gua.Core and Gua.Testing, and a runtime addon in addons/gua. The addon P/Invokes gua_runtime.dll, which owns both the Gua context and the Inspector WebSocket bridge.

Build it with:

dotnet build examples/dotnet-godot/GuaGodotSample.csproj -v:minimal

The sample uses Gua.Godot.GuaGodotRuntime from the addon, attaches it to the root Godot Control, and starts an Inspector bridge on ws://127.0.0.1:8765. The adapter collects standard controls into the semantic tree and dispatches external click requests through normal Godot button signals. examples/dotnet-monogame is still present as a future placeholder.

Godot 4.7 GDScript Addon

This is the recommended Godot integration for both GDScript projects and .NET-enabled projects. Its standard-Control adapter implements the complete Godot feature set currently documented by Gua.

The GDScript-facing adapter lives in native/gua-godot and builds a thin GDExtension wrapper over native/gua-runtime. It exposes GuaContext to GDScript without reimplementing the runtime core or the Inspector bridge:

var ui := GuaContext.new()
ui.begin_frame("title")
ui.register_node("start", "button", "Start Game", Rect2(512, 312, 256, 56), true, true)
ui.end_frame()
ui.start_inspector_bridge(8765)
ui.enqueue_click("start")

while true:
    var event := ui.poll_event()
    if event.is_empty():
        break

Build the Windows debug GDExtension:

cmake --preset windows-msvc-debug
cmake --build --preset windows-msvc-debug --target gua-godot

The build writes the debug DLL into examples/godot-gdscript/addons/gua/bin. Open examples/godot-gdscript with Godot 4.7 and run the project. The running game process starts an Inspector bridge on ws://127.0.0.1:8765, which Gua Inspector can connect to while the game is open. The addon includes plugin.cfg only for standard Godot addon packaging; Gua's runtime API is provided by gua.gdextension, not by an editor MCP.

For game scripts, instantiate the GDScript adapter through an explicit preload instead of relying on class_name registration order:

const GuaAutoAdapterScript := preload("res://addons/gua/gua_auto_adapter.gd")

var ui := GuaAutoAdapterScript.new()

The adapter resolves the native GuaContext class through ClassDB on first use and verifies that required methods such as consume_click_request exist before dispatching Inspector click requests. If that check fails, rebuild gua-godot; the stale vendored DLL is the problem, not the game script.

Run the GDScript smoke check with:

C:\Users\testk\.local\bin\Godot_v4.7-stable_win64_console.exe --headless --path examples/godot-gdscript --script res://scripts/gua_smoke.gd

Repository Layout

protocol/             Protocol specs and JSON schemas
native/gua-core/      C ABI runtime core and C++ reference implementation
native/gua-runtime/   Shared native runtime bridge for Godot C# and GDScript
native/gua-imgui/     ImGui adapter layer
native/gua-testing/   C++ testing helpers over the C ABI
native/gua-godot/     Godot GDExtension adapter for GDScript
bindings/dotnet/      .NET P/Invoke binding and C# testing helpers
bindings/dotnet/src/Gua.Testing.Godot/
                      Godot process test helpers over Gua.Testing
bindings/dotnet/src/Gua.Testing.Visual/
                      Opt-in PNG baseline comparison helpers
bindings/dotnet/src/Gua.Testing.Recording/
                      Semantic operation recording and correlated replay
packages/mcp/         Published MCP server package
packages/inspector/   Browser and Tauri desktop Inspector UI
examples/             Minimal demos and samples, including the Godot C# sample
docs/                 Native toolchain guidance

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 115 Commits
.codex/skills		.codex/skills
.github		.github
bindings		bindings
docs		docs
examples		examples
native		native
packages		packages
protocol		protocol
scripts		scripts
.editorconfig		.editorconfig
.gitattributes		.gitattributes
.gitignore		.gitignore
AGENTS.md		AGENTS.md
CMakeLists.txt		CMakeLists.txt
CMakePresets.json		CMakePresets.json
CONTRIBUTING.md		CONTRIBUTING.md
Directory.Build.props		Directory.Build.props
LICENSE		LICENSE
README.ja.md		README.ja.md
README.md		README.md
bun.lock		bun.lock
package.json		package.json
tsconfig.base.json		tsconfig.base.json

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Gua

NuGet Packages

MCP and Inspector

Scope

Native Toolchain

.NET Testing

Unity 6 Windows x64

Inspector

MCP

Release automation

Godot 4.7 C# Sample

Godot 4.7 GDScript Addon

Repository Layout

License

About

Releases

Packages

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

Gua

NuGet Packages

MCP and Inspector

Scope

Native Toolchain

.NET Testing

Unity 6 Windows x64

Inspector

MCP

Release automation

Godot 4.7 C# Sample

Godot 4.7 GDScript Addon

Repository Layout

License

About

Topics

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages