YAML-based Playwright MCP testing for Claude Code

2025年6月20日 10:12:24

How a simple YAML configuration built for Claude Code and Playwright MCP transformed our testing workflow and made automation accessible to everyone on the team

If you've ever maintained a large Playwright test suite, you know the pain. Hundreds of lines of JavaScript scattered across dozens of files. Hardcoded values that break when environments change. Test logic so complex that only the original author dares to modify it.

What if I told you there's a better way? A way to write tests that are readable by anyone, maintainable by design, and powerful enough to handle complex workflows?

Enter YAML-based Playwright testing for Claude Code — a paradigm shift that's changing how teams approach test automation by leveraging the power of Claude Code's AI capabilities and Playwright MCP's browser automation.

The Problem with Traditional Playwright Tests

Let's be honest about traditional Playwright tests:

// Traditional Playwright test - 50+ lines of code
test('complete order flow', async ({ page }) => {
  await page.goto('https://example.com');
  await page.fill('[data-testid="username"]', 'user123');
  await page.fill('[data-testid="password"]', 'pass456');
  await page.click('[data-testid="login-btn"]');
  await expect(page.locator('h1')).toContainText('Dashboard');
  
  // ... 40+ more lines of clicking, filling, asserting
  // ... hardcoded values everywhere
  // ... no reusability between tests
});

Problems:

  • ❌ Verbose and complex — Simple actions buried in boilerplate
  • ❌ Hardcoded values — Environment changes break everything
  • ❌ Poor reusability — Copy-paste leads to maintenance nightmares
  • ❌ Technical barrier — Only developers can write/modify tests
  • ❌ Scattered logic — Related tests live in different files

The YAML Revolution: Tests That Make Sense

Now imagine the same test written in YAML:

# test-cases/order.yml
tags: 
  - smoke
  - order
  - checkout

steps:
  - include: "login"
  - "Click Add to Cart button for first product"
  - "Click Add to Cart button for second product"  
  - "Click shopping cart icon in top right"
  - "Enter First Name"
  - "Enter Last Name"
  - "Enter Postal Code"
  - "Click Continue button"
  - "Click Finish button"
  - "Verify page displays Thank you for your order!"
  - include: "cleanup"

Immediate benefits:

  • ✅ Crystal clear intent — Anyone can understand what this test does
  • ✅ Natural language — Steps read like user stories
  • ✅ Reusable components — Login and cleanup are shared across tests
  • ✅ Environment agnostic — No hardcoded values in sight

The Magic Behind the Simplicity

1. Reusable Step Libraries

Common workflows become building blocks:

# steps/login.yml
steps:
  - "Open {{BASE_URL}} page"
  - "Fill username field with {{TEST_USERNAME}}"
  - "Fill password field with {{TEST_PASSWORD}}"  
  - "Click login button"
  - "Verify page displays Swag Labs"

Write once, use everywhere. No more copy-paste madness.

2. Environment Variable Magic

Different environments? No problem:

# .env.dev
BASE_URL=https://dev.example.com
TEST_USERNAME=dev_user

# .env.prod  
BASE_URL=https://example.com
TEST_USERNAME=prod_user

Same tests, different environments. Automatically.

3. Intelligent Tag Filtering

Run exactly what you need:

# Run only smoke tests
/run-yaml-test tags:smoke

# Run order AND checkout tests  
/run-yaml-test tags:order,checkout

# Run smoke OR critical tests
/run-yaml-test tags:smoke|critical

No more running the entire suite when you only changed the login flow.

4. Smart Reporting

Automatically generated HTML reports with:

  • ✅ Step-by-step execution details
  • ✅ Environment configuration
  • ✅ Screenshots and artifacts
  • ✅ Success/failure statistics

Real-World Impact: A Case Study

Before YAML testing:

  • 📊 2,000+ lines of Playwright JavaScript
  • ⏱️ 3 days to onboard new QA team members
  • 🐛 15+ test failures per environment change
  • 👥 Only 3 developers could modify tests

After YAML testing:

  • 📊 200 lines of readable YAML
  • ⏱️ 30 minutes to onboard new team members
  • 🐛 0 test failures during environment changes
  • 👥 Entire team can write and modify tests

Why This Matters for Your Team

For Developers:

  • Less time writing boilerplate, more time building features
  • Tests that actually document your application's behavior
  • No more "let me just quickly fix this test" rabbit holes

For QA Engineers:

  • Focus on test strategy, not JavaScript syntax
  • Rapid test creation and modification
  • Clear visibility into test coverage

For Product Managers:

  • Tests that read like acceptance criteria
  • Easy to verify that tests match requirements
  • Confidence that important flows are covered

For DevOps:

  • Predictable test execution across environments
  • Clear failure reporting and debugging
  • Easy integration with CI/CD pipelines

Technical Architecture: How It Works

This YAML Playwright testing framework is specifically designed for Claude Code and Playwright MCP. The framework consists of several key components:

Claude Code Integration

  • AI-Powered Execution: Claude Code's AI interprets natural language test steps and converts them to Playwright actions
  • Smart Step Recognition: Advanced understanding of testing intent from plain English descriptions
  • Context Awareness: Maintains context across test steps for more intelligent automation

Playwright MCP Foundation

  • Browser Automation: Leverages Playwright MCP for reliable cross-browser testing
  • Element Detection: Intelligent element finding and interaction
  • Screenshot & Reporting: Built-in capture and documentation capabilities

Multi-Environment Configuration

├── .env.dev          # Development environment
├── .env.test         # Test environment  
├── .env.prod         # Production environment

Reusable Step Libraries

├── steps/
│   ├── login.yml     # Authentication flows
│   ├── cleanup.yml   # Cleanup procedures
│   └── navigation.yml # Common navigation

Test Cases with Natural Language

├── test-cases/
│   ├── order.yml     # E-commerce order flow
│   ├── user.yml      # User management
│   └── search.yml    # Search functionality

Intelligent Execution Engine

The framework automatically:

  1. Loads environment-specific configuration
  2. Expands include references from step libraries
  3. Substitutes environment variables ({{BASE_URL}})
  4. Executes tests using Playwright MCP
  5. Generates comprehensive reports

Getting Started: Your First YAML Test

The beauty of YAML-based testing is its simplicity. Here's how to get started:

1. Prerequisites

# Install Claude Code (if not already installed)
# Follow instructions at: https://claude.ai/code

# Install Playwright MCP for Claude Code
claude mcp add playwright -- npx -y @playwright/mcp@latest

# Clone the YAML testing framework
git clone https://github.com/terryso/claude-code-playwright-mcp-test.git
cd claude-code-playwright-mcp-test

2. Project Structure

your-project/
├── .env.dev              # Environment config
├── steps/               # Reusable step libraries
├── test-cases/          # Your test cases
├── screenshots/         # Test artifacts
└── reports/            # Generated reports

3. Write Your First Test

# test-cases/login.yml
tags:
  - smoke
  - auth

steps:
  - "Open {{BASE_URL}} page"
  - "Fill username with {{TEST_USERNAME}}"
  - "Fill password with {{TEST_PASSWORD}}"
  - "Click login button"
  - "Verify successful login"

4. Execute and Iterate

# In Claude Code, use the built-in commands
/run-yaml-test file:test-cases/login.yml env:dev

# Or run with tag filtering
/run-yaml-test tags:smoke env:dev

Within hours, you'll have tests that are more maintainable than anything you've written before. The magic happens through Claude Code's AI understanding your natural language steps and Playwright MCP executing them as browser actions.

Advanced Features

Complex Tag Filtering

# Multiple conditions
/run-yaml-test tags:smoke,login|critical

# Environment-specific execution
/run-yaml-test tags:order env:prod

Dynamic Step Parameters

steps:
  - "Add product {{PRODUCT_NAME}} to cart"
  - "Set quantity to {{QUANTITY}}"
  - "Apply discount code {{DISCOUNT_CODE}}"

Comprehensive Reporting

  • HTML Reports: Beautiful, interactive test reports
  • JSON/XML Output: For CI/CD integration
  • Screenshot Capture: Automatic failure documentation
  • Performance Metrics: Execution timing and statistics

The Future is Readable

We're moving toward a world where:

  • Tests are documentation that executes
  • Anyone can contribute to test automation
  • Maintenance is a joy, not a chore
  • Environments are just configuration

YAML-based Playwright testing isn't just a tool — it's a philosophy. It's the belief that tests should be clear, maintainable, and accessible to everyone on the team.

Common Questions Answered

Q: How does this compare to existing solutions like Cucumber?
A: While Cucumber requires learning Gherkin syntax and step definitions, this YAML testing framework uses natural language directly with Claude Code's AI interpreting the intent. No step definition mapping needed - Claude Code understands what you want to do.

Q: What about test debugging?
A: Claude Code provides detailed execution logs, Playwright MCP captures screenshots on failure, and you get clear error messages that map back to your YAML steps. The AI context helps identify issues quickly.

Q: Can I integrate this with CI/CD?
A: Absolutely. The framework generates standard exit codes and multiple report formats (HTML, JSON, XML) for seamless CI/CD integration.

Q: How do you handle complex assertions?
A: Claude Code's AI makes natural language assertions surprisingly powerful: "Verify page contains 'Thank you'", "Verify cart total equals $43.18", "Verify 2 items in cart". The AI understands context and intent.

Take Action Today

The question isn't whether this approach is better. The question is: How much time are you willing to waste on brittle, complex tests?

Start your YAML testing journey:

  1. Get Claude Code: Install Claude Code and Playwright MCP
  2. Try the demo: Clone the project from https://github.com/terryso/claude-code-playwright-mcp-test and run your first YAML test
  3. Convert one test: Take your most complex Playwright test and rewrite it in YAML
  4. Share with your team: Show them how readable tests can be
  5. Scale gradually: Convert more tests as you see the benefits

Ready to transform your testing workflow with Claude Code and Playwright MCP? The future of test automation is readable, maintainable, and accessible to everyone.

🔗 Get Started: https://github.com/terryso/claude-code-playwright-mcp-test

What's your biggest pain point with current Playwright tests? How would YAML-based testing with Claude Code solve it for your team?

Tags:
Claude CodeMCPPlaywright