[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-04-10

[github-actions[bot]]

Tested as a brand-new user attempting to get started with GitHub Agentic Workflows for the first time. Navigated 3 key pages of the docs site and identified content bugs, clarity gaps, and positive patterns.

⚠️ Note: Playwright browser screenshots were unavailable due to network isolation between the agent container and the Playwright container. Analysis was performed via curl and source file review.

---

Summary

• Date: 2026-04-10
• Pages visited:
  • https://github.com/github/gh-aw (docs home /gh-aw/)
  • /gh-aw/setup/quick-start/
  • /gh-aw/setup/cli/
• Overall impression: The quick start guide is well-structured and the 10-minute estimated time is encouraging. However, a few typos and a broken external link create friction right in the critical first-time user flow. The CLI commands page is comprehensive but may overwhelm beginners with GHES content before they've run a single workflow.

---

🔴 Critical Issues Found

1. Broken "What's Next?" Link — Wrong Domain (github.github.com)

File: docs/src/content/docs/setup/quick-start.mdx line 121

[Peli's Agent Factory](https://github.github.com/gh-aw/blog/2026-01-12-welcome-to-pelis-agent-factory/)
```

The domain `github.github.com` is incorrect — this should almost certainly be the docs site URL (e.g., `https://github.com/github/gh-aw` or the correct blog path `/gh-aw/blog/...`). This link appears **at the very end of the Quick Start**, right when a new user is most excited to explore further, and it leads nowhere.

**Suggested fix**: Change to `/gh-aw/blog/2026-01-12-welcome-to-pelis-agent-factory/` (relative docs URL, consistent with other blog links on the page).

---

#### 2. Missing Word in Overview — "compile command **compiles** this markdown"

**File**: `docs/src/content/docs/introduction/overview.mdx` line 44

```
The `gh aw compile` command this markdown file into a hardened GitHub Actions Workflow...

The verb "compiles" is missing. A beginner reading this will be confused by the broken sentence. It should read:

The gh aw compile command compiles this markdown file into a hardened GitHub Actions Workflow...

---

🟡 Confusing Areas

3. Typo: "blacklog" instead of "backlog"

File: docs/src/content/docs/setup/quick-start.mdx line 101

Edit the section "What to include" to list things you are having trouble with regularly in your repository: your issue blacklog, your CI setup...

This is a typo — "blacklog" should be "backlog". Minor but looks unprofessional.

4. COPILOT_GITHUB_TOKEN — Why is it "a separate GitHub token"?

Quick Start, Step 2: The guide mentions that COPILOT_GITHUB_TOKEN is "a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN". For a beginner, this is confusing:

• Why are there two tokens?
• Where do I get this "separate" token?
• The parenthetical explanation is easy to miss in the middle of the list

The link to /gh-aw/reference/auth/ helps, but the inline explanation could be clearer. Consider a dedicated callout box: "⚠️ Important: You need two tokens — see Authentication for why and how."

5. CLI Page: GitHub Enterprise Server Section Before Basic Commands

The CLI reference page (/gh-aw/setup/cli/) has excellent beginner-friendly "Most Common Commands" table at the top ✅, but then dives deep into GHES configuration, version pinning, and enterprise setup before explaining what the commands actually do. A new user just looking for "how do I use gh aw compile?" has to scroll past a lot of enterprise content.

Suggestion: Move the GHES/pinning/standalone-installer sections to a collapsible <details> block or a dedicated "Advanced Installation" subsection.

6. "frontmatter" Undefined on First Use in Quick Start

In Step 4 of the Quick Start, the word "frontmatter" appears with a link but no inline definition:

If you have changed the frontmatter (the YAML configuration block between --- markers...)

The parenthetical (the YAML configuration block between --- markers...) is good — but it only appears in the Step 4 context. In the Overview page, "frontmatter" is used earlier without this explanation, leaving beginners confused.

---

🟢 What Worked Well

1. Prerequisites clearly listed — The bullet list with specific version requirements (gh v2.0.0+), OS support, and links to install each dependency is excellent for beginners.

2. Estimated time — "⏱️ Estimated time: 10 minutes" sets realistic expectations immediately.

3. Step-by-step structure — The numbered steps (Install → Add → Wait → Customize) give beginners a clear mental model of the process.

4. add-wizard command — The interactive wizard approach described in Step 2 is very beginner-friendly; it removes the need to know secrets/config details upfront.

5. "What's next?" section — Good structure for continuing the learning journey (even though the Peli's Factory link is broken).

6. CLI "Most Common Commands" table — The quick reference table at the top of the CLI page is helpful for orienting beginners.

7. Tip callouts — The > [!TIP] blocks for authentication issues and troubleshooting are well-placed.

---

Recommendations

Quick Wins (fix immediately)

1. 🔴 Fix github.github.com → correct domain in quick-start.mdx line 121
2. 🔴 Add missing "compiles" word in overview.mdx line 44
3. 🟡 Fix typo "blacklog" → "backlog" in quick-start.mdx line 101

Medium Term

4. Add a callout box explaining why COPILOT_GITHUB_TOKEN is separate from GITHUB_TOKEN
5. Move GHES/enterprise installation content to a collapsed/advanced section in the CLI page
6. Ensure "frontmatter" has an inline definition on first use in the Overview

Longer Term

7. Add a visual flow diagram to the Quick Start showing: Install CLI → Add Workflow → GitHub Actions runs → Issue created
8. Consider a "I'm lost" link on each page pointing to the FAQ/Common Issues pages

---

Screenshots

Visual screenshots were unavailable due to Playwright container network isolation. Pages were analyzed via curl and source file review.

---

References:

• Workflow run §24244702505

Generated by Documentation Noob Tester · ● 4.2M · ◷

• expires on Apr 11, 2026, 1:36 PM UTC

[github-actions[bot]]

💥 WHOOSH! The Claude Smoke Test Agent swoops in!

KA-POW! 🦸 I, the valiant smoke-test bot, have arrived to declare that Run 24248606598 completed successfully! All systems are GO — the Claude engine roars with power!

BIFF! Build passed. ZAP! Playwright captured GitHub. BOOM! MCP tools responded. WHAM! The agentic workflows stand ready!

With great smoke tests comes great reliability. 🔥

— The Claude Smoke Test Agent, 2026-04-10

💥 [THE END] — Illustrated by Smoke Claude · ● 197.2K · ◷

[github-actions[bot]]

🤖 Beep boop! The smoke test agent stopped by to say hello! 👋

I was just here running my battery of tests to make sure everything is working. The docs look thorough — found those typos already! Nice catch on "blacklog" vs "backlog" 😄

(This is an automated smoke test comment from run §24248606547)

📰 BREAKING: Report filed by Smoke Copilot · ● 1.1M · ◷

[github-actions[bot]]

🤖 Smoke test agent was here! 🔥\n\nBeep boop! Your friendly neighborhood Copilot smoke test agent swung by to verify all systems are nominal. The circuits are humming, the tokens are flowing, and the tests are... testing. 🚀\n\nCarrying on — nothing to see here, just a robot doing robot things! 🤖✨

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

• security: fix agent-stdio.log world-readable exposure and MCP gateway token leakage in redaction pipeline #25618 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

📰 BREAKING: Report filed by Smoke Copilot · ● 1.4M · ◷

[github-actions[bot]]

💥 WHOOSH! 🦸

The Smoke Test Agent swoops in, cape billowing!

"By the power of Claude Sonnet, I have arrived — Run 24248677209 is COMPLETE!"

⚡ KA-POW! All systems nominal. The agent tested GitHub MCP, Serena LSP, Make builds, Playwright, and safe-outputs — and lived to tell the tale!

[The agent disappears in a flash of tokens... until next time!] 💫

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

• security: fix agent-stdio.log world-readable exposure and MCP gateway token leakage in redaction pipeline #25618 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

💥 [THE END] — Illustrated by Smoke Claude · ● 191.4K · ◷