Skip to content

STJ migration: DesignMode/wrapper communication hangs during test execution #15577

@nohwnd

Description

@nohwnd

Problem

After migrating from Newtonsoft.Json to System.Text.Json (branch stj-migration, commit 9782a98ea), the direct CLI path works:

dotnet vstest.console.dll test.dll  →  Passed: 1, Failed: 1, Skipped: 1

But the wrapper/DesignMode path hangs:

  • Translation layer wrapper → socket → vstest.console → socket → testhost
  • 4 vstest.console + 4 testhost processes sit at near-zero CPU indefinitely
  • No error messages, no exceptions — both sides just wait for a message that never comes
  • Smoke tests and integration tests all hang in this path

What was fixed so far

  • TestRunStatistics.Stats private setter → custom converter (was causing NullReferenceException)
  • IDictionary<string, object> values deserialized as JsonElementObjectConverter unwraps to primitives (was causing InvalidCastException)

Root cause hypothesis

The DesignMode communication loop (DesignModeClient.ProcessRequests) receives messages via SocketCommunicationManager.ReceiveMessage() and processes them. If a message fails to deserialize silently (swallowed exception), the loop may continue waiting for the next message while the sender waits for a response — causing a deadlock.

The message loop should abort if it cannot process data, rather than silently continue.

Investigation needed

  1. Add --diag to wrapper-based test execution to capture the full protocol exchange
  2. Compare the messages sent/received between wrapper and vstest.console vs what the direct CLI path sends
  3. Check if any try/catch in the message processing loop swallows serialization exceptions
  4. Consider adding a message-level error response when deserialization fails, so the other side knows and can abort

Branch

stj-migration on nohwnd/vstest — commit 9782a98ea

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions