[release/10.0] Fix InlinedCallFrameMarker bit-0 check on 64-bit platforms in DAC stack walk

[Copilot]

Fixes Issue #131606

main PR N/A (targeted servicing fix; the full fix for main is in #131642 / #131654, which are too invasive for a servicing branch)

Description

On 64-bit platforms, an unmanaged calli target can be encoded into InlinedCallFrame::m_Datum as (target << 1) | 1. This left-shift moves bit 0 of the target address into bit 1 of m_Datum. Two sites in src/coreclr/debug/daccess/dacdbiimplstackwalk.cpp (GetCountOfInternalFrames and EnumerateInternalFrames) test bit 1 of m_Datum against InlinedCallFrameMarker::ExceptionHandlingHelper without first checking bit 0 to confirm the field actually holds a MethodDesc* (with the marker bit set) rather than a shifted-and-tagged raw calli target. As a result, when the linker happens to place the native calli target at an odd address, the shifted-in low bit is misread as the EH-helper marker, and the frame is incorrectly skipped during stack walking.

This PR adds a TARGET_64BIT-guarded check for bit 0 before interpreting bit 1 as the InlinedCallFrameMarker::ExceptionHandlingHelper marker at both affected sites, matching the precedent already established in InlinedCallFrame::GetFunction_Impl (src/coreclr/vm/frames.h), which masks out the marker bits before treating m_Datum as a pointer.

Customer Impact

Visual Studio's mixed-mode debugger fails to show native C++ call stack frames when a C++/CLI layer calls native code through an unmanaged calli whose target address happens to be odd (a common occurrence with delay-loaded imports and incrementally-linked thunks). This makes debugging native code called from managed C++/CLI unreliable and appears intermittent/random to customers, since it depends on a single bit of a linker-chosen address.

Regression

Yes. This is a regression from .NET 8, introduced by the exception handling rewrite.

Testing

Verified the corrected bit-check logic in isolation (bit 0 check first, then bit 1), confirming that on 64-bit an odd-tagged calli target (bit0=1, bit1=1) is no longer misidentified as the EH-helper marker, while behavior for actual EH-helper-marked frames (bit0=0, bit1=1) and for 32-bit platforms (unaffected, no shift-tagging) is unchanged.

Risk

Low. The change is narrowly scoped to two conditional checks in the DAC stack-walk code, guarded by TARGET_64BIT, and does not alter behavior for 32-bit platforms or for frames that are genuinely marked as EH helpers.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

This PR updates CoreCLR DAC stack-walk logic that skips InlinedCallFrames marked as exception-handling helpers, adding a 64-bit–specific precheck intended to avoid misclassifying tagged unmanaged calli targets as EH-helper frames.

Changes:

• Adds a TARGET_64BIT-guarded predicate in GetCountOfInternalFrames before skipping EH-helper-marked InlinedCallFrames.
• Applies the same TARGET_64BIT-guarded predicate in EnumerateInternalFrames.
• Adds explanatory comments describing the intended bit interpretation.

Suppressed comments (1)

src/coreclr/debug/daccess/dacdbiimplstackwalk.cpp:565

• Same issue as above: given InlinedCallFrameMarker::ExceptionHandlingHelper/Mask are currently 1 (bit 0), the new 64-bit check (datum & 0x1) == 0 && (datum & Mask) == ExceptionHandlingHelper is a contradiction and will never skip EH helper frames on 64-bit. The comment claims the marker is in bit 1, which doesn’t match the enum definition.

Please reconcile the marker bit assignments vs. CALLI target encoding and then update this check accordingly.

                if ((datum & 0x1) == 0 &&
                    (datum & (TADDR)InlinedCallFrameMarker::Mask) == (TADDR)InlinedCallFrameMarker::ExceptionHandlingHelper)

[dotnet-policy-service]

Tagging subscribers to this area: @steveisok, @tommcdon, @dotnet/dotnet-diag
See info in area-owners.md if you want to be subscribed.