SSLSessionContext corrupted by post-install hook JAR classloading — No new session is allowed and no existing session can be resumed

[Thiagott]

Summary

During bundle install, any Java gem that fires a post-install hook (e.g. digest-crc, ruby-maven-libs, psych, fast-rsa-engine) loads BouncyCastle JARs into the JVM under a new classloader. This corrupts the already-registered BC JCE provider's classloader context and orphans the active
SSLSessionContext. All subsequent HTTPS connections then fail with:

OpenSSL::SSL::SSLError: No new session is allowed and no existing session can be resumed

Environment

• JRuby versions tested: 9.4.13.0, 9.4.15.0, 10.0.6.0, 10.1.1.0 — all fail identically
• JDK: 8, 11, 21 — all fail identically
• jruby-openssl: 0.15.5 (also tested 0.19.0 — same failure)
• OS: Ubuntu 22.04, persistent Jenkins bare builder (long-lived JVM)
• TLS endpoint: HTTPS gem server behind nginx ingress enforcing TLS 1.3

Root Cause

jruby-openssl registers the BouncyCastle JCE provider once at startup. When a post-install hook later loads BC JARs under a different classloader (via jar-dependencies/ruby-maven), two copies of BouncyCastle exist in different classloader contexts. The security provider registry becomes stale —
the registered provider no longer matches the classloader that owns the active SSLSessionContext. Subsequent TLS session resume attempts fail because BouncyCastle cannot find or validate the cached session.

The SSLSessionContext is effectively orphaned — new sessions cannot be created (corrupted state) and existing sessions cannot be resumed (wrong classloader context), leaving the error message perfectly accurate but with no graceful fallback to a full handshake.

Failure Sequence (typical)

Installing digest-crc 0.7.0 with native extensions   ← BC JARs loaded under new classloader
Installing ruby-maven-libs 3.9.9                      ← additional JAR classloading
Fetching next-gem                                     ← SSLError, all retries fail

On heavily used builders the corruption fires even earlier:

Installing jruby-openssl 0.15.5 (java)               ← sufficient to trigger on degraded JVM state
Fetching next-gem                                     ← SSLError immediately

What Does NOT Fix It

• Any jdk.tls.* system property (jdk.tls.client.protocols, jdk.tls.disabledAlgorithms, jdk.tls.client.enableSessionTicketExtension) — BouncyCastle has its own TLS stack and ignores JSSE entirely
• JAVA_TOOL_OPTIONS — same reason
• Upgrading jruby-openssl to 0.19.0
• Upgrading JRuby (9.4.x → 10.x) — same failure on all versions tested
• Changing JDK version (8 → 11 → 21)
• Forcing TLS 1.2 via jdk.tls.disabledAlgorithms — see below

Forcing TLS 1.2 does not fix it

We confirmed via JSSE debug that adding TLSv1.3 to jdk.tls.disabledAlgorithms in java.security successfully forces TLS 1.2 — all ClientHellos advertised only TLSv1.2 and the server negotiated TLSv1.2 for every connection. Gems downloaded normally during the early phase.

Yet the session resumption error still occurred at exactly the same point — immediately after a Java gem post-install hook fired:

# JAVA_TOOL_OPTIONS=-Djava.security.properties=/tmp/java-tls12.security
# jdk.tls.disabledAlgorithms confirmed to include TLSv1.3
# All connections negotiating TLSv1.2 ✓

Installing digest-crc 0.7.0 with native extensions   ← hook fires, BC JARs loaded
Installing ruby-maven-libs 3.9.9                      ← additional JAR classloading
Fetching next-gem                                     ← SSLError — same failure, TLS version irrelevant

This confirms the issue is purely at the JVM classloader level, not the TLS protocol version. jdk.tls.disabledAlgorithms is a JSSE-level control and has zero effect on BouncyCastle's TLS stack — BouncyCastle maintains its own independent TLS implementation.

Workaround

Two-phase install separating TLS downloads from JAR classloading:

export JARS_SKIP=true
bundle package --no-install   # Phase 1: all HTTPS downloads complete before any hooks run
bundle install --local         # Phase 2: installs from cache — hooks fire but no network needed

JARS_SKIP=true additionally suppresses Maven post-install hooks.

Local Reproduction (no private gem server required)

The issue reproduces against any HTTPS gem server including rubygems.org. The only requirement is a persistent JVM with prior build history — a completely fresh JRuby install may pass on the first run.

Setup

# 1. Source RVM
source /etc/profile.d/rvm.sh

# 2. Install JRuby and create gemset
rvm install jruby-9.4.13.0
rvm use jruby-9.4.13.0@repro --create
gem install bundler

# 3. Create a working directory
mkdir ~/repro && cd ~/repro

# 4. Create the Gemfile
cat > Gemfile << 'EOF'
source 'https://rubygems.org'

gem 'digest-crc'
gem 'ruby-maven-libs'
gem 'fast-rsa-engine'
gem 'psych'
gem 'rails', '~> 7.1'
gem 'rubocop'
gem 'rspec'
gem 'cucumber'
EOF

# 5. Run the install
bundle install

If you get a JAR missing error instead of the SSL error (e.g. no such file to load -- org/snakeyaml/snakeyaml-engine/...), your gemset may be in a corrupted state from a previous run. Delete it and start fresh:

rvm gemset delete repro --force
rvm use jruby-9.4.13.0@repro --create
gem install bundler
bundle install

If the first run succeeds (fresh JVM)

Install a Java gem first to simulate accumulated classloader state, then retry:

gem install nokogiri
bundle clean --force
bundle install

Reproducing with TLS 1.2 forced (shows protocol version is not the fix)

# Copy the system java.security and add TLSv1.3 to disabledAlgorithms
cp $(find $(dirname $(dirname $(readlink -f $(which java)))) -name java.security 2>/dev/null | head -1) /tmp/java-tls12.security
sed -i 's/jdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1,/jdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1, TLSv1.3,/' /tmp/java-tls12.security

export JAVA_TOOL_OPTIONS="-Djava.security.properties=/tmp/java-tls12.security"

# Warm up JVM state if needed
gem install nokogiri
bundle clean --force

# Run — TLS 1.2 will be successfully forced but the SSLError still occurs
bundle install

The Picked up JAVA_TOOL_OPTIONS: line in the output confirms the security properties are applied. Despite TLS 1.2 being enforced end-to-end, the session resumption error fires at the same trigger point.

Why a warm-up may be needed

On a completely fresh JRuby install the JVM classloader state is clean and the first run may succeed. The corruption accumulates with prior Java class loads — which is exactly why it is consistent on long-running CI builders but intermittent on first-ever installs. Installing any gem with Java extensions
before the repro run (e.g. gem install nokogiri) replicates the degraded JVM state reliably.

Related Issues

• Double loading with 0.11 as default and 0.12.2 installed #251 — double-loading BC via classloader causing OpenSSL class corruption (same mechanism, different trigger)
• openssl fails to load when BC FIPS jars are on the JRuby classloader #361 — BC classloader conflict via FIPS jars (same mechanism, different trigger)
• No existing session to resume #97 — same error string, different cause, closed without fix

This is distinct from all three: the trigger here is any Java gem post-install hook loading BC JARs during an active bundle install, not a specific gem or FIPS configuration.

Note on commit ed95b2c: This commit addresses premature SSLSocket.session= initialization and is a different issue. It does not fix the classloader corruption described here — the corruption occurs after
initialization completes, when a post-install hook loads a second copy of BouncyCastle under a new classloader.