Same Headers, Different Responses: Beyond Copy as cURL

A feed opens in Chrome. Copy its request as cURL, paste it into a terminal, and the response changes.

The URL is right. The User-Agent says Chrome. The browser’s Accept, language, client hints and Fetch Metadata headers are there. Yet the terminal gets 406, or a security checkpoint, while the browser gets the feed.

Copy as cURL copies a request description. The new client still builds its own connection.

We investigated that gap across Chrome, curl, Python, Node and Java. Then we tested three focused ALPN changes and two browser-profile clients. The useful question was not which language can scrape a page. It was which parts of browser behavior survive a copied request, and which parts belong to the client underneath it.

We matched selected application header values across programmatic clients, not every byte of a browser request. We excluded cookies, requested uncompressed responses, and supplied a shared Connection value for HTTP/1.1. HTTP/2 does not permit that field. The browser baseline remained a separate request context.

The short version

• Food52 separated the clients clearly. Nine of twelve main configurations received its security checkpoint on both attempts. Three OkHttp configurations retrieved feed markup on both attempts.
• Newsweek changed without a code change. Four main configurations went from 406 to 200 between observations. That prevents a clean claim that an individual setting caused access.
• Browser-profile clients retrieved both feeds. Python and Node profile controls returned 200 on both attempts per target, but changed several network characteristics together.

The local connection captures showed that matching application headers did not reproduce the same ClientHello. They did not reveal either site’s exact blocking rule.

The source and observations are available in full. There are no hidden probe parameters or unreleased client configurations in this study.

What Copy as cURL carries across

A browser request has several kinds of state around it. Some are visible in the copied command:

• The URL, method and any copied request body.
• Ordinary HTTP header values, including the reported User-Agent.
• Cookies or authorization fields, when the browser includes them in the export.

Other behavior comes from the program executing the command: its TLS implementation, ClientHello offer, ALPN negotiation, HTTP/2 implementation, connection reuse and proxy configuration. Those do not become Chrome’s behavior because a header says Chrome.

JavaScript can also prepare a request or obtain session state before it is copied. A replay may work when that copied state is sufficient and still valid. The command itself does not execute the browser’s JavaScript again, maintain its runtime, or recreate its network stack. Some sites need none of those distinctions. Others can observe or act on them.

The general mechanism is not specific to an RSS endpoint. The results below are two case studies, not a claim that every site uses the same rule.

Two easy ways to compare different requests accidentally

The original terminal example ended with -I. That sends HEAD, whereas opening a feed in the browser makes a GET. A HEAD status does not tell us what body a GET would return. The curl manual documents that distinction.

It also showed this before the target response:

HTTP/1.1 200 Connection Established

That is a proxy’s CONNECT response. It is not the feed’s status. A later HTTP/2 406 or HTTP/2 429 belongs to the target exchange.

We used GET throughout the new command-line matrix. We also tested curl with its configured proxy and with --noproxy '*'. Both routes initially failed on both feeds, so the proxy did not explain away the initial mismatch. Keeping the routes separate still mattered when Newsweek changed later.

What we expected to happen

Our hypothesis was that matching selected HTTP headers would not always reproduce Chrome’s result. TLS negotiation, protocol implementation and request context can still differ underneath those values.

If a focused ALPN change altered the outcome while an unchanged control stayed stable, that would support a narrower explanation. If unchanged clients also changed outcomes, we would need to keep timing and server-side state in the explanation. The experiment was designed to distinguish those possibilities.

Methodology

We observed these two endpoints on 21 October 2026:

• Food52’s blog feed
• Newsweek’s RSS feed

The browser baseline was the existing Chrome 154 session. Each feed was reloaded with cache ignored and returned 200. A separate JavaScript fetch with credentials: 'omit' also returned feed content from each. These are different browser request contexts, recorded separately rather than presented as identical repetitions.

For programmatic clients, we used the 14 navigation-header values copied from Chrome 154. We added Accept-Encoding: identity to avoid conflating access with Brotli decoding, and a shared Connection value for HTTP/1.1 comparisons. The exact header file is included. We did not copy cookies or credentials into these clients.

The main matrix contains twelve configurations, two targets, and two observations per configuration and target: 48 GET observations. Requests were sequential with four-second gaps. The second pass reversed client order. Python and Node browser-profile controls were tested afterward, adding eight observations. Combined with four browser observations, the expanded study contains 60 public-endpoint observations.

Versions were curl 8.5.0 / OpenSSL 3.0.13, Python 3.12 / OpenSSL 3.0.13, Requests 2.31.0, Node 24.18.0 / OpenSSL 3.5.7, Zulu Java 17.0.15, Apache HttpClient 4.5.14, OkHttp 4.12.0 and Conscrypt 2.5.1. The profile clients used curl_cffi 0.16.3 and impit 0.14.6.

Check the connection, not only the application code

We sent thirteen configurations to an owned local HTTPS probe. It recorded the offered ClientHello before terminating TLS, then read the actual HTTP/1.1 headers. This let us check whether application-level changes really changed the handshake.

The probe deliberately supported HTTP/1.1 only. HTTP/2-capable clients offered their protocols and fell back to HTTP/1.1 locally. Their public-endpoint protocol observations are separate. We did not equate an HTTP/2 header block with an HTTP/1.1 byte stream.

All native configurations and the Node profile produced identical normalized header values at that probe. Their order and casing varied. The Python profile omitted Priority on HTTP/1.1 fallback, an exception retained in the evidence. We do not describe that control as perfect on-wire header equality.

Findings

1. The mismatch crossed languages

The main matrix produced three outcome patterns per target. The figure counts configurations with each observed pair, without turning two requests into a success-rate estimate.

Each native-client cell below gives the two observed statuses in order. Browser rows contain one observation per context. Profile controls were measured after the main matrix.

Food52’s 429 responses contained Vercel Security Checkpoint HTML. Its 200 responses contained RSS markup. Newsweek’s 406 responses were empty, while its successful command-line bodies parsed as RSS with 20 items.

This is not a Java-specific problem. Copying Chrome’s header values into curl, Python or Node did not reliably reproduce the browser result either. Conversely, Java could receive the feeds when configured differently. The language name does not identify the complete client behavior.

2. The clients advertised different connections

The local probe saw these offers while the shared header values stayed fixed:

These are captured list-entry counts for these versions and settings. They are not universal signatures or a measure of how authentic a client is. The full ordered lists, including supported groups and versions, are available to inspect.

Even clients using the same TLS provider differed. Apache and OkHttp both used Java TLS in one pair, but offered different cipher lists and extensions. The HTTP library participates in configuring the socket.

ALPN is an extension inside ClientHello that negotiates an application protocol. It is not an HTTP header. RFC 7301 defines the negotiation. Advertising a particular User-Agent cannot add ALPN, reorder offered cipher suites, or change an HTTP/2 implementation.

JA3 and JA4 describe ways to represent aspects of this observable connection behavior. We retained the underlying fields rather than computing a fingerprint hash or claiming access to either site’s classifier.

3. Test ALPN as a variable, not a diagnosis

We added only an http/1.1 ALPN offer to three otherwise unchanged native clients.

Python:

context.set_alpn_protocols(["http/1.1"])

Node HTTPS:

const options = {
  headers,
  ALPNProtocols: ["http/1.1"],
};

Apache’s TLS socket preparation on Java 17:

SSLParameters parameters = socket.getSSLParameters();
parameters.setApplicationProtocols(new String[]{"http/1.1"});
socket.setSSLParameters(parameters);

For each pair, the local captures showed unchanged cipher lists, supported groups and supported versions, with ALPN extension 16 added. The Python SSL documentation and Node’s TLS options document these controls.

On Food52, all three ALPN additions still returned 429. Ordinary curl already offered ALPN and negotiated HTTP/2, yet it also received the checkpoint. ALPN presence alone was not sufficient in this capture.

On Newsweek, Apache with ALPN returned 200 in both observations. But the unchanged non-ALPN Apache control also changed from 406 to 200 later. Python’s pair changed together, while Node’s pair remained rejected. That sequence does not establish that missing ALPN was the sole cause of Newsweek’s original rejection.

The useful engineering move is to add a measurable control, observe the result, and retain the unsuccessful control too. A missing extension is a hypothesis to test, not a site diagnosis inferred from a status code.

4. Changing the network profile restored access in Python and Node

We then tested two browser-profile libraries while retaining the same supplied application header values.

Python used curl_cffi’s explicit chrome150 profile:

from curl_cffi import requests

with requests.Session(trust_env=False) as session:
    response = session.get(
        url,
        headers=headers,
        impersonate="chrome150",
        default_headers=False,
        allow_redirects=False,
        timeout=25,
    )

Node used impit’s explicit chrome151 profile:

const { Impit } = require("impit");

const client = new Impit({
  browser: "chrome151",
  http3: false,
  timeout: 25000,
});
const response = await client.fetch(url, {
  method: "GET",
  headers,
  redirect: "manual",
});

Both controls returned feed content from both targets in both observations. Python’s profile negotiated HTTP/2. The impit wrapper used by this harness did not expose the negotiated HTTP version, so we did not invent one for its records.

These clients do not run Chrome or execute page JavaScript. They change network behavior. The libraries’ curl_cffi guide and impit implementation describe that purpose.

They also change several things together: TLS offers, extension behavior, protocol settings and serialization. That is a practical working alternative in this experiment, not a single-variable demonstration of which ClientHello field a server used. Their profiles were Chrome 150 and 151, while the copied header values reported Chrome 154. Neither should be called an exact Chrome 154 clone.

The local Python-profile capture also dropped Priority on HTTP/1.1 fallback despite that field being supplied. We keep that caveat explicit. The Node profile retained the full normalized header map locally. “Disable default headers” is a setting worth verifying, not a guarantee about every byte on the wire.

What the server can see beyond the copied headers

A useful way to investigate the mismatch is to separate four layers:

Copy as cURL is valuable for reproducing message-level details. It also helps expose when a browser-generated request depends on cookies, request bodies, or session state. It does not transfer the browser’s TLS engine or protocol implementation to the terminal.

HTTP/2 adds another representation boundary: it uses pseudo-headers and binary framing and prohibits connection-specific fields. Shared header values do not mean identical HTTP/1.1 and HTTP/2 serialization. RFC 9113 specifies that distinction.

The response still needs inspection. An earlier custom fetcher advertised Brotli but could not decode it, turning a successful 200 into a garbled text file. The new programmatic controls requested identity encoding and saved bytes first. Chrome and the successful profile clients also exposed a separate document issue: Food52’s feed used an undeclared snf namespace, so strict XML parsing failed despite receiving RSS markup. Access, decoding and extraction are different outcomes.

A 406 alone does not prove a particular anti-bot rule. Its defined HTTP semantics concern acceptable representations. Food52 supplied recognizable checkpoint HTML, but neither target exposed its internal decision logic to this experiment.

Limitations

The mechanism is general, but the measured outcomes belong to these two feeds, these client versions and this capture window. Two observations per programmatic configuration are not a population-level success rate. Profile controls ran after the native-client matrix rather than in randomized interleaved pairs.

Chrome used an existing browser profile. Newsweek’s navigation included a cookie, while Food52’s did not. Separate cookie-free browser fetches also succeeded, but their natural fetch headers differed from the copied navigation headers. We did not clear the user’s browser state, pin its network route, or assert that all clients reached the same edge or public egress IP. Proxy and direct curl results are reported separately.

The local fixture gives direct evidence of headers and ClientHello offers, not visibility into the request received inside a public CDN. Its SNI and server context differ from the public targets. It did not capture real Chrome’s ClientHello, HTTP/2 SETTINGS, or HTTP/3 traffic. Profile clients bundle multiple changes and cannot isolate one detection feature.

Newsweek’s ordinary direct clients sometimes changed from 406 to 200 without a code change. The earlier Java-only pilot also showed that behavior. We retained those transitions rather than selecting only failures for the comparison. A stable-looking screenshot can hide changing server-side context.

Reproduction

The expanded reproduction package contains all client source, the shared headers, pinned dependencies, 56 programmatic HTTP observations, four browser observations, and thirteen local connection captures. Cookies, challenge tokens, egress addresses and public feed bodies are excluded. None is needed to inspect the recorded comparison.

Start with the owned local fixture. Requirements are Python 3, Node, curl and OpenSSL. Java comparisons additionally need Java 17 and Maven for dependency installation.

unzip reproduction.zip
cd http-stack-study-v2
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
npm ci
python3 check_evidence.py
python3 probe_server.py --port 8443 --out captures.jsonl

In another terminal, activate the environment and compare native clients:

python3 compare.py https://localhost:8443/probe \
  --modes curl-direct,python-plain,python-alpn,requests-direct,node-plain,node-alpn \
  --out results/native

Then try Java and the profile controls:

python3 compare.py https://localhost:8443/probe \
  --modes apache,apache-alpn,jsse-h1,conscrypt-h1,conscrypt-h2,python-browser-profile,node-browser-profile \
  --out results/profiles

For an authorized remote comparison, substitute that URL and keep the request rate low. The scripts make GET requests and retain raw bodies plus JSON metadata. Local self-signed certificates are accepted only for explicit loopback hosts. Remote certificate verification stays enabled.

The Java source was compiled and exercised with the pinned JAR versions. Its Maven download bootstrap was not run in this environment because Maven was unavailable. The full native Python, Node and profile clients were executed for the measurements. The local parser supports the unfragmented ClientHello records observed here and is not a general packet dissector.

The main results, connection captures, and provenance can also be read separately. The earlier Java-only package remains available as a separate pilot, not part of the new sample counts.

Disclosure

Independent research using our own workstation and local probe, with small sets of public feed requests. No vendor supplied credits, sponsorship or affiliate arrangement for this experiment.

Copy the headers to reproduce the message. Measure the connection before assuming you reproduced the client.