URL Encoding Explained: %20 vs +, Reserved Characters, When to Encode

Percent-encoding, reserved characters and unreserved ones

A URL is a string with structure. Slashes separate path segments, a question mark starts the query, an ampersand separates parameters, a hash starts the fragment. Percent-encoding is the escape mechanism that lets a value contain one of those characters without being read as structure: the character's bytes are written as % followed by two hexadecimal digits, so a slash inside a value becomes %2F and stops behaving like a separator.

RFC 3986 splits the character set into three groups. Unreserved characters — the ASCII letters, the digits, and the four punctuation marks - . _ ~ — are safe everywhere and should never be encoded; encoding them changes nothing about how the URL is interpreted but does make two URLs that mean the same thing look different. Reserved characters are the ones with structural meaning, split into gen-delims (: / ? # [ ] @) and sub-delims (! $ & ' ( ) * + , ; =). Everything else, including the space and every non-ASCII character, has no place in a URL at all and must be encoded.

The crucial word in that middle group is "structural". A reserved character is not forbidden — it is required, in the position where it does its job. The question mark that starts the query must stay a question mark; a question mark inside a search term must become %3F. That is the whole decision procedure: are you writing the URL's structure, or a value that sits inside it? The functions in the next section differ only in which answer they assume.

%20 versus +: two different encodings

A space cannot appear in a URL, so it always has to be encoded — but there are two encodings in play and they disagree about how. Generic percent-encoding, the RFC 3986 kind, writes a space as %20 and works anywhere in a URL. Form encoding, the application/x-www-form-urlencoded serialisation that HTML forms have used since RFC 1866 defined HTML 2.0 in 1995 and that the WHATWG URL Standard specifies today, writes a space as + instead. It is used for query strings built from form data and for request bodies with that content type.

Both are correct in their own context, and both decode to a space when read by something that expects them. The problem is that + only means "space" under form encoding, which applies to the query string. In a path it is an ordinary character: the URL below has /files/a+b%20c as its path, and decoding that path yields a+b c, with the plus intact. A file genuinely named a+b will break the moment someone runs the path through a form decoder.

The mirror-image trap is decoding a query string with the wrong tool. Reading ?q=a+b%20c with URLSearchParams gives the string "a b c", because it applies form decoding and turns the + into a space. Passing the same raw value to decodeURIComponent gives "a+b c", because that function only understands percent-escapes. Neither is wrong; they answer different questions. The rule that avoids the whole class of bug is never to hand-build a query string — let URLSearchParams (or Python's urlencode) do it, so encoder and decoder agree.

const u = new URL("https://api.example.com/files/a+b%20c?q=a+b%20c");

u.pathname;                        // '/files/a+b%20c'
decodeURIComponent(u.pathname);    // '/files/a+b c'   ← the + survives
u.search;                          // '?q=a+b%20c'
u.searchParams.get("q");           // 'a b c'          ← form decoding: + is a space
decodeURIComponent("a+b%20c");     // 'a+b c'          ← percent decoding only

encodeURIComponent, encodeURI and URLSearchParams

JavaScript ships three encoders, and picking the wrong one is the most common URL bug there is. encodeURIComponent encodes a value: it escapes every reserved character, so an ampersand inside a search term becomes %26 and cannot split the query. encodeURI encodes a whole URL: it deliberately leaves the reserved characters alone so the URL keeps working, which means it is useless for values — a search term containing & passes straight through and truncates the parameter. URLSearchParams builds a query string with form encoding, so it escapes the reserved characters and writes spaces as +.

Run all three on the same input and the difference is immediate. With the string Ali & Bob's café/π? 100% #1 a+b=c, encodeURIComponent produces a value in which every separator is neutralised; encodeURI leaves &, /, ? and # exactly where they were, which would be right for a URL and is catastrophic for a query value; URLSearchParams produces the form-encoded version with plus signs for spaces and %27 for the apostrophe.

The table shows where they disagree character by character. Two rows are worth dwelling on. encodeURIComponent does not escape ! ' ( ) * — it follows the older RFC 2396 "mark" set rather than RFC 3986's unreserved set — so those five characters survive it untouched, which occasionally matters when a server-side signature expects them encoded. And URLSearchParams escapes ~ as %7E while leaving * alone, the exact opposite of encodeURIComponent, because the form-encoding serialiser has its own list. Neither is a bug; both are what their respective specifications say.

In practice: use URLSearchParams to build query strings, encodeURIComponent for a value you are splicing into a path segment or a fragment, and encodeURI essentially never — if you find yourself reaching for it, you probably have a URL that was assembled by string concatenation and should have been built with the URL constructor. Pasting a value into URL encode and decode is the quickest way to see which form you are looking at.

const raw = "Ali & Bob's café/π? 100% #1 a+b=c";

encodeURIComponent(raw);
// "Ali%20%26%20Bob's%20caf%C3%A9%2F%CF%80%3F%20100%25%20%231%20a%2Bb%3Dc"

encodeURI(raw);
// "Ali%20&%20Bob's%20caf%C3%A9/%CF%80?%20100%25%20#1%20a+b=c"
//        ↑ & / ? # survive — fine for a URL, fatal for a value

new URLSearchParams({ q: raw }).toString();
// "q=Ali+%26+Bob%27s+caf%C3%A9%2F%CF%80%3F+100%25+%231+a%2Bb%3Dc"

The same job in Python

Python's urllib.parse offers the same split with different defaults, and the default is the one that catches people. quote() is meant for path components, so it leaves / unescaped: quoting a value that contains a slash produces a string with a live path separator in it. Pass safe="" to turn that off and get the equivalent of encodeURIComponent.

quote_plus() is the form-encoding version: it escapes / as %2F and writes spaces as +. urlencode() builds a whole query string from a dict or a list of pairs and uses quote_plus on each value, so it is the direct counterpart of URLSearchParams and the right default for building a query. The round trip is exact — unquote_plus(quote_plus(s)) returns the original string, including the non-ASCII characters.

The differences from JavaScript are small but real, and they matter when a signature or a cache key is computed on both sides of a request. Python's quote leaves - . _ ~ unescaped, matching RFC 3986's unreserved set exactly, so it escapes ! ' ( ) * where encodeURIComponent does not, and it escapes * as %2A where URLSearchParams leaves it alone. If a Python service and a JavaScript client both sign a canonicalised URL, this is precisely where the signatures diverge.

from urllib.parse import quote, quote_plus, urlencode, unquote_plus

raw = "Ali & Bob's café/π? 100% #1 a+b=c"

quote(raw)
# "Ali%20%26%20Bob%27s%20caf%C3%A9/%CF%80%3F%20100%25%20%231%20a%2Bb%3Dc"
#                                 ↑ safe="/" is the default

quote(raw, safe="")
# "Ali%20%26%20Bob%27s%20caf%C3%A9%2F%CF%80%3F%20100%25%20%231%20a%2Bb%3Dc"

quote_plus(raw)
# "Ali+%26+Bob%27s+caf%C3%A9%2F%CF%80%3F+100%25+%231+a%2Bb%3Dc"

urlencode({"q": raw})
# "q=Ali+%26+Bob%27s+caf%C3%A9%2F%CF%80%3F+100%25+%231+a%2Bb%3Dc"

unquote_plus(quote_plus(raw)) == raw
# True

Non-ASCII characters are encoded as UTF-8 bytes

Percent-encoding operates on bytes, not on characters, so a non-ASCII character is first encoded to bytes and then each byte is written as its own percent-escape. Modern encoders use UTF-8, which means one character can become two, three or four escapes: é is %C3%A9, π is %CF%80, 中 is %E4%B8%AD, and an emoji is four escapes because it is four UTF-8 bytes. This is why an encoded URL can be several times longer than the text it carries, and why truncating a percent-encoded string at a fixed length can cut a character in half and produce something that will not decode.

The choice of UTF-8 is not universal history. Older systems encoded query strings in the page's own charset, so a search for café from a Latin-1 page could arrive as %E9 rather than %C3%A9. The WHATWG URL Standard settles this for the web — URLs are UTF-8 — but legacy back-ends, desktop clients and file-name schemes still produce the older forms, and a %E9 that a UTF-8 decoder rejects is the usual signature. If a value is arriving mangled and you need to see what bytes are really in it, decoding it with URL encode and decode and comparing against a Base64 encode and decode round trip of the same text will show which encoding it came from.

One consequence worth designing around: length limits are on bytes after encoding, not on characters before it. A 500-character search term in Japanese can become a 4,500-character query string, and servers do enforce URL length limits. If a value can be long or is likely to be non-ASCII, it belongs in a POST body rather than in a query string.

for (const s of ["é", "π", "中", "😀"]) {
  console.log(s, encodeURIComponent(s), new TextEncoder().encode(s).length);
}
// é  %C3%A9        2 UTF-8 bytes
// π  %CF%80        2 UTF-8 bytes
// 中 %E4%B8%AD     3 UTF-8 bytes
// 😀 %F0%9F%98%80  4 UTF-8 bytes

Double encoding, and how to recognise it

Because % is itself a character that has to be encoded, encoding an already-encoded string produces a valid but different string: a space becomes %20, and encoding that again turns the percent sign into %25, giving %2520. Decode it once and you get %20 back — a literal percent-two-zero, not a space. The value looks almost right, which is what makes the bug survive so long in production.

The visual signature is %25 followed by two more hex digits: %2520 for a space, %2526 for an ampersand, %253A for a colon. Once you know to look for %25 in a URL that should not contain a literal percent sign, double encoding becomes obvious at a glance. The usual cause is a value that is encoded by application code and then encoded again by a framework, an HTTP client or a proxy — or a redirect URL that is passed through two layers, each of which helpfully encodes it.

The fix is the same architectural rule as with HTML escaping: encode exactly once, at the boundary where the value is placed into the URL. Do not encode values as you store them, do not encode them again when reading them back, and do not run a URL you were given through an encoder before appending to it. When you inherit data that is already double-encoded, decode it as many times as it was encoded — one decode of %2520 gets you to %20, not to a space.

const once  = encodeURIComponent("a b&c");   // 'a%20b%26c'
const twice = encodeURIComponent(once);      // 'a%2520b%2526c'

decodeURIComponent(twice);                   // 'a%20b%26c'   ← still encoded
decodeURIComponent(decodeURIComponent(twice)); // 'a b&c'

// The same thing happens in Python, with + standing in for %20:
//   quote_plus(quote_plus("a b&c"))  ->  'a%2Bb%2526c'
//   unquote_plus(quote_plus(quote_plus("a b&c")))  ->  'a+b%26c'

Putting JSON in a query string

Sooner or later an API needs a structured filter in a GET request, and someone reaches for JSON in a query parameter. It works, provided the JSON is encoded as a single opaque value. Every character JSON uses structurally — the braces, the brackets, the quotes, the commas, the colons — is either reserved or not allowed in a URL, so an unencoded JSON blob in a query string will be mangled by the first proxy that normalises it, and any & inside a string value will split the parameter outright.

Serialise with JSON.stringify, hand the result to URLSearchParams, and read it back with searchParams.get followed by JSON.parse. The intermediate string is unreadable, which is the point: it survives proxies, logs and redirects unchanged. Handling it by hand is where the mistakes come from, so if you are debugging a request like this, JSON URL encode does the two steps together, and running the decoded value through the JSON formatter makes it readable again.

Two caveats. URLs have practical length limits — many servers and CDNs cut off somewhere between 4 KB and 8 KB of request line — and an encoded JSON object grows quickly, so anything beyond a modest filter belongs in a POST body. And a query string is logged, cached, stored in browser history and passed in Referer headers, which makes it the wrong place for anything sensitive no matter how thoroughly it is encoded. Encoding is not concealment.

const filter = { status: ["open", "pending"], q: "a&b" };

const qs = new URLSearchParams({ filter: JSON.stringify(filter) }).toString();
// 'filter=%7B%22status%22%3A%5B%22open%22%2C%22pending%22%5D%2C%22q%22%3A%22a%26b%22%7D'

const back = JSON.parse(new URLSearchParams(qs).get("filter"));
// { status: [ 'open', 'pending' ], q: 'a&b' }