code point
Click on the red underlined text to get to the source
... algorithm called
Bootstring, which allows strings composed from a small set of "basic"
code points to uniquely represent any string of code points drawn
from a larger set. Punycode is Bootstring with particular parameter
values ...
... Bootstring, which allows strings composed from a small set of "basic"
code points to uniquely represent any string of code points drawn
from a larger set. Punycode is Bootstring with particular parameter
values appropriate for IDNA ...
... Bootstring has been designed to have the following features:
* Completeness: Every extended string (sequence of arbitrary code
points) can be represented by a basic string (sequence of basic
code points). Restrictions on what strings are allowed, and on
...
... * Completeness: Every extended string (sequence of arbitrary code
points) can be represented by a basic string (sequence of basic
code points). Restrictions on what strings are allowed, and on
length, can be imposed by higher layers.
...
... at odds; Bootstring aims at a good balance between them.
* Readability: Basic code points appearing in the extended string
are represented as themselves in the basic string (although the
main purpose is to improve efficiency, not readability).
...
... RFC2119].
A code point is an integral value associated with a character in a
coded character set.
...
... Unicode Standard [UNICODE], Unicode code points are denoted
by "U+" followed by four to six hexadecimal digits, while a range of
...
... by "U+" followed by four to six hexadecimal digits, while a range of
code points is denoted by two hexadecimal numbers separated by "..",
with no prefixes.
...
...
Bootstring represents an arbitrary sequence of code points (the
"extended string") as a sequence of basic code points (the "basic
...
... Bootstring represents an arbitrary sequence of code points (the
"extended string") as a sequence of basic code points (the "basic
string"). This section describes the representation. Section 6
"Bootstring algorithms ...
...
The following sections describe the four techniques used in
Bootstring. "Basic code point segregation" is a very simple and
efficient encoding for basic code points ...
... code point segregation" is a very simple and
efficient encoding for basic code points occurring in the extended
string: they are simply copied all at once. "Insertion unsort
coding" encodes the non-basic code points ...
... code points occurring in the extended
string: they are simply copied all at once. "Insertion unsort
coding" encodes the non-basic code points as deltas, and processes
the code points in numerical order rather than in order of
...
... coding" encodes the non-basic code points as deltas, and processes
the code points in numerical order rather than in order of
appearance, which typically results in smaller deltas. The deltas
are represented as "generalized variable-length ...
... are represented as "generalized variable-length integers", which use
basic code points to represent nonnegative integers. The parameters
of this integer representation are dynamically adjusted using "bias
...
... Basic code point segregation ...
...
All basic code points appearing in the extended string are
represented literally at the beginning of the basic string, in their
original order, followed by a delimiter if (and only if) the number
...
... represented literally at the beginning of the basic string, in their
original order, followed by a delimiter if (and only if) the number
of basic code points is nonzero. The delimiter is a particular basic
code point, which never appears in the remainder of the basic string.
...
... of basic code points is nonzero. The delimiter is a particular basic
code point, which never appears in the remainder of the basic string.
The decoder can therefore find the end of the literal ...
... literal portion of the basic string
(excluding the last delimiter). The decoder inserts non-basic code
points, one for each delta, into the extended string, ultimately
arriving at the final decoded string.
...
... division and remainder calculations to compute the next insertion
state directly.) It is an error if the inserted code point is a
basic code point (because basic code points ...
... state directly.) It is an error if the inserted code point is a
basic code point (because basic code points were supposed to be
segregated as described in section 3.1).
...
... code point is a
basic code point (because basic code points were supposed to be
segregated as described in section 3.1).
...
... cause the decoder to construct the desired string. It can do this by
repeatedly scanning the extended string for the next code point that
the decoder would need to insert, and counting the number of state
changes ...
... decoder would need to perform, mindful of the fact that
the decoder's extended string will include only those code points
that have already been inserted. Section 6.3 "Encoding procedure"
...
... let delta = delta + (delta div numpoints)
numpoints is the total number of code points encoded/decoded so
far (including the one corresponding to this delta itself, and
including the basic code points ...
... code points encoded/decoded so
far (including the one corresponding to this delta itself, and
including the basic code points).
3. Delta is repeatedly divided until it falls within a threshold ...
...
Given a set of basic code points, one needs to be designated as the
delimiter. The base cannot be greater than the number of
distinguishable basic code points ...
... code points, one needs to be designated as the
delimiter. The base cannot be greater than the number of
distinguishable basic code points remaining. The digit-values in the
range 0 through base-1 need to be associated with distinct non-
...
... range 0 through base-1 need to be associated with distinct non-
delimiter basic code points. In some cases multiple code points need
to have the same digit-value; for example, uppercase and lowercase
...
... range 0 through base-1 need to be associated with distinct non-
delimiter basic code points. In some cases multiple code points need
to have the same digit-value; for example, uppercase and lowercase
versions ...
...
The initial value of n cannot be greater than the minimum non-basic
code point that could appear in extended strings.
The remaining five parameters (tmin, tmax, skew, damp, and the
...
...
If support for mixed-case annotation is desired (see appendix A),
make sure that the code points corresponding to 0 through tmax-1 all
have both uppercase and lowercase forms.
...
... to work well with Unicode [UNICODE] code points, which are integers
in the range 0..10FFFF (but not D800..DFFF, which are reserved for
...
... the ASCII [ASCII] code points (0..7F), of which U+002D (-) is the
delimiter, and some of the others have digit-values as follows:
...
... delimiter, and some of the others have digit-values as follows:
code points digit-values
------------ ----------------------
41..5A (A-Z) = 0 to 25, respectively
...
... can end with a hyphen-minus only if the Unicode string consists
entirely of basic code points, but IDNA forbids such strings from
being encoded. The encoded string can begin with a hyphen-minus, but
...
... explain the conditions under which they can be omitted.
Formally, code points are integers, and hence the pseudocode assumes
that arithmetic operations can be performed directly on code points ...
... code points are integers, and hence the pseudocode assumes
that arithmetic operations can be performed directly on code points.
In some programming languages, explicit conversion between code
points ...
... code points.
In some programming languages, explicit conversion between code
points and integers might be necessary.
...
... let bias = initial_bias
let output = an empty string indexed from 0
consume all code points before the last delimiter (if there is one)
and copy them to output, fail on any non-basic code point
...
... consume all code points before the last delimiter (if there is one)
and copy them to output, fail on any non-basic code point
if more than zero code points were consumed then consume one more
...
... and copy them to output, fail on any non-basic code point
if more than zero code points were consumed then consume one more
(which will be the last delimiter)
while the input is not exhausted do begin
...
... let w = 1
for k = base to infinity in steps of base do begin
consume a code point, or fail if there was none to consume
let digit = the code point's digit-value, fail if it has none
...
... consume a code point, or fail if there was none to consume
let digit = the code point's digit-value, fail if it has none
let i = i + digit * w, fail on overflow
let t = tmin if k <= bias {+ tmin}, or
...
... let n = n + i div (length(output) + 1), fail on overflow
let i = i mod (length(output) + 1)
{if n is a basic code point then fail}
insert n into output at position i
increment i
...
...
The full statement enclosed in braces (checking whether n is a basic
code point) can be omitted if initial_n exceeds all basic code points
(which is true for Punycode), because n is never less than initial_n.
...
... The full statement enclosed in braces (checking whether n is a basic
code point) can be omitted if initial_n exceeds all basic code points
(which is true for Punycode), because n is never less than initial_n.
...
... encoded string that can represent a given sequence of integers. The
only error conditions are invalid code points, unexpected end-of-
input, overflow, and basic code points encoded using deltas instead
...
... error conditions are invalid code points, unexpected end-of-
input, overflow, and basic code points encoded using deltas instead
of appearing literally. If the decoder fails on these errors as
...
... let delta = 0
let bias = initial_bias
let h = b = the number of basic code points in the input
copy them to the output in order, followed by a delimiter if b > 0
{if the input contains a non-basic code point ...
... code points in the input
copy them to the output in order, followed by a delimiter if b > 0
{if the input contains a non-basic code point < n then fail}
while h < length(input) do begin
let m = the minimum {non-basic} code point ...
... code point < n then fail}
while h < length(input) do begin
let m = the minimum {non-basic} code point >= n in the input
let delta = delta + (m - n) * (h + 1), fail on overflow
let n = m
...
... let delta = delta + (m - n) * (h + 1), fail on overflow
let n = m
for each code point c in the input (in order) do begin
if c < n {or c is basic} then increment delta, fail on overflow
if c == n then begin
...
... tmax if k >= bias + tmax, or k - bias otherwise
if q < t then break
output the code point for digit t + ((q - t) mod (base - t))
let q = (q - t) div (base - t)
end
...
... let q = (q - t) div (base - t)
end
output the code point for digit q
let bias = adapt(delta, h + 1, test h equals b?)
let delta = 0
...
...
The full statement enclosed in braces (checking whether the input
contains a non-basic code point less than n) can be omitted if all
code points less than initial_n are basic code points ...
... contains a non-basic code point less than n) can be omitted if all
code points less than initial_n are basic code points (which is true
for Punycode if code points ...
... code point less than n) can be omitted if all
code points less than initial_n are basic code points (which is true
for Punycode if code points are unsigned).
...
... code points less than initial_n are basic code points (which is true
for Punycode if code points are unsigned).
The brace-enclosed conditions "non-basic" and "or c is basic" can be
...
...
The brace-enclosed conditions "non-basic" and "or c is basic" can be
omitted if initial_n exceeds all basic code points (which is true for
Punycode), because the code point being tested is never less than
...
... omitted if initial_n exceeds all basic code points (which is true for
Punycode), because the code point being tested is never less than
initial_n.
...
... IDNA labels without overflow, because any string that needed a 27-bit
delta would have to exceed either the code point limit (0..10FFFF) or
the label length limit (63 characters). However, overflow handling
is necessary because the inputs are not necessarily valid ...
... from happening. For example, if the encoder were to verify that no
input code points exceed M and that the input length does not exceed
L, then no delta could ever exceed (M - initial_n) * (L + 1), and
hence no overflow could occur if integer ...
... state of the decoder is shown
as a sequence of hexadecimal values, representing the code points in
the extended string. An asterisk appears just after the most
recently inserted code point ...
... code points in
the extended string. An asterisk appears just after the most
recently inserted code point, indicating both n (the value preceeding
the asterisk) and i (the position of the value just after the
asterisk). Other numerical values are decimal.
...
...
In the following traces, code point values are hexadecimal, while
other numerical values are decimal.
...
... input is:
4ED6 4EEC 4E3A 4EC0 4E48 4E0D 8BF4 4E2D 6587
there are no basic code points, so no literal portion
next code point ...
... code points, so no literal portion
next code point to insert is 4E0D
needed delta is 19853, encodes as "ihq"
bias becomes 21
...
... needed delta is 19853, encodes as "ihq"
bias becomes 21
next code point to insert is 4E2D
needed delta is 64, encodes as "wc"
bias becomes 20
...
... needed delta is 64, encodes as "wc"
bias becomes 20
next code point to insert is 4E3A
needed delta is 37, encodes as "rb"
bias becomes 13
...
... needed delta is 37, encodes as "rb"
bias becomes 13
next code point to insert is 4E48
needed delta is 56, encodes as "4c"
bias becomes 17
...
... needed delta is 56, encodes as "4c"
bias becomes 17
next code point to insert is 4EC0
needed delta is 599, encodes as "v8a"
bias becomes 32
...
... needed delta is 599, encodes as "v8a"
bias becomes 32
next code point to insert is 4ED6
needed delta is 130, encodes as "8d"
bias becomes 23
...
... needed delta is 130, encodes as "8d"
bias becomes 23
next code point to insert is 4EEC
needed delta is 154, encodes as "qg"
bias becomes 25
...
... needed delta is 154, encodes as "qg"
bias becomes 25
next code point to insert is 6587
needed delta is 46301, encodes as "056p"
bias becomes 84
...
... needed delta is 46301, encodes as "056p"
bias becomes 84
next code point to insert is 8BF4
needed delta is 88531, encodes as "qjye"
bias becomes 90
...
... input is:
0033 5E74 0042 7D44 91D1 516B 5148 751F
basic code points (0033, 0042) are copied to literal portion: "3B-"
next code point ...
... code points (0033, 0042) are copied to literal portion: "3B-"
next code point to insert is 5148
needed delta is 62042, encodes as "ww4c"
bias becomes 27
...
... needed delta is 62042, encodes as "ww4c"
bias becomes 27
next code point to insert is 516B
needed delta is 139, encodes as "5e"
bias becomes 24
...
... needed delta is 139, encodes as "5e"
bias becomes 24
next code point to insert is 5E74
needed delta is 16683, encodes as "180e"
bias becomes 67
...
... needed delta is 16683, encodes as "180e"
bias becomes 67
next code point to insert is 751F
needed delta is 34821, encodes as "575a"
bias becomes 82
...
... needed delta is 34821, encodes as "575a"
bias becomes 82
next code point to insert is 7D44
needed delta is 14592, encodes as "65l"
bias becomes 67
...
... needed delta is 14592, encodes as "65l"
bias becomes 67
next code point to insert is 91D1
needed delta is 42088, encodes as "sy2b"
bias becomes 84
...
... IDNA can disregard this appendix.
Basic code points can use mixed case directly, because the decoder
copies them verbatim, leaving lowercase code points ...
... code points can use mixed case directly, because the decoder
copies them verbatim, leaving lowercase code points lowercase, and
leaving uppercase code points uppercase. Each non-basic code point ...
... copies them verbatim, leaving lowercase code points lowercase, and
leaving uppercase code points uppercase. Each non-basic code point
is represented by a delta, which is represented by a sequence of
...
... code points lowercase, and
leaving uppercase code points uppercase. Each non-basic code point
is represented by a delta, which is represented by a sequence of
basic code points ...
... code point
is represented by a delta, which is represented by a sequence of
basic code points, the last of which provides the annotation. If it
is uppercase, it is a suggestion to map the non-basic code point to
...
... basic code points, the last of which provides the annotation. If it
is uppercase, it is a suggestion to map the non-basic code point to
uppercase (if possible); if it is lowercase, it is a suggestion to
map the non-basic code point ...
... code point to
uppercase (if possible); if it is lowercase, it is a suggestion to
map the non-basic code point to lowercase (if possible).
These annotations do not alter the code points ...
... code point to lowercase (if possible).
These annotations do not alter the code points returned by decoders;
the annotations are returned separately, for the caller ...
... caller to use or
ignore. Encoders can accept annotations in addition to code points,
but the annotations do not alter the output, except to influence the
uppercase/lowercase form of ASCII letters ...
... Unicode to Punycode. The input */
/* is represented as an array of Unicode code points (not code */
/* units; surrogate pairs are not allowed), and the output */
/* will be represented as an array of ASCII ...
... /* units; surrogate pairs are not allowed), and the output */
/* will be represented as an array of ASCII code points. The */
/* output string is *not* null-terminated; it will contain */
/* zeros if and only if the input contains zeros. (Of course */
...
... /* the caller can leave room for a terminator and add one if */
/* needed.) The input_length is the number of code points in */
/* the input. The output_length is an in/out argument: the */
/* caller ...
... /* the input. The output_length is an in/out argument: the */
/* caller passes in the maximum number of code points that it */
/* can receive, and on successful return it will contain the */
...
...
/* can receive, and on successful return it will contain the */
/* number of code points actually output. The case_flags array */
/* holds input_length boolean values, where nonzero suggests that */
...
... /* after being decoded (if possible), and zero suggests that */
/* it be forced to lowercase (if possible). ASCII code points */
/* are encoded literally, except that ASCII letters are forced */
...
... /* uppercase flags. If case_flags is a null pointer then ASCII */
/* letters are left as they are, and other code points are */
/* treated as if their uppercase flags were zero. The return */
/* value can be any of the punycode_status values defined above */
...
... Unicode. The input is */
/* represented as an array of ASCII code points, and the output */
/* will be represented as an array of Unicode code points ...
... code points, and the output */
/* will be represented as an array of Unicode code points. The */
/* input_length is the number of code points in the input. The */
...
... Unicode code points. The */
/* input_length is the number of code points in the input. The */
/* output_length is an in/out argument: the caller passes in */
...
... /* output_length is an in/out argument: the caller passes in */
/* the maximum number of code points that it can receive, and */
/* on successful return it will contain the actual number of */
/* code points ...
... code points that it can receive, and */
/* on successful return it will contain the actual number of */
/* code points output. The case_flags array needs room for at */
/* least output_length values, or it can be a null pointer if the */
/* case information is not needed. A nonzero flag suggests that */
...
... caller (if possible), while zero suggests that it be */
/* forced to lowercase (if possible). ASCII code points are */
/* output already in the proper case, but their flags will be set */
/* appropriately so that applying the flags would be harmless. */
...
... initial_bias = 72, initial_n = 0x80, delimiter = 0x2D };
/* basic(cp) tests whether cp is a basic code point: */
#define basic(cp) ((punycode_uint)(cp) < 0x80)
...
... }
/* encode_digit(d,flag) returns the basic code point whose value */
/* (when used for representing integers) is d, which needs to be in */
/* the range ...
... }
/* flagged(bcp) tests whether a basic code point is flagged */
/* (uppercase). The behavior is undefined if bcp is not a */
/* basic code point ...
... code point is flagged */
/* (uppercase). The behavior is undefined if bcp is not a */
/* basic code point. */
#define flagged(bcp) ((punycode_uint)(bcp) - 65 < 26)
...
...
/* encode_basic(bcp,flag) forces a basic code point to lowercase */
/* if flag is zero, uppercase if flag is nonzero, and returns */
/* the resulting code point ...
... code point to lowercase */
/* if flag is zero, uppercase if flag is nonzero, and returns */
/* the resulting code point. The code point is unchanged if it */
/* is caseless. The behavior is undefined if bcp is not a basic */
...
... /* if flag is zero, uppercase if flag is nonzero, and returns */
/* the resulting code point. The code point is unchanged if it */
/* is caseless. The behavior is undefined if bcp is not a basic */
/* code point ...
... code point is unchanged if it */
/* is caseless. The behavior is undefined if bcp is not a basic */
/* code point. */
static char encode_basic(punycode_uint bcp, int flag)
...
... bias = initial_bias;
/* Handle the basic code points: */
for (j = 0; j < input_length; ++j) {
...
... }
/* else if (input[j] < n) return punycode_bad_input; */
/* (not needed for Punycode with unsigned code points) */
}
...
... h = b = out;
/* h is the number of code points that have been handled, b is the */
/* number of basic code points, and out is the number of characters */
...
... /* h is the number of code points that have been handled, b is the */
/* number of basic code points, and out is the number of characters */
/* that have been output. */
...
...
while (h < input_length) {
/* All non-basic code points < n have been */
/* handled already. Find the next larger one: */
...
... bias = initial_bias;
/* Handle the basic code points: Let b be the number of input code */
/* points before the last delimiter, or 0 if there is none, then */
/* copy the first b code points ...
... code points: Let b be the number of input code */
/* points before the last delimiter, or 0 if there is none, then */
/* copy the first b code points to the output. */
for (b = j = 0; j < input_length; ++j) if (delim(input[j])) b = j;
...
... /* Main decoding loop: Start just after the last delimiter if any */
/* basic code points were copied; start at the beginning otherwise. */
...
...
/* in is the index of the next character to be consumed, and */
/* out is the number of code points in the output array. */
/* Decode a generalized variable-length ...
... fprintf(stderr,
"\n"
"%s -e reads code points and writes a Punycode string.\n"
"%s -d reads a Punycode string and writes code points.\n"
...
... "%s -e reads code points and writes a Punycode string.\n"
"%s -d reads a Punycode string and writes code points.\n"
"\n"
"Input and output are plain text in the native character set ...
... "Input and output are plain text in the native character set.\n"
"Code points are in the form u+hex separated by whitespace.\n"
"Although the specification allows Punycode strings to contain\n"
"any characters from the ASCII ...
... int c;
/* Read the input code points: */
input_length = 0;
...