KPS 9566

The ASCII character set originated in the United States in 1963, and was revised in 1967 to the form it has today.^[8] ASCII also became accepted as an international standard in 1967, becoming ECMA-6,^[8] designated ISO/IEC 646 by the International Organization for Standardization.^[9] It is presently designated ANSI X3.4-1986 and ISO 646:1991.^[10] ASCII was a 7-bit, single-byte encoding including 94 graphical characters, the space, and 33 control codes, which provided basic support for representing American English text as a series of bytes.^[8][10]

The next edition of ISO 646, published in 1972, revised the standard to introduce the concept of national versions of the code, allowing countries to replace a few less commonly used codes with their own required characters. At the same time, work on defining extension mechanisms for ASCII was underway, with the intention of being applicable to both 7-bit and 8-bit environments. This was completed in 1973 and published as JIS X 0202, ECMA-35 and ISO 2022.^[11] ISO 2022 specifies mechanisms for using single-byte and multiple-byte character sets with a certain structure in both 7-bit and 8-bit environments, and for declaring and switching between them in a standard fashion using shift codes and escape sequences.^[12]

Countries in East Asia, due to using large repertoires of Chinese characters, introduced standardised double-byte encodings (DBCS) for their writing systems, since the number of characters representable in a single-byte code was not sufficient. In an ISO 2022 compliant DBCS, every character can be represented with two ASCII printing character bytes; the location of a character can be referenced by these byte values, or by two numbers from 1 to 94 (a kuten), equal to the respective bytes minus 32.^[13] The first registered ISO 2022 compliant DBCS, and the first East Asian DBCS to be established as a national standard, was the first edition of JIS X 0208 (Japan), published in 1978.^[14][15] This was followed by GB 2312 (Mainland China) in 1980, and by Wansung code (South Korea; first designated KS C 5601-1987) in 1987.^[16][15] Big5 (Taiwan), defined in 1984, did not follow the ISO 2022 structure.^[16] When used in an 8-bit (rather than 7-bit) environment, GB 2312 and Wansung code were usually used with the eighth bit set, with ASCII or a similar SBCS used with the eighth bit unset; these encoding schemes are known as EUC-CN and EUC-KR, respectively.^[17]

Although the Korean writing system includes individual symbols (jamo) for consonants and vowels, serving as an alphabet, Korean text is properly typeset with these symbols composed into blocks for each syllable. Wansung code included individual Korean syllable blocks separately, treating them as a large set of characters similarly to Hanja,^[18] and was first defined by the third edition of the South Korean standard KS C 5601. The first edition had defined an encoding of individual jamo which allowed syllable blocks to be encoded as sequences, which was named N-byte Hangul, and had not been adopted as widely as intended.^[19][20]

Wansung code did not encode all possible modern Korean syllables, only a selection of the 2350 most common,^[2] although it allowed them to be specified using combining sequences, which often were not supported.^[18] An alternative encoding, also South Korean, named Johab did, and served as a competitor to Wansung for some time.^[19] Unified Hangul Code (UHC), introduced by Microsoft with Windows 95, extended EUC-KR, allowing the use of invalid EUC double-byte codes to represent all other syllables available in Johab.^[18] A similar approach was taken by the Mainland Chinese GBK encoding, extending GB 2312 with support for Traditional Chinese and for less common Chinese characters by encoding them to double-byte codes invalid in EUC-CN.^[16]

South Korea was not the only country developing an ISO 2022 DBCS for Korean: the Mainland Chinese GB 12052 was published in 1989. This was not closely related to Wansung code, although it also included composed syllables. Instead, it corresponded to GB 2312 with Korean syllables (and 94 Hanja) replacing the Chinese characters, except for the inclusion of a dollar sign in place of a yuan sign. It was developed for use by the Korean minority in north-eastern China.^[2]

Likewise, North Korea developed KPS 9566. Although North Korea and South Korea both use Korean Chosŏn'gŭl (Hangul) as their primary writing system, they use different lexicographical orders.^[21] Hence, character ordering differs between Wansung code and KPS 9566.^[4]

KPS 9566 has undergone several revisions, including editions of 1997 and 2003,^[22] mainly to enhance compatibility with Unicode. These are commonly indicated by specifying the year (e.g. KPS 9566-97, 9566-2003). The current edition as of the release of Red Star OS 3.0 appears to be KPS 9566-2011, which adds Kim Jong Un to the list of leaders.^[3] The publicly available code chart for the 1997 edition of KPS 9566 shows a ISO 2022 94×94 plane.^[23] The more recent editions, from what sources of information are available outside of North Korea itself, appear to define additional allocations outside of the EUC plane (similarly to GBK or UHC).^[3]

Due to the interoperability issues arising from the use of multiple national standard and platform- or font-specific proprietary character encodings, the Unicode standard was developed with the intent of allowing all representable text to be interchanged in a single, universal format. The first edition of Unicode was published in 1991 and 1992,^[24] and ISO/IEC 10646 was established in sync with Unicode in 1993.^[25] Unicode formats are preferred for international use on the World Wide Web, where legacy character encodings are treated as partial encodings of Unicode by means of mapping files.^[26][27]

In principle, KPS 9566 is similar to the Wansung character set defined by the South Korean KS X 1001 standard, although the two are not compatible. Both encode a section of punctuation, symbols, jamo, kana and alphabetical characters, followed by a subset of the possible modern Chosŏn'gŭl syllables, followed by a section of Hanja.^[2] However, KPS 9566 uses a different ordering of jamo and syllables to conform with North Korean lexicographical ordering standards.^[4] KPS 9566 also includes 28 explicitly rotated punctuation characters for vertical typography, which KS X 1001 does not, and encodes each Hanja only once, whereas KS X 1001 encodes several Hanja with multiple readings multiple times.^[2]

KPS 9566-97 encodes a total of 2679 Chosŏn'gŭl syllables and 4653 Hanja. This provides better coverage than the 2350 syllables encoded by Wansung code: for instance, the 똠 character used in the name of 똠방각하, a noted Korean literary work, does not have an assigned Wansung codepoint, but has one (38-02) in KPS 9566.^[2] The Hanja section includes 4652 characters from the Unified Repertoire and Ordering and one from CJK Unified Ideographs Extension A. The entirety of row 15, the latter half of row 44 (after the syllables block) and the latter half of row 94 (after the Hanja block) may be used for user-defined purposes.^[23][2]

KPS 9566 is especially distinguished by its inclusion of several special characters from North Korean political life. Specifically, it includes the hammer, sickle and brush emblem of the Workers' Party of Korea, both uncircled and circled^[7] (code points 12-01 and 12-02),^[23] and two groups of three special-purpose characters which spell out the names of the North Korean leaders Kim Il Sung (김일성) and Kim Jong Il (김정일) in a special decorative font (code points 04-72 to 04-74 and 04-75 to 04-77, respectively).^[28] The syllables for Kim and Il, which are identical in the spelling of both names, are encoded twice. KPS 9566-2011 additionally includes the name of Kim Jong Un (김정은) as code points 04-78 to 04-80.^[3][5]

Due to these special characters, there is currently no full round-trip compatibility between KPS 9566 and Unicode, unless unsupported characters are mapped to the Private Use Area.^[1]

Unicode's initial coverage of Korean syllables, added in version 1.0, was based on Wansung code. In Unicode version 2.0, a new block of Korean syllables (the present Hangul Syllables block) was added, based on the syllable repertoire available in Johab, and the previous block was deleted (it is now occupied by CJK Unified Ideographs Extension A). This was done under the assumption that no Unicode-encoded Korean data existed yet, but became known as the "Korean mess", and the responsible committees pledged not to make such an incompatible change in the future,^[38] a pledge codified by the Unicode Stability Policy.^[39]

The code chart for KPS 9566-97, published April 1997,^[2] was submitted to the ISO International Register of Coded Character Sets for registration for use with ISO/IEC 2022. It was registered in June 1998 with the number ISO-IR-202. This code chart is publicly available from the Information Processing Society of Japan.^[23]

In August 1999, the North Korean national body submitted a document to WG2 (ISO/IEC JTC 1/SC 2 Working Group 2), the ISO body responsible for ISO/IEC 10646, the international standard corresponding to Unicode. This document requested the addition of the KPS 9566 codes to the existing cross-references from the CJK Unified Ideographs charts, the addition of 80 symbol characters from KPS 9566 which did not have existing Unicode mappings, a resolution to the difference in collation order between KPS 9566 and Unicode (due to the order of the characters in Unicode following the South Korean encodings) and the addition of 8 combining jamo. It also requested for WG2 to edit the existing Unicode character and block names to use the term "Korean character" rather than "Hangul".^[40] An expanded version of this proposal, broken into several documents, was submitted as a work item in December 1999.^[41]

A detailed response was submitted by the Swedish representative in March 2000, opposing several of the points and elaborating on Sweden's vote against the proposal. This response stated that changing the encoding of the Korean characters again would cause major disruption, even more so than the first time, which was done when comparatively few implementations existed, but which in retrospect should not have been done. It explained that that few or no languages can be collated correctly by code point value, and that a tailoring for the Unicode Collation Algorithm or ISO/IEC 14651 (then being drafted) should be used for that purpose, and that normative names of characters already assigned cannot be changed, due to the stability policy, although non-normative translations to other languages can be employed. It suggested that a machine-readable mapping file between Unicode and KPS 9566 could be provided by the North Korean body itself, and would be more useful than a printed cross-reference in the standard document. Regarding the proposed additional characters, the response stated that characters which would have compatibility decompositions in Unicode should not be added and that logos, including those of political parties, and special characters for names of particular people should not be added.^[42]

In May 2000, the North Korean body submitted a proposal to the Ideographic Rapporteur Group (IRG), which operates under WG2, to include KPS 9566 and KPS 10721 mappings for characters in the Unified Repertoire and Ordering, CJK Unified Ideographs Extension A, Extension B and Extension C.^[43] This accompanied a table of mappings between KPS 10721 and CJK Extension C for 339 hanja,^[44] and a copy of the combined bibliography list of KPS 9566 and KPS 10721.^[29]

In July 2000, the North Korean body wrote to WG2, accusing them of developing both versions of the Unicode encoding for Korean on the basis of South Korean proposals only, without consulting North Korea, accusing them putting the commercial interests of companies and fears of international confusion over respect to North Korea's sovereignty, and stating that North Korea would regard further refusal to change the name and order of the Korean characters in Unicode as an insult to their sovereign dignity and as compromising the ISO's claims to impartiality. They re-iterated their demand for WG2 and Unicode to "correct" the order of the Korean characters, and to "correct" the names "Hangul Jamo" and "Hangul Syllable" to "Korean Alphabet" and "Korean Syllable".^[4]

In August 2000, the North Korean national body submitted a more detailed version of their requests in a series of five consecutive proposals. These requested the addition of 14 additional jamo characters,^[45] the addition of 82 symbol characters,^[46] and the use of the term "Korean alphabet" instead of "Hangul",^[47] provided supporting evidence for the North Korean collation order,^[21] and requested addition of the North Korean Hanja repertoire.^[48] These proposals were discussed in two meetings between North Korean, South Korean, Swedish and other WG2 representatives in September 2000, in which the North Korean body was asked to provide manuscript evidence for the additional jamo characters, to resubmit their symbols proposal with symbols which had already been accepted into Unicode removed, and to consider using ISO/IEC 14651, then at final draft stage, for collation purposes.^[49]

In September 2001, the North Korean national body submitted a revised series of proposals requesting the addition of several KPS 9566 and KPS 10721 characters, including 70 symbol characters, to Unicode.^[50][51] In this version of the proposal, a section of document excerpts demonstrating use of several characters and short explanations of their purpose was included. The Workers' Party of Korea symbol was named the "Hammer and Sickle and Brush",^[50] renamed from "Mark of the Workers' Party of Korea" in earlier versions of the proposal,^[46] and justified as being used as an identifying symbol on maps.^[50] As justification for the proposed characters for leaders' names, they explained that the leaders' names often appear with a different size and font weight in North Korean publications for the purpose of emphasis.^[50] A follow-up by South Korean WG2 representatives requested evidence, names in Korean and justifications for adding certain of these characters, and noted that non-emphasised versions of the characters for the leaders' names already existed.^[52]

A meeting of North and South Korean representatives from WG2 was convened in October 2001, which recommended 47 of the symbol characters for adding to Unicode, and suggested that the leaders' names and WPK symbols be raised for further discussion by WG2.^[53] Later that month, the North Korean body submitted a set of tables of mappings between Unicode and KPS 9566 and 10721, covering hanja only, to the IRG.^{[54][55][56][57]} In November 2001, the Unicode Consortium and the United States national body requested that the North Korean and South Korean representatives prepare an additional mapping table between KS X 1001, and KPS 9566 and 10721.^[58]

A subsequent feedback document from February 2002 regarding the North Korean proposed additions requested that the "tea" symbol for a tea house be accepted as a more general "hot beverage" symbol, equating it with symbols used in guidebooks to denote hot or non-alcoholic beverages. It also recommended that the reference glyph for the existing codepoint for an umbrella without rain be modified to harmonise with the proposed reference glyph for the umbrella with rain, equating them to the "keep dry" symbols used on packaging, and raised the question of which lightning bolt and high voltage warning symbols in existing symbol collections could be unified with the proposed "high voltage" character.^[59] All three of these characters were accepted into Unicode in version 4.0.^[60] It also recommended that the horizontal-barred fractions and the left-up pointing scissors be encoded using a variation selector, since the scissors did not accompany a differently-oriented pair of scissors, and since the existing Unicode fraction codepoints unified the skewed and horizontal forms.^[59]

In November 2002, the South Korean body published a complete set of three-way tables mapping characters between the KPS 9566, KS X 1001 (as EUC-KR) and ISO/IEC 10646 standards as they existed in 2000. These tables had been prepared without input from North Korea.^[61]

In August 2004, a pair of mapping tables between KPS 9566-2003 and Unicode were submitted to the OpenOffice.org project by an individual using the name "ooprojlover", who stated that they represented the updated version of the KPS 9566 standard and requested that support be added.^[22] These files mapped the characters unavailable in Unicode to the Private Use Area, and included additional encoded forms for other syllable blocks outside of the main ISO-IR-202 plane. A mapping table was later published by the Unicode Consortium in 2011, based on this mapping data but with errors corrected with reference to the ISO-IR chart.^[1]

Copies of Red Star OS 3.0 include fonts for a more recent edition of KPS 9566, appearing to be KPS 9566-2011. The mapping table used by Red Star OS internally has been successfully extracted. Besides adding Kim Jong Un to the list of leaders, KPS 9566-2011 amends the mappings of certain vertical forms compared to the 2003 mappings (taking advantage of the Vertical Forms block added in Unicode 4.1), and also includes several additional Hanja and symbols encoded outside of the ISO-IR-202 plane. Several of these additional symbols are also mapped to the Private Use Area; however, their identity is not known, since no names or reference glyphs for those characters are known outside of North Korea.^[3]

The 1997 edition of KPS 9566 was registered with the International Register of Coded Character Sets for Use with Escape Sequences as ISO-IR-202,^[23] and can therefore be encoded using ISO/IEC 2022. It is a 94ⁿ multiple-byte G-set, i.e. if it is used in a 7-bit ISO 2022 code (analogous to ISO-2022-JP or ISO-2022-KR), characters will be encoded with pairs of bytes between 0x21 and 0x7E when in the appropriate mode.

The documented mappings between KPS 9566 and Unicode for the 2003^[22][1] and 2011^[3] editions of KPS 9566 use an encoding resembling an adaptation of Unified Hangul Code (UHC) to encode KPS 9566 rather than Wansung code, with their updated versions of the ISO-IR-202 plane being encoded using pairs of bytes between 0xA1 and 0xFE, and with other two-byte codes used for syllables not present in ISO-IR-202. The order of the extended syllables follows usual KPS 9566 order. Similarly to UHC, they use lead bytes 0x81 and above, and trail bytes from the ranges 0x41–0x5A, 0x61–0x7A and 0x81–0xFE, excluding the range 0xA1–0xFE if the lead byte is 0xA1 or above.^[3]

The 2011 edition also includes several additional Hanja and symbols encoded outside of the ISO-IR-202 plane, after the range used for the extended syllable blocks.^[3] This approach is similar to that taken by GBK, but with the trail bytes remaining in the UHC-style ranges: like the extended syllables with lead bytes 0xA1 and above, these all use the trail byte ranges 0x41–0x5A, 0x61–0x7A and 0x81–0xA0. Extended Hanja are encoded with lead bytes between 0xC8 and 0xDC, extended symbols are encoded using lead bytes between 0xE0 and 0xEA, and extended codes with lead bytes between 0xEC and 0xFE are mapped, without gaps, to the Private Use Area^[3] (compare the user-defined ranges in GBK). Several of the characters in the extended symbols section and three in the Hanja section are also mapped to the Unicode Private Use Area; unlike the PUA-mapped symbols in the main ISO-IR-202 plane, the identity of these characters is unknown.^[3]

This chart details the overall layout of the main plane of the KPS 9566 character set by lead byte.^[23] For lead bytes used for characters other than composed Chosŏn'gŭl syllables or Hanja, links are provided to charts on this page listing the characters encoded under that lead byte. For lead bytes used for Hanja, links are provided to the appropriate section of Wiktionary's Hanja index.

Where two hexadecimal numbers are given, the value below 0x7F is used in a 7-bit encoding,^[a] and the larger value (between 0xA1 and 0xFE) is used in an 8-bit EUC-style encoding.^[17] The extended UHC-style 8-bit encodings defined by the 2003 edition onwards likewise use the larger byte values, between 0xA1 and 0xFE inclusive, for the main ISO-IR-202-based plane.^[1][3]

Precomposed Chosŏn'gŭl syllable clusters are allocated code points in a continuous sorted block between code points 16-01 and 44-47 inclusive. Not all possible clusters are allocated code points.^[82] Compare the different ordering and availability in KS X 1001.

The encoded form documented for KPS 9566-2003 encodes the KPS 9566 plane on GR (0xA1-0xFE) and additionally encodes the remaining syllable clusters using lead bytes in the range 0x80-0xC2 and trail bytes in the ranges 0x41-0x5A, 0x61-0x7A and 0x81-0xFE (where at most one byte is in the range 0xA1-0xFE),^[1] similarly to Unified Hangul Code but with the omitted clusters from and sorting order of KPS 9566, not KS X 1001.