1 .HTML "Hello World or Καλημέρα κόσμε or こんにちは 世界
18 rob,ken@plan9.bell-labs.com
21 Originally appeared, in a slightly different form, in
23 Proc. of the Winter 1993 USENIX Conf.,
28 Plan 9 from Bell Labs has recently been converted from ASCII
29 to an ASCII-compatible variant of the Unicode Standard, a 16-bit character set.
30 In this paper we explain the reasons for the change,
31 describe the character set and representation we chose,
32 and present the programming models and software changes
33 that support the new text format.
34 Although we stopped short of full internationalization\(emfor
35 example, system error messages are in Unixese, not Japanese\(emwe
36 believe Plan 9 is the first system to treat the representation
37 of all major languages on a uniform, equal footing throughout all its
43 The world is multilingual but most computer systems
44 are based on English and ASCII.
45 The first release of Plan 9 [Pike90], a new distributed operating
46 system from Bell Laboratories, seemed a good occasion
47 to correct this chauvinism.
48 It is easier to make such deep changes when building new systems than
49 by refitting old ones.
51 The ANSI C standard [ANSIC] contains some guidance on the matter of
52 `wide' and `multi-byte' characters but falls far short of
53 solving the myriad associated problems.
54 We could find no literature on how to convert a
56 to larger character sets, although some individual
59 This paper reports what we discovered as we
60 explored the problem of representing multilingual
61 text at all levels of an operating system,
62 from the file system and kernel through
63 the applications and up to the window system
66 Plan 9 has not been `internationalized':
67 its manuals are in English,
68 its error messages are in English,
69 and it can display text that goes from left to right only.
70 But before we can address these other problems,
71 we need to handle, uniformly and comfortably,
72 the textual representation of all the major written languages.
73 That subproblem is richer than we had anticipated.
77 Our first step was to select a standard.
78 At the time (January 1992),
79 there were only two viable options:
80 ISO 10646 [ISO10646] and Unicode [Unicode].
81 The documents describing both proposals were still in the draft stage.
83 The draft of ISO 10646 was not
84 very attractive to us.
85 It defined a sparse set of 32-bit characters,
88 and have punitive storage requirements.
89 Also, the draft attempted to
90 mollify national interests by allocating
91 16-bit subspaces to national committees
92 to partition individually.
93 The suggested mode of use was to
94 ``flip'' between separate national
95 standards to implement the international standard.
96 This did not strike us as a sound basis for a character set.
97 As well, transmitting 32-bit values in a byte stream,
98 such as in pipes, would be expensive and hard to implement.
99 Since the standard does not define a byte order for such
100 transmission, the byte stream would also have to carry
101 state to enable the values to be recovered.
103 The Unicode Standard is a proposal by a consortium of mostly American
104 computer companies formed
105 to protest the technical
106 failings of ISO 10646.
107 It defines a uniform 16-bit code based on the
108 principle of unification:
109 two characters are the same if they look the
110 same even though they are from different
112 This principle, called Han unification,
113 allows the large Japanese, Chinese, and Korean
114 character sets to be packed comfortably into a 16-bit representation.
116 We chose the Unicode Standard for its technical merits and because its
117 code space was better defined.
119 the Unicode Consortium was derailing the
122 ISO 10646 is a standard
123 with one 16-bit group defined,
124 which is almost exactly the Unicode Standard.
125 As most people expected, the two standards bodies
126 reached a détente and
127 ISO 10646 and Unicode represent the same character set.)
129 The Unicode Standard defines an adequate character set
130 but an unreasonable representation.
131 It states that all characters
132 are 16 bits wide and are communicated and stored in
134 It also reserves a pair of characters
135 (hexadecimal FFFE and FEFF) to detect byte order
136 in transmitted text, requiring state in the byte stream.
137 (The Unicode Consortium was thinking of files, not pipes.)
138 To adopt this encoding,
139 we would have had to convert all text going
140 into and out of Plan 9 between ASCII and Unicode, which cannot be done.
141 Within a single program, in command of all its input and output,
142 it is possible to define characters as 16-bit quantities;
143 in the context of a networked system with
144 hundreds of applications on diverse machines
145 by different manufacturers,
148 We needed a way to adapt the Unicode Standard to the tools-and-pipes
149 model of text processing embodied by the Unix system.
151 needed an ASCII-compatible textual
152 representation of Unicode characters for transmission
154 In the draft ISO standard there was an informative
158 that provided a byte stream encoding
159 of the 32-bit ISO code.
160 The encoding uses multibyte sequences composed
161 from the 190 printable characters of Latin-1
162 to represent character values larger
165 The UTF encoding has several good properties.
166 By far the most important is that
167 a byte in the ASCII range 0-127 represents
169 Thus UTF is backward compatible with ASCII.
171 UTF has other advantages.
172 It is a byte encoding and is
173 therefore byte-order independent.
174 ASCII control characters appear in the byte stream
175 only as themselves, never as an element of a sequence
176 encoding another character,
177 so newline bytes separate lines of UTF text.
180 function applied to UTF strings preserves the ordering of Unicode characters.
182 To encode and decode UTF is expensive (involving multiplication,
183 division, and modulo operations) but workable.
184 UTF's major disadvantage is that the encoding
185 is not self-synchronizing.
186 It is in general impossible to find the character
187 boundaries in a UTF string without reading from
188 the beginning of the string, although in practice
189 control characters such as newlines,
190 tabs, and blanks provide synchronization points.
193 X-Open circulated a proposal for another UTF-like
194 byte encoding of Unicode characters.
195 Their major concern was that an embedded character
197 (in particular a slash)
198 could be part of an escape sequence in UTF and
199 therefore confuse a traditional file system.
200 Their proposal would allow all 7-bit ASCII characters
201 to represent themselves
202 .I "and only themselves"
204 Multibyte sequences would contain only characters
205 with the high bit set.
206 We proposed a modification to the new UTF that
207 would address our synchronization problem.
208 Our proposal, which was originally known informally as UTF-2 and FSS-UTF,
209 is now referred to as UTF-8 and has been approved by ISO to become
210 Annex P to ISO 10646.
212 The model for text in Plan 9 is chosen from these
215 * ``That's the nice thing about standards\(emthere's so many to choose from.'' \- Andy Tannenbaum (no, the other one)
217 the Unicode character set encoded as a byte stream by
219 (soon to be) Annex P of ISO 10646.
220 Although this mixture may seem like a precarious position for us to adopt,
221 it is not as bad as it sounds.
222 ISO 10646 and the Unicode Standard have converged,
223 other systems such as Linux have adopted the same character set and encoding,
224 and the general feeling seems to be that Unicode and UTF-8 will be accepted as the way
225 to exchange text between systems.
226 The prognosis for wide acceptance is good.
228 There are a couple of aspects of the Unicode Standard we have not faced.
229 One is the issue of right-to-left text such as Hebrew or Arabic.
230 Since that is an issue of display, not representation, we believe
231 we can defer that problem for the moment without affecting our
232 ability to solve it later.
233 Another issue is diacriticals and `combining characters',
234 which cause overstriking of multiple Unicode characters.
235 Although necessary for some scripts, such as Thai, Arabic, and Hebrew,
236 such characters confuse the issues for Latin languages because they
237 generate multiple representations for accented characters.
238 ISO 10646 describes three levels of implementation;
239 in Plan 9 we decided not to address the issue.
240 Again, this can be labeled as a display issue and its finer points are still being debated,
241 so we felt comfortable deferring. Mañana.
243 Although we converted Plan 9 in the altruistic interests of
244 serving foreign languages, we have found the large character
245 set attractive for other reasons. The Unicode Standard includes many
246 characters\(emmathematical symbols, scientific notation,
247 more general punctuation, and more\(emthat we now use
248 daily in our work. We no longer test our imaginations
249 to find ways to include non-ASCII symbols in our text;
252 when you can use the character ☺?
253 Most compelling is the ability to absorb documents
254 and data that contain non-ASCII characters; our browser for the
255 Oxford English Dictionary
256 lets us see the dictionary as it really is, with pronunciation
257 in the IPA font, foreign phrases properly rendered, and so on,
260 In the rest of this paper, except when
261 stated otherwise, the term `UTF' refers to the UTF-8 encoding
262 of Unicode characters as adopted by Plan 9.
266 The first program to be converted to UTF
268 There are two levels of conversion.
269 On the syntactic level,
270 input to the C compiler
271 is UTF; on the semantic level,
272 the C language needs to define
273 how compiled programs manipulate
276 The syntactic part is simple.
277 The ANSI C language standard defines the
278 source character set to be ASCII.
279 Since UTF is backward compatible with ASCII,
280 the compiler needs little change.
281 The only places where a larger character set
282 is allowed are in character constants, strings, and comments.
283 Since 7-bit ASCII characters can represent only
285 the compiler does not have to be careful while looking
286 for the termination of a string or comment.
288 The Plan 9 compiler extends ANSI C to treat any Unicode
289 character with a value outside of the ASCII range as
291 To a Greek programmer or an English mathematician,
292 α is a sensible and now valid variable name.
294 On the semantic level, ANSI C allows,
295 but does not tie down,
298 and admits string and character constants
300 We chose the wide character type to be
303 In the libraries, the word
311 used to signify a Unicode character.
313 There are surprises; for example:
318 \&'ÿ' \f1is -1, stdio \fPEOF\f1 (if \fPchar\f1 is signed)\fP
319 L'\f1α\fP' \f1is 945\fP
320 \&'\f1α\fP' \f1is illegal\fP
322 In the string constants,
327 the former is an array of
331 while the latter is an array of
335 with 8 elements and a null
338 The Plan 9 library provides an output conversion function,
350 produces text, its output is always UTF.
351 The character conversion
353 (lower case) masks its argument
354 to 8 bits before converting to UTF.
364 will print as the Unicode
365 character with decimal value 177.
366 The character conversion
368 (upper case) masks its argument
369 to 16 bits before converting to UTF.
374 will print correctly under
384 and copies UTF sequences up to a null byte.
387 (upper case) expects a pointer to
392 conversions until a null
396 Another problem in format conversion
399 does the number refer to bytes or characters?
400 We decided that such formats were most
401 often used to align output columns and
402 so made the number count characters.
403 Some programs, however, use the count
404 to place blank-padded strings
405 in fixed-sized arrays.
406 These programs must be found and corrected.
408 Here is a complete example:
412 char c[] = "\f(Jpこんにちは 世界\fP";
413 Rune s[] = L"\f(Jpこんにちは 世界\fP";
417 print("%d, %d\en", sizeof(c), sizeof(s));
426 and then two identical lines of
432 are rare in programs; one reason is
433 that most formatted I/O is done in unconverted UTF.
437 All programs in Plan 9 now read and write text as UTF, not ASCII.
438 This change breaks two deep-rooted symmetries implicit in most C programs:
440 A character is no longer a
443 The internal representation (Rune) of a character now differs from its
444 external representation (UTF).
446 In the sections that follow,
447 we show how these issues were faced in the layers of
448 system software from the operating system up to the applications.
449 The effects are wide-reaching and often surprising.
453 Since UTF is the only format for text in Plan 9,
454 the interface to the operating system had to be converted to UTF.
455 Text strings cross the interface in several places:
458 user names (people can log in using their native name),
460 and miscellaneous minor places such as commands to the I/O system.
461 Little change was required: null-terminated UTF strings
462 are equivalent to null-terminated ASCII strings for most purposes
463 of the operating system.
464 The library routines described in the next section made that
465 change straightforward.
467 The window system, once called
469 is now rightfully called
474 A header file included by all programs (see [Pike92]) declares
477 type to hold 16-bit character values:
479 typedef unsigned short Rune;
481 Also defined are several constants relevant to UTF:
485 UTFmax = 3, /* maximum bytes per rune */
486 Runesync = 0x80, /* can't appear in UTF sequence (<) */
487 Runeself = 0x80, /* rune==UTF sequence (<) */
488 Runeerror = 0x80, /* decoding error in UTF */
491 (With the original UTF,
493 was hexadecimal 21 and
498 to hold the UTF encoding of any Unicode character.
499 Characters of value less than
501 only appear in a UTF string as
502 themselves, never as part of a sequence encoding another character.
503 Characters of value less than
505 encode into single bytes
507 Finally, when the library detects errors in UTF input\(embyte sequences
508 that are not valid UTF sequences\(emit converts the first byte of the
509 error sequence to the character
511 There is little a rune-oriented program can do when given bad data
512 except exit, which is unreasonable, or carry on.
513 Originally the conversion routines, described below,
514 returned errors when given invalid UTF,
515 but we found ourselves repeatedly checking for errors and ignoring them.
516 We therefore decided to convert a bad sequence to a valid rune
517 and continue processing.
518 (The ANSI C routines, on the other hand, return errors.)
520 This technique does have the unfortunate property that converting
521 invalid UTF byte strings in and out of runes does not preserve the input,
522 but this circumstance only occurs when non-textual input is
523 given to a textual program.
524 The Unicode Standard defines an error character, value FFFD, to stand for
525 characters from other sets that it does not represent.
528 character is a different concept, related to the encoding rather than the character set, so we
529 chose a different character for it.
531 The Plan 9 C library contains a number of routines for
533 The first set converts between runes and UTF strings:
535 extern int runetochar(char*, Rune*);
536 extern int chartorune(Rune*, char*);
537 extern int runelen(long);
538 extern int fullrune(char*, int);
543 to a UTF sequence and returns the number of bytes produced.
545 goes the other way, reporting how many bytes were consumed.
547 returns the number of bytes in the UTF encoding of a rune.
549 examines a UTF string up to a specified number of bytes
550 and reports whether the string begins with a complete UTF encoding.
551 All these routines use the
553 character to work around encoding problems.
555 There is also a set of routines for examining null-terminated UTF strings,
556 based on the model of the ANSI standard
567 extern int utflen(char*);
568 extern char* utfrune(char*, long);
569 extern char* utfrrune(char*, long);
570 extern char* utfutf(char*, char*);
573 returns the number of runes in a UTF string;
575 returns a pointer to the first occurrence of a rune in a UTF string;
578 a pointer to the last.
580 searches for the first occurrence of a UTF string in another UTF string.
581 Given the synchronizing property of UTF-8,
585 if the arguments point to valid UTF strings.
587 It is a mistake to use
591 unless searching for a 7-bit ASCII character, that is, a character
595 We have no routines for manipulating null-terminated arrays of
597 Although they should probably exist for completeness, we have
598 found no need for them, for the same reason that
604 Most Plan 9 programs use a new buffered I/O library, BIO, in place of
606 BIO contains routines to read and write UTF streams, converting to and from
613 the next character in the UTF input stream;
615 takes a rune and writes its UTF representation.
617 puts a rune back into the input stream for rereading.
619 Plan 9 programs use a simple set of macros to process command line arguments.
620 Converting these macros to UTF automatically updated the
621 argument processing of most programs.
623 argument flag names can no longer be held in bytes and
624 arrays of 256 bytes cannot be used to hold a set of flags.
626 We have done nothing analogous to ANSI C's locales, partly because
627 we do not feel qualified to define locales and partly because we remain
628 unconvinced of that model for dealing with the problems.
629 That is really more an issue of internationalization than conversion
630 to a larger character set; on the other hand,
631 because we have chosen a single character set that encompasses
632 most languages, some of the need for
633 locales is eliminated.
636 that translates between UTF and other character sets.)
638 There are several reasons why our library does not follow the ANSI design
639 for wide and multi-byte characters.
640 The ANSI model was designed by a committee, untried, almost
641 as an afterthought, whereas
642 we wanted to design as we built.
643 (We made several major changes to the interface
644 as we became familiar with the problems involved.)
645 We disagree with ANSI C's handling of invalid multi-byte sequences.
646 Also, the ANSI C library is incomplete:
647 although it contains some crucial routines for handling
648 wide and multi-byte characters, there are some serious omissions.
649 For example, our software can exploit
650 the fact that UTF preserves ASCII characters in the byte stream.
651 We could remove that assumption by replacing all
657 (Because of the weaker properties of the original UTF,
658 we have actually done so.)
660 the standard says nothing about the representation, so portable code should
664 yet there is no ANSI equivalent to
666 ANSI C simultaneously invalidates
668 and offers no replacement.
670 Finally, ANSI did nothing to integrate wide characters
671 into the I/O system: it gives no method for printing
673 We therefore needed to invent some things and decided to invent
675 In the end, some of our entry points do correspond closely to
676 ANSI routines\(emfor example
684 Plan 9's library defines more functionality, enough
685 to write real applications comfortably.
689 The source for our tools and applications had already been converted to
690 work with Latin-1, so it was `8-bit safe', but the conversion to the Unicode
691 Standard and UTF is more involved.
692 Some programs needed no change at all:
695 interprets its argument strings, delivered in UTF,
696 as file names that it passes uninterpreted to the
699 and then just copies bytes from its input to its output;
700 it never makes decisions based on the values of the bytes.
703 has no options such as
705 to complicate matters.)
706 Most programs, however, needed modest change.
709 find automatically the places that need attention,
713 Software that uses the libraries conscientiously can be searched
714 for calls to library routines that examine bytes as characters:
719 Replacing these by calls to
724 is enough to fix many programs.
725 Few tools actually need to operate on runes internally;
726 more typically they need only to look for the final slash in a file
727 name and similar trivial tasks.
728 Of the 170 C source programs in the top levels of
730 only 23 now contain the word
735 store runes internally
736 are mostly those whose
739 is character manipulation:
747 (the window system and terminal emulator),
749 To decide whether to compute using runes
750 or UTF-encoded byte strings requires balancing the cost of converting
751 the data when read and written
752 against the cost of converting relevant text on demand.
753 For programs such as editors that run a long time with a relatively
754 constant dataset, runes are the better choice.
755 There are space considerations too, but they are more complicated:
756 plain ASCII text grows when converted to runes; UTF-encoded Japanese
759 Again, it is hard to automate the conversion of a program from
763 It is not enough just to change the type of variables; the assumption
764 that bytes and characters are equivalent can be insidious.
765 For instance, to clear a character array by
767 memset(buf, 0, BUFSIZE)
771 is changed from an array of
775 Any program that indexes tables based on character values needs
779 which originally used multiple 256-byte arrays for the mapping.
780 The naïve conversion would yield multiple 65536-rune arrays.
783 saves space by building in effect
784 a run-encoded version of the map.
787 has related problems.
788 The cooperation of UTF and
790 means that a simple sort\(emone with no options\(emcan be done
791 on the original UTF strings using
793 With sorting options enabled, however,
795 may need to convert its input to runes: for example,
798 requires searching for alphas in the input text to
799 crack the input into fields.
802 refers to 2 runes beyond the third field.
803 Some of the other options are hopelessly provincial:
804 consider the case-folding and dictionary order options
805 (Japanese doesn't even have an official dictionary order) or
807 which compares by case-insensitive English month name.
808 Handling these options involves the
809 larger issues of internationalization and is beyond the scope
810 of this paper and our expertise.
813 works sensibly with options that make sense relative to the input.
814 The simple and most important options are, however, usually meaningful.
817 sorts UTF into the same order that
821 Regular expression-matching algorithms need rethinking to
822 be applied to UTF text.
823 Deterministic automata are usually applied to bytes;
824 converting them to operate on variable-sized byte sequences is awkward.
825 On the other hand, converting the input stream to runes adds measurable
827 and the state tables expand
828 from size 256 to 65536; it can be expensive just to generate them.
829 For simple string searching,
830 the Boyer-Moore algorithm works with UTF provided the input is
831 guaranteed to be only valid UTF strings; however, it does not work
832 with the old UTF encoding.
833 At a more mundane level, even character classes are harder:
834 the usual bit-vector representation within a non-deterministic automaton
835 is unwieldy with 65536 characters in the alphabet.
838 An existing library for compiling and executing regular expressions
839 was adapted to work on runes, with two entry points for searching
840 in arrays of runes and arrays of chars (the pattern is always UTF text).
841 Character classes are represented internally as runs of runes;
844 marks the end of the class.
847 utilities that use regular expressions\(emeditors,
850 etc.\(emexcept the shell, whose notation
851 was grandfathered, were converted to use the library.
852 For some programs, there was a concomitant loss of performance,
853 but there was also a strong advantage.
854 To our knowledge, Plan 9 is the only Unix-like system
855 that has a single definition and implementation of
856 regular expressions; patterns are written and interpreted
857 identically by all the programs in the system.
859 A handful of programs have the notion of character built into them
860 so strongly as to confuse the issue of what they should do with UTF input.
861 Such programs were treated as individual special cases.
864 is, by default, unchanged in behavior and output; a new option,
866 counts the number of correctly encoded runes\(emvalid UTF sequences\(emin
869 the number of invalid sequences.
871 It took us several months to convert all the software in the system
872 to the Unicode Standard and the old UTF.
873 When we decided to convert from that to the new UTF,
874 only three things needed to be done.
875 First, we rewrote the library routines to encode and decode the
876 new UTF. This took an evening.
877 Next, we converted all the files containing UTF
879 We wrote a trivial program to look for non-ASCII bytes in
880 text files and used a Plan 9 program called
882 (translate character set) to change encodings.
883 Finally, we recompiled all the system software;
884 the library interface was unchanged, so recompilation was sufficient
885 to effect the transformation.
886 The second two steps were done concurrently and took an afternoon.
887 We concluded that the actual encoding is relatively unimportant to the
888 software; the adoption of large characters and a byte-stream encoding
891 are much deeper issues.
895 Plan 9 provides only minimal support for plain text terminals.
896 It is instead designed to be used with all character input and
897 output mediated by a window system such as
899 The window system and related software are responsible for the
900 display of UTF text as Unicode character images.
901 For plain text, the window system must provide a user-settable
903 that provides a (possibly empty) picture for each Unicode character.
904 Fancier applications that use bold and Italic characters
905 need multiple fonts storing multiple pictures for each
907 All the issues are apparent, though,
908 in just the problem of
909 displaying a single image for each character, that is, the
910 Unicode equivalent of a plain text terminal.
911 With 128 or even 256 characters, a font can be just
912 an array of bitmaps. With 65536 characters,
913 a more sophisticated design is necessary. To store the ideographs
914 for just Japanese as 16×16×1 bit images,
915 the smallest they can reasonably be, takes over a quarter of a
916 megabyte. Make the images a little larger, store more bits per
917 pixel, and hold a copy in every running application, and the
918 memory cost becomes unreasonable.
920 The structure of the bitmap graphics services is described at length elsewhere
922 In summary, the memory holding the bitmaps is stored in the same machine that has
923 the display, mouse, and keyboard: the terminal in Plan 9 terminology,
924 the workstation in others'.
925 Access to that memory and associated services is provided
926 by device files served by system
927 software on the terminal. One of those files,
929 interprets messages written upon it as requests for actions
930 corresponding to entry points in the graphics library:
931 allocate a bitmap, execute a raster operation, draw a text string, etc.
933 acts as a multiplexer that mediates access to the services
934 and resources of the terminal by simulating in each client window
935 a set of files mirroring those provided by the system.
936 That is, each window has a distinct
939 and so on through which applications drive graphical
942 One of the resources managed by
944 and the terminal is the set of active
946 Each subfont holds the
947 bitmaps and associated data structures for a sequential set of Unicode
949 Subfonts are stored in files and loaded into the terminal by
952 For example, one subfont
953 might hold the images of the first 256 characters of the Unicode space,
954 corresponding to the Latin-1 character set;
955 another might hold the standard phonetic character set, Unicode characters
956 with value 0250 to 02E9.
957 These files are collected in directories corresponding to typefaces:
958 .CW /lib/font/bit/pelm
959 contains the Pellucida Monospace character set, with subfonts holding
960 the Latin-1, Greek, Cyrillic and other components of the typeface.
961 A suffix on subfont files encodes (in a subfont-specific
962 way) the size of the images:
963 .CW /lib/font/bit/pelm/latin1.9
964 contains the Latin-1 Pellucida Monospace characters with lower
965 case letters 9 pixels high;
966 .CW /lib/font/bit/jis/jis5400.16
967 contains 16-pixel high
968 ideographs starting at Unicode value 5400.
970 The subfonts do not identify which portion of the Unicode space
971 they cover. Instead, a
972 font file, in plain text,
973 describes how to assemble subfonts into a complete
975 The font file is presented as an argument to the window system
976 to determine how plain text is displayed in text windows and
978 Here is the beginning of the font file
979 .CW /lib/font/bit/pelm/jis.9.font ,
980 which describes the layout of a font covering that portion of
981 the Unicode Standard for which we have characters of typical
982 display size, using Japanese characters
983 to cover the Han space:
986 0x0000 0x00FF latin1.9
987 0x0100 0x017E latineur.9
989 0x0386 0x03F5 greek.9
990 0x0400 0x0475 cyrillic.9
991 0x2000 0x2044 ../misc/genpunc.9
992 0x2070 0x208E supsub.9
993 0x20A0 0x20AA currency.9
994 0x2100 0x2138 ../misc/letterlike.9
995 0x2190 0x21EA ../misc/arrows
996 0x2200 0x227F ../misc/math1
997 0x2280 0x22F1 ../misc/math2
998 0x2300 0x232C ../misc/tech
999 0x2500 0x257F ../misc/chart
1000 0x2600 0x266F ../misc/ding
1003 0x3000 0x303f ../jis/jis3000.16
1004 0x30a1 0x30fe ../jis/katakana.16
1005 0x3041 0x309e ../jis/hiragana.16
1006 0x4e00 0x4fff ../jis/jis4e00.16
1007 0x5000 0x51ff ../jis/jis5000.16
1010 The first two numbers set the interline spacing of the font (18
1011 pixels) and the distance from the baseline to the top of the
1013 When characters are displayed, they are placed so as best
1014 to fit within those constraints; characters
1015 too large to fit will be truncated.
1016 The rest of the file associates subfont files
1017 with portions of Unicode space.
1018 The first four such files are in the Pellucida Monospace typeface
1019 and directory; others reside in other directories. The file names
1020 are relative to the font file's own location.
1022 There are several advantages to this two-level structure.
1023 First, it simultaneously breaks the huge Unicode space into manageable
1024 components and provides a unifying architecture for
1025 assembling fonts from disjoint pieces.
1026 Second, the structure promotes sharing.
1027 For example, we have only one set of Japanese
1028 characters but dozens of typefaces for the Latin-1 characters,
1029 and this structure permits us to store only one copy of the
1030 Japanese set but use it with any Roman typeface.
1031 Also, customization is easy.
1032 English-speaking users who don't need Japanese characters
1033 but may want to read an on-line Oxford English Dictionary can
1034 assemble a custom font with the
1035 Latin-1 (or even just ASCII) characters and the International
1036 Phonetic Alphabet (IPA).
1037 Moreover, to do so requires just editing a plain text file,
1038 not using a special font editing tool.
1039 Finally, the structure guides the design of
1040 caching protocols to improve performance and memory usage.
1042 To load a complete Unicode character set into each application
1044 much memory and, particularly on slow terminal lines, would take
1046 Instead, Plan 9 assembles a multi-level cache structure for
1048 An application opens a font file, reads and parses it,
1049 and allocates a data structure.
1050 A message written to
1052 allocates an associated structure held in the terminal, in particular,
1053 a bitmap to act as a cache
1054 for recently used character images.
1055 Other messages copy these images to bitmaps such as the screen
1056 by loading characters from subfonts into the cache on demand and
1057 from there to the destination bitmap.
1058 The protocol to draw characters is in terms of cache indices,
1059 not Unicode character number or UTF sequences.
1060 These details are hidden from the application, which instead
1061 sees only a subroutine to draw a string in a bitmap from a
1062 given font, functions to discover character size information,
1063 and routines to allocate and to free fonts.
1066 subfonts are opened by the graphics library, read, and then downloaded
1068 They are held open by the library in an LRU-replacement list.
1069 Even when the program closes a subfont, it is retained
1070 in the terminal for later use.
1071 When the application opens the subfont, it asks the terminal
1072 if it already has a copy to avoid reading it from the file
1074 This level of cache has the property that the bitmaps for, say,
1075 all the Japanese characters are stored only once, in the terminal;
1076 the applications read only size and width information from the terminal
1077 and share the images.
1079 The sizes of the character and subfont caches held by the
1080 application are adaptive.
1081 A simple algorithm monitors the cache miss rate to enlarge and
1082 shrink the caches as required.
1083 The size of the character cache is limited to 2048 images maximum,
1084 which in practice seems enough even for Japanese text.
1085 For plain ASCII-like text it naturally stays around 128 images.
1087 This mechanism sounds complicated but is implemented by only about
1088 500 lines in the library and considerably less in each of the
1089 terminal's graphics driver and
1091 It has the advantage that only characters that are
1092 being used are loaded into memory.
1093 It is also efficient: if the characters being drawn
1094 are in the cache the extra overhead is negligible.
1095 It works particularly well for alphabetic character sets,
1096 but also adapts on demand for ideographic sets.
1097 When a user first looks at Japanese text, it takes a few
1098 seconds to read all the font data, but thereafter the
1099 text is drawn almost as fast as regular text (the images
1100 are larger, so draw a little slower).
1101 Also, because the bitmaps are remembered by the terminal,
1102 if a second application then looks at Japanese text
1103 it starts faster than the first.
1106 building a `font server'
1107 to cache character images and associated data
1108 for the applications, the window system, and the terminal.
1109 We rejected this design because, although isolating
1110 many of the problems of font management into a separate program,
1111 it didn't simplify the applications.
1112 Moreover, in a distributed system such as Plan 9 it is easy
1113 to have too many special purpose servers.
1114 Making the management of the fonts the concern of only
1115 the essential components simplifies the system and makes
1116 bootstrapping less intricate.
1120 A completely different problem is how to type Unicode characters
1121 as input to the system.
1122 We selected an unused key on our ASCII keyboards
1123 to serve as a prefix for multi-keystroke
1124 sequences that generate Unicode characters.
1125 For example, the character
1127 is generated by the prefix key
1132 followed by a double quote and a lower-case
1134 When that character is read by the application, from the file
1136 it is of course presented as its UTF encoding.
1137 Such sequences generate characters from an arbitrary set that
1138 includes all of Latin-1 plus a selection of mathematical
1139 and technical characters.
1140 An arbitrary Unicode character may be generated by typing the prefix,
1141 an upper case X, and four hexadecimal digits that identify
1144 These simple mechanisms are adequate for most of our day-to-day needs:
1145 it's easy to remember to type `ALT 1 2' for ½\^ or `ALT accent letter'
1146 for accented Latin letters.
1147 For the occasional unusual character, the cut and paste features of
1149 serve well. A program called (perhaps misleadingly)
1151 takes as argument a hexadecimal value, and prints the UTF representation of that character,
1152 which may then be picked up with the mouse and used as input.
1155 are clearly unsatisfactory when working in a non-English language.
1156 In the native country of such a language
1157 the appropriate keyboard is likely to be at hand.
1158 But it's also reasonable\(emespecially now that the system handles Unicode characters\(emto
1159 work in a language foreign to the keyboard.
1161 For alphabetic languages such as Greek or Russian, it is
1162 straightforward to construct a program that does phonetic substitution,
1163 so that, for example, typing a Latin `a' yields the Greek `α'.
1164 Within Plan 9, such a program can be inserted transparently
1165 between the real keyboard and a program such as the window system,
1166 providing a manageable input device for such languages.
1168 For ideographic languages such as Chinese or Japanese the problem is harder.
1169 Native users of such languages have adopted methods for dealing with
1170 Latin keyboards that involve a hybrid technique based on phonetics
1171 to generate a list of possible symbols followed by menu selection to
1172 choose the desired one.
1174 effective, but their design must be rooted in information about
1175 the language unknown to non-native speakers.
1177 a Chinese terminal emulator built by and for
1178 Chinese programmers,
1179 employs such a technique
1181 Although the technical problem of implementing such a device
1182 is easy in Plan 9\(emit is just an elaboration of the technique for
1183 alphabetic languages\(emour lack of familiarity with such languages
1184 has restrained our enthusiasm for building one.
1186 The input problem is technically the least interesting but perhaps
1187 emotionally the most important of the problems of converting a system
1188 to an international character set.
1189 Beyond that remain the deeper problems of internationalization
1190 such as multi-lingual error messages and command names,
1191 problems we are not qualified to solve.
1192 With the ability to treat text of most languages on an equal
1193 footing, though, we can begin down that path.
1194 Perhaps people in non-English speaking countries will
1195 consider adopting Plan 9, solving the input problem locally\(emperhaps
1196 just by plugging in their local terminals\(emand begin to use
1197 a system with at least the capacity to be international.
1201 Dennis Ritchie provided consultation and encouragement.
1202 Bob Flandrena converted most of the standard tools to UTF.
1203 Brian Kernighan suffered cheerfully with several
1204 inadequate implementations and converted
1207 Rich Drechsler converted his Postscript driver to UTF.
1208 John Hobby built the Postscript ☺.
1213 [ANSIC] \f2American National Standard for Information Systems \-
1214 Programming Language C\f1, American National Standards Institute, Inc.,
1218 ISO/IEC DIS 10646-1:1993
1219 \f2Information technology \-
1220 Universal Multiple-Octet Coded Character Set (UCS) \(em
1221 Part 1: Architecture and Basic Multilingual Plane\fP.
1223 [Pike90] R. Pike, D. Presotto, K. Thompson, H. Trickey,
1224 ``Plan 9 from Bell Labs'',
1225 UKUUG Proc. of the Summer 1990 Conf.,
1229 [Pike91] R. Pike, ``8½, The Plan 9 Window System'', USENIX Summer
1230 Conf. Proc., Nashville, 1991, reprinted in this volume.
1232 [Pike92] R. Pike, ``How to Use the Plan 9 C Compiler'', this volume.
1234 [Pong and Zhang] Man-Chi Pong and Yongguang Zhang, ``cxterm:
1235 A Chinese Terminal Emulator for the X Window System'',
1237 Software\(emPractice and Experience,
1239 Vol 22(1), 809-926, October 1992.
1242 \f2The Unicode Standard,
1243 Worldwide Character Encoding,
1244 Version 1.0, Volume 1\f1,
1245 The Unicode Consortium,