RFC 2640
Internationalization of the File Transfer Protocol
Internationalization of the File Transfer Protocol 🎀
Okay bestie, here's the vibe check on this one ✨: it's RFC 2640, written by B. Curtin at the Defense Information Systems Agency in July 1999 📅. Its category is Proposed Standard 💜. It updates RFC 959, and it doesn't list anything it obsoletes 🫶. It's from the Network Working Group, so the Internet crew helped build it 🌐.
Status of this Memo ✨
Okay so this doc is an Internet standards track protocol 🦋 for the whole Internet community, and it's literally asking for discussion and suggestions for improvements 🫶. For where this protocol stands, check the current edition of the "Internet Official Protocol Standards" (STD 1) 👑. Distribution of this memo is unlimited 💖.
Copyright Notice 💜
Copyright (C) The Internet Society (1999). All Rights Reserved.Abstract 🌸
Okay bestie, FTP, as defined in RFC 959 [RFC959] and RFC 1123 Section 4 [RFC1123], is one of the oldest and most widely used protocols on the Internet 🕰️. Its main character set, 7 bit ASCII, carried it through the early growth years of the Internet like a champ 💅. But the Internet is getting more global, so we need character sets beyond 7 bit ASCII 🌍.
This doc is all about the internationalization (I18n) of FTP 🌈. That means supporting the many character sets and languages used across the Internet community ✨. It's achieved by extending the FTP specification and giving recommendations for proper internationalization support 🎀.
Table of Contents 📋
ABSTRACT.......................................................1
1 INTRODUCTION.................................................2
1.1 Requirements Terminology..................................2
2 INTERNATIONALIZATION.........................................3
2.1 International Character Set...............................3
2.2 Transfer Encoding Set.....................................4
3 PATHNAMES....................................................5
3.1 General compliance........................................5
3.2 Servers compliance........................................6
3.3 Clients compliance........................................7
4 LANGUAGE SUPPORT.............................................7
4.1 The LANG command..........................................8
4.2 Syntax of the LANG command................................9
4.3 Feat response for LANG command...........................11
4.3.1 Feat examples.........................................11
5 SECURITY CONSIDERATIONS.....................................12
6 ACKNOWLEDGMENTS.............................................12
7 GLOSSARY....................................................13
8 BIBLIOGRAPHY................................................13
9 AUTHOR'S ADDRESS............................................15
ANNEX A - IMPLEMENTATION CONSIDERATIONS.......................16
A.1 General Considerations...................................16
A.2 Transition Considerations................................18
ANNEX B - SAMPLE CODE AND EXAMPLES............................19
B.1 Valid UTF-8 check........................................19
B.2 Conversions..............................................20
B.2.1 Conversion from Local Character Set to UTF-8..........20
B.2.2 Conversion from UTF-8 to Local Character Set..........23
B.2.3 ISO/IEC 8859-8 Example................................25
B.2.4 Vendor Codepage Example...............................25
B.3 Pseudo Code for Translating Servers......................26
Full Copyright Statement......................................271 Introduction ✨
As the Internet spreads around the world, supporting character sets beyond the ASCII [ASCII] / Latin-1 [ISO-8859] set gets more and more urgent 🌏. For FTP there's a huge installed base, so it's absolutely paramount that this happens without breaking existing clients and servers 😭. This document tackles that by defining a solution that still lets the installed base interoperate with new clients and servers 🫶.
This doc levels up FTP by removing the 7-bit restrictions on pathnames used in client commands and server responses 🔥. It RECOMMENDs the use of a Universal Character Set (UCS) ISO/IEC 10646 [ISO-10646], RECOMMENDs a UCS transformation format (UTF) UTF-8 [UTF-8], and defines a new command for language negotiation 💖.
All of this lines up with the IETF policy on character sets and languages in RFC 2277 [RFC2277] ✨.
1.1. Requirements Terminology 🎀
Here's the key-word tea 🍓: the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [BCP14].
2 Internationalization 🌍
Back when FTP was made, the big character sets were 7 bit ASCII and 8 bit EBCDIC 🕰️. Those just can't cover the huge range of characters that multinational systems need 😩. There are plenty of character sets in use with more characters than 7-bit ASCII, so it makes sense to pick a neat way to represent the union of all those possibilities 🧚. To work globally you either support a pile of character sets and convert between them, or you use one preferred character set 🌈. For global interoperability this doc RECOMMENDs the latter approach and defines a single character set, in addition to NVT ASCII and EBCDIC, that every system can understand 💜. For FTP this character set SHALL be ISO/IEC 10646:1993. For global compatibility it is STRONGLY RECOMMENDED that clients and servers use UTF-8 encoding when exchanging pathnames 🫶. Clients and servers are, however, under no obligation to convert the contents of a file for operations like STOR or RETR 💅.
The character set used to store files SHALL remain a local decision and MAY depend on what the local operating systems can do 🖥️. Before pathnames get exchanged they SHOULD be converted into an ISO/IEC 10646 format and UTF-8 encoded ✨. This approach lets pathnames be exchanged internationally while staying backward compatible with older systems 🦋, because the code set positions for ASCII characters are identical to the one byte sequence in UTF-8 💖.
Sections 2.1 and 2.2 give a quick rundown of the international character set and transfer encoding RECOMMENDED by this document 🎀. For the full deep dive on UTF-8, ISO/IEC 10646, and UNICODE [UNICODE], check RFC 2279 [RFC2279] 🤓.
2.1 International Character Set 🌏
The character set for international FTP support SHALL be the Universal Character Set as defined in ISO 10646:1993 as amended 👑. It pulls in the character sets from tons of existing international, national, and corporate standards 💅. ISO/IEC 10646 defines two alternate encodings, UCS-4 and UCS-2 ✨.
UCS-4 is a four byte (31 bit) encoding with 2**31 code positions, divided into 128 groups of 256 planes. Each plane has 256 rows of 256 cells 🧩. UCS-2 is a 2 byte (16 bit) character set made of plane zero, the Basic Multilingual Plane (BMP) 🫧. Right now, no codesets have been defined outside the 2 byte BMP 🙈.
The Unicode standard version 2.0 [UNICODE] is consistent with the UCS-2 subset of ISO/IEC 10646 🌈. It includes the IS 10646 repertoire, amendments 1-7 of IS 10646, and the editorial and technical corrigenda 🦋.
2.2 Transfer Encoding 💌
UCS Transformation Format 8 (UTF-8), once called UTF-2 or UTF-FSS, SHALL be used as the transfer encoding to transmit the international character set 🫶. It's a file safe encoding that avoids byte values with special meaning while parsing pathname strings 🛡️. UTF-8 is an 8 bit encoding of the UCS characters 💜. Its perks are that it's compatible with 7 bit ASCII, so programs that give special meanings to ASCII characters aren't affected 🙌. It's immune to synchronization errors 🔐, its encoding rules make identification easy 🔍, and it has enough room for a large number of character sets 🌟.
UTF-8 represents each UCS character as a sequence of 1 to 6 bytes ✨. For sequences of one byte the most significant bit is ZERO. For sequences of more than one byte, count the ONE bits at the start of the first byte, from the most significant bit position. That count is the number of bytes in the UTF-8 sequence, and it's followed by a ZERO bit. For example, the first byte of a 3 byte UTF-8 sequence has 1110 as its most significant bits 🧮. Each continuing byte has a ONE bit followed by a ZERO bit as its most significant bits, and the remaining free bit positions identify the UCS characters 💖. Here's how the UCS and UTF-8 relationship breaks down 👇:
UCS-4 range(hex) UTF-8 byte sequence(binary)
00000000 - 0000007F 0xxxxxxx
00000080 - 000007FF 110xxxxx 10xxxxxx
00000800 - 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
00010000 - 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
00200000 - 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx
10xxxxxx
04000000 - 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx
10xxxxxx 10xxxxxxA fave thing about UTF-8 is that its single byte sequence is consistent with the ASCII character set 🥰. That's what lets old ASCII-only clients keep interoperating with new servers that support UTF-8 encoding during the transition 🦋.
Another perk is that the encoding rules make it super unlikely that a character sequence from a different character set gets mistaken for UTF-8 🕵️. Clients and servers can use a simple routine to determine whether the character set being exchanged is valid UTF-8. Section B.1 shows a code example of this check 💻.
3 Pathnames 🗂️
3.1 General compliance ✨
- The 7-bit restriction for pathnames exchanged is dropped. 🙅♀️
Okay so, a lot of operating systems let you use spaces <SP>, carriage returns <CR>, and line feeds <LF> inside a pathname 😭 The problem is that FTP commands with pathnames look like this, so those special characters can get parsed wrong:
COMMAND <SP> <pathname> <CRLF>To allow pathnames with these characters, the pathname definition changes from this:
<pathname> ::= <string> ; in BNF formatto this:
pathname = 1*(%x01..%xFF) ; in ABNF format [ABNF].To keep these characters from being mistaken for command syntax, the following rules apply. 🧠
There MUST be only one <SP> between an FTP command and the pathname. Implementations MUST treat any <SP> characters after that first one as part of the pathname. For example, the pathname in STOR <SP><SP><SP>foo.bar<CRLF> is <SP><SP>foo.bar. 🎀
Current implementations might allow several <SP> characters between the command and the pathname. Those MUST make sure they follow this single <SP> convention. Note: implementations that treat 3-character commands (for example CWD, MKD, and so on) as a fixed 4-character command by padding the command with a trailing <SP> do NOT comply with this specification. 🙈
When a <CR> shows up as part of a pathname, it MUST be padded with a <NUL> character before the command is sent. When a pathname containing a <CR><NUL> sequence is received, the <NUL> MUST be stripped away. This approach is described in the Telnet protocol [RFC854] on pages 11 and 12. For example, to store a pathname foo<CR><LF>boo.bar, the pathname becomes foo<CR><NUL><LF>boo.bar before the command STOR <SP>foo<CR><NUL><LF>boo.bar<CRLF> is sent. When the altered pathname is received, the <NUL> right after the <CR> gets stripped away, and you're back to the original pathname. 🦋
- Conforming clients and servers MUST support UTF-8 for the transfer and receipt of pathnames. Clients and servers MAY also give users a choice of specifying how pathnames are interpreted in another encoding. Note that configuring clients and servers to use character sets or encodings other than UTF-8 is outside the scope of this document. The document admits that this might be desirable in certain operational scenarios, but it is left as a quality of implementation and operational issue. 🌈
- Pathnames are sequences of bytes. The encoding of names that are valid UTF-8 sequences is assumed to be UTF-8. The character set of other names is undefined. Unless they've been configured to support a specific native character set, clients and servers MUST check for a valid UTF-8 byte sequence to determine whether the pathname being presented is UTF-8. 🔍
- To avoid data loss, clients and servers SHOULD use UTF-8 encoded pathnames when they can't convert them to a usable code set. 💾
- Sometimes the code set or encoding presented to the server or client can't be determined. In those cases, the raw bytes SHOULD be used. 🫶
3.2 Servers compliance 👑
- Servers MUST support the UTF-8 feature in response to the FEAT command [RFC2389]. The UTF-8 feature is a line containing the exact string
"UTF8". That string is not case sensitive, but it SHOULD be transmitted in upper case. The response to a FEAT command SHOULD be:
C> feat
S> 211- <any descriptive text>
S> ...
S> UTF8
S> ...
S> 211 endThe ellipses are placeholders where other features may be included, but they are NOT REQUIRED. The one-space indentation of the feature lines is mandatory [RFC2389]. 🌸
- Mirror servers may want to reflect the site they're mirroring exactly. In those cases, servers MAY store and present the exact pathname bytes they received from the main server. 🪞
3.3 Clients compliance 💖
- Clients that don't need to display pathnames have no obligation to do so. Non-display clients don't need to conform to the requirements associated with display. 😌
- Clients that are presented with UTF-8 pathnames by the server SHOULD parse UTF-8 correctly and try to display the pathname within the limits of the resources available. ✨
- Clients MUST support the FEAT command and recognize the
"UTF8"feature (defined in 3.2 above) to figure out whether a server supports UTF-8 encoding. 🔎 - The character semantics of other names remain undefined. If a client detects that a server isn't UTF-8, it SHOULD change its display appropriately. How a client implementation handles non-UTF-8 is a quality of implementation issue. It MAY try to assume some other encoding, give the user a chance to pick one, or save encoding assumptions for a server from one FTP session to another. 🧩
- Glyph rendering is outside the scope of this document. How a client presents characters it cannot display is a quality of implementation issue. This document RECOMMENDS that octets corresponding to non-displayable characters SHOULD be presented in URL
%HHformat, as defined in RFC 1738 [RFC1738]. They MAY, however, be displayed as question marks, with their UCS hexadecimal value, or in any other suitable fashion. 🔤 - Many existing clients interpret 8-bit pathnames as being in the local character set. They MAY continue to do so for pathnames that are not valid UTF-8. 🕰️
4. Language Support 🌍
The Character Set Workshop Report [RFC2130] suggests that clients and servers SHOULD negotiate a language for "greetings" and "error messages". This specification reads the term "error message", as used by RFC 2130, to mean any explanatory text string returned by server-PI in response to a user-PI command. 💬
Implementers SHOULD note that FTP commands and numeric responses are protocol elements. As such, their use is not affected by any guidance expressed by this specification. 🤓
Language support for greetings and command responses shall be the default language supported by the server, or the language supported by the server and selected by the client. 🗣️
It may be possible to achieve language support through a virtual host, as described in [MLST]. However, an FTP server might not support virtual servers, or virtual servers might be configured for an environment that doesn't care about language. To allow language negotiation, this specification defines a new LANG command. Clients and servers that comply with this specification MUST support the LANG command. 🫧
4.1 The LANG command 🏳️🌈
A new command, "LANG", is added to the FTP command set so the server-FTP process can determine which language to use for server greetings and the textual part of command responses. The parameter for the LANG command SHALL be one of the language tags defined in RFC 1766 [RFC1766]. If a LANG command is issued without a parameter, the server's default language is used. 📝
Greetings and responses issued before language negotiation SHALL be in the server's default language. Paragraph 4.5 of [RFC2277] states that this "default language MUST be understandable by an English-speaking person". This specification RECOMMENDS that the server's default language be English, encoded using ASCII. That text may be augmented by text from other languages. Once negotiated, server-PI MUST return server messages and the textual part of command responses in the negotiated language, encoded in UTF-8. Server-PI MAY wish to re-send previously issued server messages in the newly negotiated language. 🌷
The LANG command only affects the presentation of greeting messages and explanatory text associated with command responses. No attempt should be made by the server to translate protocol elements (FTP commands and numeric responses) or data transmitted over the data connection. 🚫
User-PI MAY issue the LANG command at any time during an FTP session. To get the full benefit of this command, it SHOULD be presented before authentication. In general, it will be issued after the HOST command [MLST]. Note that issuing a HOST or REIN command [RFC959] will negate the effect of the LANG command. User-PI SHOULD be capable of supporting UTF-8 encoding for the negotiated language. Guidance on the interpretation and rendering of UTF-8, defined in section 3, SHALL apply. 🌊
Although NOT REQUIRED by this specification, user-PI SHOULD issue a FEAT command [RFC2389] before a LANG command. This lets the user-PI find out whether the server supports the LANG command and which language options are available. 🔭
To help the server figure out whether a connection has been established with a client that conforms to this specification or an older one, user-PI MUST send a HOST [MLST] and/or LANG command before issuing any other command (other than FEAT [RFC2389]). If user-PI issues a HOST command and the server's default language is acceptable, it doesn't need to issue a LANG command. However, if the implementation does not support the HOST command, a LANG command MUST be issued. Until server-PI receives either a HOST or LANG command, it SHOULD assume that the user-PI does not comply with this specification. 🕵️♀️
4.2 Syntax of the LANG command ✨
okay bestie, here's how the LANG command is spelled out 🎀
lang-command = "Lang" [(SP lang-tag)] CRLF
lang-tag = Primary-tag *( "-" Sub-tag)
Primary-tag = 1*8ALPHA
Sub-tag = 1*8ALPHA
lang-response = lang-ok / error-response
lang-ok = "200" [SP *(%x00..%xFF) ] CRLF
error-response = command-unrecognized / bad-argument /
not-implemented / unsupported-parameter
command-unrecognized = "500" [SP *(%x01..%xFF) ] CRLF
bad-argument = "501" [SP *(%x01..%xFF) ] CRLF
not-implemented = "502" [SP *(%x01..%xFF) ] CRLF
unsupported-parameter = "504" [SP *(%x01..%xFF) ] CRLFthe word "lang" is case independent, so you can type it however you want 💅 "LANG", "lang", "Lang", and "lAnG" are all equivalent commands, no cap.
the OPTIONAL "Lang-tag" parameter lists the primary language tag plus zero or more sub-tags, as defined in [RFC1766]. as [RFC1766] explains, language tags are treated as case insensitive 🦋 if the parameter is left out, the server-PI MUST use the server's default language. no exceptions, periodt 👑
Server-FTP answers the "Lang" command with either "lang-ok" or "error-response" 🌸 a "lang-ok" MUST be sent if Server-FTP supports the "Lang" command and can support some form of the "lang-tag". the level of support SHOULD go like this:
- if server-FTP gets "Lang" with no parameters, it SHOULD return messages and command responses in the server default language 🧚
- if server-FTP gets "Lang" with only a primary tag argument (e.g. en, fr, de, ja, zh, etc.) that it can support, it SHOULD return messages and command responses in the language tied to that primary tag. it's possible that server-FTP will only support the primary tag when it's paired with a sub-tag (e.g. en-US, en-UK, etc.). in those cases, server-FTP MAY work out the right variant to use during the session. how server-FTP makes that call is outside the scope of this specification. if server-FTP can't figure out whether a sub-tag variant is appropriate, it SHOULD return an "unsupported-parameter" (504) response 💀
- if server-FTP gets "Lang" with a primary tag and sub-tag(s) argument that it implements, it SHOULD return messages and command responses that support the language argument 💜 it's possible that server-FTP supports the primary tag of the "Lang" argument but not the sub-tag(s). in that case server-FTP MAY return messages and command responses in the most appropriate variant of the primary tag that it has implemented. how server-FTP makes that determination is outside the scope of this specification. if server-FTP can't determine whether a sub-tag variant is appropriate, it SHOULD return an "unsupported-parameter" (504) response.
for example, if client-FTP sends a "LANG en-AU" command and server-FTP has implemented the language tags en-US and en-UK, it may decide the most appropriate tag is en-UK and return 200 en-AU not supported. Language set to en-UK. the numeric response is a protocol element and can not be changed 🔥 the string attached to it is for illustrative purposes only.
Clients and servers that conform to this specification MUST support the LANG command 🫶 Clients SHOULD, however, expect a 500 or 502 command response, in cases where older or non-compliant servers don't recognize or haven't implemented the "Lang". a 501 response SHOULD be sent if the argument to the "Lang" command isn't syntactically correct. a 504 response SHOULD be sent if the "Lang" argument is syntactically correct but isn't implemented. as noted above, an argument may count as a lexicon match even though it isn't an exact syntax match 🌈
4.3 Feat response for LANG command
a server-FTP process that supports the LANG command, and has language support for messages and command responses, MUST include a feature line in its response to the FEAT command [RFC2389]. that line says the LANG command is supported and gives a fact list of the supported language tags 📝 a response to a FEAT command SHALL look like this:
Lang-feat = SP "LANG" SP lang-fact CRLF
lang-fact = lang-tag ["*"] *(";" lang-tag ["*"])
lang-tag = Primary-tag *( "-" Sub-tag)
Primary-tag= 1*8ALPHA
Sub-tag = 1*8ALPHAthe lang-feat response has the string "LANG" followed by a language fact 🦋 that string isn't case sensitive, but it SHOULD be sent in upper case, as [RFC2389] recommends. the initial space in the Lang-feat response is REQUIRED by the FEAT command, and it MUST be a single space character. more or fewer space characters are not allowed 🙅♀️ the lang-fact SHALL include the lang-tags that server-FTP can support. at least one lang-tag MUST be included in the FEAT response. the lang-tag SHALL be in the form described earlier in this document. the OPTIONAL asterisk, when it shows up, SHALL indicate the current lang-tag that server-FTP is using for messages and responses ✨
4.3.1 Feat examples
C> feat
S> 211- <any descriptive text>
S> ...
S> LANG EN*
S> ...
S> 211 endin this example server-FTP can only support English, which is the current language (shown by the asterisk) the server uses for messages and command responses 🌸
C> feat
S> 211- <any descriptive text>
S> ...
S> LANG EN*;FR
S> ...
S> 211 end
C> LANG fr
S> 200 Le response sera changez au francais
C> feat
S> 211- <quelconque descriptif texte>
S> ...
S> LANG EN;FR*
S> ...
S> 211 endin this one, server-FTP supports both English and French, as the first FEAT response shows 🇫🇷 the asterisk means English is the current language in use by server-FTP. after a LANG command switches the language to French, the next FEAT response shows French as the current language in use 💖
in the examples above, ellipses are placeholders for where other features might go, but they are NOT REQUIRED 🧁
5 Security Considerations 🚨
this document covers support for character sets wider than 1 byte, plus a new language negotiation command. conformance to this document should not induce a security risk 🔐
6 Acknowledgments 🫶
the following people contributed to this document:
- D. J. Bernstein
- Martin J. Duerst
- Mark Harris
- Paul Hethmon
- Alun Jones
- Gregory Lundberg
- James Matthews
- Keith Moore
- Sandra O'Donnell
- Benjamin Riefenstahl
- Stephen Tihor
(and others from the FTPEXT working group) 💜
7 Glossary 📖
- BIDI - abbreviation for Bi-directional, a reference to mixed right-to-left and left-to-right text.
- Character Set - a collection of characters used to represent textual information in which each character has a numeric value
- Code Set - (see character set).
- Glyph - a character image represented on a display device.
- I18N - "I eighteen N", the first and last letters of the word "internationalization" and the eighteen letters in between.
- UCS-2 - the ISO/IEC 10646 two octet Universal Character Set form.
- UCS-4 - the ISO/IEC 10646 four octet Universal Character Set form.
- UTF-8 - the UCS Transformation Format represented in 8 bits.
- TF-16 - A 16-bit format including the BMP (directly encoded) and surrogate pairs to represent characters in planes 01-16; equivalent to Unicode.
8 Bibliography 📚
- [ABNF] Crocker, D. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", RFC 2234, November 1997.
- [ASCII] ANSI X3.4:1986 Coded Character Sets - 7 Bit American National Standard Code for Information Interchange (7-bit ASCII)
- [ISO-8859] ISO 8859. International standard -- Information processing -- 8-bit single-byte coded graphic character sets -- Part 1:Latin alphabet No. 1 (1987) -- Part 2: Latin alphabet No. 2 (1987) -- Part 3: Latin alphabet No. 3 (1988) -- Part 4: Latin alphabet No. 4 (1988) -- Part 5: Latin/Cyrillic alphabet (1988) -- Part 6: Latin/Arabic alphabet (1987) -- Part : Latin/Greek alphabet (1987) -- Part 8: Latin/Hebrew alphabet (1988) -- Part 9: Latin alphabet No. 5 (1989) -- Part10: Latin alphabet No. 6 (1992)
- [BCP14] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
- [ISO-10646] ISO/IEC 10646-1:1993. International standard -- Information technology -- Universal multiple-octet coded character set (UCS) -- Part 1: Architecture and basic multilingual plane.
- [MLST] Elz, R. and P. Hethmon, "Extensions to FTP", Work in Progress.
- [RFC854] Postel, J. and J. Reynolds, "Telnet Protocol Specification", STD 8, RFC 854, May 1983.
- [RFC959] Postel, J. and J. Reynolds, "File Transfer Protocol (FTP)", STD 9, RFC 959, October 1985.
- [RFC1123] Braden, R., "Requirements for Internet Hosts -- Application and Support", STD 3, RFC 1123, October 1989.
- [RFC1738] Berners-Lee, T., Masinter, L. and M. McCahill, "Uniform Resource Locators (URL)", RFC 1738, December 1994.
- [RFC1766] Alvestrand, H., "Tags for the Identification of Languages", RFC 1766, March 1995.
- [RFC2130] Weider, C., Preston, C., Simonsen, K., Alvestrand, H., Atkinson, R., Crispin, M. and P. Svanberg, "Character Set Workshop Report", RFC 2130, April 1997.
- [RFC2277] Alvestrand, H., " IETF Policy on Character Sets and Languages", RFC 2277, January 1998.
- [RFC2279] Yergeau, F., "UTF-8, a transformation format of ISO 10646", RFC 2279, January 1998.
- [RFC2389] Elz, R. and P. Hethmon, "Feature Negotiation Mechanism for the File Transfer Protocol", RFC 2389, August 1998.
- [UNICODE] The Unicode Consortium, "The Unicode Standard - Version 2.0", Addison Westley Developers Press, July 1996.
- [UTF-8] ISO/IEC 10646-1:1993 AMENDMENT 2 (1996). UCS Transformation Format 8 (UTF-8).
9 Author's Address 🎀
Bill Curtin JIEO Attn: JEBBD Ft. Monmouth, N.J. 07703-5613
EMail: curtinw@ftm.disa.mil
Annex A - Implementation Considerations 🧚
A.1 General Considerations 🎀
- okay bestie, when you're supporting the extended character set, your code needs to account for stuff like using a NULL character to end a string or no longer being able to steal the high order bit for internal use 💀
- implementers should be aware that non UTF-8 pathnames might accidentally parse as valid UTF-8 🙈 the odds are low for some encodings and statistically zero to zero for others. a recent non-scientific analysis found EUC encoded Japanese words had a 2.7% false reading, SJIS had a 0.0005% false reading, and encodings like ASCII or KOI-8 had a 0% false reading. the chance is highest for short pathnames and drops as they get longer. you might look for signs that a pathname which parses as UTF-8 isn't really valid UTF-8, like multiple local character sets showing up in a short pathname. hopefully as more implementations use UTF-8 transfer encoding, there'll be less need to guess the encoding 🦋
- client developers should know that pathnames can mix characters, like
//Latin1DirectoryName/HebrewFileName🌈 they should be ready to handle bi-directional (BIDI) display, so right to left for the directory and left to right for the filename. bi-directional display is outside the scope of this document and way more complicated than that example, but you can find an algorithm for it in the UNICODE 2.0 [UNICODE] standard. also, pathnames can have different byte ordering and still be logically and display-wise equivalent, because BIDI control characters can be inserted at different points during composition. mixed character sets can also cause problems with font swapping 🔥 - a server that copies pathnames transparently from a local filesystem can keep doing that. then it's up to the local file creators to use UTF-8 pathnames 💅
- servers can support charset labeling of files and/or directories, so different pathnames can have different charsets. the server should try to convert all pathnames to UTF-8, but if it can't, it should leave that name in its raw form ✨
- some server OSes don't mandate character sets but let admins configure one in the FTP server. these servers should be set up to use a particular mapping table, either external or built-in. this gives you the flexibility to define different charsets for different directories 🧚
- if the server's OS doesn't mandate the character set and the FTP server can't be configured, the server should just use the raw bytes in the file name. they might be ASCII or UTF-8 🌸
- if the server is a mirror and wants to look exactly like the site it's mirroring, it should store the exact file name bytes it received from the main server 🫶
A.2 Transition Considerations 🦋
- servers that support this specification can nearly always tell, when they get a pathname from an old client (one that doesn't support this specification), whether it's UTF-8 (see B.1) or some other code set 👀 to support these older clients, servers may want to default to a non UTF-8 code set. how a server supports non UTF-8 is outside the scope of this specification 💜
- newer clients can figure out whether the server supports UTF-8 (meaning it supports this specification) by checking if the server supports the FEAT command and the UTF8 feature (defined in 3.2). if a newer client finds out the server doesn't support UTF-8, it may want to default to a different code set. client developers should keep in mind that pathnames on older servers might be stored in UTF-8 ⭐ how a client supports non UTF-8 is outside the scope of this specification too 💅
- clients and servers can transition to UTF-8 either by converting to and from the local encoding, or by having users store UTF-8 filenames 🍓 the first approach is easier on tightly controlled file systems (e.g. PCs and MACs), and the second is easier on more free form file systems (e.g. Unix) 🌈
- for interactive use, focus on the user interface and ease of use. non-interactive use needs consistent and controlled behavior 🎀
- lots of applications reference files by their old raw pathname (e.g. linked URLs), and changing the pathname to UTF-8 will make access to the old URL fail 😭 one fix is for the server to act as if the file has 2 different pathnames. it could do this internally on controlled file systems, or with symbolic links on free form systems. this might work for non-interactive transfer of a single file, but a non-interactive transfer of all the files in a directory will produce duplicates, and interactive users may see lists of files that are double the actual number of files 💀
Annex B - Sample Code and Examples ✨
B.1 Valid UTF-8 check 🌸
The following routine checks whether a byte sequence is valid UTF-8. It does this by checking that the first and following bytes are tagged properly so they match the UTF-8 format. Then it checks that the data part of the UTF-8 sequence falls within the range the encoding allows. Note: this routine won't detect characters that haven't been assigned and therefore don't exist 🙈
int utf8_valid(const unsigned char *buf, unsigned int len)
{
const unsigned char *endbuf = buf + len;
unsigned char byte2mask=0x00, c;
int trailing = 0; // trailing (continuation) bytes to follow
while (buf != endbuf)
{
c = *buf++;
if (trailing)
if ((c&0xC0) == 0x80) // Does trailing byte follow UTF-8 format?
{if (byte2mask) // Need to check 2nd byte for proper range?
if (c&byte2mask) // Are appropriate bits set?
byte2mask=0x00;
else
return 0;
trailing--; }
else
return 0;
else
if ((c&0x80) == 0x00) continue; // valid 1 byte UTF-8
else if ((c&0xE0) == 0xC0) // valid 2 byte UTF-8
if (c&0x1E) // Is UTF-8 byte in
// proper range?
trailing =1;
else
return 0;
else if ((c&0xF0) == 0xE0) // valid 3 byte UTF-8
{if (!(c&0x0F)) // Is UTF-8 byte in
// proper range?
byte2mask=0x20; // If not set mask
// to check next byte
trailing = 2;}
else if ((c&0xF8) == 0xF0) // valid 4 byte UTF-8
{if (!(c&0x07)) // Is UTF-8 byte in
// proper range?
byte2mask=0x30; // If not set mask
// to check next byte
trailing = 3;}
else if ((c&0xFC) == 0xF8) // valid 5 byte UTF-8
{if (!(c&0x03)) // Is UTF-8 byte in
// proper range?
byte2mask=0x38; // If not set mask
// to check next byte
trailing = 4;}
else if ((c&0xFE) == 0xFC) // valid 6 byte UTF-8
{if (!(c&0x01)) // Is UTF-8 byte in
// proper range?
byte2mask=0x3C; // If not set mask
// to check next byte
trailing = 5;}
else return 0;
}
return trailing == 0;
}B.2 Conversions 🌈
The code examples in this section closely follow the algorithm in ISO 10646, and they might not be the most efficient way to convert to and from UTF-8 encoding. if efficiency matters to you, implementers should use the appropriate bitwise operators 💅
You can find more code examples and loads of mapping tables at the Unicode site, HTTP://www.unicode.org or FTP://unicode.org 🧚
Heads up: the conversion examples below assume the local character set supported by the operating system is something other than UCS2/UTF-16. some operating systems already support UCS2/UTF-16, notably Plan 9 and Windows NT 👑 on those systems, no conversion is needed from the local character set to the UCS ✨
B.2.1 Conversion from Local Character Set to UTF-8
Okay so converting from the local filesystem character set into UTF-8 is a two-step glow-up 💅. First you convert the local set into the UCS, then you convert the UCS into UTF-8 ✨.
Step one can be done with a mapping table that pairs each local character code with its matching UCS code. For example, the ISO/IEC 8859-8 [ISO-8859] code for the Hebrew letter "VAV" is 0xE4 🍓, and its matching 4 byte ISO/IEC 10646 code is 0x000005D5 🌸.
Step two is turning the UCS character code into UTF-8. This routine figures out how many bytes are needed based on the UCS-4 character code and encodes them:
unsigned int ucs4_to_utf8 (unsigned long *ucs4_buf, unsigned int
ucs4_len, unsigned char *utf8_buf)
{
const unsigned long *ucs4_endbuf = ucs4_buf + ucs4_len;
unsigned int utf8_len = 0; // return value for UTF8 size
unsigned char *t_utf8_buf = utf8_buf; // Temporary pointer
// to load UTF8 values
while (ucs4_buf != ucs4_endbuf)
{
if ( *ucs4_buf <= 0x7F) // ASCII chars no conversion needed
{
*t_utf8_buf++ = (unsigned char) *ucs4_buf;
utf8_len++;
ucs4_buf++;
}
else
if ( *ucs4_buf <= 0x07FF ) // In the 2 byte utf-8 range
{
*t_utf8_buf++= (unsigned char) (0xC0 + (*ucs4_buf/0x40));
*t_utf8_buf++= (unsigned char) (0x80 + (*ucs4_buf%0x40));
utf8_len+=2;
ucs4_buf++;
}
else
if ( *ucs4_buf <= 0xFFFF ) /* In the 3 byte utf-8 range. The
values 0x0000FFFE, 0x0000FFFF
and 0x0000D800 - 0x0000DFFF do
not occur in UCS-4 */
{
*t_utf8_buf++= (unsigned char) (0xE0 +
(*ucs4_buf/0x1000));
*t_utf8_buf++= (unsigned char) (0x80 +
((*ucs4_buf/0x40)%0x40));
*t_utf8_buf++= (unsigned char) (0x80 + (*ucs4_buf%0x40));
utf8_len+=3;
ucs4_buf++;
}
else
if ( *ucs4_buf <= 0x1FFFFF ) //In the 4 byte utf-8 range
{
*t_utf8_buf++= (unsigned char) (0xF0 +
(*ucs4_buf/0x040000));
*t_utf8_buf++= (unsigned char) (0x80 +
((*ucs4_buf/0x10000)%0x40));
*t_utf8_buf++= (unsigned char) (0x80 +
((*ucs4_buf/0x40)%0x40));
*t_utf8_buf++= (unsigned char) (0x80 + (*ucs4_buf%0x40));
utf8_len+=4;
ucs4_buf++;
}
else
if ( *ucs4_buf <= 0x03FFFFFF )//In the 5 byte utf-8 range
{
*t_utf8_buf++= (unsigned char) (0xF8 +
(*ucs4_buf/0x01000000));
*t_utf8_buf++= (unsigned char) (0x80 +
((*ucs4_buf/0x040000)%0x40));
*t_utf8_buf++= (unsigned char) (0x80 +
((*ucs4_buf/0x1000)%0x40));
*t_utf8_buf++= (unsigned char) (0x80 +
((*ucs4_buf/0x40)%0x40));
*t_utf8_buf++= (unsigned char) (0x80 +
(*ucs4_buf%0x40));
utf8_len+=5;
ucs4_buf++;
}
else
if ( *ucs4_buf <= 0x7FFFFFFF )//In the 6 byte utf-8 range
{
*t_utf8_buf++= (unsigned char)
(0xF8 +(*ucs4_buf/0x40000000));
*t_utf8_buf++= (unsigned char) (0x80 +
((*ucs4_buf/0x01000000)%0x40));
*t_utf8_buf++= (unsigned char) (0x80 +
((*ucs4_buf/0x040000)%0x40));
*t_utf8_buf++= (unsigned char) (0x80 +
((*ucs4_buf/0x1000)%0x40));
*t_utf8_buf++= (unsigned char) (0x80 +
((*ucs4_buf/0x40)%0x40));
*t_utf8_buf++= (unsigned char) (0x80 +
(*ucs4_buf%0x40));
utf8_len+=6;
ucs4_buf++;
}
}
return (utf8_len);
}B.2.2 Conversion from UTF-8 to Local Character Set
Going the other way is the reverse procedure, like rewinding your favourite song 🎀. First the UTF-8 encoding gets transformed into the UCS-4 character set, and then the UCS-4 gets converted into the local character set using a mapping table (the opposite of the table used to form the UCS-4 character code) 🧚.
To convert from UTF-8 to UCS-4, the free bits (the ones that don't define the UTF-8 sequence size or signify continuation bytes) get concatenated as a bit string. Those bits are then distributed into a four-byte sequence starting from the least significant bits, and any bits that don't get assigned a spot in the four-byte sequence are padded with ZERO bits 💜. Here's the routine that converts UTF-8 encoding to UCS-4 character codes:
int utf8_to_ucs4 (unsigned long *ucs4_buf, unsigned int utf8_len,
unsigned char *utf8_buf)
{
const unsigned char *utf8_endbuf = utf8_buf + utf8_len;
unsigned int ucs_len=0;
while (utf8_buf != utf8_endbuf)
{
if ((*utf8_buf & 0x80) == 0x00) /*ASCII chars no conversion
needed */
{
*ucs4_buf++ = (unsigned long) *utf8_buf;
utf8_buf++;
ucs_len++;
}
else
if ((*utf8_buf & 0xE0)== 0xC0) //In the 2 byte utf-8 range
{
*ucs4_buf++ = (unsigned long) (((*utf8_buf - 0xC0) * 0x40)
+ ( *(utf8_buf+1) - 0x80));
utf8_buf += 2;
ucs_len++;
}
else
if ( (*utf8_buf & 0xF0) == 0xE0 ) /*In the 3 byte utf-8
range */
{
*ucs4_buf++ = (unsigned long) (((*utf8_buf - 0xE0) * 0x1000)
+ (( *(utf8_buf+1) - 0x80) * 0x40)
+ ( *(utf8_buf+2) - 0x80));
utf8_buf+=3;
ucs_len++;
}
else
if ((*utf8_buf & 0xF8) == 0xF0) /* In the 4 byte utf-8
range */
{
*ucs4_buf++ = (unsigned long)
(((*utf8_buf - 0xF0) * 0x040000)
+ (( *(utf8_buf+1) - 0x80) * 0x1000)
+ (( *(utf8_buf+2) - 0x80) * 0x40)
+ ( *(utf8_buf+3) - 0x80));
utf8_buf+=4;
ucs_len++;
}
else
if ((*utf8_buf & 0xFC) == 0xF8) /* In the 5 byte utf-8
range */
{
*ucs4_buf++ = (unsigned long)
(((*utf8_buf - 0xF8) * 0x01000000)
+ ((*(utf8_buf+1) - 0x80) * 0x040000)
+ (( *(utf8_buf+2) - 0x80) * 0x1000)
+ (( *(utf8_buf+3) - 0x80) * 0x40)
+ ( *(utf8_buf+4) - 0x80));
utf8_buf+=5;
ucs_len++;
}
else
if ((*utf8_buf & 0xFE) == 0xFC) /* In the 6 byte utf-8
range */
{
*ucs4_buf++ = (unsigned long)
(((*utf8_buf - 0xFC) * 0x40000000)
+ ((*(utf8_buf+1) - 0x80) * 0x010000000)
+ ((*(utf8_buf+2) - 0x80) * 0x040000)
+ (( *(utf8_buf+3) - 0x80) * 0x1000)
+ (( *(utf8_buf+4) - 0x80) * 0x40)
+ ( *(utf8_buf+5) - 0x80));
utf8_buf+=6;
ucs_len++;
}
}
return (ucs_len);
}B.2.3 ISO/IEC 8859-8 Example
Let's watch this one go all the way round 🔁. This example maps the ISO/IEC 8859-8 character set to UTF-8 and then back to ISO/IEC 8859-8. As noted earlier, the Hebrew letter "VAV" is converted from the ISO/IEC 8859-8 character code 0xE4 to the matching 4 byte ISO/IEC 10646 code of 0x000005D5 by a simple lookup in a conversion/mapping file 🫶.
The UCS-4 character code gets transformed into UTF-8 using the ucs4_to_utf8 routine from earlier:
- Because the UCS-4 character is between
0x80and0x07FF, it will map to a 2 byte UTF-8 sequence. - The first byte is defined by
(0xC0 + (0x000005D5 / 0x40)) = 0xD7. - The second byte is defined by
(0x80 + (0x000005D5 % 0x40)) = 0x95.
The UTF-8 encoding gets transferred back to UCS-4 using the utf8_to_ucs4 routine from earlier:
- Because the first byte of the sequence, when the
&operator with a value of0xE0is applied, will produce0xC0(0xD7 & 0xE0 = 0xC0), the UTF-8 is a 2 byte sequence. - The four byte UCS-4 character code is produced by
(((0xD7 - 0xC0) * 0x40) + (0x95 - 0x80)) = 0x000005D5.
Finally, the UCS-4 character code is converted to the ISO/IEC 8859-8 character code (using the mapping table that matches ISO/IEC 8859-8 to UCS-4) to produce the original 0xE4 code for the Hebrew letter "VAV" 🎉.
B.2.4 Vendor Codepage Example
Now for vendor codepages 💅. This example shows how a codepage maps to UTF-8 and back to a vendor codepage. Mapping between vendor codepages can be done in a very similar way to what we just did. For instance, both the PC and Mac codepages reflect the character set from the Thai standard TIS 620-2533 🌈. The character code on both platforms for the Thai letter "SO SO" is 0xAB. That character can then be mapped into UCS-4 by way of a conversion/mapping file, giving the UCS-4 code 0x0E0B.
The UCS-4 character code gets transformed into UTF-8 using the ucs4_to_utf8 routine from earlier:
- Because the UCS-4 character is between
0x0800and0xFFFF, it will map to a 3 byte UTF-8 sequence. - The first byte is defined by
(0xE0 + (0x00000E0B / 0x1000) = 0xE0. - The second byte is defined by
(0x80 + ((0x00000E0B / 0x40) % 0x40))) = 0xB8. - The third byte is defined by
(0x80 + (0x00000E0B % 0x40)) = 0x8B.
The UTF-8 encoding gets transferred back to UCS-4 using the utf8_to_ucs4 routine from earlier:
- Because the first byte of the sequence, when the
&operator with a value of0xF0is applied, will produce0xE0(0xE0 & 0xF0 = 0xE0), the UTF-8 is a 3 byte sequence. - The four byte UCS-4 character code is produced by
(((0xE0 - 0xE0) * 0x1000) + ((0xB8 - 0x80) * 0x40) + (0x8B - 0x80) = 0x0000E0B.
Finally, the UCS-4 character code is converted to either the PC or Mac codepage character code (using the mapping table that matches the codepage to UCS-4) to get back the original 0xAB code for the Thai letter "SO SO" 🍓.
B.3 Pseudo Code for a High-Quality Translating Server
Last but not least, here's the pseudo code for a high-quality translating server 🧠✨:
if utf8_valid(fn)
{
attempt to convert fn to the local charset, producing localfn
if (conversion fails temporarily) return error
if (conversion succeeds)
{
attempt to open localfn
if (open fails temporarily) return error
if (open succeeds) return success
}
}
attempt to open fn
if (open fails temporarily) return error
if (open succeeds) return success
return permanent errorFull Copyright Statement ✨
Okay so the legal bit, bestie 💅 Copyright (C) The Internet Society (1999). All Rights Reserved. 🎀
This document and translations of it can be copied and handed out to others. Derivative works that comment on or explain it, or help with implementing it, can also be made, copied, published and distributed, in whole or in part, with no real restrictions at all. The one condition is that the copyright notice and this paragraph have to be on every copy and derivative work 🫶 However, the document itself may not be modified in any way. That means no removing the copyright notice or the references to the Internet Society or other Internet organizations. The exceptions are when it's needed for developing Internet standards (then the copyright procedures in the Internet Standards process must be followed), or when it's translated into languages other than English 🌈
The permissions above are perpetual, so the Internet Society and its successors or assigns won't revoke them 💜
Now the disclaimer, which is serious even though I'm being cute about it 😭
This document and the information in it are provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgement 🌸
Funding for the RFC Editor function is currently provided by the Internet Society. Literally the reason this whole document exists and gets published, so big thanks to them 🧚✨
📦 where this page comes from
- ietf: rfc2640 1999-07
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms