Chapter 9 Lexical structure

19. Lexical structure

29.1 Programs

3A C# program consists of one or more source files, known formally as compilation units (§16.1). A source

4file is an ordered sequence of Unicode characters. Source files typically have a one-to-one correspondence

5with files in a file system, but this correspondence is not required.

6Conceptually speaking, a program is compiled using three steps:

71. Transformation, which converts a file from a particular character repertoire and encoding scheme into a

8sequence of Unicode characters.

92. Lexical analysis, which translates a stream of Unicode input characters into a stream of tokens.

103. Syntactic analysis, which translates the stream of tokens into executable code.

11Conforming implementations shall accept Unicode source files encoded with the UTF-8 encoding form

12(as defined by the Unicode standard), and transform them into a sequence of Unicode characters.

13Implementations can choose to accept and transform additional character encoding schemes (such as

14UTF-16, UTF-32, or non-Unicode character mappings).

15[Note: It is beyond the scope of this standard to define how a file using a character representation other

16than Unicode might be transformed into a sequence of Unicode characters. During such transformation,

17however, it is recommended that the usual line-separating character (or sequence) in the other character

18set be translated to the two-character sequence consisting of the Unicode carriage-return character

19followed by Unicode line-feed character. For the most part this transformation will have no visible

20effects; however, it will affect the interpretation of verbatim string literal tokens (§9.4.4.5). The purpose

21of this recommendation is to allow a verbatim string literal to produce the same character sequence

22when its source file is moved between systems that support differing non-Unicode character sets, in

23particular, those using differing character sequences for line-separation. end note]

249.2 Grammars

25This specification presents the syntax of the C# programming language using two grammars. The lexical

26grammar (§9.2.1) defines how Unicode characters are combined to form line terminators, white space,

27comments, tokens, and pre-processing directives. The syntactic grammar (§9.2.2) defines how the tokens

28resulting from the lexical grammar are combined to form C# programs.

299.2.1 Lexical grammar

30The lexical grammar of C# is presented in §9.2.3, §9.4, and §9.5. The terminal symbols of the lexical

31grammar are the characters of the Unicode character set, and the lexical grammar specifies how characters

32are combined to form tokens (§9.4), white space (§9.3.3), comments (§9.3.2), and pre-processing directives

33(§9.5).

34Every source file in a C# program shall conform to the input production of the lexical grammar (§9.2.3).

359.2.2 Syntactic grammar

36The syntactic grammar of C# is presented in the clauses, subclauses, and appendices that follow this

37subclause. The terminal symbols of the syntactic grammar are the tokens defined by the lexical grammar,

38and the syntactic grammar specifies how tokens are combined to form C# programs.

39Every source file in a C# program shall conform to the compilation-unit production (§16.1) of the syntactic

40grammar.

63