Enhanced #line directives

Summary

The compiler applies the mapping defined by #line directives to diagnostic locations and sequence points emitted to the PDB.

Currently only the line number and file path can be mapped while the starting character is inferred from the source code. The proposal is to allow specifying full span mapping.

Motivation

DSLs that generate C# source code (such as ASP.NET Razor) can't currently produce precise source mapping using #line directives. This results in degraded debugging experience in some cases as the sequence points emitted to the PDB can't map to the precise location in the original source code.

For example, the following Razor code

@page "/"
Time: @DateTime.Now

generates code like so (simplified):

#line hidden
void Render()
{
   _builder.Add("Time:");
#line 2 "page.razor"
   _builder.Add(DateTime.Now);
#line hidden
}

The above directive would map the sequence point emitted by the compiler for the _builder.Add(DateTime.Now); statement to the line 2, but the column would be off (16 instead of 7).

The Razor source generator actually incorrectly generates the following code:

#line hidden
void Render()
{
   _builder.Add("Time:");
   _builder.Add(
#line 2 "page.razor"
      DateTime.Now
#line hidden
);
}

The intent was to preserve the starting character and it works for diagnostic location mapping. However, this does not work for sequence points since #line directive only applies to the sequence points that follow it. There is no sequence point in the middle of the _builder.Add(DateTime.Now); statement (sequence points can only be emitted at IL instructions with empty evaluation stack). The #line 2 directive in above code thus has no effect on the generated PDB and the debugger won't place a breakpoint or stop on the @DateTime.Now snippet in the Razor page.

Issues addressed by this proposal: https://github.com/dotnet/roslyn/issues/43432 https://github.com/dotnet/roslyn/issues/46526

Detailed design

We amend the syntax of line_indicator used in pp_line directive like so:

Current:

line_indicator
    : decimal_digit+ whitespace file_name
    | decimal_digit+
    | 'default'
    | 'hidden'
    ;

Proposed:

line_indicator
    : '(' decimal_digit+ ',' decimal_digit+ ')' '-' '(' decimal_digit+ ',' decimal_digit+ ')' whitespace decimal_digit+ whitespace file_name
    | '(' decimal_digit+ ',' decimal_digit+ ')' '-' '(' decimal_digit+ ',' decimal_digit+ ')' whitespace file_name
    | decimal_digit+ whitespace file_name
    | decimal_digit+
    | 'default'
    | 'hidden'
    ;

That is, the #line directive would accept either 5 decimal numbers (start line, start character, end line, end character, character offset), 4 decimal numbers (start line, start character, end line, end character), or a single one (line).

If character offset is not specified its default value is 0, otherwise it specifies the number of UTF-16 characters. The number must be non-negative and less then length of the line following the #line directive in the unmapped file.

(start line, start character)-(end line, end character) specifies a span in the mapped file. start line and end line are positive integers that specify line numbers. start character, end character are positive integers that specify UTF-16 character numbers. start line, start character, end line, end character are 1-based, meaning that the first line of the file and the first UTF-16 character on each line is assigned number 1.

The implementation would constraint these numbers so that they specify a valid sequence point source span:

• start line - 1 is within range [0, 0x20000000) and not equal to 0xfeefee.
• end line - 1 is within range [0, 0x20000000) and not equal to 0xfeefee.
• start character - 1 is within range [0, 0x10000)
• end character - 1 is within range [0, 0x10000)
• end line is greater or equal to start line.
• start line is equal to end line then end character is greater than start character.

Note that the numbers specified in the directive syntax are 1-based numbers but the actual spans in the PDB are zero-based. Hence the -1 adjustments above.

The mapped spans of sequence points and the locations of diagnostics that #line directive applies to are calculated as follows.

Let d be the zero-based number of the unmapped line containing the #line directive. Let span L = (start: (start line - 1, start character - 1), end: (end line - 1, end character - 1)) be zero-based span specified by the directive.

Function M that maps a position (line, character) within the scope of the #line directive in the source file containing the #line directive to a mapped position (mapped line, mapped character) is defined as follows:

M(l, c) =

l == d + 1 => (L.start.line + l – d – 1, L.start.character + max(c – character offset, 0))

l > d + 1 => (L.start.line + l – d – 1, c)

The syntax constructs that sequence points are associated with are determined by the compiler implementation and not covered by this specification. The compiler also decides for each sequence point its unmapped span. This span may partially or fully cover the associated syntax construct.

Once the unmapped spans are determined by the compiler the function M defined above is applied to their starting and ending positions, with the exception of the ending position of all sequence points within the scope of the #line directive whose unmapped location is at line d + 1 and character less than character offset. The end position of all these sequence points is L.end.

Example [5.i] demonstrates why it is necessary to provide the ability to specify the end position of the first sequence point span.

The above definition allows the generator of the unmapped source code to avoid intimate knowledge of which exact source constructs of the C# language produce sequence points. The mapped spans of the sequence points in the scope of the #line directive are derived from the relative position of the corresponding unmapped spans to the first unmapped span.

Specifying the character offset allows the generator to insert any single-line prefix on the first line. This prefix is generated code that is not present in the mapped file. Such inserted prefix affects the value of the first unmapped sequence point span. Therefore the starting character of subsequent sequence point spans need to be offset by the length of the prefix (character offset). See example [2].

Examples

For clarity the examples use spanof('...') and lineof('...') pseudo-syntax to express the mapped span start position and line number, respectively, of the specified code snippet.

1. First and subsequent spans

Consider the following code with unmapped zero-based line numbers listed on the right:

#line (1,10)-(1,15) "a" // 3
  A();B(                // 4
);C();                  // 5
    D();                // 6

d = 3 L = (0, 9)..(0, 14)

There are 4 sequence point spans the directive applies to with following unmapped and mapped spans: (4, 2)..(4, 5) => (0, 9)..(0, 14) (4, 6)..(5, 1) => (0, 15)..(1, 1) (5, 2)..(5, 5) => (1, 2)..(1, 5) (6, 4)..(6, 7) => (2, 4)..(2, 7)

2. Character offset

Razor generates _builder.Add( prefix of length 15 (including two leading spaces).

Razor:

@page "/"                                  
@F(() => 1+1, 
   () => 2+2
) 

Generated C#:

#line hidden
void Render()            
{ 
#line spanof('F(...)') 15 "page.razor"  // 4
  _builder.Add(F(() => 1+1,            // 5
  () => 2+2                            // 6
));                                    // 7
#line hidden
}
);
}

d = 4 L = (1, 1)..(3,0) character offset = 15

Spans:

• _builder.Add(F(…)); => F(…): (5, 2)..(7, 2) => (1, 1)..(3, 0)
• 1+1 => 1+1: (5, 23)..(5, 25) => (1, 9)..(1, 11)
• 2+2 => 2+2: (6, 7)..(6, 9) => (2, 7)..(2, 9)

3. Razor: Single-line span

Razor:

@page "/"
Time: @DateTime.Now

Generated C#:

#line hidden
void Render()
{
  _builder.Add("Time:");
#line spanof('DateTime.Now') 15 "page.razor"
  _builder.Add(DateTime.Now);
#line hidden
);
}

4. Razor: Multi-line span

Razor:

@page "/"                                  
@JsonToHtml(@"
{
  ""key1"": "value1",
  ""key2"": "value2"
}") 

Generated C#:

#line hidden
void Render()
{
  _builder.Add("Time:");
#line spanof('JsonToHtml(@"...")') 15 "page.razor"
  _builder.Add(JsonToHtml(@"
{
  ""key1"": "value1",
  ""key2"": "value2"
}"));
#line hidden
}
);
}

5. Razor: block constructs

i. block containing expressions

In this example, the mapped span of the first sequence point that is associated with the IL instruction that is emitted for the _builder.Add(Html.Helper(() => statement needs to cover the whole expression of Html.Helper(...) in the generated file a.razor. This is achieved by application of rule [1] to the end position of the sequence point.

@Html.Helper(() => 
{
    <p>Hello World</p>
    @DateTime.Now
})

#line spanof('Html.Helper(() => { ... })') 13 "a.razor"
_builder.Add(Html.Helper(() => 
#line lineof('{') "a.razor"
{
#line spanof('DateTime.Now') 13 "a.razor"
_builder.Add(DateTime.Now);
#line lineof('}') "a.razor"
}
#line hidden
)

ii. block containing statements

Uses existing #line line file form since

a) Razor does not add any prefix, b) { is not present in the generated file and there can't be a sequence point placed on it, therefore the span of the first unmapped sequence point is unknown to Razor.

The starting character of Console in the generated file must be aligned with the Razor file.

@{Console.WriteLine(1);Console.WriteLine(2);}

#line lineof('@{') "a.razor"
  Console.WriteLine(1);Console.WriteLine(2);
#line hidden

iii. block containing top-level code (@code, @functions)

Uses existing #line line file form since

a) Razor does not add any prefix, b) { is not present in the generated file and there can't be a sequence point placed on it, therefore the span of the first unmapped sequence point is unknown to Razor.

The starting character of [Parameter] in the generated file must be aligned with the Razor file.

@code {
    [Parameter]
    public int IncrementAmount { get; set; }
}

#line lineof('[') "a.razor"
    [Parameter]
    public int IncrementAmount { get; set; }
#line hidden

6. Razor: @for, @foreach, @while, @do, @if, @switch, @using, @try, @lock

Uses existing #line line file form since a) Razor does not add any prefix. b) the span of the first unmapped sequence point may not be known to Razor (or shouldn't need to know).

The starting character of the keyword in the generated file must be aligned with the Razor file.

@for (var i = 0; i < 10; i++)
{
}
@if (condition)
{
}
else
{
}

#line lineof('for') "a.razor"
 for (var i = 0; i < 10; i++)
{
}
#line lineof('if') "a.razor"
 if (condition)
{
}
else
{
}
#line hidden