Looking under the cover (How does it work?)

The source code of the project is available in the F# Community Samples project at CodePlex (under the MS-PL license). In this section, I'll write little information about the implementation of the tool. This may be interesting if you want to implement something similar yourself (anything to do with F# color highlighting or IntelliSense) or if you want to contribute to the tool - it is open-source and you are more than welcome to make it even better!

The implementation is easier than you would expect, because the information about types as well as the tokenizer (used for colorization) is available as a service of the F# compiler (more specifically, the FSharp.Compiler.dll assembly). The only problem is that the functionality is not quite documented (there are some comments in the source code distributed with the F# CTP) and that the functionality is marked as internal (with several InternalsVisibleTo attributes that expose it to F# integration for Visual Studio).

Internal F# language service API

To call the internal API, I'm using a couple of wrapper types that provide a typed access to the F# compiler API. Under the cover, the wrappers use .NET Reflection to call the F# compiler services. To implement the wrappers, I first create my implementation of the dynamic invoke operator (<expr>?<ident>) that calls a method or reads a property of any .NET object using Reflection.

The implementation of the dynamic invoke operator is quite interesting, but it would require a whole separate blog post, so we won't discuss it here (although I may write the blog post about it sooner or later and you can find it in the source code). Instead, we'll look at a part of the wrapper for the most important type. The InteractiveChecker type represents an F# compiler service running in the background that can be called to parse a source code (Visual Studio does that when the source changes) and to get information about a parsed F# file. There are two kinds of information that we can get from the background service:

• Untyped parse results (the UntypedParseInfo type) - this type represents information that the F# compiler obtains after running initial parsing of the source code (but before the type-checking has started). It can be used to get the names of identifiers in the file (e.g. to display in navigation bars in Visual Studio) etc. In F# Web Snippets we don't need anything from this type directly. However, we still need to get it before we can call a method to get the second kind of information...
• Typed parse results (the TypeCheckAnswer type) - this type is returned as a result of type-checking (and so it may take some time before we get it back). It can be used to get information about types of various identifiers and expressions in the source code and also to implement auto-completion. Note that we don't get a full AST of the source code, but simply an object that can be used for getting some specific IntelliSense information.

The following snippet shows a part of the wrapper for the InteractiveChecker type. It shows a method used for creating it and two methods that are used to get the untyped and typed information about F# source file. Finally, there is also a method that tries to get both of the information from a cache (which can be used by IDE to get information quickly). As you can see, the dynamic access operator makes the implementation quite easy:

 1: type InteractiveChecker(wrapped:obj) =
 2:   /// Crate a new instance of a wrapped InteractiveChecker object
 3:   static member Create (dirty:FileTypeCheckStateIsDirty) =
 4:     InteractiveChecker(FSharpCompiler.InteractiveChecker?Create(dirty))
 5:     
 6:   /// Parse a source code file, returning a handle that can be used for obtaining navigation 
 7:   /// bar information; to get the full information, call 'TypeCheckSource' method on the result
 8:   member x.UntypedParse(filename:string, source:string, options:CheckOptions) : UntypedParseInfo =
 9:     UntypedParseInfo(wrapped?UntypedParse(filename, source, options.Wrapped))
10: 
11:   /// Typecheck a source code file, returning a handle to the results of the parse including
12:   /// the reconstructed types in the file. Returns 'None' if the background builder is not yet 
13:   /// done preparing the type check results for the antecedent to the file.
14:   member x.TypeCheckSource
15:       ( parsed:UntypedParseInfo, filename:string, fileversion:int, 
16:         source:string, options:CheckOptions, (IsResultObsolete f)) = (...)
17: 
18:   /// Try to get recent type check results for a file. This may arbitrarily refuse to 
19:   /// return any results if the 'InteractiveChecker' would like a chance to recheck the 
20:   /// file, in which case 'UntypedParse' and 'TypeCheckSource' should be called. If the 
21:   /// source of the file has changed the results returned by this function may be out 
22:   /// of date, though may still be usable for generating intellsense menus and information.
23:   member x.TryGetRecentTypeCheckResultsForFile(filename:string, options:CheckOptions) =
24:     let res = wrapped?TryGetRecentTypeCheckResultsForFile(filename, options.Wrapped) : obj
25:     if res = null then None else
26:       let tuple = res?Value
27:       Some(UntypedParseInfo(tuple?Item1), TypeCheckResults(tuple?Item2), int tuple?Item3)F# Web Snippets

The ? operator can be used in various ways. In the Create method, we use it to call a static method of a type (the InteractiveChecker property of the FSharpCompiler type returns a value of type System.Type that represents the type whose static method we want to call). In the UntypedParse method, we call an instance method of the wrapped object. In both of the cases, the parameters are written as a tuple and the dynamic invoke operator extracts elements of the tuple and passes it to the method using Reflection. Note that we have to provide type annotations (for both parameters and return type), because the compiler doesn't have any hints about the types. Without type annotations, everything would be of type obj, but we want to define a typed wrapper. We also wrap the result of method call into other wrapper types (e.g. UntypedParseInfo) that are implemented similarly to InteractiveChecker.

Finally, the TryGetRecentTypeCheckResultsForFile method shows one complication of this approach. When a method returns an F# option type (or a tuple or list), we don't know the exact type of the result (because the actual type argument is some internal type). As a result, we cannot use the option type (or a tuple) in an untyped manner. For option type, we simply check if the value is null, because this is how None is represented and otherwise use the Value property to get the actual value. The situation with tuples is quite similar - we need to dynamically access the properties Item1, Item2, etc. to get the values of the tuple.

Obtaining typed information

Once we have the wrapper, we can use it to parse F# source code and get typed information for IntelliSense. The following code shows a part of a type declaration that implements the source file processing. It starts by creating an instance of InteractiveChecker and constructing object that represents F# compiler options. Then we parse the source code to get an untyped information. This is relatively fast operation that doesn't fail. Getting typed information may fail (the checker needs to process all referenced files first), so we use asynchronous workflow to invoke the operation repeatedly:

 1: type SourceFile(file, source, lines:string[], ?options, ?defines) = 
 2:   (construction of interactive checker and compiler options omitted)
 3: 
 4:   // Run first parsing phase - parse source into AST without type information 
 5:   let untypedInfo = checker.UntypedParse(file, source, opts) 
 6: 
 7:   /// Type-checking takes some time and doesn't return information on the
 8:   /// first call, so this function creates workflow that tries repeatedly
 9:   let rec getTypeCheckInfo() = async {
10:     let obs = IsResultObsolete(fun () -> false)
11:     let info = checker.TypeCheckSource(untypedInfo, file, 0, source, opts, obs) 
12:     match info with
13:     | TypeCheckSucceeded(res) when res.TypeCheckInfo.IsSome ->
14:         let errs = (copying of errors omitted)
15:         return res.TypeCheckInfo.Value, errs
16:     | _ -> 
17:         do! Async.Sleep(500)
18:         return! getTypeCheckInfo() }
19: 
20:   /// Runs type checking and allows specifying a timeout
21:   member x.RunTypeCheck(?timeout) =
22:     Async.RunSynchronously(getTypeCheckInfo(), ?timeout = timeout)F# Web Snippets

The asynchronous workflow implements a simple loop that calls TypeCheckSource. When the result of the call is a value representing successful parsing and it contains a value with the actual result then we return the value (together with a list of errors reported by F# compiler). If the operation fails, we simply wait 500ms and then try again. The workflow can be started from the RunTypeCheck method which allows specification of a timeout and blocks until the operation completes (the application doesn't hang, because the whole processing is done in background).

As you can see from the tool tips in the snippet above, the object that we get as the result has a type TypeCheckInfo. This type exposes various methods that can be used to get information that might be used to implement IntelliSense for F#. In the next section, we'll see how to use it to get tool tip for a specified location in the source.

Processing of source code lines

Once the compiler successfully parses and type-checks our F# source code, we process the entire source code, colorize keywords and literals and generate tool tips for identifiers. The processing is implemented using SourceTokenizer type, which is relatively easy to use. It gives us a list of tokens for each line. We will not look at working with this type (you can find it in the source code) and instead we'll look at post processing of lines that calculates color of tokens and finds tool tips using the TypeCheckInfo object:

 1: let rec processLine island tokens = seq {
 2:   match tokens with 
 3:   | [] -> ()
 4:   | (str, (tok:TokenInformation))::rest ->
 5:     (updating of long identifier information omitted)
 6:     let tip =
 7:       // If we're processing an identfier, see if it has any tool tip
 8:       if (tok.TokenName = "IDENT") then
 9:         let island = island |> List.rev
10:         let pos = (line, tok.LeftColumn + 1)
11:         let tip = checkInfo.GetDataTipText(pos, lines.[line], island, identToken)
12:         match ToolTip.TryCreate(tip) with
13:         | Some(_) as res -> res
14:         | _ when island.Length > 1 -> (alternative attempt omitted)
15:         | _ -> None
16:       elif tok.TokenName.StartsWith("OMIT") then (...)
17:       else None
18: 
19:     // Find color for the current token
20:     let color = 
21:       if tok.TokenName.StartsWith("OMIT") then Some("omitted")
22:       else Colors.colorMap.TryFind(tok.ColorClass)
23:     // Return all information about token and continue
24:     yield { Token = tok; Text = str; Color = color; Tip = tip }
25:     yield! processLine island rest }F# Web Snippets

The snippet shows a function processLine that recursively processes a list of tokens (tokens) on a single line. It also maintains a state with the current long name (named island), which is needed for getting tool tip information. An example of a long name is Collections.List.map - when a mouse pointer is over List, we want to show information about the module, but when the pointer moves to map, we want to show type signature of a function. When getting the IntelliSense information, the F# language service wants a current identifier as an input (so that it doesn't show outdated information based just on the location when the source code changes).

In the recursive processing, we get a TokenInformation value. When the name of the token is IDENT, we want to see if there is any tool tip information for the token, so we use the GetDataTipText method to query the F# language service. Alternatively, the token name may be OMIT - this is a special name generated by our pre-processing that deals with (*[omit:...]*) comments. In this case, we return a tool tip with the collapsed text.

Finally, once we determine the tool tip information, we also want to get a color of the current token. This is quite easy, because the color is reported from the tokenizer (the property name is ColorClass). We first handle our special OMIT token and for all other (standard) tokens, we use a simple lookup into a table that returns a CSS class name. You can find the possible CSS class names in a sample CSS file template (see downloads below).

Summary

In this article, I presented a tool for generating nice HTML snippets from F# source code called F# Web Snippets. The most interesting thing about the tool is that it also generates tool tips with type information for F# identifiers. This way, the readers of your blog get almost the same user experience as when reading F# code in Visual Studio. The additional information makes the code much easier to follow. You can see some examples of formatted F# code in this article and also in my three recent articles on parallel programming. To use the tool, you can either try a (somewhat limited and slow) web based version or download a standalone WinForms application (see links below). If you want to use the tool on your blog, you'll need to add a simple JavaScript and CSS snippets to your blog template (you can get those below too).

The second part of the article also talked briefly about the implementation - the tool invokes F# compiler service under the cover. This is done using Reflection, because the services are internal, but it provides a reasonably comfortable way of implementing colorization or IntelliSense tools for F#. I demonstrated how to initialize the service and how to use it to get tool tip information - hopefully, you can use this information to build some interesting F# tools yourself!

References & source code

• Download the source code from F# Community Samples project at CodePlex (go to Samples/WebSnippets).
• For a more detailed user guide, see the documentation
• Try web based version or download WinForms based version
• Download JavaScript and CSS templates for your blog or web