Intro
For the last year, I’ve been working quite extensively in the dotnet/fsharp codebase.
I’ve picked up a couple of habits and created some helper scripts along the way.
This is some stuff that works for me and it might be insightful for you. Or not.
In this blog post, I’m going over some of my frequently used scripts.
Some are general purpose, others are very specific to the F# compiler codebase.
PowerShell profile
I’ve always been a Windows guy and thus pwsh is my go-to. (pwsh is cross-platform though)
Having multiple terminal windows open is the norm and I frequently add (or alias) functions in my $PROFILE.
I’ll go over some of my frequently used functions in my $PROFILE, in no particular order.
Copy current location
function Copy-CurrentLocation() {
$pwd.Path | clip
}
Set-Alias -Name "ccp" -Value Copy-CurrentLocation
This copied the current location of your terminal session to your clipboard.
Remove recursively and forced
function Remove-ForceRecurse($path) {
Remove-Item $path -Recurse -Force
}
Set-Alias -Name "rmf" -Value Remove-ForceRecurse
Self-explanatory, I use this a lot.
Sync git fork
function Sync-Master(){
git checkout master
git fetch upstream
git rebase upstream/master
git push
}
function Sync-Main(){
git checkout main
git fetch upstream
git rebase upstream/main
git push
}
This is based on the Updating your fork section from the F# dev guide.
Format changed
function Format-Changed(){
$root = git rev-parse --show-toplevel
Push-Location $root
$files = git status --porcelain | Where-Object { ($_.StartsWith(" M") -or $_.StartsWith("AM")) -and (Test-FSharpExtension $_) } | ForEach-Object { $_.substring(3) }
& "dotnet" "fantomas" $files
Pop-Location
}
Format the files you have touched in a git repository with the local Fantomas installation.
Surface area
function Surface-Area() {
$env:TEST_UPDATE_BSL=1
dotnet test tests/FSharp.Compiler.Service.Tests/FSharp.Compiler.Service.Tests.fsproj --filter "SurfaceAreaTest"
dotnet test tests/FSharp.Compiler.Service.Tests/FSharp.Compiler.Service.Tests.fsproj --filter "SurfaceAreaTest"
dotnet test tests/FSharp.Compiler.Service.Tests/FSharp.Compiler.Service.Tests.fsproj --filter "SurfaceAreaTest" -c Release
dotnet test tests/FSharp.Compiler.Service.Tests/FSharp.Compiler.Service.Tests.fsproj --filter "SurfaceAreaTest" -c Release
}
When you change certain bits in the FSharp.Compiler.Service project, you can break the public API of the binary.
For me, this happens frequently when I touch SyntaxTree.fsi.
It is a pain to update the baseline, so I wrapped it into a function that can be called at the repository root.
Kill dotnet
function Kill-DotNet() {
Get-Process -Name "dotnet" | Kill
Get-Process -Name "msbuild" | Kill
}
The F# compiler build scripts can potentially create some dotnet or msbuild ghost processes. This can lock certain dll files and can interrupt a git clean -xdf.
When I had enough of that, I just Kill-DotNet.
Watch tools
function Watch-Tools() {
Push-Location "C:\Users\nojaf\Projects\fantomas-tools"
& dotnet fsi build.fsx -p "Fantomas-Git"
& dotnet fsi build.fsx -p "Watch"
Pop-Location
}
I use Fantomas Tools a lot. You can view the untyped AST in one of the tabs, which is quite useful.
ls but with full paths
function Get-ChildItemFullPath($path) {
Get-ChildItem $path | Select-Object -ExpandProperty FullName
}
Set-Alias -Name "lsf" -Value Get-ChildItemFullPath
Instead of ls my-dir, use lsf my-dir and I get the full paths.
Ready to run the local compiler
function ReadyToRun() {
& dotnet publish .\src\fsc\fscProject\fsc.fsproj -c Release -r win-x64 -p:PublishReadyToRun=true -f net7.0 --no-self-contained
}
This will create the fsc.exe as it will be shipped with the dotnet SDK. Execute in the repository root.
It also creates fsc.dll, which you can plug into your local fsproj file:
<PropertyGroup>
<!-- local compiler, picked up by MSBuild -->
<DotnetFscCompilerPath>C:\Users\nojaf\Projects\fsharp\artifacts\bin\fsc\Release\net7.0\win-x64\fsc.dll</DotnetFscCompilerPath>
</PropertyGroup>
typically combined with some specific compiler flags, these can be set by:
<PropertyGroup>
<DotnetFscCompilerPath>C:\Users\nojaf\Projects\fsharp\artifacts\bin\fsc\Release\net7.0\win-x64\fsc.dll</DotnetFscCompilerPath>
<!-- Special flags, that could only be relevant to my local compiler -->
<!-- Checkout https://github.com/dotnet/fsharp/pull/14494 to learn more about parallel type-checking in compilation -->
<OtherFlags>--test:GraphBasedChecking --test:DumpCheckingGraph --deterministic-</OtherFlags>
</PropertyGroup>
Create F# compiler arguments
function Get-ArgsFile($project) {
& dotnet fsi "C:\Users\nojaf\Projects\scripts\fsharp\args-file.fsx" $project
}
Sometimes you want to just run fsc.exe for an existing project. The Get-ArgsFile function will invoke an F# script that will scrape the arguments from an existing dotnet build run.
args-file.fsx looks like:
#r "nuget: CliWrap, 3.6.0"
#r "nuget: MSBuild.StructuredLogger, 2.1.746"
open System.IO
open Microsoft.Build.Logging.StructuredLogger
open CliWrap
/// Create a text file with the F# compiler arguments scrapped from a binary log file.
/// Run `dotnet build --no-incremental -bl` to create the binlog file.
/// The --no-incremental flag is essential for this scraping code.
let mkCompilerArgsFromBinLog file =
let build = BinaryLog.ReadBuild file
let projectName =
build.Children
|> Seq.choose (
function
| :? Project as p -> Some p.Name
| _ -> None
)
|> Seq.distinct
|> Seq.exactlyOne
let message (fscTask: FscTask) =
fscTask.Children
|> Seq.tryPick (
function
| :? Message as m when m.Text.Contains "fsc" -> Some m.Text
| _ -> None
)
let mutable args = None
build.VisitAllChildren<Task>(fun task ->
match task with
| :? FscTask as fscTask ->
match fscTask.Parent.Parent with
| :? Project as p when p.Name = projectName -> args <- message fscTask
| _ -> ()
| _ -> ()
)
match args with
| None -> printfn "Could not process the binlog file. Did you build using '--no-incremental'?"
| Some args ->
let content =
let idx = args.IndexOf "-o:"
args.Substring(idx)
let directory = FileInfo(file).Directory.FullName
let argsPath =
Path.Combine(directory, $"{Path.GetFileNameWithoutExtension(projectName)}.args.txt")
File.WriteAllText(argsPath, content)
printfn "Wrote %s" argsPath
let project = Array.last fsi.CommandLineArgs
if not (File.Exists project) then
failwithf "%s does not exist" project
if not (project.EndsWith(".fsproj")) then
failwithf "%s is not an fsharp project file" project
Cli
.Wrap("dotnet")
.WithArguments($"build {project} -bl --no-incremental")
.ExecuteAsync()
.Task.Wait()
let binLogFile = Path.Combine(FileInfo(project).DirectoryName, "msbuild.binlog")
mkCompilerArgsFromBinLog binLogFile
The main gist is that you build your project and create a msbinlog file. Open that file and look for the fsc arguments.
Once you have an MyProject-args.txt file you can pass it to your local fsc.exe invocation:
# Create arguments file
Get-ArgsFile .\Fantomas.Core.fsproj
# In PowerShell, mind the `&`
& "C:\Users\nojaf\Projects\fsharp\artifacts\bin\fsc\Release\net7.0\win-x64\publish\fsc.exe" "@./Fantomas.Core.args.txt" --times
Also notice that you need to put an @ character before the path to your args.txt file. (Otherwise, the compiler doesn’t pick this up)
--times is an additional argument that shows you more timings of each phase in the compilation:
Microsoft (R) F# Compiler version 12.5.0.0 for F# 7.0
Copyright (c) Microsoft Corporation. All Rights Reserved.
--------------------------------------------------------------------------------------------------------
|Phase name |Elapsed |Duration| WS(MB)| GC0 | GC1 | GC2 |Handles|Threads|
|------------------------------------|--------|--------|-------|-------|-------|-------|-------|-------|
warning FS0075: The command-line option 'times' is for test purposes only
|Import mscorlib+FSharp.Core | 0.1732| 0.1350| 129| 0| 0| 0| 268| 28|
|Parse inputs | 0.2423| 0.0624| 167| 0| 0| 0| 349| 49|
|Import non-system references | 0.2691| 0.0227| 192| 0| 0| 0| 349| 49|
|Typecheck | 1.4108| 1.1378| 814| 2| 1| 1| 411| 69|
|Typechecked | 1.4151| 0.0003| 814| 0| 0| 0| 411| 69|
|Write Interface File | 1.4186| 0.0000| 814| 0| 0| 0| 411| 69|
|Write XML doc signatures | 1.4239| 0.0019| 814| 0| 0| 0| 411| 69|
|Write XML docs | 1.4294| 0.0021| 814| 0| 0| 0| 411| 69|
|Encode Interface Data | 1.4857| 0.0528| 842| 0| 0| 0| 412| 69|
|Optimizations | 1.7403| 0.2507| 1010| 0| 0| 0| 550| 69|
|Ending Optimizations | 1.7444| 0.0000| 1010| 0| 0| 0| 550| 69|
|Encoding OptData | 1.7543| 0.0063| 1011| 0| 0| 0| 550| 69|
|TAST -> IL | 2.3593| 0.6014| 1343| 0| 0| 0| 552| 69|
|>Write Started | 2.3677| 0.0031| 1345| 0| 0| 0| 554| 69|
|>Module Generation Preparation | 2.3738| 0.0021| 1346| 0| 0| 0| 554| 69|
|>Module Generation Pass 1 | 2.3945| 0.0172| 1354| 0| 0| 0| 554| 69|
|>Module Generation Pass 2 | 2.4642| 0.0659| 1398| 0| 0| 0| 554| 69|
|>Module Generation Pass 3 | 2.4720| 0.0039| 1399| 0| 0| 0| 554| 69|
|>Module Generation Pass 4 | 2.4768| 0.0013| 1400| 0| 0| 0| 554| 69|
|>Finalize Module Generation Results | 2.4814| 0.0003| 1400| 0| 0| 0| 554| 69|
|>Generated Tables and Code | 2.4874| 0.0025| 1400| 0| 0| 0| 554| 69|
|>Layout Header of Tables | 2.4909| 0.0001| 1400| 0| 0| 0| 554| 69|
|>Build String/Blob Address Tables | 2.5071| 0.0114| 1405| 0| 0| 0| 554| 69|
|>Sort Tables | 2.5118| 0.0000| 1405| 0| 0| 0| 554| 69|
|>Write Header of tablebuf | 2.5193| 0.0038| 1407| 0| 0| 0| 554| 69|
|>Write Tables to tablebuf | 2.5229| 0.0000| 1407| 0| 0| 0| 554| 69|
|>Layout Metadata | 2.5267| 0.0000| 1407| 0| 0| 0| 554| 69|
|>Write Metadata Header | 2.5302| 0.0001| 1407| 0| 0| 0| 554| 69|
|>Write Metadata Tables | 2.5336| 0.0001| 1407| 0| 0| 0| 554| 69|
|>Write Metadata Strings | 2.5369| 0.0000| 1407| 0| 0| 0| 554| 69|
|>Write Metadata User Strings | 2.5406| 0.0002| 1407| 0| 0| 0| 554| 69|
|>Write Blob Stream | 2.5450| 0.0010| 1408| 0| 0| 0| 554| 69|
|>Fixup Metadata | 2.5487| 0.0002| 1408| 0| 0| 0| 554| 69|
|>Generated IL and metadata | 2.5536| 0.0016| 1408| 0| 0| 0| 554| 69|
|>Layout image | 2.5616| 0.0040| 1409| 0| 0| 0| 555| 69|
|>Writing Image | 2.5663| 0.0010| 1409| 0| 0| 0| 554| 69|
|>Signing Image | 2.5697| 0.0000| 1409| 0| 0| 0| 554| 69|
|>Write Started | 2.5746| 0.0004| 1410| 0| 0| 0| 555| 69|
|>Module Generation Preparation | 2.5798| 0.0018| 1411| 0| 0| 0| 555| 69|
|>Module Generation Pass 1 | 2.5963| 0.0131| 1417| 0| 0| 0| 555| 69|
|>Module Generation Pass 2 | 2.7748| 0.1748| 1528| 0| 0| 0| 555| 69|
|>Module Generation Pass 3 | 2.7818| 0.0028| 1529| 0| 0| 0| 555| 69|
|>Module Generation Pass 4 | 2.7867| 0.0014| 1530| 0| 0| 0| 555| 69|
|>Finalize Module Generation Results | 2.7910| 0.0001| 1531| 0| 0| 0| 555| 69|
|>Generated Tables and Code | 2.7947| 0.0002| 1531| 0| 0| 0| 555| 69|
|>Layout Header of Tables | 2.7983| 0.0001| 1531| 0| 0| 0| 555| 69|
|>Build String/Blob Address Tables | 2.8131| 0.0115| 1535| 0| 0| 0| 555| 69|
|>Sort Tables | 2.8169| 0.0000| 1535| 0| 0| 0| 555| 69|
|>Write Header of tablebuf | 2.8244| 0.0039| 1536| 0| 0| 0| 555| 69|
|>Write Tables to tablebuf | 2.8281| 0.0000| 1536| 0| 0| 0| 555| 69|
|>Layout Metadata | 2.8319| 0.0000| 1536| 0| 0| 0| 555| 69|
|>Write Metadata Header | 2.8353| 0.0001| 1536| 0| 0| 0| 555| 69|
|>Write Metadata Tables | 2.8397| 0.0001| 1536| 0| 0| 0| 555| 69|
|>Write Metadata Strings | 2.8431| 0.0000| 1536| 0| 0| 0| 555| 69|
|>Write Metadata User Strings | 2.8470| 0.0006| 1537| 0| 0| 0| 555| 69|
|>Write Blob Stream | 2.8515| 0.0012| 1538| 0| 0| 0| 555| 69|
|>Fixup Metadata | 2.8556| 0.0000| 1539| 0| 0| 0| 555| 69|
|>Generated IL and metadata | 2.8718| 0.0130| 1540| 0| 0| 0| 555| 69|
|>PDB: Defined 31 documents | 2.8809| 0.0053| 1540| 0| 0| 0| 555| 69|
|>PDB: Sorted 9932 methods | 2.9174| 0.0312| 1550| 0| 0| 0| 555| 69|
|>PDB: Created | 2.9288| 0.0076| 1550| 0| 0| 0| 555| 69|
|>Layout image | 2.9344| 0.0015| 1551| 0| 0| 0| 555| 69|
|>Writing Image | 2.9382| 0.0003| 1551| 0| 0| 0| 554| 69|
|>Generate PDB Info | 2.9422| 0.0005| 1551| 0| 0| 0| 555| 69|
|>Finalize PDB | 2.9455| 0.0000| 1551| 0| 0| 0| 555| 69|
|>Signing Image | 2.9495| 0.0002| 1551| 0| 0| 0| 554| 69|
|Write .NET Binary | 2.9529| 0.5896| 1551| 0| 0| 0| 554| 69|
--------------------------------------------------------------------------------------------------------
Get the mvid
function Get-Mvid($dll) {
& dotnet fsi "C:\Users\nojaf\Projects\scripts\fsharp\mvid-reader.fsx" $dll
}
When working with reference assemblies, it is useful to inspect the mvid of a binary.
mvid-reader.fsx:
#r "nuget: System.Reflection.Metadata"
open System
open System.IO
open System.Reflection.Metadata
open System.Reflection.PortableExecutable
let getMvid refDll =
use embeddedReader = new PEReader(File.OpenRead refDll)
let sourceReader = embeddedReader.GetMetadataReader()
let loc = sourceReader.GetModuleDefinition().Mvid
let mvid = sourceReader.GetGuid(loc)
printfn "%s at %s" (mvid.ToString()) (DateTime.Now.ToString())
let dll : string = Array.last fsi.CommandLineArgs
if not (dll.EndsWith(".dll")) then
failwithf "Expected %s to have .dll extension" dll
if not (File.Exists dll) then
failwithf "%s does not exist on disk" dll
getMvid dll
Get-Mvid .\bin\Debug\netstandard2.0\Fantomas.Core.dll
4e838868-adae-c4d5-445f-480b444a96f7 at 2/2/2023 12:00:56 PM
Closing thoughts
Again, all this stuff works for me, it might not be for you.
I felt like sharing, that’s all folks.
Until next time,
Florian
Photo by Jack Prommel on Unsplash