Testing Microsoft Foundry Local with C#

I today had the chance to test Microsoft's new Foundry Local AI platform that is able to work without any cloud connections. The first version went GA (Globally Available) in April, and since then, I've looked at for a suitable time to test it.

Firstly, a definition of Foundry Local. This is what Microsoft says:

"Foundry Local is an end-to-end local AI solution in a compact package that is small enough to bundle directly inside your application installer without meaningfully impacting download size. The small size with zero dependencies lets you ship a fully self-contained AI-powered app the same way you would ship any other desktop or edge application, and keeps your CI/CD artifacts lean."

Now, how do you use this thing? To install and get started, you need nothing more than one NuGet package, and then another if you want to enable logging. The NuGet package you need is Microsoft.AI.Foundry.Local.

So, you can install it using Visual Studio's NuGet Package Manager, or from the command with a command like this:

dotnet add package Microsoft.AI.Foundry.Local

This does not download any models yet, just installs the runtime. For pure Windows-only use, there's a Windows-and-GPU optimized version: Microsoft.AI.Foundry.Local.WinML.

Here's a simple C# console application with .NET 10 that can send a chat message to a local AI model, and then get the response back. My code is an edit from Microsoft's own tutorial, but apparently NuGet package interfaces have slightly changed since Microsoft published their code. My code uses the version 1.24, published about two weeks ago. I've also added a little bit of logging to the console, so be sure to install the package Microsoft.Extensions.Logging.Console as well.

using Microsoft.AI.Foundry.Local;
using Microsoft.Extensions.Logging;
using Betalgo.Ranul.OpenAI.ObjectModels.RequestModels;
using Betalgo.Ranul.OpenAI.ObjectModels.ResponseModels;

CancellationToken ct = new();
Configuration config = new()
{
    AppName = "foundry_local_samples",
    LogLevel = Microsoft.AI.Foundry.Local.LogLevel.Information
};

// Initialize the foundry with logging to console.
ILogger logger = LoggerFactory.Create(builder => builder.AddConsole()).CreateLogger("FoundryLocalSamples");
await FoundryLocalManager.CreateAsync(config, logger);
FoundryLocalManager mgr = FoundryLocalManager.Instance;

// Discover available execution providers and their registration status.
EpInfo[] eps = mgr.DiscoverEps();
int maxNameLen = 30;
Console.WriteLine("Available execution providers:");
Console.WriteLine($"  {"Name".PadRight(maxNameLen)}  Registered");
Console.WriteLine($"  {new string('─', maxNameLen)}  {"──────────"}");
foreach (EpInfo ep in eps)
{
    Console.WriteLine($"  {ep.Name.PadRight(maxNameLen)}  {ep.IsRegistered}");
}

// Download and register all execution providers with per-EP progress.
// EP packages include dependencies and may be large.
// Download is only required again if a new version of the EP is released.
// For cross platform builds there is no dynamic EP download and this will return immediately.
Console.WriteLine("\nDownloading execution providers:");
if (eps.Length > 0)
{
    string currentEp = "";
    await mgr.DownloadAndRegisterEpsAsync((epName, percent) =>
    {
        if (epName != currentEp)
        {
            if (currentEp != "")
            {
                Console.WriteLine();
            }
            currentEp = epName;
        }
        Console.Write($"\r  {epName.PadRight(maxNameLen)}  {percent,6:F1}%");
    });
    Console.WriteLine();
}
else
{
    Console.WriteLine("No execution providers to download.");
}


// Get the model catalog
ICatalog catalog = await mgr.GetCatalogAsync();

// Get a model using an alias.
IModel model = await catalog.GetModelAsync("qwen2.5-0.5b") ?? throw new Exception("Model not found");

// Download the model (the method skips download if already cached)
await model.DownloadAsync(progress =>
{
    Console.Write($"\rDownloading model: {progress:F2}%");
    if (progress >= 100f)
    {
        Console.WriteLine();
    }
});

// Load the model
Console.Write($"Loading model {model.Id}...");
await model.LoadAsync();
Console.WriteLine("Loading done.");
Console.WriteLine();
Console.WriteLine("====================================");
Console.WriteLine();

// Get a chat client
OpenAIChatClient chatClient = await model.GetChatClientAsync();

// Create a chat message
List messages =
[
    new ChatMessage { Role = "user", Content = "Why is the sky blue?" }
];
Console.WriteLine($"Asked the AI model: {messages[0].Content}");

// Get a streaming chat completion response
Console.WriteLine("Chat completion response:");
IAsyncEnumerable streamingResponse = chatClient.CompleteChatStreamingAsync(messages, ct);
await foreach (ChatCompletionCreateResponse chunk in streamingResponse)
{
    Console.Write(chunk.Choices[0].Message.Content);
    Console.Out.Flush();
}
Console.WriteLine();
Console.WriteLine("====================================");
Console.WriteLine();

// Tidy up - unload the model
await model.UnloadAsync();

When you run this code, it will download the model, which takes maybe a minute. In my case, the model was qwen2.5-0.5b. So, to the code's question of "Why is the sky blue?" I got the following answer:

"Good question! The sun's rays create the first light of day and cause it to glow in colors that we see today. Our eyes detect this light and interpret its color to determine what our visual receptors are seeing.

The most important reason why the sky can appear blue is due to scattering of sunlight. When sunlight passes through the Earth's atmosphere at 50 miles per hour, it collides with air molecules which scatter and diffract the sunlight.

Each scattering incident on an object causes less than one part of the incoming solar radiation to be reflected back towards the observer."

If you look closely, you will see a little physics problem: light doesn't travel at 50 miles per hour... So, a little more physics training is needed for this model, but that of course is not a Foundry or C# coding problem, just a model problem.

Happy hacking!