Prometheus data from PowerShell

At work we have a system which is dependent on fetching email from an on-premises Exchange server via Exchange Web Services (EWS). Apparently there’s a “wontfix” bug in Exchange, where a request to fetch email over this interface that can silently fail. The known workaround is to move email between folders in the affected account, which for some reason causes a state change of some kind that suddenly allows email to be handled again.

As usual when providing a service to someone, we want to be aware of any issue before our customers notice it, and that means monitoring, but how do we know that email is gathering dust in a specific mailbox? The most obvious answer is to start looking at the Get-MailboxFolderStatistics cmdlet. Of course the returned value ItemsInFolder is completely unusable for this purpose, so my method was to log the size of specifically the Inbox folder. Also - of course - an Exchange mailbox can contain multiple folders of type Inbox, so my script assumes - correctly in my environment, at least - that the first one is the actual Inbox that interests us.

The next question I faced was how to present this information back to our monitoring/observability tool. We currently use a monitoring stack based on Grafana, and that gives us some interesting possibilities. As I’ve already been using log monitoring via Loki, my first naïve attempt to get alerts was based on logging my script’s result into a file and letting the Grafana agent scrape it. I would then set up an alerting rule in Grafana if a log line indicated that an Inbox had grown beyond a certain size. As I should have understood before I tried it, each individual log line with information about an oversized Inbox was treated as an individual alert trigger. So that was a no-go.

But Grafana is also able to work with Prometheus metrics. That should be a better fit. Along with the Grafana agent’s ability to parse Prometheus data from text files and report it back in, the problem was reduced to outputting the data obtained with the Get-MailboxFolderStatistics in a suitable format, and finding out where the Grafana agent expects to find the resulting .prom files.

Simplified example:

Add-PSSnapin Microsoft.Exchange.Management.PowerShell.SnapIn
$prometheusingestpath="C:\Program Files\Grafana Agent\textfile_inputs"
$promfile="$prometheusingestpath\inboxsizes.prom"

$inboxes=(Get-Mailbox -Database "MailboxDB")

Set-Content -Path $promfile -Encoding UTF8 -NoNewline -Value ""
Add-Content -Path $promfile -Encoding UTF8 -NoNewline -Value "# HELP Inbox_size Number of bytes currently in inbox.`n"
Add-Content -Path $promfile -Encoding UTF8 -NoNewline -Value "# TYPE Inbox_size gauge`n"

foreach ($inbox in $inboxes) {
    $mbstats=($inbox | Get-MailboxFolderStatistics -FolderScope Inbox)
    $foldersize=$mbstats.FolderSize[0].ToKB()
    Add-Content -Path $promfile -Encoding UTF8 -NoNewline -Value "Inbox_size{mailbox=`"$mailbox`"} $foldersize`n"
}