The Case of Multicast Message Loss (again)

I have written about trouble-shooting multicast issues several times before, but multicast is a gift that keeps on giving.

The Problem

The application in question would miss a substantial number of messages. A trace on the connected switch showed that all packets had been put on the wire. Tracing with Microsoft Message Analyzer on the machine showed these same messages missing, so our application probably was not at fault. Additionally, it did work on other machines just fine.

The Analysis

So I went back to the drawing board, reviewed and double-checked everything I had learned about high-throughput multicast messaging and

  • set appropriately large socket receive buffer sizes in the multicast message receiving application,
  • activated all TCP/UDP Rx/Tx offloads in the NIC configuration,
  • activated receive side scaling (RSS) and picked the maximum number of RSS queues,
  • set the NICs’ receive buffers to their maximum values,
  • disabled flow-control,
  • turned off all power-saving features in NIC and operating system,
  • used the most aggressive interrupt moderation setting, and
  • updated the NIC driver top the latest version.

In order to check NIC settings, I keep the following PowerShell snippet handy. It gives me the current, all valid and maximum values for each parameter of each NIC in the NIC team. And it doesn’t even require admin privileges.

(Get-NetLbfoTeam "MyNicTeam").Members | Get-NetAdapterAdvancedProperty | ft DisplayName,DisplayValue,ValidDisplayValues,NumericParameterMaxValue

Other useful sources:

But even after ensuring all parameter were at their optimal values, the problems persisted. So I spent some time setting up perfmon with these network-related performance counters.

One counter immediately jumped out: Packets Received Discarded was pretty much constant on the machines our application worked on. But on the machines where we noticed packet loss, this number was growing fast.

This Technet blog post has a good explanation of that performance counter and tips on how to gather it from multiple machines remotely using PowerShell.

The Cause

It turns out the machines experiencing multicast message loss had substantially smaller receive buffers (512) compared to the machines that were working fine (2048 and 4096). Even though our setup script had correctly configured the maximum value for this parameter, that was apparently still insufficient.

So we ended up upgrading the NICs on the cluster experiencing the problems and the multicast messages loss went away.

Upon closer examination we also noticed TCP packet loss while our multicast application was running. But because resends were mostly successful, only introducing small delay this had gone unnoticed before.


The Case of a Delphi Application hanging in ExitProcess

A colleague came to me with a Delphi application that would not shut down, but just hang. The application in question had been refactored such that one module was extracted into a DLL to be reused in another application. When this extracted module was loaded into the application and the application was closed, it would hang. If the module was not loaded, the application would shut down normally.


Debugging the application was initially unsuccessful. Stepping through our code we verified that the shutdown logic executed normally with destructors running as expected. Interestingly, it was not possible to break into the application once it had become unresponsive. Trying to pause the hung program from within the IDE would simply cause the IDE to hang as well.

Thus we used Process Explorer instead to look at the application’s threads and their callstacks.

There we saw that there was one thread stuck on a call to WaitForSingleObject which originated in our DLL code. Higher up the callstack was ExitProcess. I looked at the documentation for ExitProcess to see look for ways in which it could deadlock. One sentence looked promising: “If one of the terminated threads in the process holds a lock and the DLL detach code in one of the loaded DLLs attempts to acquire the same lock, then calling ExitProcess results in a deadlock.” But since there was only one thread, this could not be it.

Looking next at what happens exactly inside ExitProcess, two other things jumped at me:

  • All threads are terminated (except for the one calling ExitProcess).
  • All DLLs are unloaded.


It turns out, the initial analysis that “the shutdown logic executed normally” was wrong. One of the shared units compiled into the DLL (through several layers of indirections), had a finalization section. In this finalization section, a background thread that had been created in the corresponding initialization section, was being destroyed. As part of the destructor code, the thread class was waiting for an event that was set when the thread had stopped executing.

Holzscheite (4)

This finalization section was running as part of the “all DLLs are unloaded” step by ExitProcess. Unfortunately, all threads (including the one created in the initialization) had already been terminated. I am not quite sure how that was accomplished, but it apparently circumvented the normal thread termination logic which set the event that the thread had stopped executing.

This is different for code in the main application, where finalization sections are run while the application is still in working order.


Instead of waiting for the thread to set its “stopped executing” event, I wait on the thread handle to check if the thread was even there to set the event. When run from the DLL’s finalization section, this detects the thread’s absence and just returns.

The Case of Outlook not downloading RSS content because of a problem connecting to the server

The Problem

I had been using Outlook to subscribe to RSS feeds for many years. Mostly because I like to have my email and news in one place with the ability to add them to my Outlook to-do list for follow-up.

Over time, however, I noticed that one feed after the other stopped updating. Even feeds from Microsoft sites such as TechNet and MSDN which I had assumed simply had to be compatible with Microsoft’s own RSS reader. Trying to remove and then re-add them gave the following error:


I eventually found out that all feeds that no longer updated had been switched from HTTP to HTTPS, which seems to be unsupported by Outlook.

The Cause

This support document describes the scenarios in which Outlook supports authenticated RSS feeds. Basically, support is extremely limited. Normally, HTTPS is not supported and the workaround using Internet Explorer’s Common Feed List did not work for me.

Initially, Internet Explorer would download the RSS feed and they would show up as unread items in Outlook, but after the initial download I wouldn’t get any new items in Outlook, even though they did show up in Internet Explorer.

The Solution

Searching online for a solution did not turn up anything. Either people described workarounds that did not work for me, or recommended simply abandoning Outlook in favor of some other feed reader.

So I implemented my own solution and wrote a very simple proxy server that runs a local web server via HTTP that fetches the requested feed via HTTPS from the original source. By configuring Outlook to use this proxy, I now receive updates for feeds in Outlook even when the source uses HTTPS.

This RSS-via-HTTP Proxy runs as a service in the background. Its source code and some instructions on how to configure it are available on GitHub.

The Case of Multicast Message Loss on Windows Server 2012 R2

I have worked quite a bit with applications using UDP/multicast messaging recently. And I’ve run into a few issues along the way, such as multicast messages not being received on a Windows Fail-over Cluster.

So by now I have a solid checklist of things to configure on our servers and ensure in our applications that consume multicast messages to make sure everything runs smoothly and message loss is kept at acceptable levels. Yet, on our latest Windows Server 2012 R2 machines I had applications experiencing serious datagram loss as the amount of network traffic (in general, not just multicast) on the machine increased increased.

I researched the problem online and got the tips you’d expect: get latest NIC drivers, increase NIC receive buffer sizes, turn on offloads, turn on receive-side scaling, fine-tune receive-side scaling, increase socket buffer sizes etc. Of course, I had already tried all those things, and none of them had worked.

Solution: Exempting multicast traffic from Base Filtering Engine

Eventually I found this support document: Datagram loss when you run a multicast receiver application in Windows 8 and in Windows Server 2012. The problem description matched perfectly with what I was seeing on our server. Unfortunately, the document describes an issue in Windows Server 2012 and the hotfix available there cannot be installed on Windows Server 2012 R2. Fortunately, it doesn’t have to be. You can just set the registry key and the Base Filtering Engine supports it out of the box.

New-ItemProperty HKLM:\System\CurrentControlSet\services\Tcpip\Parameters\ -Name UdpExemptPortRange -Value "XXXX-YYYY" -PropertyType MultiString -Force

I haven’t found any official documentation on this, and prior to this post, there were just four results when searching Google for UdpExemptPortRange. But it works as far as I can tell.

The Case of the Not Enough Storage Error in a Delphi Application

The Problem

After a few weeks Delphi applications running on Windows Server (2008 or later) will fail to start or create new windows with error 8: Not enough storage is available to process this command.

Possible Cause 1: Desktop Heap Exhaustion

Session 0 which service applications are running in gets allocated substantially less desktop heap than interactive sessions, so you might be running out of it. There is a system event log entry with event ID 243 or 244 when the desktop heap gets exhausted.

By increasing the values in the SharedSection part of the data in registry value “Windows” in key “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems”, you can allocate more to the service session, as described in this Microsoft support article or this Stack Overflow answer.

Note that Microsoft’s Desktop Heap Monitor tool no longer works on current versions of Windows Server.

Possible Cause 2: Atom Leak

Applications compiled with Delphi XE2 or earlier call RegisterWindowMessage in the initialization section of Controls.pas to allocate a unique window message ID. This takes up one slot in the session’s atom table. Unfortunately, there are only around 16 000 slots in the atom table. With no way to unregister a window message, you will eventually run out of them. It has been said that on earlier versions of Windows, the counter wrapped around so you never ran out as slots got overwritten/reused.

For more details, see this bug report on Embarcadero’s Quality Central and this Stack Overflow answer.

I fixed this by editing the Controls.pas unit replacing this line

RM_GetObjectInstance := RegisterWindowMessage(PChar(ControlAtomString));

with this one

RM_GetObjectInstance := RegisterWindowMessage(PChar('DelphiRM_GetObjectInstance'));

This way all Delphi applications share the same atom instead of creating their own. There is also a more involved fix by Andreas Hausladen, but this solution sufficed for me.

Analyzing the Atom Leak

Jordi Corbilla has written the super useful Atom Table Monitor which dumps entries from the atom table. I made a C# version of it, the Atom Table Dumper, because (a) I didn’t want to diagnose issues with my Delphi applications using another Delphi application that had the same issue and (b) I needed an easier to read and process output format.

Note that your analysis tool has to run in the same session as the application getting the error.

Further note that not all of the atoms listed under RegisteredWindowMessage were necessarily allocated by RegisterWindowsMessage. RegisterClass is another function that allocates atoms in this table. However, as the documentation points out “All window classes that an application registers are unregistered when it terminates.“ so having many of them (temporarily) isn’t necessarily a problem.

The Case of Running Turbo Pascal 7

After getting Delphi 2007 to run on Windows 10 and seeing some of my earlier Delphi programming again, I was intrigued to try if I couldn’t get Turbo Pascal 7 working too. This was the development environment that I first started programming with in school in 2000. Of course, Windows 10 on an x64 machine has no way of running 16 bit DOS applications, so I needed a different solution.

Installing on hardware: fail

Turbo Pascal Floppy DisksI do in fact own the full set of Borland Pascal 7 installation floppy disks as well as the manuals. Of course, I can’t install the software from them anymore, as I don’t have a floppy drive. Also, I doubt, they would even be readable anymore. But I made a backup of the disks’ contents and kept it was as a ZIP file.

I also still have my 1GHz Pentium III powered laptop running Windows ME from 2001. I remember fondly many a night spent sitting on that laptop working on some Turbo Pascal project I had given myself.

Unfortunately, the installer would not work as it was complaining about “insufficient memory”. This is funny as the 256 MB of RAM my laptop has were substantially more than computers had when Turbo Pascal 7 was released.

Turbo Pascal BooksMaybe the problem wasn’t that there was too little memory, but rather  too much of it. For instance, there is an integer overflow error in the Borland Database Engine causing it to report insufficient disk space when it encounters free space close to a multiple of 2GB.

Hyper-V: better

So next, I tried setting up a virtual machine in Hyper-V. This would allow me to tweak the amount of memory the installer was seeing.

While in college I had free access to Virtual PC 2004 and so ended up with one virtual machine for each version of Windows I had owned up to that point. Unfortunately, all of the machines running Windows 9x/ME would fail to boot in Hyper-V. The earliest version to run was Windows XP.

Unfortunately, Windows XP demanded I activate before I was able to even log in. But without an opportunity to install drivers for the virtual network adaptor, it wasn’t able to get an internet connection and for some reason the phone activation failed as well.

I was, however, able to boot Windows into safe mode with command prompt, allowing me to execute the following command to “rearm” the activation and buy me some more time to get the registration issues sorted out, because of course, this was a licensed version of Windows.

C:\windows\system32\rundll32.exe syssetup,SetupOobeBnk

After I did that and could log on normally, I still had trouble with getting a network connection between host and VM. So in order to get the Turbo Pascal installer onto the VM, I had to copy it to the VM’s virtual hard drive (VHD). Luckily, Windows 10 makes it really easy to work with VHD files: double-clicking mounts them as a drive and you can copy any files you want onto them just like with a real disk.

Now I had all the pieces on the VM and was ready to install. However, no matter how much or how little memory I assigned to the VM, the installer’s unzip component would always complain about it being insufficient. I considered writing a replacement for the unzip.exe which would not have this limitation. But it didn’t come to that, as I remembered having an installed version of Turbo Pascal 7 on a VM that I had cloned from the PC I used when I first started programming. I copied it to this VM, and boom there I was looking at white and yellow text on blue background: the Turbo Pascal IDE.


A few more steps

What you see above is part of the source code to TaRech, an RPN-based calculator that I wrote as my first project while learning Turbo Pascal.

Compilation, however, failed, because some essential components were still missing: object files for the graphics components (EGAVGA and the fonts used by my application). I found a solution in an old physics textbook on Google books. This command line uses the binobj.exe utility from Turbo Pascal’s bin folder to convert bgi and chr files into obj files:

binobj ..\BGI\egavga.bgi ..\OBJ\egavga.obj EGAVGADriverProc

The next problem was that once compiled my program failed to run, crashing during startup with error 200: division by zero. I learned that one of the central pieces of code, the unit crt, has a bug that causes this error when running on very fast computers (i.e. clock speeds greater than 200MHz). Luckily, that article pointed me to a fix published by c’t, a German PC magazine, in an issue of theirs from 1997. That code is still available from their FTP server.


With all these issues sorted out, I was finally able to see my application running again.; including what at the time seemed like fancy transitions; Unfortunately, the timing in the virtualized environment is off and they were now running substantially slower.





TaRech (named after “Taschen-Rechner”, the German word for calculator) supports basic arithmetic as well as trig and other functions. You can choose a fixed number of decimal places or switch to scientific notation. The boxes on the right show the contents of the calculator’s four registers.




It took me a few hours, but eventually I was really happy to see code and program again which I had been really proud of at the time. That code is available on GitHub.

Looking back at it now, 15 years later, I am a bit ashamed of the quality of that code, however: there are no useful comments, almost all all variables have global scope, values for positioning elements on screen are hard-coded, identifiers have terrible names and I apparently couldn’t make up my mind whether to choose English or German.

But no matter whether I look at code I wrote 15 years ago or code I wrote 15 months ago, it’s all terrible. I think this is a good sign, as it means that I keep improving and honing my skills.

The Case of Installing Delphi 2006 on Windows 10

tl;dr I did not get Delphi 2006 to install on Windows 10. Delphi 2007 with the right patches seems to work fine, though.

Getting started

The first thing the Delphi 2006 installer checks is the presence of the .NET Framework 1.1, as parts of the IDE are implemented in .NET (J# I believe).

Unfortunately, according to MSDN Windows 7 is the latest version of Windows that the .NET Framework 1.1 is supported on.

Note that it says, Windows 8.1 isn’t supported either. However, I found several people claiming they got .NET 1.1 to install on Windows 8.1. Figuring that Windows 10 was internally close to Windows 8.1 I set out to install .NET 1.1 on Windows 10.

The installation seemed to be going fine for a while, until eventually the installer ran into an error and everything was rolled back. I checked the application event log and saw a large number of warnings with Source “MsiInstaller” such as this one:

Product: Microsoft .NET Framework 1.1. The application tried to modify a protected Windows registry key \Software\Classes\CLSID\…

Trying to figure it out

Interestingly, the error came pretty late in the installation process, at which point the .NET Framework installation directory C:\Windows\Microsoft.NET\Framework\v1.1.4322\ already contained many or most of the framework’s files.

I figured this might be just enough for Delphi to install and run. So I just killed the installer process as soon as it had hit the error but before it got a chance to begin the rollback. It seems, though, that Windows Installer was prepared for this, as the rollback was executed automatically even after the installer process had been killed.

I imagine Windows Installer has a process in background waiting on the installer process handle to undo its work in case of a crash.

So I thought what if I just suspended the installer? This seemed to do the trick, as no rollback was taking place. Of course, this would not be a permanent solution, but at this point I just wanted to figure out how this thing was operating. With the installer suspended and the rollback staved off, I wanted to compile a Hello World program.

And lo and behold using the command line compiler csc.exe from the aforementioned framework directory worked. Executing the resulting program, however, did not, as it crashed immediately. I guess there is more to the .NET framework than was contained in that directory. Too bad.

At this point, I gave up on running Delphi 2006 on Windows 10. Using the .NET Framework cleanup tool I found recommended in one post seemed to risky to me, given all the caveats regarding the impact on newer .NET frameworks in this MSDN blog post.

Delphi 2007 instead

But lucky for me, I also own a copy of Delphi 2007 which no longer needs .NET 1.1. So until .NET 2.0 isn’t supported anymore, I have an alternative.

And Delphi 2006 and 2007 are binary compatible, hopefully making the switch relatively painless.

Since there had been a number of updates since I bought Delphi 2007, I installed with the CodeGear RAD Studio 2007 ISO (Dec 2007) instead of the original install disk.

That ISO is 4.2GB, but alas I don’t have Delphi 2007 R2, which would entitle me to use this smaller web installer instead.

I also got the Help Update 4 and the unofficial debugger patch based on this helpful response on Stack Overflow.

Not mentioned there is Andreas Hausladen’s IdeFixPack 4.4 for Delphi 2007 on Windows 10 which came highly recommended, so I installed that one, too.

Why do this?

So why did I go through all this trouble of installing a ten year old IDE?

I had been interested in distributed version control for a while, but only read about it and never actually used it. So when I came across a Delphi 2006 project that I had begun in school and improved upon while in college, I wanted to take it as an opportunity to try out GitHub and keep the project there.

With Delphi 2007 installed now, I can hopefully get started on making this project ready to be shared. So far, though, I’ve only registered the personalnexus username on GitHub. Stay tuned.