The Case of Multicast Message Loss on Windows Server 2012 R2

I have worked quite a bit with applications using UDP/multicast messaging recently. And I’ve run into a few issues along the way, such as multicast messages not being received on a Windows Fail-over Cluster.

So by now I have a solid checklist of things to configure on our servers and ensure in our applications that consume multicast messages to make sure everything runs smoothly and message loss is kept at acceptable levels. Yet, on our latest Windows Server 2012 R2 machines I had applications experiencing serious datagram loss as the amount of network traffic (in general, not just multicast) on the machine increased increased.

I researched the problem online and got the tips you’d expect: get latest NIC drivers, increase NIC receive buffer sizes, turn on offloads, turn on receive-side scaling, fine-tune receive-side scaling, increase socket buffer sizes etc. Of course, I had already tried all those things, and none of them had worked.

Solution: Exempting multicast traffic from Base Filtering Engine

Eventually I found this support document: Datagram loss when you run a multicast receiver application in Windows 8 and in Windows Server 2012. The problem description matched perfectly with what I was seeing on our server. Unfortunately, the document describes an issue in Windows Server 2012 and the hotfix available there cannot be installed on Windows Server 2012 R2. Fortunately, it doesn’t have to be. You can just set the registry key and the Base Filtering Engine supports it out of the box.

New-ItemProperty HKLM:\System\CurrentControlSet\services\Tcpip\Parameters\ -Name UdpExemptPortRange -Value "XXXX-YYYY" -PropertyType MultiString -Force

I haven’t found any official documentation on this, and prior to this post, there were just four results when searching Google for UdpExemptPortRange. But it works as far as I can tell.

Lighting Design

The sub-title of this blog is “Travel and Technology” my two passions in life. This post, however, is about my third passion: theater. Specifically: lighting design.

I’ve been involved with drama groups since the 6th grade, when I joined the group at school as a stagehand. Many years later, I’m still working with some of these people, now mostly focusing on lighting design aspects.

This year has been the first time that I had an all-white stage to light. A white stage comes with a few challenges, but it is also a great opportunity to experiment with colors.

In this production the light design was very simple – only half a dozen different cues – but still accomplished setting the right moods for the show.

Full Stage

Weißes Frontlicht

Center Sidelight

Seitenlicht

Narrow Backlight for Intimate Scenes

Gasse Hinten

Low Green Backlight for Mysterious Scenes

Grünes Gegenlicht

Bright, Colorful Band Scenes

Bunt

Solo Musician

Solo Links

Osthaus-Museum Hagen

Last year when M was home for Christmas, we saw an exhibition at the Museum Folkwang in Essen. This year, we went to the Osthaus-Museum in Hagen.

The exhibition was called “Photorealistic Paintings” and showed off some incredible work with magnificent attention to detail.

Being situated in an old villa, the exhibition space itself is also quite beautiful.

Brunnen von oben

Brunnen von vorne

Bibliothek

I particularly enjoyed the Spieltisch 1987 (“play table”) containing building blocks in various colors, some with letters, that let’s you create your own art.

Spieltisch

111 Places to see in Düsseldorf, Part 2

Five years ago, I got the book 111 Düsseldorfer Orte, die man gesehen haben muss (English: 111 Places to see in Düsseldorf). I posted an article detailing to which of these places I had already been at the time.

Today I decided to revisit that list and travel around town to see a few of those I hadn’t been to before. Pictures from today’s trip are at the end of this post. I’ve also created the hashtag #111DüsseldorferOrte on Instagram to tag pictures I take whenever I visit a place from the list.

Below is the list where I have marked all the places that as of 07-Nov-2016

  1. I have seen: 75 (up from 46 in 2011),
  2. I know or have at least heard of but haven’t seen: 17 (down from 24 in 2011),
  3. I have only read about in the book: 19 (down from 41 in 2011).
1 Ackerstraße Been there
2 Alte Golzheimer Friedhof Been there
3 Apsis von St. Margareta Know of
4 Aschlökchen Know of
5 Atelierhaus Walzwerkstraße Never heard of
6 Berger Kirche Been there
7 Bilker Sternwarte Been there
8 Blumenlädchen am Barbarossaplatz Been there
9 Blutkapelle Never heard of
10 Bolkerstraße Been there
11 Bootshafen am Unterbacher See Know of
12 Botanischer Garten Been there
13 Burghof-Mauer Been there
14 Carlsplatz Been there
15 Carsch-Haus Been there
16 Düsseldorfer Malerschule im museum kunst palast Been there
17 Düsselquelle Know of
18 Ecke Tußmann- und Moltkestraße Been there
19 Eisstadion an der Brehmstraße Been there
20 Ende der Zollstraße Been there
21 Engländerwiese Been there
22 Esskastanienallee Been there
23 Fähre nach Zons Been there
24 Filmmuseum und die Black Box Been there
25 Flinger Broich Know of
26 Frauensteine Never heard of
27 Fußgängerbrücke im Medienhafen Been there
28 Galerie Simonis Never heard of
29 GAP 15 Been there
30 Gehry-Bauten Been there
31 Gnadenauge der Maxkirche Never heard of
32 Goethe-Museum im Schloss Jägerhof Been there
33 Goldene Brücke im Hofgarten Been there
34 Grabbeplatz Been there
35 Grabmal Wilhelm des Reichen Been there
36 Greifweg Never heard of
37 Hand von St. Suitbertus Been there
38 Haus der Ey Been there
39 Haus Zum Neuen Schelfisch Never heard of
40 Heine Haus Been there
41 Hetjens-Museum Been there
42 Am Heyebad Know of
43 Himmelgeister Kastanie Been there
44 Hungerturm Been there
45 Hunsrückstraße 16 Know of
46 Internationale Bushaltestelle Been there
47 Japan Store Never heard of
48 “Junges Rheinland” im Stadtmuseum Never heard of
49 K20 Been there
50 K21 Been there
51 Kaiserpfalz Been there
52 Kaiserswerther Deich Been there
53 Karlrobert-Kreiten-Stolperstein Know of
54 Killepitsch-Stube Know of
55 KIT Been there
56 Kölner Straße Been there
57 Konrad Fischer Galerie Know of
58 Konsortium Never heard of
59 Kreuzherreneck Know of
60 Kreuzstation am Lindenplatz Been there
61 Kunstakademie Know of
62 Kunsthalle Been there
63 Lantz’scher Park Been there
64 Lassalle-Gedächtnisstätte Been there
65 Malkasten-Park Know of
66 marokkanische Viertel Know of
67 Mausoleum in der Andreaskirche Been there
68 Münstertherme Been there
69 Napoleonsecke im Schiffchen Never heard of
70 Neandertal Been there
71 Nitribitt-Grab Never heard of
72 NRW-Forum im Ehrenhof Been there
73 Oberbilker Markt Been there
74 Park am Spee’schen Graben Been there
75 Rahmenmuseum Been there
76 Ratinger Straße Been there
77 Ratinger Tor Been there
78 Raum 20 Never heard of
79 Reeser Platz Been there
80 Reinraum Never heard of
81 Reiterstandbild auf dem Marktplatz Been there
82 Rheinuferpromenade Been there
83 Rheinwiesen Been there
84 Rochuskirche Been there
85 Salon des Amateurs Know of
86 Schloss in Benrath Been there
87 Schloss Mickeln Been there
88 Schlosstreppe Been there
89 Schlossturm Been there
90 Schöne Aussicht Been there
91 Schwanenmarkt Been there
92 Schwarzwald-Christel Never heard of
93 Seufzer-Allee im Hofgarten Been there
94 Siedlung Freie Erde im Eller Forst Never heard of
95 Souterrain im Cafè Muggel Been there
96 Stadterhebungsmonument Been there
97 Stindermühle Never heard of
98 Stoffler Kapellchen Never heard of
99 Stresemannplatz Been there
100 Tausendfüßler Been there
101 Trödelmarkt am Aachener Platz Been there
102 Uerige Know of
103 Urdenbacher Kämpe Been there
104 Volksgarten Been there
105 Volkshaus Been there
106 Yede-Gör Been there
107 zakk Know of
108 ZERO-Atelier Never heard of
109 Zicke Been there
110 Zoo Been there
111 Zwischen Luegplatz und Belsenplatz Been there

One of the stranger places on the list is Reeser Platz which is this huge square that is unused except it has on one side a memorial erected by the Nazis in 1939 glorifying soldiers and the wars they went off to fight in.

Reeser Platz (2)

Reeser Platz (1)

Not a whole lot of traffic this morning.

Leere Straße

Lantz’scher Park, with its old trees, numerous sculptures and well-designed lines of sight would would be a beautiful place to relax, if it wasn’t for the thunder of planes every two minutes to and from Düsseldorf airport.

Lantz'scher Park Klein

Kapelle Lantz'scher Park

Passing the by Kaiserpfalz in Kaiserswerth.

An der Kaiserpfalz Klein

Schloss Kalkum reflecting in its water-filled moat.

Schloss Kalkum

The Case of the Not Enough Storage Error in a Delphi Application

The Problem

After a few weeks Delphi applications running on Windows Server (2008 or later) will fail to start or create new windows with error 8: Not enough storage is available to process this command.

Possible Cause 1: Desktop Heap Exhaustion

Session 0 which service applications are running in gets allocated substantially less desktop heap than interactive sessions, so you might be running out of it. There is a system event log entry with event ID 243 or 244 when the desktop heap gets exhausted.

By increasing the values in the SharedSection part of the data in registry value “Windows” in key “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems”, you can allocate more to the service session, as described in this Microsoft support article or this Stack Overflow answer.

Note that Microsoft’s Desktop Heap Monitor tool no longer works on current versions of Windows Server.

Possible Cause 2: Atom Leak

Applications compiled with Delphi XE2 or earlier call RegisterWindowMessage in the initialization section of Controls.pas to allocate a unique window message ID. This takes up one slot in the session’s atom table. Unfortunately, there are only around 16 000 slots in the atom table. With no way to unregister a window message, you will eventually run out of them. It has been said that on earlier versions of Windows, the counter wrapped around so you never ran out as slots got overwritten/reused.

For more details, see this bug report on Embarcadero’s Quality Central and this Stack Overflow answer.

I fixed this by editing the Controls.pas unit replacing this line

RM_GetObjectInstance := RegisterWindowMessage(PChar(ControlAtomString));

with this one

RM_GetObjectInstance := RegisterWindowMessage(PChar('DelphiRM_GetObjectInstance'));

This way all Delphi applications share the same atom instead of creating their own. There is also a more involved fix by Andreas Hausladen, but this solution sufficed for me.

Analyzing the Atom Leak

Jordi Corbilla has written the super useful Atom Table Monitor which dumps entries from the atom table. I made a C# version of it, the Atom Table Dumper, because (a) I didn’t want to diagnose issues with my Delphi applications using another Delphi application that had the same issue and (b) I needed an easier to read and process output format.

Note that your analysis tool has to run in the same session as the application getting the error.

Further note that not all of the atoms listed under RegisteredWindowMessage were necessarily allocated by RegisterWindowsMessage. RegisterClass is another function that allocates atoms in this table. However, as the documentation points out “All window classes that an application registers are unregistered when it terminates.“ so having many of them (temporarily) isn’t necessarily a problem.

Museum Insel Hombroich (again)

After my previous visit to Museum Insel Hombroich was cut short by rain, I went again this week when the weather was nicer.

Obviously, it was a much more enjoyable visit. Museum Insel Hombroich just has this wonderful combination of art…

Statuen

Quader

…architecture…

Außenansicht Ecke

Rundes Haus

Fenster Klein

…and nature.

Fluss

Stufen

Teich

Powerlines I came across while walking back to Neuss. #symmetry

Strommast Totale

Strommast Klein