The Case of Outlook not downloading RSS content because of a problem connecting to the server

The Problem

I had been using Outlook to subscribe to RSS feeds for many years. Mostly because I like to have my email and news in one place with the ability to add them to my Outlook to-do list for follow-up.

Over time, however, I noticed that one feed after the other stopped updating. Even feeds from Microsoft sites such as TechNet and MSDN which I had assumed simply had to be compatible with Microsoft’s own RSS reader. Trying to remove and then re-add them gave the following error:

image

I eventually found out that all feeds that no longer updated had been switched from HTTP to HTTPS, which seems to be unsupported by Outlook.

The Cause

This support document describes the scenarios in which Outlook supports authenticated RSS feeds. Basically, support is extremely limited. Normally, HTTPS is not supported and the workaround using Internet Explorer’s Common Feed List did not work for me.

Initially, Internet Explorer would download the RSS feed and they would show up as unread items in Outlook, but after the initial download I wouldn’t get any new items in Outlook, even though they did show up in Internet Explorer.

The Solution

Searching online for a solution did not turn up anything. Either people described workarounds that did not work for me, or recommended simply abandoning Outlook in favor of some other feed reader.

So I implemented my own solution and wrote a very simple proxy server that runs a local web server via HTTP that fetches the requested feed via HTTPS from the original source. By configuring Outlook to use this proxy, I now receive updates for feeds in Outlook even when the source uses HTTPS.

This RSS-via-HTTP Proxy runs as a service in the background. Its source code and some instructions on how to configure it are available on GitHub.

Reading up on Blockchain and Distributed Ledger Technology

Last year, I was reading up on Bitcoin, blockchain and beyond. Since then, there have been several interesting developments in distributed ledger technology (DLT).

I also came cross this great article defining criteria to avoid pointless blockchain projects and its follow-up on four genuine blockchain use cases.

R3 Corda

For one, R3, which I thought then and still think today shows a lot of promise, has released the code for Corda, its distributed ledger project. They also published a non-technical whitepaper as an introduction and two webinar videos: Introduction to Corda and Corda Developers’ Tutorial. There is alos this excellent non-technical 18 second definition of DLT by Richard Gendal Brown, CTO of R3.

R3 also offered its code to the Hyperledger project.

Hyperledger

Hyperledger isn’t a distributed ledger, per se, but contains multiple DLT projects, e.g. Fabric, which is backed by IBM. While you can run Hyperledger Fabric on your own machines, IBM also gives developers an opportunity to play with the technology in their cloud Bluemix.

Unlike Corda, which was built from the ground up for the financial services industry, finance is only one of the industries Hyperledger is targeting. There are, however, a number of projects underway in the financial services that use Hyperledger, as their proof of concept tracker shows.

One of those projects was undertaken by Germany’s central bank Deutsche Bundesbank and the country’s largest exchange operator Deutsche Börse. A November 2016 speech by Carl-Ludwig Thiele, member of the executive board of Deutsche Bundesbank contained mostly questions about the new technology. His speech from January 2017 already presented a prototype to handle simple settlement, payment and corporate actions.

Conclusion

There are a number of interesting projects underway to apply distributed ledger technology to finance.

Still, a lot of questions to be addressed regarding distributed ledger technology, as this position paper by SWIFT and Accenture from last year points out.

The Germany IT industry association Bitkom looks at some of these, e.g. legal ramifications of distributed ledger technology in banking (in German).

It is interesting to see though that regulators and central banks are already actively involved even though distributed ledger technology is still in its infancy in the financial services industry.

The Case of Multicast Message Loss on Windows Server 2012 R2

I have worked quite a bit with applications using UDP/multicast messaging recently. And I’ve run into a few issues along the way, such as multicast messages not being received on a Windows Fail-over Cluster.

So by now I have a solid checklist of things to configure on our servers and ensure in our applications that consume multicast messages to make sure everything runs smoothly and message loss is kept at acceptable levels. Yet, on our latest Windows Server 2012 R2 machines I had applications experiencing serious datagram loss as the amount of network traffic (in general, not just multicast) on the machine increased increased.

I researched the problem online and got the tips you’d expect: get latest NIC drivers, increase NIC receive buffer sizes, turn on offloads, turn on receive-side scaling, fine-tune receive-side scaling, increase socket buffer sizes etc. Of course, I had already tried all those things, and none of them had worked.

Solution: Exempting multicast traffic from Base Filtering Engine

Eventually I found this support document: Datagram loss when you run a multicast receiver application in Windows 8 and in Windows Server 2012. The problem description matched perfectly with what I was seeing on our server. Unfortunately, the document describes an issue in Windows Server 2012 and the hotfix available there cannot be installed on Windows Server 2012 R2. Fortunately, it doesn’t have to be. You can just set the registry key and the Base Filtering Engine supports it out of the box.

New-ItemProperty HKLM:\System\CurrentControlSet\services\Tcpip\Parameters\ -Name UdpExemptPortRange -Value "XXXX-YYYY" -PropertyType MultiString -Force

I haven’t found any official documentation on this, and prior to this post, there were just four results when searching Google for UdpExemptPortRange. But it works as far as I can tell.

The Case of the Not Enough Storage Error in a Delphi Application

The Problem

After a few weeks Delphi applications running on Windows Server (2008 or later) will fail to start or create new windows with error 8: Not enough storage is available to process this command.

Possible Cause 1: Desktop Heap Exhaustion

Session 0 which service applications are running in gets allocated substantially less desktop heap than interactive sessions, so you might be running out of it. There is a system event log entry with event ID 243 or 244 when the desktop heap gets exhausted.

By increasing the values in the SharedSection part of the data in registry value “Windows” in key “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems”, you can allocate more to the service session, as described in this Microsoft support article or this Stack Overflow answer.

Note that Microsoft’s Desktop Heap Monitor tool no longer works on current versions of Windows Server.

Possible Cause 2: Atom Leak

Applications compiled with Delphi XE2 or earlier call RegisterWindowMessage in the initialization section of Controls.pas to allocate a unique window message ID. This takes up one slot in the session’s atom table. Unfortunately, there are only around 16 000 slots in the atom table. With no way to unregister a window message, you will eventually run out of them. It has been said that on earlier versions of Windows, the counter wrapped around so you never ran out as slots got overwritten/reused.

For more details, see this bug report on Embarcadero’s Quality Central and this Stack Overflow answer.

I fixed this by editing the Controls.pas unit replacing this line

RM_GetObjectInstance := RegisterWindowMessage(PChar(ControlAtomString));

with this one

RM_GetObjectInstance := RegisterWindowMessage(PChar('DelphiRM_GetObjectInstance'));

This way all Delphi applications share the same atom instead of creating their own. There is also a more involved fix by Andreas Hausladen, but this solution sufficed for me.

Analyzing the Atom Leak

Jordi Corbilla has written the super useful Atom Table Monitor which dumps entries from the atom table. I made a C# version of it, the Atom Table Dumper, because (a) I didn’t want to diagnose issues with my Delphi applications using another Delphi application that had the same issue and (b) I needed an easier to read and process output format.

Note that your analysis tool has to run in the same session as the application getting the error.

Further note that not all of the atoms listed under RegisteredWindowMessage were necessarily allocated by RegisterWindowsMessage. RegisterClass is another function that allocates atoms in this table. However, as the documentation points out “All window classes that an application registers are unregistered when it terminates.“ so having many of them (temporarily) isn’t necessarily a problem.

Watch out mixing String and PChar in Delphi

Rudy Velthuis has written an excellent article about PChar in Delphi. Whenever someone comes up to me with a PChar/string question, I refer them to that article.

There are, however, a few caveats one needs to be aware of when mixing string and PChar that he does not mention. This is what I want to cover in this brief article.

The problems mixing string and PChar described here are caused by null characters (or #0 in Delphi syntax). Since Delphi strings have a length counter, It is perfectly legal for them to contain #0. Since PChar are null-terminated, by definition they cannot contain #0.

However, a few Delphi routines that seem to operate on strings internally actually operate on PChar. They will therefore not work as expected when the string they are given contains #0.

You cannot replace #0 in a string

If you call StringReplace to replace #0, it simply won’t work.

function RemoveNull(AInput: string): string;
begin
  Result := StringReplace(AInput, #0, '', [rfReplaceAll]); // Won’t work
end;

It seems that all replacing routines internally cast to PChar making them useless for this. Instead you will have to do a character by character comparison as in this routine by David Heffernan.

StrPCopy stops at #0

The StrPCopy routine takes a string as an input and copies it into a PChar. I used to use this routine in a TStringBuilder-like class which used a PChar to refer to an internal buffer containing the string being built. It also had a length counter, so one would not have to scan the PChar to figure out its length.

But even though StrPCopy takes a string as its input and could thus have access to its length, it does not copy all character, but stops at the first #0.

If you really want to copy all characters, you’ll have to use a routine such as Move that copies bytes not characters as in this example (again by David Heffernan).

Beware of implicit conversions

Since the compiler supports implicitly converting between PChar and string, you will often get away with passing a PChar where a routine takes a string parameter. The compiler will just generate a temporary string from the PChar and pass that instead.

Hence, this code will compile and work just fine:

function GetSubstring(AInput: PChar; AStart, ASubstringLength: Integer): string;
begin
  Result := Copy(AInput, AStart, ASubstringLength); // Works, but is slow
end;

Unfortunately, it has horrible performance, because the compiler needs to know the length of the PChar in order to built that temporary string. And the only way to figure out the length of a PChar is to check every character until the first #0.

See this question I asked on Stack Overflow how to fix this with a PChar-equivalent for the Copy routine.

Reading up on Bitcoin, Blockchain and Beyond

Digital Gold by Nathaniel Popper is neither a tech nor a finance book, but it provided me with a great introduction to Bitcoin. It describes at length the varied backgrounds and motivations of the people that were instrumental in getting it off the ground.

After you’ve read the book and learned the basics, Coindesk is a great source of current news on all things blockchain, cryptocurrency and so on. For specific questions, there is also a Bitcoin Stack Exchange Q&A site.

Of course, one of the most interesting things to come out of Bitcoin is not the cryptocurrency itself, but the blockchain, a distributed alternative to the centralized ledgers used in finance everywhere, it is built upon.

Blockchain, Distributed Ledgers

Unfortunately, a blockchain as used by Bitcoin has features that make it unsuitable for certain uses; poor scalability and slow speed of processing and confirming transactions being two of them.

There are, however, several projects that try to address these issues, such as Juno which specifically targets greater throughput performance.

Microsoft even has a blockchain project for its cloud computing service Azure called Project Bletchley.

But, the most interesting, in my opinion, distributed ledger project right now is R3 Corda. This is a joint venture between a number of banks aiming to built a distributed ledger specifically for financial transactions. What makes Corda stand out among the many blockchain/distributed ledger projects is that it is not a technology project looking for applications. Instead they are taking real requirements from the world of finance and try to mold the technology to fit them. As Richard Gendal Brown, R3’s Chief Technology Officer, writes in their introduction:

Every successful project I’ve worked on started with the requirements, not some cool piece of technology.

Richard Gendal Brown, by the way, also has his own blog Thoughts on the Future of Finance, on which he discusses technology in the financial sector in general (e.g. Apple Pay), not just distributed ledgers. Highly recommended.

Smart Contracts

Looking beyond finance, there is Ethereum which has built a smart contract platform on its blockchain. One example of such a smart contract is The DAO, a kind of investment company bound by bylaws written in code instead of legal prose. Since the company exists only virtually, it’s still unclear which country’s laws govern it. Its creators say a DAO is self-governing, i.e. its actions determined only by its coding and thereby not subject to outside influence such as a regulator.

Because of a coding error in the code of The DAO, an attacker was (almost) able to rob it of millions of dollars on 17-Jun-2016. This article has a good summary of how it happened. The Slock.it blog also has a number of articles worth reading on the attack and its aftermath. On the topic of entrusting your investment money to an autonomous organization bound only by smart contracts, their COO Stephan Tual writes:

[I]t is very challenging to write smart contracts that are both complex in nature and 100% safe. Therefore, it’s fair to say that the discussion of any Ethereum-based project should be handled with the great care it deserves.

Speaking of Slock.it: they have used the Ethereum blockchain as the backbone of a neat IoT project: the Ethereum computer powering smart locks called slocks.

Reading up on Concurrent Programming – Reloaded

In 2011 I posted Reading up on Concurrent Programming. Now, five years later, I thought it a good time to review that article, fix a couple of dead links and add a few new ideas I have come across since.

Basics

These are some general explanations a programmer should be familiar with regardless of the platform they are working on.

.NET and Windows

A good book to get you started is Concurrent Programming on Windows by Joe Duffy. It describes various aspects of locking and lock-free programming, the synchronization primitives available on Windows and much more. It’s certainly not easy-reading, but well worth it when you really want to learn these things in detail. The book is for .NET as well as Win32 programmers, as everything is explained in terms of both C# and Visual C++.

If your target platform is .NET, you must check out Threading in C# by Joseph Albahari which contains everything one could possibly want to know about, well, threading in C#. My personal advice, however, when doing multithreading in C#: in 99% of all cases don’t bother about creating your own threads, but use the Task Parallel Library (TPL) extensively. Don’t get me wrong, it’s still important to know these other things, though. And if you are really serious about optimizing your code, you will have to resort to some of the advanced techniques described there.

Another good resource for applications targeting Windows is the Parallel Computing page on MSDN containing links to a bunch of documentation, sample code tutorials and videos. While there is some information there about the Async technology preview, you might also want to check out this separate Visual Studio Asynchronous Programming page. I particularly liked Anders Hejlsberg’s introduction video. This document on the Task-based Asynchronous Pattern provides a great overview of all those scenarios that will become a lot easier to implement with Async.

The PFX team’s blog also has some interesting posts about various parallel programming topics. I particularly recommend Know Thine Implicit Allocations. Applying some of these techniques has made my code a lot more “garbage collector friendly”, meaning fewer interruptions by the GC and thus more reliable execution times.

I also have a separate article on Writing a High-Performance C# Application containing links to more such articles.

In 2011 Raymond Chen did a series on lock-free algorithms on his blog The Old New Thing.

Java

If your focus is Java, you have to take a look at Disruptor, a “hard-core” (their words, not mine) concurrent programming framework for the JVM. To get started, you should check-out this video of a presentation by the developers and/or read their technical paper. There is also a description of the larger architecture that Disruptor is a part of on Martin Fowler’s website.

But even if you are not programming in Java, you should look at this, as it contains some interesting pointers to what one could do in other languages to improve the performance of concurrent code.

The Mechanical Sympathy blog by one of the developers contains a bunch of additional background information regarding Disruptor’s implementation.

Other

Finally, here’s a couple of questions from Stack Overflow and the Programmers Stack Exchange that I think have generated some interesting responses: