Since I was a kid, my family has been going to the village of Obereggenen near Schliengen in the black forest for fall vacation. So for the 30th anniversary of our first vacation, we all spent the weekend there again.
A colleague came to me with a Delphi application that would not shut down, but just hang. The application in question had been refactored such that one module was extracted into a DLL to be reused in another application. When this extracted module was loaded into the application and the application was closed, it would hang. If the module was not loaded, the application would shut down normally.
Debugging the application was initially unsuccessful. Stepping through our code we verified that the shutdown logic executed normally with destructors running as expected. Interestingly, it was not possible to break into the application once it had become unresponsive. Trying to pause the hung program from within the IDE would simply cause the IDE to hang as well.
Thus we used Process Explorer instead to look at the application’s threads and their callstacks.
There we saw that there was one thread stuck on a call to WaitForSingleObject which originated in our DLL code. Higher up the callstack was ExitProcess. I looked at the documentation for ExitProcess to see look for ways in which it could deadlock. One sentence looked promising: “If one of the terminated threads in the process holds a lock and the DLL detach code in one of the loaded DLLs attempts to acquire the same lock, then calling ExitProcess results in a deadlock.” But since there was only one thread, this could not be it.
Looking next at what happens exactly inside ExitProcess, two other things jumped at me:
- All threads are terminated (except for the one calling ExitProcess).
- All DLLs are unloaded.
It turns out, the initial analysis that “the shutdown logic executed normally” was wrong. One of the shared units compiled into the DLL (through several layers of indirections), had a finalization section. In this finalization section, a background thread that had been created in the corresponding initialization section, was being destroyed. As part of the destructor code, the thread class was waiting for an event that was set when the thread had stopped executing.
This finalization section was running as part of the “all DLLs are unloaded” step by ExitProcess. Unfortunately, all threads (including the one created in the initialization) had already been terminated. I am not quite sure how that was accomplished, but it apparently circumvented the normal thread termination logic which set the event that the thread had stopped executing.
This is different for code in the main application, where finalization sections are run while the application is still in working order.
Instead of waiting for the thread to set its “stopped executing” event, I wait on the thread handle to check if the thread was even there to set the event. When run from the DLL’s finalization section, this detects the thread’s absence and just returns.
As my tour of Meyer Werft wasn’t until the afternoon, I had a few hours in Papenburg to walk along the canals and take in the beautiful scenery of this quaint little town.
Last Saturday I attended the CIM Lingen conference (again). Since I already was in the region, I figured why not add one more day to the trip and visit the Meyer Werft shipyards in nearby Papenburg as well.
The World Dream docked outside the huge Dockhalle 2 factory building as the ship is being prepped for its maiden voyage later this year.
Most of the tour at Meyer Werft was spent in the visitors center, watching video of their 220+ year history and looking at model of some of the 700+ ships they built in this time.
One of the ships with the most interesting history is the Graf Goetzen. Built in 1913 at the shipyard in Papenburg for the German Empire, it was immediately disassembled after it was finished and its parts shipped in boxes to Africa. After brief use as a naval vessel and being sunk, it was raised years later, renovated and continues to be operated on Lake Tanganyika to this day.
But the highlight came at the end of the two-hour tour: a peek into the two factory buildings Dockhalle 1 and Dockhalle 2.
Pictured above is the smaller Dockhalle 1 where they built parts for final assembly in Dockhalle 2. The latter is so large, that it is impossible to fully take it in through the small window in the visitors area.
Interesting detail: one of the cranes in Dockhalle 1 was actually made in East Germany (“Deutsche Demokratische Republik”). Apparently, the East German buyers of one of the ships built by Meyer did not have enough hard currency to pay for it and delivered equipment to make up the difference.
The company logo, then and now.
Coming home from work on Friday of last week, I decided I needed to get out of the city for the weekend. Destination: Wangerooge, an island in the North Sea.
The weather wasn’t ideal on Saturday, but it was nonetheless the best travel experience I’ve had in a long time. And on Sunday, shortly before heading home, the sky cleared up creating these perfect vistas.
You just don’t get to see sunsets like this in the city.
I woke up early on Sunday and thus got to enjoy the most beautiful sunrise I’ve ever seen.
Seals in the distance (poor quality, because I had to shoot through the ferry window).
One last look back to Wangerooge before it was time to take the train back home.
There are many business intelligence solutions large and small for knowledge workers to choose from. But due to its ubiquity, I assume many (myself included) just use Microsoft Excel to interactively analyze data.
A lot of data on the web is available via RESTful APIs returning JSON, e.g. the REST countries service I’ll use in this example.
In the 2016 release Microsoft has vastly expanded the data analysis capabilities of Excel compared to previous versions. And with the free PowerQuery plugin, these capabilities are available to Excel 2013 users as well.
Getting JSON into Excel
One of these capabilities is to retrieve JSON data from the web and turn it into an Excel table. Just go to Data > From Web and enter the URL, e.g.
Excel will figure out whether there is a regular web page at the given URL (and offer to extract HTML tables from it) or JSON. In the latter case, a list of records is displayed in the Query Editor.
There is a Convert To Table button conveniently placed at the top left. But with every JSON document I’ve come across, this has required several additional steps to create a proper table.
Making a table
Instead I recommend going to View > Advanced Editor and add a manual conversion step by changing the query to this.
let Source = Json.Document(Web.Contents("http://restcountries.eu/rest/v2/all")), Table = Table.FromRecords(Source) in Table
The same can be achieved by adding a manual step in the Query Settings sidebar and adding the bold text as the function.
Voilà, you now have that JSON as a table with one record per country.
You may want to add some more steps such as right-click the topLevelDomain column and Extract Values… to get the list of domains comma-separated in a single cell.
Great usability for non-programmers
This is just one of the many conversion and transformations available via easy to use (but numerous) commands in the context-menu and ribbon.
The great thing about Excel’s handling of all of these conversion steps is that they are applied non-destructively. In the Query Settings sidebar, you can see each one and click it to see the intermediate results it produces.
Not so great: M language
There is one, thing, however, that is a bit disappointing: the programming language behind all of this is the M language inspired by F#. This is unfortunate as it means any previous skills you’ve had working with data in Excel macros is useless, as even basic things such as if statements are syntactically so different I needed to look it up.
For 31 days starting in the beginning of July I posted one new picture each afternoon. All of these pictures were created using the same basic shapes following the rules set out here.
It was interesting to see, how despite their similarities some pictures scored substantially more likes than others. One thing that was abundantly clear from the start was that using many popular hashtags significantly increased my chances of getting likes. Every Instagram guide will tell you that.
What I am interested in is whether there is a way to forecast how many likes a given picture might get. I’m planning on writing a little application to analyze the dataset gathered in July for correlations. This would be an interesting opportunity to try out TensorFlow or Azure Machine Learning and Cognitive Services.