Windows Service Gotchas

• the current directory is not the location of the executable, it’s System32! But that’s easy to change.
• anything logged to the console will be lost. You can redirect anything you output with System.Diagnostics.Trace. That includes for instance the internal debugging output from log4net (useful when you don’t know why your service is not logging anything).

Quoted from log4net documentation:

 

There are 2 different ways to enable internal debugging in log4net. These are listed below. The preferred method is to specify the log4net.Internal.Debug option in the application's config file.

• Internal debugging can also be enabled by setting a value in the application's configuration file (not the log4net configuration file, unless the log4net config data is embedded in the application's config file). The log4net.Internal.Debug application setting must be set to the value true. For example:

  <?xml version="1.0" encoding="utf-8" ?>
  <configuration>
      <appSettings>
          <add key="log4net.Internal.Debug" value="true"/>
      </appSettings>
  </configuration>

  This setting is read immediately on startup an will cause all internal debugging messages to be emitted.

• To enable log4net's internal debug programmatically you need to set the log4net.Util.LogLog.InternalDebugging property to true. Obviously the sooner this is set the more debug will be produced.

Internal debugging messages are written to the console and to the System.Diagnostics.Trace system. If the application does not have a console the messages logged there will be lost. Note that an application can redirect the console stream by setting the System.Console.Out. The Trace system will by default send the message to an attached debugger (where the messages will appear in the output window). If the process does not have a debugger attached then the messages are sent to the system debugger. A utility like DebugView from http://www.sysinternals.com may be used to capture these messages.

As log4net internal debug messages are written to the System.Diagnostics.Trace system it is possible to redirect those messages to a local file. You can define a trace listener by adding the following to your application's .config file:

<configuration>
    ...
    
    <system.diagnostics>
        <trace autoflush="true">
            <listeners>
                <add 
                    name="textWriterTraceListener" 
                    type="System.Diagnostics.TextWriterTraceListener" 
                    initializeData="C:\tmp\log4net.txt" />
            </listeners>
        </trace>
    </system.diagnostics>

    ...
</configuration>

Make sure that the process running your application has permission to write to this file.

Your Home Address on the Web

 

I’ve always thought privacy concerns over Facebook were exaggerated -they usually come from your parents who have been publishing their home address and phone number on the white pages for decades anyway. There is much much worse.

The mere act of voting in the UK ends up in having your home address published on the web.

If you registered to vote on the Electoral Roll and ticked the wrong box, the one with the confusing caption, something like “tick here if you don’t want your details to appear in the private listing”, then all your past addresses are on the web, year by year. Yippee!

It’s not just your home address, but also the list of people who lived with you. That must be the most blatant breach of privacy I’ve ever seen. It’s like Facebook for flatmates with privacy settings set to ‘Everyone’.

Look at this. Only the first part of the postcode is available. To get the full info all you have to do is register at 192.com.

Currently Reading or Watching…

• Authentication, authorization and impersonation in WCF: WCF Training Kit (Safari Books Online)
• Herb Sutter: Writing modern C++ code: how C++ has evolved over the years (video from the //build/ event)
• Active, dormant and tombstoned apps: Multithreading in Mango

Current Reading About Hedging

Where to invest in case of a catastrophe?

If you don’t own gold, now is a good time to buy

How Bill Gates is betting on inflation

Crisis Investing: Hedging for and Against Catastrophe

TeamCity, msbuild and Version Numbers

How to automatically set all your solution’s projects to the same version number with msbuild and TeamCity?

We have a number of solutions mixing native and managed projects.

The mechanism we use to ensure all DLLs and EXEs have the same version number relies on modifying 2 files:

• A VersionInfo.h file for native projects
• A VersionInfo.cs file for managed projects

The .h file contains something like this:

#define PROJECT_VERSION 3,9,0,1234
#define PROJECT_VERSION_STR “3.9.0.1234”

All native projects in the solution have a resource file (.rc) that includes this unique VersionInfo.h.  This is how the exe/dll obtain their version info.

On the .NET side, the VersionInfo.cs file contains:

[assembly: Assemblyversion(“3.9.0.1234”)]
[assembly: AssemblyFileversion(“3.9.0.1234”)]

All managed projects in the solution compile this VersionInfo.cs file (using a soft link, no duplication necessary).

So all we have to do in order to ensure that the 30+ binaries have exactly the same version number is to get our build script to modify the .h and the .cs just before building the solution.

Sounds easy!

Then there is the question of where to store the “3.9.0.1234” version. It should be in one single place of course. An obvious choice seems to be the TeamCity build number.

From Major.Minor.Build.Revision, we can set Major.Minor.Build by hand in TeamCity and have the Revision set automatically to the current SVN revision number, which is a handy way to relate builds to SVN revisions.

In the TeamCity project settings, the build number format field looks like this:

3.9.0.%build.vcs.number.MyProject_SVN_Root%

Then within our msbuild XML file we refer to the build number set in TeamCity like this:

$(BUILD_NUMBER)

The following msbuild lines update the version number stored in the VersionInfo.cs.

    <FileUpdateFiles Files="VersionInfo.cs"
      Regex="(\d+)\.(\d+)\.(\d+)\.(\d+)"
      ReplacementText="$(BUILD_NUMBER)"/>

This regex does wonders to replace “1.2.3.4” with “3.9.0.1234”. But now, more tricky: when doing the string replace in VersionInfo.h (it’s always the native projects that cause trouble ), how do you replace

#define PROJECT_VERSION 1,2,3,4

with

#define PROJECT_VERSION 3,9,0,1234

given the string in $(BUILD_NUMBER) “3.9.0.1234”.

Can’t do anything a bit subtle with msbuild without having to download and install third-party tasks… So I’m currently looking into the TextString task from the MsBuild Extension Pack to tokenise the version string and then concatenate it back to the appropriate format. It’s all a bit complicated, there must be an easier way to do this…

Update (16/10/2011):

It’s actually not that bad, here is the code that formats $(BUILD_NUMBER) and updates the .h /.cs VersionInfo files.

    <TextString TaskAction="Replace"
                OldString="$(BUILD_NUMBER)"
                OldValue="."
                NewValue=",">
      <Output PropertyName="FormattedBuildNumber"
              TaskParameter="NewString"/>
    </TextString>
    <Message Text="Replacing version in VersionInfo.h with $(FormattedBuildNumber) and $(BUILD_NUMBER)">
    </Message>
    <FileUpdateFiles Files="VersionInfo.h"
      Regex="(\d+)\.(\d+)\.(\d+)\.(\d+)"
      ReplacementText="$(BUILD_NUMBER)"/>
    <FileUpdateFiles Files="VersionInfo.h"
      Regex="(\d+)\,(\d+)\,(\d+)\,(\d+)"
      ReplacementText="$(FormattedBuildNumber)"/>
    <Message Text="Replacing version in VersionInfo.cs with $(BUILD_NUMBER)">
    </Message>
    <FileUpdateFiles Files="VersionInfo.cs"
      Regex="(\d+)\.(\d+)\.(\d+)\.(\d+)"
      ReplacementText="$(BUILD_NUMBER)"/>

Getting Ready For FOWA London 2011

This is the schedule for tomorrow’s first day of FOWA (Forum of Web Apps) at the Brewery between Shoreditch and Whitechapel.

http://futureofwebapps.com/london-2011/schedule/

Body of Lies

I just saw Body of Lies by Ridley Scott. At minute 1h38’ there is a scene where Di Caprio talks on the phone to the bad guys who kidnapped his girlfriend. The call is recorded so to make that obvious to the audience, there is a massive old-school tape recorder spinning in the background. Cell phones, drones and satellites… You can never go wrong with good old reel-to-reel tape recording.

I Own a Domain Name I Can’t Use!

I bought a domain name through Blogger a few months ago, got my receipt through Google Checkout and went on holiday. The Google Checkout receipt has a link to retrieve the Google Apps account to manage the domain: that link doesn’t work any more and the domain does not appear in Blogger.

My domain fell into a black hole: I can prove I bought it, but I can’t use it!

Currently reading… (Read It Later)

• Windows 8: A bad picture is worth a thousand long discussions
• Economy: Grece, questions sur une interminable crise (Le Point)
• DB: Mythbusting: Concurrent Update/Insert Solutions
• Misc: Italian scientists on trial for failing to predict earthquake

Fire-proof Software #2: Designing Fault Tolerant Systems

Follow-up from part 1.

This is a few of the failure conditions we see with real production software, along with some approaches to deal with them. The general idea is to define activities.

• Each activity is a unit of work that may either complete or fail and that can be retried if necessary.
• An activity has a start and an end, as well as a status (Complete/Failed).
• An activity also has success criteria that allow a process to check the activity actually completed.

The database contains a schedule of those activities along with a MaxRunTime value and a flag indicating if it's currently running. The MaxRunTime allows processes to spot an activity that timed out.

Input data not available (Failure of one of the upstream systems)

You can’t bet on the fact that upstream systems will work, regardless of the reason why, they will fail. There is no point trying to imagine the reasons behind the potential failure, what matters is the impact of the failure on your system and on the business.

• Have a retry policy for each activity (retry period + retry window)
• Use a stale version of the data (previous day for instance) if acceptable with the business.

Notification not available

• Back-up the notification mechanism with an activity schedule: an activity should automatically start at a defined time if it hasn’t already.
• If an activity has already started following a notification, the DB flag in the activity schedule table will guarantee that it doesn't start again.

Process crash

• Have a Windows service detect the process crash and restart the process.
• If the process crashes following an unhandled exception, the currently running activity will be automatically marked as failed. When the process starts again it should attempt to schedule or start the failed activity.
• If the process is simply killed with no opportunity to mark the current activity as failed the activity will still appear as running in the ActivitySchedule. When the process starts again it will see that its activity is currently running and will simply schedule a check at StartTime + MaxRunTime. Obviously the check will fail and the process will restart the activity.

Process stalls

• Kill it with a watchdog thread: this is a pattern used by embedded systems on real-time OSs to reset a stalled CPU. Inside the app server process the main thread –the one that does all the work- should periodically reset the watchdog flag. If the watchdog flag is not reset after a defined period of time the watchdog thread kills the process after failing the current activity.

Server down or unreachable

• Use 2 servers, one primary and one backup.
• Each server runs identical processes scheduling the same activities. Both processes will attempt to start their activity at the same time however the ActivitySchedule table in the DB will allow only one process to actually start the activity.
• If one server goes down, all processes will simply attempt to reschedule their activities upon restart.
• If one server goes down while an activity is running, the activity will still appear as running in the ActivitySchedule. Upon restart processes will see the activity is marked as running and will schedule a check at the expected completion time.

Database not available

• Avoid maintaining a single DB connection for too long, that reduces the opportunities of the system to reconnect.
• If you can detect the connection failure and don’t want the complexity of implementing a retry mechanism, at least fail the current activity to take advantage of the activity’s retry policy.