Scriptico

Add to Google Reader or Homepage

Debugging with WinDbg. Deadlocks in Applications.

In the following article I will show how to debug with windbg and detect deadlocks in applications. 

First of all to demonstrate this problem we need a sample application. I took one from MSDN Magazine (open it in a new window) to save the time, and in the end of the article you will find a complete Visual Studio project.

Ok, in the steps below we will try to detect the problem. In our sample the deadlock is obvious, but in large commercial products sometimes developers are spending hours to find the problem. We need a few additional instruments to achieve our goal. So, the first instrument is WinDbg. This application is a part of Debugging tools for Windows ( open additional Info and download link in a new window ).
The second instrument is SOSEX ( More info and download link ). Sosex is a new debugging extension for managed code and this stuff is amazing! I found a problem in huge production application within minutes!

At this point you should download and install these instruments in your environment.

0. Start the sample if it is not started yet.

C:\Users\JohnDoe>cd C:\dev\DeadlockSample\DeadlockSample\bin\Debug\
C:\Users\JohnDoe>DeadlockSample.exe
Inside methodA
Inside methodB
methodB: Inside LockB

1. Open WinDbg
start -> run... -> WinDbg
( screenshot )

2. Attach to the application process
File -> Attach to a process
( screenshot )

3. Load the sosex assembly
.load sosex
( screenshot )

Make sure that the sosex assembly is placed in the WinDbg installation directory.

4. Run the following command
!dlk
( screenshot )

5. Analyze the result!

Examining SyncBlocks...
Scanning for ReaderWriterLocks...
Scanning for lock holders on ReaderWriterLocks...
Scanning for threads waiting on SyncBlocks...
*** WARNING: Unable to verify checksum for C:\dev\DeadlockSample\DeadlockSample\bin\Debug\DeadlockSample.exe
Scanning for threads waiting on ReaderWriterLocks...
Deadlock detected:
CLR thread 0x3 owns SyncBlock 00283ef4 OBJ:01ad8644[System.Object]
is waiting for SyncBlock 00283ec4 OBJ:01ad8650[System.Object]
CLR thread 0x4 owns SyncBlock 00283ec4 OBJ:01ad8650[System.Object]
is waiting for SyncBlock 00283ef4 OBJ:01ad8644[System.Object]
CLR Thread 0x3 is waiting at DeadlockSample.DeadlockSampleClass.MethodA()(+0x3c IL)(+0x78 Native) [C:\dev\DeadlockSample\DeadlockSample\DeadlockSampleClass.cs, @ 24,17]
CLR Thread 0x4 is waiting at DeadlockSample.DeadlockSampleClass.MethodB()(+0x3c IL)(+0x78 Native) [C:\dev\DeadlockSample\DeadlockSample\DeadlockSampleClass.cs, @ 41,17]

1 deadlock detected.

Now, how about a case when you are not able to connect a process? For example, you are a tech support guy and your customer has suspicious about a deadlock in some application. Well, not a big deal! You can analyze a problem just with a crash dump of the application. How to achieve it? Instead of the second step just complete the following:

1. Open the task manager and get an id number of the application process.

2. Open command prompt (cmd), and create a full crash dump of the application with adplus utility ( More Info ).
This utility is located in the installation directory of Debugging Tools for Windows (in my case the path is C:\Program Files\Debugging Tools for Windows (x86)). Ok, the command is shown below:
adplus –crash –p -o "Path_to_dumps".
process_pid is an id of our process, and Path_to_dumps is a path where dump will be stored. Make sure that this folder is exist in your environment. ( Creating Process Dumps with ADPlus )

3. Now you have a complete crash dump of your application.
The way how to load this dump in WinDbg is shown below.
File -> Open Crash Dump

Analyze result is not so hard, and what you should to do is just read carefuly the output in WinDbg.

CLR thread 0x3 owns SyncBlock 00283ef4 OBJ:01ad8644[System.Object]
is waiting for SyncBlock 00283ec4 OBJ:01ad8650[System.Object]
CLR thread 0x4 owns SyncBlock 00283ec4 OBJ:01ad8650[System.Object]
is waiting for SyncBlock 00283ef4 OBJ:01ad8644[System.Object]
CLR Thread 0x3 is waiting at DeadlockSample.DeadlockSampleClass.MethodA()(+0x3c IL)(+0x78 Native) [C:\dev\DeadlockSample\DeadlockSample\DeadlockSampleClass.cs, @ 24,17]
CLR Thread 0x4 is waiting at DeadlockSample.DeadlockSampleClass.MethodB()(+0x3c IL)(+0x78 Native) [C:\dev\DeadlockSample\DeadlockSample\DeadlockSampleClass.cs, @ 41,17]

 

Thread with id 0x3 (3) owns a block with object 01ad8644 and it is waiting for the object 01ad8650; moreover, the different thread with id 0x4 (4) owns 01ad8650 and it is waiting for the object 01ad8644! Bingo! Now, just open the code and fix the problem! As we can see the code has a problem in the line 24 and on the line 41. Done!

BTW, do you know a simpler solution? Have you ever try Concurrency Visualizer in Visual Studio 2010?

 

    Resources

Category: C#, Development, Multithreading

Tagged:

Comments are closed.