Post-Mortem Debugging Your Application with Minidumps and Visual Studio .NET

有时间翻译一下。
source: http://www.codeproject.com/KB/debug/postmortemdebug_standalone1.aspx

Contents

What Is a Minidump?
Creating a Minidump
Build Issues
Writing a Minidump with
MiniDumpWriteDump
Reading a Minidump with Visual
Studio .NET
How Microsoft Uses Minidumps
Further Improvements
Conclusion

What Is a Minidump?

A minidump is a file containing the most important parts of a
crashed application. It is written on the user’s machine and then the customer
can submit it to the developer. The developer can load the dump to help
determine the cause of the crash and develop a fix.

Since the early days of Windows NT, the Dr. Watson program has
been able to generate crash dump files, denoted with the .dmp extension.
However, they were not as useful as they should have been because of two
problems:

1. They were huge. The dump of an application included every byte
   of the entire process space, so a crash in something simple like Notepad would
   be several megabytes in size, and a crash in something like Word can be many
   hundreds of megabytes. The files were just too big to send by e-mail or FTP.
2. Their content was not necessarily useful. Dr. Watson was, in
   fact, a just-in-time (JIT) debugger, and it is difficult for a debugger to get
   the full path to a loaded module. Full debuggers such as the Visual Studio
   debugger complete a number of steps to get the paths, but Dr. Watson did not.
   This usually resulted in unhelpful module names such as MOD0000 and so on.

Minidumps were designed to fix these problems in a number of
ways:

• Instead of saving the entire process space, only
  certain sections are saved. There is no point in saving copies of modules such
  as Kernel32.dll; if the Version number is included, it is easy to get a copy from
  a Windows CD. The actual memory heap of the application is by default not saved
  in a minidump; it is not required to debug a surprisingly high percentage of
  crashes. You can save the heap if you need to, however.
• The minidump save code works to get accurate and full
  information for modules, including their names, paths, Version information, and
  internal timestamps.
• The minidump save code also gets the list of threads,
  their contexts (that is, register sets), and the memory behind their stacks.
• The whole file is compressed, further reducing its
  size. A minidump for Notepad is around 6K on Windows XP, almost 300 times
  smaller than the previous crash dump of the same process.

Note   Kernel-mode
minidumps, which are generated by Windows XP after a computer stops responding,
also exist, but this article discusses more common user-mode minidumps.

Creating a Minidump

There are three ways to create a minidump:

• Add code to your own application to write a minidump if
  the application has an unhandled exception.
• In the integrated development environment of Visual
  Studio .NET, click Save Dump on the Debug menu when debugging the
  application.
• Do nothing.

The first option is discussed in more detail in the following
sections.

The second option works only on a workstation with the debugger
already configured, which is likely useful only within an organization (for
example, with another developer or tester). If you debug a crash with Visual
Studio .NET, you can then click Save Dump As on the Debug menu. You can
save as a Minidump or Minidump With Heap. You do not need any symbols or PDBs
configured to save a dump file; you will need them later, however, to read it
back.

The third option works only in Windows XP, which automatically
creates a minidump if an application has an unhandled exception and no JIT
debugger has been set up. Also, the minidump is submitted directly to
Microsoft, so you will not have the opportunity to determine why.

Build Issues

To configure your application to create dumps when it stops
responding, you must configure your builds to generate full debugging
information, especially for retail builds. After generating a PDB, you also
must archive every binary that ships to customers and its matching PDB; you
will need the PDB later to debug any minidumps that your customers submit.

Also, ensure that you set the Version information correctly on
your binaries. Every release of every component that you ship should have a
different Version, so you can match them to the minidump. The Version field
assists you in matching the bits. The debugger itself does not use the Version
information, however; it matches binaries based on the internal timestamp in
the PE header.

In terms of output, generating debug information for release
builds has a small effect. A PDB, taking up some space on a build machine, is
generated, and the binary will be a few hundred bytes larger, to record the
name of the PDB in the debug directory within the PE file. You should not
provide customers with the PDB; this could allow customers to more easily
reverse-engineer your application.

Writing a Minidump with MiniDumpWriteDump

The key API to saving a minidump is MiniDumpWriteDump,
which is exported from Dbghelp.dll, a redistributable DLL that comes with the
Platform SDK. Make sure that you are using the Windows XP Version 5.1.2600;
earlier beta and release candidate versions had problems with the API, and
versions 5.0.x are from Windows 2000 and do not export the function. If you
have a Version earlier than 5.0, then it comes from the Systems Debugger
package (including WinDbg and so on) and is not redistributable.

To call the API, you need to catch the crash by setting an
unhandled exception handler with the SetUnhandledExceptionFilter
API. This allows the filter function to be called at almost any time an
unhandled exception occurs in an application. In certain unhandled exceptions,
such as a double stack fault, the operating system will immediately terminate
the application without calling the filter or a JIT debugger.

In your filter function, you must load Dbghelp.dll. This is not
as simple as calling LoadLibrary; as with Windows 2000,
you will access the one from the System32 directory, which does not have the
right export. The sample code in the attached file tries to load from the same
location as the EXE. Install the correct Version of Dbghelp.dll into the same
directory as the EXE; if this does not work, then the code reverts to a plain LoadLibrary, which will only work if the application is
running on Windows XP.

After loading the DLL, it then checks for the named export; if
correct, it then creates a file with an appropriate name, such as saving to the
Temp directory and using the application name with the .dmp extension. This
handle is then passed to the API itself, with some additional information such
as the process ID and the type of dump file. The sample uses MiniDumpNormal. You may want to or-in the flag
value of MiniDumpWithDataSegs, which is the equivalent
to the Minidump With Heap option in the Visual Studio
debugger. This does result in substantially larger dump files.

Once the .dmp file is created, the application informs the user
where it is stored. The user can then e-mail it or use FTP to get you the file
to investigate.

To use the sample code provided, add the mdump.h file and declare
one MiniDumper object at global scope. Its constructor
takes an argument, which should be the basename of the minidump file. Add
mdump.cpp to your project. You need to have the correct Dbghelp.dll file in the
same directory as the EXE to run successfully.

You cannot debug the code that writes the minidump with a
debugger (in the sample code, Minidumper::TopLevelFilter).
If a debugger is attached to a process, then the unhandled exception filter
will never be called. If you encounter problems, you will need to use
MessageBox debugging.

Reading
a Minidump with Visual Studio .NET

This section uses an example of a manually created minidump from
Notepad in Windows 2000 and debugging in Windows XP.

Start Visual Studio .NET and, on the File menu, click Open
Solution. Change the Files of type drop-down menu to Dump Files (*.dmp;
*.mdmp), navigate to the minidump, and create a default project by clicking
Open.

To launch the dump with the debugger, press F5. This will provide
you with information to get started. The debugger creates a fake process; in
the Output window, various module load messages are displayed. The debugger is
recreated the crashed processes state only. After displaying a warning that the
EXE contains no debug information, the debugger stops with the user's crash,
such as an access violation. If you then examine the Call Stack window, you
will notice the lack of symbols and a lack of useful information.

Figure 1. Initial Stack with No Symbols

To read a minidump, you typically need copies of the binaries
involved. To find the right binaries, open the Modules window.

Figure 2. Initial Modules with No Binaries

Figure 2 shows the Notepad example and demonstrates two things.
First, the paths to the binaries are marked with asterisks, which denotes the
path on the user's workstation but the binary could not be found at that
location on your machine. Second, the messages read “No matching binary found”
in the Information field. The key to finding matching binaries is to look at
the Version field and the file name. In this example, the Version of most
system files is 2195, which indicates Windows 2000. It does not immediately
indicate the exact service pack (SP) or quality fix engineering (QFE), however.
For more information, see the DLL Help database at http://support.microsoft.com/servicedesks/fileversion/dllinfo.asp.

At this point, you will need to find a Windows operating system
CD or a user's workstation with the correct versions and copy the correct
versions into a directory. It is not normally necessary to find every binary
that was in the process, but it is important to find each binary that features
on every relevant call stack. This will often involve both operating system
binaries (such as Kernel32.dll) and your application binaries (in this example,
Notepad.exe).

When you have found the bits and copied them to a local
directory, on the Debug menu, click Stop Debugging. In Solution Explorer,
right-click the project icon and click Properties on the shortcut menu. This
will take you to the Debugging page. Fill in the Command Arguments to be
MODPATH, followed by an equal sign, and followed by a semicolon-delimited list
of places to look for binaries. In this example, it is:

MODPATH=m:/sysbits

After setting the path, press F5 to reload the minidump. MODPATH
reuses the command argument to get the value to the debugger; a future Version
of Visual Studio .NET may have a better way of setting this, perhaps as an
option in the Properties dialog box.

Although finding the binaries is unlikely to improve the call
stack, it should resolve some issues in the Modules window, as shown in Figure
3.

Figure 3. Modules with Binaries

Instead of saying “No matching binary found”, it now says a
combination of “Cannot find or open a required DBG file” or “No symbols
loaded”. The first message occurs with system DLLs that use DBGs for their
debugging information. The second message occurs for DLLs that use PDBs.
Getting matching binaries will not necessarily improve your call stack; you
also need debug information that matches them.

Method A: Symbols the Hard Way

To completely analyze a minidump, you should find debug
information for everything, but to save time, you can find only the information
you need. The example stack lists User32.dll and Kernel32.dll, so they need
matching debug information.

Matching Debug Information

The best place to locate system symbols is at http://www.microsoft.com/ddk/debugging.
You can also locate system symbols on the Support compact disc that ships with
Windows NT Server and Windows 2000 Server operating systems. In this example,
they were copied to the location of the binaries. In real instances, you will
have non-Microsoft binaries listed, so you will need to have PDBs for them.
Also in this example, the DBG and PDB for Notepad were also copied, because it
was the sample application used.

After you click Stop Debugging on the Debug menu, pressing F5
will list a call stack, as shown in Figure 4. You may find that, as you add new
binaries and debug information, your call stack will change. This is to be
expected; a call stack can only be walked accurately with debug information, so
as you add more information, your stack will get more accurate, which will
often expose additional frames that were not in the original.

In this example, there was no crash. In real instances, you would
have sufficient information to determine the cause of around 70 percent of your
user crashes. In addition, this stack was created with the stripped symbols
that Microsoft ships for system component, so there was no line number
information. For your own binaries with full PDBs, you will get an even richer
stack.

Figure 4. Call Stack with Symbols and Binaries

Symbol Servers

If you are dealing with many minidumps and performing general
debugging, storing and accessing all the binary and PDB/DBG files can be
difficult. Windows NT developed a technology known as a symbol server, which
was originally conceived as storage for symbols but was extended to support
finding binary files. The Windows NT debugger was the first tool to support
this, but Visual Studio .NET also does so as an undocumented feature. For
more information on symbol server, see http://www.microsoft.com/ddk/debugging/symbols.asp.

You can also retrieve symbols from the Microsoft symbol server.
These symbols will be cached and indexed locally for you.

Method B: Symbols The Easy Way: Using Symbol Server

First, go to http://www.microsoft.com/ddk/debugging and
download the debugging tools. You need Symsrv.dll, which needs to be on the
path. You can either copy it next to devenv.exe or into your System32
directory, to allow Visual Studio .NET access to it. Once you have copied
Symsrv.dll, you can safely uninstall the debugging tools. You will also need to
make a local directory. For this example, create a local directory as
C:/localstore.

In the Project Properties dialog box, set the Symbol Path on the
Debugging page to:

SRV*c:/localstore*http://msdl.microsoft.com/download/symbols

This string tells the debugger to use the symbol server to get
symbols and create a local symbol server, where they will be copied. Now, when
you press F5 on the minidump, the symbols are copied from the Microsoft Web
site, copied to the local store, and then used by the debugger. After the first
time you do this, your performance will be quicker, because they will be
retrieved from the local store rather than from the Web.

When debugging a non-Microsoft application, you should use a
combination of methods A and B. Use A to get the system components and add
paths to your bits and symbols by separating them with semicolons, such as:

c:/drop/build/myapp;SRV*c:/localstore*http://msdl.microsoft.com/download/symbols

Because the symbol server is an undocumented feature of Visual Studio .NET,
there is no error reporting. If the syntax is incorrect or Symsrv.dll is not on
the path, the symbols will fail to load with the error "No symbols
loaded". You can also use symbol servers to store and retrieve binaries,
but the MODPATH syntax needs to use symsrv*symsrv.dll*
instead of SRV*.

Note   The Microsoft
symbol server does not contain binary files, but any symbol servers that you
create can.

Symbol servers work for "live" debugging, as well, not
only for minidumps. To do this, you will need to set the Symbol Path on the
Debugging page properly.

How
Microsoft Uses Minidumps

Microsoft has been using minidumps to improve the quality of its
applications for more than a year. Microsoft Internet Explorer 5.5 and
Microsoft Office XP were the first products to ship with the new Dr. Watson, a
utility that catches when an application stops responding, creates a minidump,
and asks the user if he or she wants to submit the information to Microsoft.

If the user clicks Send Error Report, then Dr. Watson creates and
submits the minidump to a server on the Microsoft Web site. The utility
performs the user prompting and minidump saving in a different process, which
is spawned from the crashing process. This places less demand on the crashed
application, increasing the chances of getting a valid minidump from the user.

Further Improvements

On the server side, the minidumps are analyzed into areas of
similar crashes, based on where the crash occurred and in what component. This
gives product teams statistics of how often the application crashed and how
often a given crash occurs. The teams also receive the actual minidumps of the
crashes for further investigation. In some cases where the crash is fully
understood, users are automatically directed to a Web page that contains
information about a known workaround or contains a fix to correct the problem.
Since the release of Internet Explorer 5.5 and Office XP, many other product
teams use similar technology to gather crash information. It is also a standard
part of Windows XP.

The sample discussed in this article provides a basis for
understanding minidumps. The sample does not discuss retrieving the dump file
from the user. In the simplest case, the user can be prompted to send the
minidump by e-mail. Also, potential privacy issues can occur, because user data
may be present on the stack and is guaranteed to be present in a minidump that
includes the full heap. This should be made clear to users. The Microsoft data
collection policy, which also contains additional information about the exact
contents of minidumps, is at http://watson.microsoft.com/dw/1033/dcp.asp.

In addition, the creation of minidumps from a Windows service
that has an unhandled exception poses additional challenges, to do with desktop
access (for example, if no one is using the console, then you cannot prompt
them) and security contexts.

Conclusion

Minidumps are a new technology to port-mortem debug crashes on
user workstations. It is easy to add code to existing applications to
automatically create them when an unhandled exception occurs. Visual
Studio .NET can easily read them back to recreate the crash and allow the
developer to debug the problem. Symbol server can be used to make it much
easier to find system symbols to aid in this analysis.