[Previous entry: "Are you voting with Bill?"] [Next entry: "Internet Security Guard"]

10/28/2004: "Microsoft Software Development Process"

A full test pass is where we run and analyze every single automated test we have currently built for the product, and then also manually run every test we plan to automate before the product ships. My team currently has 102,000 test cases which test 505,000 different test scenarios – so our full test passes tend to be fairly involved things.
All code checkins must always be peer-reviewed prior to checking in (this is even true when the most senior developer checks in). Code changes must then run through a few hours of checkin suites that provide base-level unit test coverage of the product prior to checkin. For ASP.NET Whidbey, these checkin suites currently yield a block-level code coverage of about 64% (meaning 64% of all code blocks are exercised and run during the tests). We then run more exhaustive nightly tests over the product to catch issues in the latest builds of the product.
...
There is a balance/trade-off between running more or less tests during checkin – the more tests the more coverage, but the more tests the longer it takes to checkin. If we ran our entire automated test bed it would take about 12 hours for each code change prior to checkin – which is why we’ve trimmed the checkin suites to be more of a key subset that provides broad scenario coverage (and then rely on nightly and full automation runs to be comprehensive). We then continually add/remove checkin suites during the product cycle in reaction to regressions that get missed.


http://weblogs.asp.net/scottgu/archive/2004/10/23/246709.aspx