Sure, it might be nice to build a higher quality product. But how much quality can we really afford? Well, let's break out our Cost of Quality Calculator and try out the numbers.

What is Our Cost of Quality Today?

First the basics: Let's say we're going to write a system of 25,000 Lines of Code, and our data tells us that we inject about 50 defects per 1000 lines of code. So along with those 25 KLOC, we will also write 1250 defects.

			Defect Injection		Defect Removal		Defects in Product		Cost of Quality
Activity	Size		Rate	#	Yield	#	#	/KLOC	Rate	Hours
Design & Code	25	KLOC	50	1250			1250	50.00
Totals				1250		0	1250	50.00		0

For a system of that size, we usually end up with a 20-page requirements document and a 40-page high-level design specification. And since we usually inject 5 defects per page in our documents, that gives us 100 defects in the requirements and 200 defects in the design bringing our total to 1550 defects.

			Defect Injection		Defect Removal		Defects in Product		Cost of Quality
Activity	Size		Rate	#	Yield	#	#	/KLOC	Rate	Hours
Write Requirements	20	Pgs	5	100			100	4.00
High Level Design	40	Pgs	5	200			300	12.00
Design & Code	25	KLOC	50	1250			1550	62.00
Totals				1550		0	1550	62.00		0

Of course, we always review the requirements. We review about 10 pages of requirements per hour, finding 40% of the defects in the document. We incur 2 hours of Cost of Quality to find and fix 40 defects.

			Defect Injection		Defect Removal		Defects in Product		Cost of Quality
Activity	Size		Rate	#	Yield	#	#	/KLOC	Rate	Hours
Write Requirements	20	Pgs	5	100			100	4.00
Review Requirements					40%	40	60	2.40	10	2
High Level Design	40	Pgs	5	200			260	10.40
Design & Code	25	KLOC	50	1250			1510	60.40
Totals				1550		40	1510	60.40		2

And of course, we test. Our data tells us that we find 50% of the existing defects in Unit test and that we can do it at the rate of 5 defects found and fixed per hour. We find 40% of the existing defects in Integration (at a rate of 2 hours per defect found & fixed, or 0.5 defects per hour), and 30% in System test (at a rate of 10 hours per defect found and fixed, or 0.1 defects per hour). Unfortunately, we also inject about 2.5 new defects for every 100 we fix.

			Defect Injection		Defect Removal		Defects in Product		Cost of Quality
Activity	Size		Rate	#	Yield	#	#	/KLOC	Rate	Hours
Write Requirements	20	Pgs	5	100			100	4.00
Review Requirements					40%	40	60	2.40	10	2
High Level Design	40	Pgs	5	200			260	10.40
Design & Code	25	KLOC	50	1250			1510	60.40
Unit Test			2.5%	19	50%	755	774	30.96	5	151
Integration Test			2.5%	8	40%	310	472	18.88	0.5	619
System Test			2.5%	4	30%	142	334	13.36	0.1	1416
Totals				1580		1246	334	13.36		2188

So this spreadsheet represents our normal 25KLOC software project. Over a person-year of effort spent in Cost of Quality work, and hundreds of defects delivered. More than 13 defects per 1000 lines of code is pretty bad.

How much will it cost us to improve it?

What if we were to do a more formal inspection of the requirements? Let's say we could only inspect about 2 pages per hour, but would find 65% of the existing defects.

			Defect Injection		Defect Removal		Defects in Product		Cost of Quality
Activity	Size		Rate	#	Yield	#	#	/KLOC	Rate	Hours
Write Requirements	20	Pgs	5	100			100	4.00
Inspect Requirements					65%	65	35	1.40	2	10
High Level Design	40	Pgs	5	200			235	9.40
Design & Code	25	KLOC	50	1250			1485	59.40
Unit Test			2.5%	19	50%	743	761	30.44	5	149
Integration Test			2.5%	8	40%	304	464	18.57	0.5	609
System Test			2.5%	3	30%	139	328	13.14	0.1	1393
Totals				1580		1251	328	13.14		2160

We found more defects, but look! Our cost of Quality went down! Why did that happen? Because those defects we removed early in the inspection didn't get found later in testing when they are more expensive. Cool! Let's try inspecting our design document, too!

			Defect Injection		Defect Removal		Defects in Product		Cost of Quality
Activity	Size		Rate	#	Yield	#	#	/KLOC	Rate	Hours
Write Requirements	20	Pgs	5	100			100	4.00
Inspect Requirements					65%	65	35	1.40	2	10
High Level Design	40	Pgs	5	200			235	9.40
HL Design Inspection					65%	153	82	3.29	2	20
Design & Code	25	KLOC	50	1250			1332	53.29
Unit Test			2.5%	17	50%	666	683	27.31	5	133
Integration Test			2.5%	7	40%	273	416	16.66	.05	546
System Test			2.5%	3	30%	125	295	11.79	0.1	1249
Totals				1577		1282	295	11.79		1959

Yes! Same again! Fewer defects and less time spent in Cost of Quality! Our coding is our biggest source of defects. What if we inspect the code as well? Let's plan for the same 65% yield and say we can inspect code at 100 LOC per hour. (That's gonna be a lot of hours, but here goes!)

			Defect Injection		Defect Removal		Defects in Product		Cost of Quality
Activity	Size		Rate	#	Yield	#	#	/KLOC	Rate	Hours
Write Requirements	20	Pgs	5	100			100	4.00
Inspect Requirements					65%	65	35	1.40	2	10
High Level Design	40	Pgs	5	200			235	9.40
HL Design Inspection					65%	153	82	3.29	2	20
Design & Code	25	KLOC	50	1250			1332	53.29
Design & Code Inspections					65%	866	466	18.65	100	250
Unit Test			2.5%	6	50%	233	239	9.56	5	47
Integration Test			2.5%	2	40%	96	146	5.83	0.5	191
System Test			2.5%	1	30%	44	103	4.13	0.1	437
Totals				1559		1456	103	4.13		955

Jackpot! We improved our defect level by almost a factor of three! And we cut our Cost of Quality by more than half! We have achieved a respectable defect level of less than 5 defects per 1000 lines of code while investing less than half of the quality effort we started with.

So, What Do The Numbers Mean?

Of course all of this is theoretical and based on some basic averages that many organizations have reported. This exercise will be most interesting when you plug in your own organization's numbers.

What? You don't have these numbers? Better start measuring your Cost of Quality today!

P.S. If you want to use my spreadsheet instead of building your own, just download it here or from my website at www.ASKProcess.com/resources/CostOfQuality.xls.

Related Resources
Our Requirements Walkthrough Checklist can help you organize a productive inspection of requirements. Make sure everyone knows what they should look for when reviewing requirements.

Comments

Nicely Done! I fully agree with the ROI of conducting Requirements and Design Formal Inspections (or whatever kind of inspection works best in your enviroment). I've had some luck with modeling this kind of early defect removal performance and generally see a gain of 300% for these early defect removal efforts. In other words, roughly, for every solid defect you remove early (Requiements or Design) you'll reduce the count of solid defects you find later (in formal test) by 3 defects. For more on this visit my website. http://peerreviewprocessroi.com/