Are Tests really Specifications?

Martin Fowler, Jeff Langr and Bob Martin seem very keen to defend the idea that tests should be regarded as specifications.  To me, this seems very strange, almost bizarre.  

Don't get me wrong, I recognize the value of automated test suites, but it never occurred to me that anyone would regard them as specifying the software they test.  This idea seems to show a fundamental misunderstanding of the relationships between specifications, programs and tests, and it blurs important distinctions.

Fowler uses the term 'Specification by Example'  but what he is really talking about is exploring the specification by generating examples.  This is a good and valid activity, but it think it is incorrect to call the generated examples the specification. 

The exponentiation example that Langr presents is great as a suite of tests, but as a specification it is rather a poor one.  Firstly, it is Java-specific (it would have to be rewritten to apply it to a Python program) but the concept of exponentiation is independent of any programming language.  Secondly, although Langr criticises the Sun specification for being too long, his is about the same length and doesn't even cover floating point exponents as the Sun one does.  Thirdly, his specification fails to actually define what exponentiation is (this is in spite of him criticising the Sun one for the same failing).  He gives lots examples (2⁴ = 16, (-2)³ = -8, and so on), but these do not amount to a definition.  If you want to define exponentiation you really have to use equations, for instance: a⁰ = 1, aⁿ = a*a^n-1 and so on.  Incidentally, this enables you to define floating point exponents in terms of integer exponents quite simply by using the equivalence between between y = a^m/n and y^m = aⁿ (for example:  y = 10^3.14159 is equivalent to y¹⁰⁰⁰⁰⁰ = a ³¹⁴¹⁵⁹).

Fowler, Langr and Martin seem to be trying to use 'Tests are Specifications'  to support Test-Driven Design (TDD).  I think this is a mistake on two counts: firstly, tests are very poor specifications and, secondly, TDD does not need this support - it stands perfectly well without it. 

Calling tests 'specifications' is contrived, misleading, and devalues the concepts of specification and testing.  Why then should obviously very intelligent and experienced people want to do this?  I can only assume that it is a result of political forces within the XP/Agile movement.  Maybe their clients asked for written specs, so the Agilists looked around for the nearest thing that they actually do write, found tests, and decided to call them specifications.

Comments

[Jeff L]

You're right, tests aren't specifications. The main thrust of "tests as specifications (by example)," or whatever, is that they are intended to document class capabilities. They are intended to diminish the need for vague, ambiguous, and otherwise poorly written English-language documents (including comments).

As far as the tests not "defining" what exponentiation is, in general most developers understand examples far better than written specs.

You're missing the main point by arguing about semantics. If you're doing TDD and producing unit tests to cover the code, they are not of much value unless they are readable. Hence the emphasis on pushing them towards an ability to act as specs.