Added test coverage measurement to build

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Added test coverage measurement to build

Attila Szegedi
Hi folks,

I've added code coverage measurement capabilities to our test suite in the  
SVN. Just run "ant test-coverage-report", and then point your browser at  
build/coverage/report/index.html. I added it specifically as I am writing  
tests for some functionality I'm writing right now, and want to focus my  
tests on covering as much of the code as possible.

We're using EMMA: <http://emma.sourceforge.net> for the task. EMMA is very  
nice - I'm using it in production projects I work on. It creates an  
instrumented copy of the code that is only incrementally updated as  
original classes are recompiled, so it's very low overhead during build  
time. (It only builds when "ant test" runs, of course - adds about 100ms  
overhead to "ant test" on my machine). It probably makes our tests run few  
percent slower, but again, they run in about 5 seconds on my machine, so  
it's not exactly a concern.

Of course, we must not be deceived by a false sense of security -- EMMA  
measures block coverage. A block is a stretch of code without a branch  
instruction. It measures whether a block was entered at least once. It is  
less comprehensive than either branch coverage (which also measures what  
proportion of all branches in/out of a block were taken) or path coverage  
(which measures what *combinations* of branches were taken), but those  
tend to explode exponentially compared to block coverage (which is linear  
to code size) and in the end, we aren't writing code for controlling  
nuclear reactors here, so having block coverage information is already  
great.

Tell me what you think.

Attila.

--
home: http://www.szegedi.org
weblog: http://constc.blogspot.com

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
FreeMarker-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/freemarker-devel
Reply | Threaded
Open this post in threaded view
|

Re: Added test coverage measurement to build

Daniel Dekany
Monday, May 7, 2007, 4:04:56 PM, Attila Szegedi wrote:

> Hi folks,
>
> I've added code coverage measurement capabilities to our test suite in the
> SVN. Just run "ant test-coverage-report", and then point your browser at
> build/coverage/report/index.html. I added it specifically as I am writing
> tests for some functionality I'm writing right now, and want to focus my
> tests on covering as much of the code as possible.
>
> We're using EMMA: <http://emma.sourceforge.net> for the task. EMMA is very
> nice - I'm using it in production projects I work on. It creates an  
> instrumented copy of the code that is only incrementally updated as  
> original classes are recompiled, so it's very low overhead during build
> time. (It only builds when "ant test" runs, of course - adds about 100ms
> overhead to "ant test" on my machine). It probably makes our tests run few
> percent slower, but again, they run in about 5 seconds on my machine, so
> it's not exactly a concern.
>
> Of course, we must not be deceived by a false sense of security -- EMMA
> measures block coverage. A block is a stretch of code without a branch
> instruction. It measures whether a block was entered at least once. It is
> less comprehensive than either branch coverage (which also measures what
> proportion of all branches in/out of a block were taken) or path coverage
> (which measures what *combinations* of branches were taken), but those
> tend to explode exponentially compared to block coverage (which is linear
> to code size) and in the end, we aren't writing code for controlling  
> nuclear reactors here, so having block coverage information is already
> great.
>
> Tell me what you think.

Somehow I like these analytical things, even if they are often useless
in practice. :) So I will certainly integrate it into my new projects
as well. I have even started to use CheckStyle, and treat CheckStyle
warnings like compilation errors. That's that you call fascism, right?
:) Don't worry, I will not ask that in FM. (BTW, if you can off-hand
recommend a similar free stuff for project size/complexity measurement
(whatever that means), tell me.)

For those who are interested, the summary of the results:

---------------+------------------+--------------------+---------------------
class, %       | method, %        | block, %           | line, %
---------------+------------------+--------------------+---------------------
72%  (417/583) | 52%  (1871/3595) | 54%  (53583/98703) | 53%  (11044,9/20788)

total packages: 22
total executable files: 274
total classes:  583
total methods:  3595
total executable lines: 20788

(Note that EMMA ignores the lines that doesn't really compile to
executable code... or something like that. Like in debuggers,
basically. So we have 20788 lines of code without comments, imports,
method declarations and such "overhead" stuff. That you can't see from
the above, is that about the 65% of lines is in the code and template
subpackages, which is a little bit higher test coverage than the
average. It reinforces my feeling that the core FreeMarker package
incorporates some stuff that rather should be separate projects from
project manager's viewpoint.)

--
Best regards,
 Daniel Dekany


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
FreeMarker-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/freemarker-devel