Recently in testing Category

I am excited to announce that I have completed my next grant milestone! I recently increased test coverage of extend_vtable.c to over 95% ( 95.5% to be exact), achieving the milestone with a half percent buffer. It definitely wasn't easy, but I changed the way I was approaching writing tests and it resulted in a huge burst of productivity.

I went through a test coverage report and wrote down, on an actual piece of paper, every function that had no test coverage. This allowed me to circle the functions that I thought would be easiest to write tests for, and quickly got those out of the way. I then went for uncovered functions that were similar to already covered functions, and then finally I got to the hard functions.

This was a fruitful exercise, because it was decided by Parrot developers that some VTABLE functions escaped accidentally and that they should be removed from the public API. Whiteknight++ removed Parrot_PMC_destroy (extra points for humor), which I was using incorrectly in the extend_vtable tests and which was actually coredumping Parrot, but only on certain platforms. I then removed Parrot_PMC_mark and Parrot_PMC_invoke, the first being an implementation detail of the garbage collector, and Parrot_PMC_invoke because it was the only function that returned a '''Parrot_Opcode_t*''' and basically not fit for public consumption.

I also created a ticket (TT#2126) for a bug in the Parrot_PMC_morph function, which has some possibly buggy but definitely unspecified behavior.

The remaining, untested functions in extend_vtable are clone_pmc, cmp_pmc, get_pointer_keyed_int, get_pointer_keyed_str, remove_vtable_override, set_pointer_keyed and set_pointer_keyed_str. I leave the testing of these functions as an exercise to the interested reader :)

Grant Refactoring

This reminds me of a saying, I can't remember it exactly, but it is something about the best laid plans of camels and butterflies often taste like onions. Anyway, since I wrote my grant, the Parrot Embed API was deprecated and replaced with a shinier and better documented system. After talking with cotto++ and whiteknight++ on IRC, it was decided that working on test coverage for the new embed API was a better use of resources than writing tests for the old embed API that my original grant referred to, which will most likely be removed from Parrot soon.

The new embed API is called src/embed/api.c and the plan is to replace my grant milestone of 95% coverage of embed.c with 95% coverage of embed/api.c, which is currently at 72% coverage.

To summarize, I have two grant milestones left, increasing extend.c (currently at 61% ) and embed/api.c to 95% coverage.

Given the lessons learned from testing extend_vtable and based on the fact that I have already made some headway, my new estimate for these milestones is three weeks each. To make this more definite, I plan to be done with this grant work by July 15th.

This is the home stretch! I can feel it in my bones.

Parrot Embed Grant Update #2

| | Comments (1) | TrackBacks (0)

I have slowly but surely been increasing test coverage of src/extend_vtable.c, which when I started my grant was at 5% and is now sitting at 43%. I estimated that it would take about two weeks to get it to 50% and then another two weeks to get to 95%, but this was a bit optimistic. When I look back, giving myself a month for each would have been appropriate.

I do believe that my development velocity has increased recently, because I wrote a convenience funciton called extend_vtable_output_is(), which greatly reduces the number of lines of code that each extend_vtable test requires. The function is basically a wrapper around already existing functions that automates the process of compiling C source code use the Parrot C API, but it removes the need for roughly 40 lines of boilerplate stuff, such as including the proper header files, defining convenience functions that do basic error checking and data structure creation.

The whole reason this subsystem is undertested is because no function like extend_vtable_output_is() existed. Now each of my extend_vtable tests is around 10 lines, instead of 50. I've only had the new function for about half the time of the grant, so tests are now getting written much faster.

I have added about 700 lines of lines of tests since this grant started (using the spiffy new extend_vtable_output_is() function), and that has caused a 38% increase in test coverage. With the power of mathematics, we can figure out that (.38)*N = 700, which means N = 700/.38 = ~1842, where N is the number of lines of tests needed to cover 100% of the file (roughly).

My new estimate for getting to 95% coverage of extend_vtable is one month from now, Feb 24th, and then another two months for the rest of the grant.

The reason I say two months is that I am actually trying to hit a moving target, and the code coverage of extend.c and embed.c have actually *gone down* since I started the grant. Currently extend.c is about 6% lower and embed.c is about 14% lower. This means my grant is actually getting harder to complete.

I also learned about a very useful yet undocument environment variable called POSTMORTEM which makes Parrot test functions leave various intermediate files around for debugging purposes if it is set to true, which greatly helps in developing these tests. I plan to add documentation about this to the Parrot developer documentation.

Given the new test function and the new coverage numbers, I estimate that I will be able to complete this grant by late April 2011.

Test::More eating your memory?

| | TrackBacks (0)
It always humors me when I am having some issue with a Perl module, thinking that I am the only one, and then reading my RSS feed and seeing someone having exactly the same issue.

I wrote an eXtended Test (xt/verify_strong.t) for Math::Primality which verifies that it correctly identified all strong pseudoprimes less than 1e15. That is 1,000,000,000,000,000 ! There end up being  419489 such numbers, so I thought nothing of writing

use Test::More tests => 419489;

Needless to say, I was a bit worried to see my test process constantly eating RAM, ending up utilizing about 650MB of RAM before exiting. I thought that maybe there was a memory leak in Math::GMPz, which Math::Primality uses to access the GNU MultiPrecision (GMP) library. But after making this small patch to roll my own ok() method, my memory usage plummeted to about 50MB, over a 90% memory usage decrease!

The very next day I see garu talking about the very same issue. Evidently it is "working as designed" according to Schwern and it comes down to Test::Builder saving the history/output of every test, which is not currently configurable. This is one of the many things that Test::Builder2 will hopefully solve.

About this Archive

This page is a archive of recent entries in the testing category.

swig is the previous category.

Find recent content on the main index or look in the archives to find all content.

Clicky Web Analytics 42