Recently in grant Category

A Final TPF Parrot Embed/Extend Grant Update


Really TLDR: The Parrot has landed.

It brings me great joy to announce that I have completed all milestones for my TPF grant regarding the Parrot Embed/Extend subsystems! Not only that, but all of my grant work was included in the most recent release of Parrot, 3.5.0 "Menelaus".

The actual TLDR of this update is "many tests were written, code coverage is above 95% for all systems described in the grant, docs were improved, many Parrot Trac tickets were created and many a blarg toast was cooked.

For those of you that have a thirst for knowledge unquenched (I know who you are), you are welcome to pondiferously peruse the Impending Technical Details.

The Deets

The last portion of this grant definitiely challenged me to think in new ways about testing and I am now only beginning to reap the benefits. I was charged with adding code coverage a few rarely-if-ever-used C functions in Parrot's embed/exted subsystem, which allows you embed Parrot into other applications and other funky stuff.

Whiteknight++ greatly helped me write a test for Parrot_sub_new_from_c_func which takes a C function and a string that describes the function signature of the C function and returns a NCI PMC, which can be invoked.

I also learned many lessons about code coverage during the final stage of this grant, even though I thought I was at such a level of expertness that it would be hard to learn drastically new and important perspectives on testing. This pattern of thinking is always wrong.

Lesson 1

Sometimes you are the underdog and you have to interpret the rules in a new way in order to have a chance at winning. You need to be Ender Wiggins from Ender's Game: continually inventing new tactics to keep a winning edge over the competition.

I noticed that a large portion (about 80%) of the uncovered code in one file was a macro that was copy-and-pasted into two places. I refactored this into a single macro called POP_CONTEXT, which reduced the total number of lines in the file by roughly 10, while simultaneously decreased the number of uncoverd lines in the file by ~20 lines, which had a combined effect of pushing the code coverage over the necessary 95% mark.

This change definitely increases the maintainability and modularity of the code, but it feels a bit like gaming the system. Nonetheless, it saved the day.

Lesson 2

The simplest useful test that you are avoiding is the most valuable next test to write, because it has the best ROI (Return on Investment, where investment is the time it takes to write the test, and the return is having an automated way of verifying that the feature works.

Lesson 3

Software developers are very optimistic about time estimates. We forget about all the possible things that could go wrong and often quote estimates on something approaching "base case scenario". As a rule of thumb, I think all software developers should think long and hard about a time estimate for a given project, write down the estimate, then multiply that time estimate by pi for a REAL estimate.

I theorize that pi is the factor of time it takes to debug and write tests for behavior of hard-to-recreate edge cases.

I originally thought my grant would take about 3 months, but it ended up taking about 9 or ten. QED.

Finally, I would like to thank my grant manager Makoto Nozaki for providing lots of feedback, support and encouragement, as well as everyone else at the The Perl Foundation for funding this grant.

I am excited to announce that I have completed my next grant milestone! I recently increased test coverage of extend_vtable.c to over 95% ( 95.5% to be exact), achieving the milestone with a half percent buffer. It definitely wasn't easy, but I changed the way I was approaching writing tests and it resulted in a huge burst of productivity.

I went through a test coverage report and wrote down, on an actual piece of paper, every function that had no test coverage. This allowed me to circle the functions that I thought would be easiest to write tests for, and quickly got those out of the way. I then went for uncovered functions that were similar to already covered functions, and then finally I got to the hard functions.

This was a fruitful exercise, because it was decided by Parrot developers that some VTABLE functions escaped accidentally and that they should be removed from the public API. Whiteknight++ removed Parrot_PMC_destroy (extra points for humor), which I was using incorrectly in the extend_vtable tests and which was actually coredumping Parrot, but only on certain platforms. I then removed Parrot_PMC_mark and Parrot_PMC_invoke, the first being an implementation detail of the garbage collector, and Parrot_PMC_invoke because it was the only function that returned a '''Parrot_Opcode_t*''' and basically not fit for public consumption.

I also created a ticket (TT#2126) for a bug in the Parrot_PMC_morph function, which has some possibly buggy but definitely unspecified behavior.

The remaining, untested functions in extend_vtable are clone_pmc, cmp_pmc, get_pointer_keyed_int, get_pointer_keyed_str, remove_vtable_override, set_pointer_keyed and set_pointer_keyed_str. I leave the testing of these functions as an exercise to the interested reader :)

Grant Refactoring

This reminds me of a saying, I can't remember it exactly, but it is something about the best laid plans of camels and butterflies often taste like onions. Anyway, since I wrote my grant, the Parrot Embed API was deprecated and replaced with a shinier and better documented system. After talking with cotto++ and whiteknight++ on IRC, it was decided that working on test coverage for the new embed API was a better use of resources than writing tests for the old embed API that my original grant referred to, which will most likely be removed from Parrot soon.

The new embed API is called src/embed/api.c and the plan is to replace my grant milestone of 95% coverage of embed.c with 95% coverage of embed/api.c, which is currently at 72% coverage.

To summarize, I have two grant milestones left, increasing extend.c (currently at 61% ) and embed/api.c to 95% coverage.

Given the lessons learned from testing extend_vtable and based on the fact that I have already made some headway, my new estimate for these milestones is three weeks each. To make this more definite, I plan to be done with this grant work by July 15th.

This is the home stretch! I can feel it in my bones.

Parrot Embed Grant Update #5 : Zen Pebbles

| | TrackBacks (0)

I am still on the path of increasing test coverage in src/extend_vtable.c. It is much like a zen study, where you methodically concentrate on many tiny little pebbles, one at a time, moving them in the sand, to just the right place. According to the latest code coverage statistics, we are now at 72% code coverage, which is an increase of about 8% since my last report.

Many, many more tests involving Key PMCs were added. For an intro to what they are, take a look at my previous grant update. Many of the tests are clusters of related tests, because most VTABLEs have many similar forms which take integer, string or PMC-flavored keys. I ran into some platform-specific bugs which only manifest on Darwin machines, which were reported by Jim Keenan in TT# 2098 and which I then fixed by querying with a non-empty Key, which is more prudent.

I also ran into some actual bugs which I reported as Trac Tickets. First is that the cmp_pmc VTABLE does not seem to be working correctly from extend_vtable, which was reported in TT #2103. Then I fell into a "hole" in the VTABLE API, where ResizablePMCArray does not respond to defined_keyed(), which it should. This is described in TT #2094.

In retrospect, this was one of the most productive periods of my grant work. I estimate that I will be very close to the 95% milestone by my next grant update at this pace, which is very exciting.

Parrot Embed Grant Update #4 : The Journey Continues

| | TrackBacks (0)

This wayward son is still on his treacherous journey to increase test coverage in src/extend_vtable.c. When we last left off our traveler, he explained what the mythical VTABLE beast looked like, and we shall continue with the study of this chimerical fauna.

According to the latest code coverage statistics, we are now at 64% code coverage, which is an increase of about 10% since my last report. Most of this grant work concentrated on vtables that required Key PMCs. A Key PMC is an object that can be used to look something up in a Hash PMC or other aggregated object that supports "keyed access". It is very much similar to a "hash key" that can be used to look up the appropriate value.

One of the lessons that I have learned in working on these tests is that it is very easy to write tests that pass on gcc, but which absolutely explode with g++. This has to do with gcc not being as strict when some questionable type casting is done. I have learned my lesson and I promise not to break the test suite anymore. I will use g++ in my testing from now on, promise!

My productivity was definitely hampered by moving to a new house and having a two week business trip in the last month, but my new home office is finally set up, so I expect productivity to approach previous levels of adding a few dozen tests per week.

Parrot Embed Grant Update #2

| | Comments (1) | TrackBacks (0)

I have slowly but surely been increasing test coverage of src/extend_vtable.c, which when I started my grant was at 5% and is now sitting at 43%. I estimated that it would take about two weeks to get it to 50% and then another two weeks to get to 95%, but this was a bit optimistic. When I look back, giving myself a month for each would have been appropriate.

I do believe that my development velocity has increased recently, because I wrote a convenience funciton called extend_vtable_output_is(), which greatly reduces the number of lines of code that each extend_vtable test requires. The function is basically a wrapper around already existing functions that automates the process of compiling C source code use the Parrot C API, but it removes the need for roughly 40 lines of boilerplate stuff, such as including the proper header files, defining convenience functions that do basic error checking and data structure creation.

The whole reason this subsystem is undertested is because no function like extend_vtable_output_is() existed. Now each of my extend_vtable tests is around 10 lines, instead of 50. I've only had the new function for about half the time of the grant, so tests are now getting written much faster.

I have added about 700 lines of lines of tests since this grant started (using the spiffy new extend_vtable_output_is() function), and that has caused a 38% increase in test coverage. With the power of mathematics, we can figure out that (.38)*N = 700, which means N = 700/.38 = ~1842, where N is the number of lines of tests needed to cover 100% of the file (roughly).

My new estimate for getting to 95% coverage of extend_vtable is one month from now, Feb 24th, and then another two months for the rest of the grant.

The reason I say two months is that I am actually trying to hit a moving target, and the code coverage of extend.c and embed.c have actually *gone down* since I started the grant. Currently extend.c is about 6% lower and embed.c is about 14% lower. This means my grant is actually getting harder to complete.

I also learned about a very useful yet undocument environment variable called POSTMORTEM which makes Parrot test functions leave various intermediate files around for debugging purposes if it is set to true, which greatly helps in developing these tests. I plan to add documentation about this to the Parrot developer documentation.

Given the new test function and the new coverage numbers, I estimate that I will be able to complete this grant by late April 2011.

About this Archive

This page is a archive of recent entries in the grant category.

git is the previous category.

gsoc is the next category.

Find recent content on the main index or look in the archives to find all content.

Clicky Web Analytics 42