Results tagged “parrot” from dukeleto.pl

A Final TPF Parrot Embed/Extend Grant Update

|

Really TLDR: The Parrot has landed.

It brings me great joy to announce that I have completed all milestones for my TPF grant regarding the Parrot Embed/Extend subsystems! Not only that, but all of my grant work was included in the most recent release of Parrot, 3.5.0 "Menelaus".

The actual TLDR of this update is "many tests were written, code coverage is above 95% for all systems described in the grant, docs were improved, many Parrot Trac tickets were created and many a blarg toast was cooked.

For those of you that have a thirst for knowledge unquenched (I know who you are), you are welcome to pondiferously peruse the Impending Technical Details.

The Deets

The last portion of this grant definitiely challenged me to think in new ways about testing and I am now only beginning to reap the benefits. I was charged with adding code coverage a few rarely-if-ever-used C functions in Parrot's embed/exted subsystem, which allows you embed Parrot into other applications and other funky stuff.

Whiteknight++ greatly helped me write a test for Parrot_sub_new_from_c_func which takes a C function and a string that describes the function signature of the C function and returns a NCI PMC, which can be invoked.

I also learned many lessons about code coverage during the final stage of this grant, even though I thought I was at such a level of expertness that it would be hard to learn drastically new and important perspectives on testing. This pattern of thinking is always wrong.

Lesson 1

Sometimes you are the underdog and you have to interpret the rules in a new way in order to have a chance at winning. You need to be Ender Wiggins from Ender's Game: continually inventing new tactics to keep a winning edge over the competition.

I noticed that a large portion (about 80%) of the uncovered code in one file was a macro that was copy-and-pasted into two places. I refactored this into a single macro called POP_CONTEXT, which reduced the total number of lines in the file by roughly 10, while simultaneously decreased the number of uncoverd lines in the file by ~20 lines, which had a combined effect of pushing the code coverage over the necessary 95% mark.

This change definitely increases the maintainability and modularity of the code, but it feels a bit like gaming the system. Nonetheless, it saved the day.

Lesson 2

The simplest useful test that you are avoiding is the most valuable next test to write, because it has the best ROI (Return on Investment, where investment is the time it takes to write the test, and the return is having an automated way of verifying that the feature works.

Lesson 3

Software developers are very optimistic about time estimates. We forget about all the possible things that could go wrong and often quote estimates on something approaching "base case scenario". As a rule of thumb, I think all software developers should think long and hard about a time estimate for a given project, write down the estimate, then multiply that time estimate by pi for a REAL estimate.

I theorize that pi is the factor of time it takes to debug and write tests for behavior of hard-to-recreate edge cases.

I originally thought my grant would take about 3 months, but it ended up taking about 9 or ten. QED.

Finally, I would like to thank my grant manager Makoto Nozaki for providing lots of feedback, support and encouragement, as well as everyone else at the The Perl Foundation for funding this grant.

Parrot Embed Grant Update #6 : Still Hackin', Less Slackin'

|

I am excited to announce that I have completed my next grant milestone! I recently increased test coverage of extend_vtable.c to over 95% ( 95.5% to be exact), achieving the milestone with a half percent buffer. It definitely wasn't easy, but I changed the way I was approaching writing tests and it resulted in a huge burst of productivity.

I went through a test coverage report and wrote down, on an actual piece of paper, every function that had no test coverage. This allowed me to circle the functions that I thought would be easiest to write tests for, and quickly got those out of the way. I then went for uncovered functions that were similar to already covered functions, and then finally I got to the hard functions.

This was a fruitful exercise, because it was decided by Parrot developers that some VTABLE functions escaped accidentally and that they should be removed from the public API. Whiteknight++ removed Parrot_PMC_destroy (extra points for humor), which I was using incorrectly in the extend_vtable tests and which was actually coredumping Parrot, but only on certain platforms. I then removed Parrot_PMC_mark and Parrot_PMC_invoke, the first being an implementation detail of the garbage collector, and Parrot_PMC_invoke because it was the only function that returned a '''Parrot_Opcode_t*''' and basically not fit for public consumption.

I also created a ticket (TT#2126) for a bug in the Parrot_PMC_morph function, which has some possibly buggy but definitely unspecified behavior.

The remaining, untested functions in extend_vtable are clone_pmc, cmp_pmc, get_pointer_keyed_int, get_pointer_keyed_str, remove_vtable_override, set_pointer_keyed and set_pointer_keyed_str. I leave the testing of these functions as an exercise to the interested reader :)

Grant Refactoring

This reminds me of a saying, I can't remember it exactly, but it is something about the best laid plans of camels and butterflies often taste like onions. Anyway, since I wrote my grant, the Parrot Embed API was deprecated and replaced with a shinier and better documented system. After talking with cotto++ and whiteknight++ on IRC, it was decided that working on test coverage for the new embed API was a better use of resources than writing tests for the old embed API that my original grant referred to, which will most likely be removed from Parrot soon.

The new embed API is called src/embed/api.c and the plan is to replace my grant milestone of 95% coverage of embed.c with 95% coverage of embed/api.c, which is currently at 72% coverage.

To summarize, I have two grant milestones left, increasing extend.c (currently at 61% ) and embed/api.c to 95% coverage.

Given the lessons learned from testing extend_vtable and based on the fact that I have already made some headway, my new estimate for these milestones is three weeks each. To make this more definite, I plan to be done with this grant work by July 15th.

This is the home stretch! I can feel it in my bones.

What is M0 ?

|

I met with fellow Parrot hackers allison++, cotto++ and chromatic++ recently in Portland, OR (it was jokingly called YAPC::OR on IRC) to talk about what we call M0. M0 stands for "magic level 0" and it is a refactoring of Parrot internals in a fundamental way.

cotto++ and I have been hacking on a detailed spec (over 35 pages now!) and a "final prototype" in Perl 5 in the last few weeks. M0 is as "magic level 0", which means it consists of the most basic building blocks of a virtual machine, which the rest of the VM can be built with. The term "magic" means high-level constructs and conveniences, such as objects, lexical variables, classes and their associated syntax sugar. M0 is not meant to be written by humans, except during bootstrapping. In the future, M0 will be probably be generated from Parrot Intermediate Representation (PIR), Not Quite Perl 6 (NQP) or other High Level Languages (HLLs).

The most important reason for M0 is to correct the fact that too much of Parrot internals are written in C. Parrot internals is constantly switching between code written in PIR, other HLL's such as NQP and C. Many types of optimizations go right out the window when you cross a language boundary. It is best for a virtual machine to minimize crossing language boundaries if an efficient JIT compiler is wanted, which we definitely desire. Since many hotpaths in Parrot internals cross between PIR and C, they can't be inlined or optimized as much as we would like.

A few years back, Parrot had a JIT compiler, from which many lessons were learned. I am sure some people were frustrated when we removed it in 1.7.0 but sometimes, it is best to start from a clean slate with many more lessons learned under your belt. Our old JIT did support multiple architectures but required maintaining a "JIT version" of every opcode on each architecture supported. Clearly, this method was not going to scale or be maintainable.

I will venture to say that M0 is the culmination of the lessons learned from our failed JIT. I should note that "failure" does not have a negative connotation in my mind. Indeed, only through failure are we truly learning. If you do something absolutely perfectly, you aren't learning.

We are at an exciting time in Parrot's history, in that for a long time, we wanted an elegant JIT, using all the latest spiffy techniques, but it was always an abstract idea, "just over there", but not enough to grab a-hold of. A new JIT that meets these goals absolutely requires something like M0, and is the driving force for its design. M0 will pave the way for an efficient JIT to be implemented on Parrot.

M0 currently consists of under 40 opcodes from which (we wager) all the rest of Parrot can be built upon. This is radically different from how Parrot currently works, where all of the deepest internals of Parrot are written in heavily macroized ANSI 89 C.

M0 has a source code, i.e. textual form and a bytecode form. chromatic++ brought up a good point at the beginning of the meeting about the bytecode file containing a cryptographic hash of the bytecode. This will allow one to distribute bytecode which can then be cryptographically verified by whoever eventually runs the bytecode. This is a very "fun" application of cryptography that I will be looking into further.

allison++ brought up some good questions about how merging bytecode files would be done. We hadn't really thought about that, so it lead to some fruitful conversation about how Parrot Bytecode (PBC) is currently merged, what it does wrong, and how M0 can do it less wronger.

We then talked about what exactly a "Continuation" in M0 means, and tried to clear up some definitions between what is actually meant by Context, State and Continuation.

chromatic++ also mentioned that an optional optimization for the garbage collector (GC) would be for it to create a memory pool solely to store Continuations, since they will be heavily used and many of them will be short-lived and reference each other, so having them in a small confined memory region will reduce cache misses. We are filing this under "good to know and we will do that when we get there."

Next we turned to concurrency, including how we would emulate the various concurrency models of the languages we want to support, such as Python's Global Interpreter Lock (GIL). We decided that M0 will totally ignorant of concurrency concepts, since it is a "magical" concept that will be implemented at a higher level. We have started to refer to the level above M0 as M1 and everything above M0 as M1+.

allison++ also mentioned that many innovations and optimizations are possible in storing isolated register sets for each Continuation (a.k.a call frame). This area of Parrot may yield some interesting surprises and perhaps some publishable findings.

We all agreed that M0 should be as ignorant about the GC as possible, but the GC will most likely learn about M0 as optimizations are implemented. The pluggability of our GC's were also talked about. allison++ raised the question "Are pluggable GC's easier to maintain/implement if they are only pluggable at compile-time?" Indeed, they probably are, but then we run into the issue that our current "make fulltest" runs our test suite under different GC's, which would require multiple compiles for a single test suite run. chromatic++ made a suggestion that we could instead make GC's pluggable at link-time (which would require a decent amount of reorganization) which would still allow developers to easily test different GC's without recompiling all of Parrot. chromatic++'s estimate is that removing runtime pluggability of GC's would result in an across the board speed improvement of 5%.

This conversation then turned toward the fact that M0 bytecode might depend on what GC was used when it was generated, i.e. the same M0 source code run under two different GC's would generate two different bytecode representations. This would happen if the M0 alloc() opcode assumes C calling conventions. This was generally deemed distasteful, so our alloc() opcode will not "bake in C assumptions", which is a good general principle, as well. This will be a fun test to write.

allison++ brought up the fact that we may need a way to tell the GC "this is allocated but uninitialized memory", a.k.a solve the "infant mortality" problem. chromatic++ suggested that we could add some kind of lifespan flag to our alloc opcode (which currently has an arbitrary/unused argument, since all M0 opcodes take 3 arguments for symmetry and performance reasons). This could be as simple as hints that a variable is local or global, or a more detailed delineation using bit flags.

It was also decided that we didn't need an invoke opcode and that invoke properly belongs as a VTABLE method on invokables.

We also talked about the fact that register machines greatly benefit from concentrating VM operations on either the caller or the callee side. Looking for more references about this. It seems that the callee side seems to be what we will try for, but I am not quite sure why.

We finally talked about calling conventions and decided that goto_chunk should roughly be equivalent to a jmp (assembly unconditional jump to address) and the invoke VTABLE would setup a return continuation (i.e. make a copy of the program counter), do a goto_chunk, and let the callee handle the rest, such as looking up a return continuation and invoking it.

After the main M0 meeting, cotto++, allison++ and I sat down at a coffee shop and came up with a list of next actions for M0:

  • Write a recursive version of 'calculate the n-th Fibonacci number' in M0
  • Write a simple checksum algorithm in M0 (suggestions?)
  • Create a working PMC in M0
  • M0 disassembler
  • Create a "glossary brochure for Github cruisers"
  • Implement function calls and returns
  • Make sure each M0 opcode is tested via Devel::Cover
  • Convert the M0 assembler to C
  • Convert the M0 interpreter to C
  • Link M0 into libparrot (no-op integration)

I have been talking to cotto++ on IRC while typing up these notes and we have come to the conclusion that a "bytecode verifier" should also be put on that list. A verifier is a utility that detects invalid bytecode and prevent attacks via malicious bytecode. This is something that happens at runtime, where as a bytecode checksum happens before runtime, or at the end of compile time. They provide different kinds of insurance. The bytecode checksum feature will be an instrinsic feature that is not optional, since it prevents Parrot from running known-bad bytecode. But a bytecode verifier adds a significant amount of overhead. This overhead is reasonable if you are running untrusted code, but it is unreasonable when your are running trusted bytecode (i.e. bytecode that you created), so the verifier will have an option to be turned off.

We obviously have a lot of fun stuff to work on, so if any of it sounds fun, come ask cotto++ or me (dukeleto) on #parrot on irc://irc.parrot.org for some M0 stuff to do. We especially need help with writing tests and documentation.

There will be a Parrot hackathon at YAPC::NA this year, where I am sure some M0-related hacking will be happening. If you have never been to a hackathon before, I highly recommend them as a way to join a project and/or community. Meatspace is still the best medium for some things :)

(UPDATE: Some factual errors about our old JIT were pointed out by rafl++ and corrected)

Parrot Embed Grant Update #5 : Zen Pebbles

|

I am still on the path of increasing test coverage in src/extend_vtable.c. It is much like a zen study, where you methodically concentrate on many tiny little pebbles, one at a time, moving them in the sand, to just the right place. According to the latest code coverage statistics, we are now at 72% code coverage, which is an increase of about 8% since my last report.

Many, many more tests involving Key PMCs were added. For an intro to what they are, take a look at my previous grant update. Many of the tests are clusters of related tests, because most VTABLEs have many similar forms which take integer, string or PMC-flavored keys. I ran into some platform-specific bugs which only manifest on Darwin machines, which were reported by Jim Keenan in TT# 2098 and which I then fixed by querying with a non-empty Key, which is more prudent.

I also ran into some actual bugs which I reported as Trac Tickets. First is that the cmp_pmc VTABLE does not seem to be working correctly from extend_vtable, which was reported in TT #2103. Then I fell into a "hole" in the VTABLE API, where ResizablePMCArray does not respond to defined_keyed(), which it should. This is described in TT #2094.

In retrospect, this was one of the most productive periods of my grant work. I estimate that I will be very close to the 95% milestone by my next grant update at this pace, which is very exciting.

Parrot Embed Grant Update #3 : Now with Dragons

|

The quest to improve test coverage for src/extend_vtable.c has continued. Some dragons were slayed, a few trolls were paid tolls to cross creaky bridges of abstraction and many siren calls to hack on other code were dutifully ignored (mostly).

This TPF grant has forced me to become very familiar with Parrot vtables (virtual tables), which is basically an API for talking to Parrot PMCs (really just objects with a funny name). PMC can stand for Parrot Magic Cookie or PolyMorphic Container. Take your pick.

Firstly, vtable is already slang for "vtable function", which expands to "virtual table function." What the junk is a "virtual table function" ? I find that the simplest way to think about it is that every PMC has slots or buckets with standardized names such as get_bool (get Boolean value) or elements (how many elements does this PMC have?)

All PMCs inherit sensible defaults for most vtables, but they are allowed to override them. Why would you want to override them? As a simple example, let us assume there is a vtable called length (there isn't actually, but it makes an easy example to explain these concepts). Our length vtable will act just like elements and tell us how many elements a PMC has. If we had a complex number PMC that was really just an FixedFloatArray PMC of two numbers underneath, the length would always return 2 for every complex number. Not very useful.

A much more useful length vtable would use the coefficients a and b from a + b*i and compute the Euclidean distance (length from the origin) sqrt(a^2 + b^2). Hopefully you now have a taste for what what vtables are about. Parrot PMCs have over 100 vtables that can be overridden to provide custom functionality.

I recently ran across the hashvalue vtable and couldn't find any tests for it in Parrot core (outside of the test that I had written for it in extend_vtable.t) or any use of it in Rakudo Perl 6. Oh noes! It seemed like an unused/untested feature, so I created a Trac Ticket to mark it as deprecated so it could be removed in a future release.

The discussion about the ticket was fierce. NotFound++ explained why the vtable was important and the mighty coding robot known as bacek++ manifested tests quickly.

Yet another case of this grant work having a positive impact on the Parrot codebase, even outside the embed/extend interface. I also improved an error message in the PMCProxy PMC, which arises when something goes bad during a partial re-compile. Yay for improved debuggability!

According to the current code coverage statistics, extend_vtable.c is up to 54% coverage from 43%, which is not quite where I predicted from my last update. No doubt this has something to do with me packing and preparing to move to a new house this month. My velocity didn't decrease so much as the amount of time that I had to work on this grant.

I am greatly enjoying working on this grant and even if it is going a bit slower than I had planned, I am very confident that it will be completed in the next few months and hopefully sooner.

Parrot Embed Grant Update #1

|

My work on a TPF grant to improve documentation and test coverage of the Parrot Embedd API is going well. I have added extensive examples of Parrot function and type signatures, as well beginning to increase test coverage of this subsystem.

I am working in the leto/embed_grant branch on Github, which has already been merged to master just before the release of Parrot 2.11.0.

My first merge of this topic branch included about 15 commits, which are about 2/3rds documentation and 1/3rd tests. I clarified some points about edge cases of Parrot function signatures, such as void input and void output, which is the emtpy string concatenated to both sides of an arrow ->, and gave an expanded description of what Pi means (PMC, invocant). Many examples of diverse kinds of function signatures were also added.

I gave the first user-visible documentation for many constants in the Embed subsystem, such as debug flags when creating interpreter objects and inline descriptions of different runcores that can be used with interpreter objects.

The tests that I added include the first coverage of returning Float PMCs and numeric constants from our embedding subsytem, as well as additional coverage for returning a ResizablePMCArray consisting of Numeric PMCs. I also added tests for creating multiple interpreter objects and added a TODO test for Trac Ticket 1880.

I also fixed a bug in the Parrot test suite, where tests in t/src were not skipped properly if src/parrot_config.o did not exist.

Most of this work was done between Thanksgiving and holiday travel, so I expect that development pace will pick up in the next few days. Currently, one of four inchstones has been achieved, and I will concentrate on raising the code coverage of extend_vtable.c in the next two weeks.

I would like to thank The Perl Foundation, and Karen Pauley in particular, for funding this very important grant to the Parrot and Rakudo Perl 6 communities.

I'm excited to announce that Parrot Foundation and The Perl Foundation have been accepted as organizations in Google Code-In 2010!

Google Code-In is a contest, similar to Google Summer of Code, where Google pays students aged 13-18 to do tasks designed by open source communities, while learning about open source. Google pays for the work to be done, and we get new members to our communities, while students learn useful skills. It is a big win for everyone.

In 2010, Google Summer of Code was a great success for Perl and Parrot. We got amazing new features in Parrot, Perl 5 and Perl 6 . In 2009, we had similarly spectacular results.

For the students, the benefits are huge. They get mentored by some of the best minds in open source and get "street cred" in the community. This contest also acts as a stepping stone for Google Summer of Code, so students that excel at Code-In will most likely be sought after for future Google Summer of Code involvement. It's also fantastic experience to put on a résumé. I see many Google Summer of Code students get snapped up by respected companies, or accepted to prestigious academic institutions.

The more well-documented tasks we have before that, the more students we will have the potential to attract. I can attest that these kind of contests attract some of the smartest students in the world, so the Perl and Parrot communities have much to gain by being involved.

I expect great results for Code-In as well, but we need your help. The Google Code-In contest opens up for students on:

  • November 22, 2010 at 12:00 AM Pacific Time / 08:00 UTC.

How Can You Get Involved?

  • Add a task to our task list There is a template that you can copy and paste, as well as many examples. Any task related to Perl 5, Perl 6 or Parrot is fair game.
  • Improve the description of an existing task. The more specific a task and the more documentation and links you provide, the easier it is for a student to choose and complete a task.
  • Volunteer to mentor a student on a task. You apply to be a mentor here. Please join the tpf-gsoc-students mailing list and introduce yourself. Provide a brief description of why you are interested when you sign up, so we know you aren't a bot :) Please also join the #gci channel on irc.perl.org.
  • Tell potential students about Google Code-In and how we are involved. Here is a link to the timeline and FAQ that you can send them, as well as flyers to post.

GSoC 2010 Mentor Summit and Git Together Wrap Up

|

So many amazing things happened at the Google Summer of Code Mentor summit 2010! I will try to jot a few of them down, before they leave for warmer climates. For those that just want to read all the session notes, you can find them here. Also, if you haven't yet read about how The Perl Foundation and Parrot Foundation fared this summer, you can read about it on the Google Open Source Blog.

It began by arriving a bit early to work with some awesome people on improving the GSoC Mentor Manual by adding a chapter for Organization Admins (there is actually documentation now!) and writing a GSoC Student Manual. This "book sprint" was facilitated by Adam Hyde of FLOSSManuals, and they were written with a completely open source software stack, as well as being released under a Creative Commons license. They are free for anyone to read online and are easily exportable to many formats. Read the Student Manual or the Mentor+Org Admin Manual online now! We even bound 60 copies of the books and handed them out to mentors attending the summit.

Parrot on RTEMS hacking with Chris Johns and Ralf from RTEMS. We used Centos 5 RPMS on Ubuntu 10.04 with rpm2cpio piped to cpio, which was a trick to get around the fact that RTEMS does not have debian packages. It worked remarkably well. I had a cross-compilation environment setup after a few minutes. I think they will be adding these intructions to their wiki. Now that I have the RTEMS toolchain on my netbook, I will be much more productive with regard to Parrot on RTEMS.

Chromatic, Chris Johns and I sat in a room and talked shop about how Parrot and RTEMS can play nicely together. There are still some feature voids on the Parrot side to fill: Parrot calls exit() in various places, which reboots RTEMS i.e. a syntax error reboots the OS. Not Fun. Parrot also needs a C probe to detect RTEMS, which already has a ticket in our bug tracker. A real-time garbage collector will be needed for long-running processes, but for short-lived applications, disabling the GC with the -G command line argument to Parrot will work.

I gave a session with Selena Deckelmann and Bart Massey introducing Troll University, which aims to educate organizations, corporations and open source communities about what motivations and principles trolls use and how to protect against them. We are working on some Trollcasts, so stay tuned!

I also gave a session called Dynamic Language Interoperability, which has been held for the last few years, to my knowledge. The consensus seemed to be that every dynamic language has the same interop problems, and Parrot VM seems to be the only project working hard to solve these complex issues in a general manner. This gives me a lot of hope that people will soon realize that Parrot is full of WIN.

It also came to my attention during the conference that Github hired the student that mentored under them this year to work on libgit2. This is one example of the amazing opportunities that students have after completing a Google Summer of Code. The sky really is the limit. And just in case you think this is an isolated incident, it isn't.

As if writing some books and going the Mentor Summit wasn't enough to totally drain me, I am currently attending the last day of the GIT Together 2010, which is the yearly Git developer and user meetup/unconferencey thing. I have learned so much that I can't even begin to describe it, but if you want to look at session notes, you can find them here.

Google Summer of Code 2010 Final Summary

|

Google Summer of Code is a global program that offers student developers summer stipends to write code for various open source software projects. Google Summer of Code 2010 went by quickly, and much was accomplished. The Perl Foundation and Parrot Foundation took part this year, and we were lucky to get proposals from very bright and capable students. We started the summer with 10 students and had 8 students pass their final evaluations. The passing projects include:

Ryan Jendoubi -- Ctypes for Perl
Mentor: Reini Urban
Blog: http://blogs.perl.org/users/doubi/
Repo: http://gitorious.org/perl-ctypes/perl-ctypes

This project is exciting many Perl developers, because it would minimize the need to use XS, which will make many more pure-Perl modules possible. This improves portability, becaue XS-based modules are notorious for being fragile across operating systems and compiler versions. This adds up to a whole lot of WIN.

Nat Tuck -- Hybrid Threads for Parrot
Mentor: Andrew Whitworth
Blog: http://parrot.org/blog/836
Repo: http://github.com/parrot/parrot/tree/gsoc_threads

Threads allow a single program to use more than one CPU, which is becoming increasingly important these days. Even mobile phones are multicore! This work aimed at adding threading support to Parrot Virtual Machine. Much was accomplished, but this effort is still on-going. So-called "green threads" were implemented, which is a necessary step to get Hybrid threads working.

Tyler Curtis -- A PAST Optimization Framework for Parrot
Mentor: chromatic
Blog: http://parrot.org/blog/839
Repo: http://github.com/ekiru/tree-optimization

This project is about providing a framework for optimizing PASTs (Parrot Abstract Syntax Trees). This will be used by language implementors when optimizing their HLLs (High Level Languages). This framework allows all languages on Parrot to benefit from optimizations that are written once, instead of each language implementor writing their own optimizations.

Daniel Arbelo Arrocha -- NFG and single-representation strings for Parrot
Mentor: Allison Randal
Blog: https://www.parrot.org/darbelo
Repo: http://github.com/parrot/parrot/tree/gsoc_nfg

NFG stands for Normal Form Grapheme, and basically means having a standard internal representation of Unicode strings, so that very expensive conversions do not have to repeatedly take place. This makes string-heavy computations much faster and unifies a lot of code.

Carl Masak -- Add support for binary data in Rakudo
Mentor: Jonathan Worthington
Blog: http://use.perl.org/~masak/journal/
Repo: http://github.com/rakudo/rakudo

Rakudo Perl 6 now supports various binary data formats that were implemented as part of this project. Many relevant tests were also added to the Perl 6 Spec Test Suite as well as improvements and clarifications to the Perl 6 Specification.

Muhd Khairul Syamil Hashim -- Instrumentation Tool for Parrot
Mentor: Christoph Otto
Blog: http://www.parrot.org/blog/841
Repo: http://github.com/khairulsyamil/parrot-instrument

This instrumentation tool for Parrot allows one to dynamically peek into the execution of Parrot op-codes. This allows for writing profiling tools that can answer questions like "who calls functions X" and "how many Objects of type X were created."

John Harrison -- Improvements to NCI/LLVM Stack Frame Builder for Parrot
Mentor: Peter Lobsinger
Blog: http://www.parrot.org/ash
Repo: http://github.com/ashgti/parrot

This project is a prerequisite for a JIT (Just In Time compilation) framework, which is an important goal for the Parrot community, since Parrot decided that our old JIT subsystem was broken beyond repair and removed it. Parrot has decided to use the very popular LLVM project in our rewrite of our JIT, and this project brings us a step closer on our journey.

Pawel Murias -- Mildew and SMOP on CPAN
Mentor: Daniel Ruoso
Repo: http://github.com/perl6/mu

This project involved working on Mildew and SMOP. Mildew is a Perl 6 implementation, and SMOP is the library Mildew uses for meta-object programming. You can think of Mildew as a sister to Rakudo Perl 6. Having many implemenations of Perl 6 helps to better define the Perl 6 specification. Updated versions of SMOP and Mildew are now available on CPAN.

The failing projects were:

Justin Hunter -- Rework Catalyst framework instance initialization code
Mentor: Florian Ragwitz

Mirko Westermeier -- Bulletproofing the Mojolicious test suite
Mentor: Marcus Ramberg 

Both of these projects passed their midterms, but due to circumstances outside of the program, these students were not able to complete their goals for their final evaluation. Sometimes Real Life throws you a curve ball, like starting a new job, moving to a new city, having a baby and similar things. We wish these students the best of luck, and hope that they complete their projects outside the structure of GSoC.

I am very proud and humbled by all the students and mentors that I worked with this year. I am constantly reminded that there are very intelligent developers that are very young, and The Perl Foundation and Parrot Foundation is very lucky to attract them and have them in our communities. I firmly believe that the passing GSoC 2010 projects have made a large positive impact on our codebases and many people will benefit from them for years to come.

Rock on and keep spreading the Open Source love!

Rakudo Perl 6 in Your PostgreSQL Database!

|

It has been a very exciting few weeks in the Perl 6 world with regard to database access. mberends++ just wrote a nice blog post about how Perl 6 support for DBI is ramping up with work on MiniDBI (formerly FakeDBI). Multiple developers recently made great progress on PostgreSQL drivers and have been improving the Parrot interface to libpq, Pg.pir .

I have also been dutifully hacking on PL/Perl6, which embeds Rakudo Perl 6 into your PostgreSQL datbase via PL/Parrot, and just recently merged the plperl6 branch, which adds the first basic support for PL/Perl 6. Here is how it currently works:

  • You must first install a recent version of Parrot and Rakudo Perl 6. I develop with Parrot trunk and Rakudo master because I often need very recent changes/bugfixes
  • You will need a moderately recent version of PostgreSQL. PL/Parrot has been known to work with versions as old as 8.3.8, but I mostly develop on the master branch, so 9.x will probably work best
  • When you type "make install" for Rakudo Perl 6, you will see that it installs a file called "perl6.pbc" into the lib/$version/languages/perl6 directory of the Parrot installation that Rakudo was configured with.

To tell PL/Parrot that you would also like PL/Perl6, you set the environment variable PERL6PBC to the full path of perl6.pbc like so:

export PERL6PBC=/Users/leto/git/rakudo/parrot_install/lib/2.5.0-devel/languages/perl6/perl6.pbc

A PBC file is Parrot bytecode, and the perl6.pbc file basically bundles all of Rakudo Perl 6 into a single file. You can play around with the Rakudo Perl 6 REPL by running:

$ parrot $PERL6PBC

which is pretty much the exact same thing as running the perl6 binary in your $PATH.

When you compile PL/Parrot with the $PERL6PBC environment variable set, it automatically creates a separate Parrot interpreter with the perl6.pbc bytecode loaded, so that Perl 6 code can be executed.

To run PL/Parrot tests, you type "make test". You can also do this the PostgreSQL way with "make installcheck", which will run the tests and verify that the output matches a certain, known output. "make tests" just generates the TAP output.

Currently PL/Perl 6 tests are not run by default, they have their own Makefile target: test_plperl6 This is what the output of "make test_plperl6" looks like currently, on commit 44a985f123 of the perl6_args branch.

We run a total of 9 tests, 6 of which correctly run Perl 6 code and pass. You will notice that all the failing tests are relating to passing arguments into Rakudo from PostgreSQL, which is what I am currently trying to get to work. Currently I am passing a Parrot ResizablePMCArray of the arguments to a Rakudo function and executing it, but the function can't seem to see it. My guess is that Rakudo wants native datatypes and not Parrot datatypes. If you know how to create Rakudo datatypes from PIR, please let me know :) I promise you will -Ofun.

What does PL/Perl 6 look like? Here is a nice example of a PL/Perl6 function which calculates the sum of all Fibonacci numbers <= 100:

CREATE FUNCTION test_fibonacci_plperl6(integer) RETURNS int LANGUAGE plperl6 AS $$
[+] (1, 1, *+* ... 100)
$$;

You will notice three spiffy new Perl 6 operators in there, the summation operator [+], the new range operator ... (which used to be .. in Perl 5), and the *+* operator. What exactly is the *+* operator? pmichaud++ jokingly referred to it as the "cheerleading plus", but it is actually just a plain old infix "+" operator, sandwiched by two "*" (a.k.a Whatever) operands. It basically takes the previous two elements in a list, sums them together and returns the sum, which is exactly how the Fibonacci sequence is defined.

If you are interested in hacking on PL/Perl 6, PL/Parrot or anything related, come join us in #plparrot on freenode, join the mailing list and fork us on github !

PL/Parrot Flies

|
PL/Parrot recently started passing all tests and marshalling three basic data types of integers, floats and strings, between PostgreSQL and Parrot Virtual Machine.

One of the important next steps for PL/Parrot is to improve the security subsystem, which will require changes to Parrot core. Parrot has only a high-level Parrot Developer Doc (PDD) that describes how security should work, but still lacks an implementation. Thus, PL/Parrot finds itself in the situation of being able to drive the development of security features in Parrot. This is very exciting and I am researching various hardening solutions. Various ideas have been thrown around, but removing or replacing certain opcodes at the lowest level is the most secure solution. How to do this, properly at run-time, is the fun part.

I will be talking about PL/Parrot at PGCon 2010, which will be very exciting. I am hoping to meet many people in the PostgreSQL community and show some PostgreSQL internals hackers why hacking on PL/Parrot is so fun and so important.

PL/Parrot is so important because PL's are hard to write and maintain. My vision is that most of the hard work goes into PL/Parrot, and then languages built on top of Parrot (HLL's), will have a very simple time piggy-backing on top of that infrastructure. Currently, only PL/PIR exists. Stored procedures can be written in PIR. The next steps will be getting a HLL working on PL/Parrot, such as NotQuitePerl (NQP) or Rakudo Perl 6. If you want to get your favorite HLL working on PL/Parrot, I surely won't stop you.

There are also many benefits to writing stored procedures in PIR. PIR will have the closest possible speed to C than any HLL run on Parrot, so if speed is your main concern, you will want PL/PIR. Parrot has a garbage collector, so you don't have to do memory management in PL/PIR. This prevents crashing the server instance due to double-free errors and fun stuff like that. PIR is also platform-independent: no more need to figure out how to get your stored procedures written in C compiling on a new platform.

If you have tuits or ideas for implementing PL/NQP or PL/Rakudo or any other comments about PL/Parrot, I would love to hear them. Feel free to fork the github repo, join us in #plparrot on irc.freenode.net or on our mailing list. We also need a website and mascot. There are many ways to help, but patches are always welcome :)

Google Summer of Code 2010

|
I am working on the application for The Perl Foundation and Parrot
Foundation
to participate in Google Summer of Code 2010. GSoC is a
program where Google funds eligible students to hack on open source
projects for a summer. It is a great opportunity for the students and
the communities that mentor them. You also may be interested in this
summary of our involvement last year . Our application will be
submitted by the end of this week.

Please join us in getting prepared for this year. There is a page for
possible mentors to volunteer as well as a page for 
project ideas . If you would like to help with the wiki, our 
main GSoC page is the best place to start. You are also invited to join 
our mailing list  and come ask questions in #soc-help on irc.perl.org .

Parrot In Your Database

|
Wouldn't it be awesome to write performance-critical stored procedures in a fast platform-independent language? That is the vision for PL/Parrot, a Postgres Language that will allow you to write stored procedures in PIR or NQP. Our current plan is to embed the Parrot VM inside each postgres backend. If you have a better idea, let us know!

Come hang out on #plparrot on irc.freenode.net or checkout the github repo if you want to help! You may also want to visit #parrot on irc.perl.org for Parrot-specific help. It is very much in a prototype stage right now, but we have lots of smart people interested in making this work, so I am confident that we will have something working very soon.

If you are a Postgres internals hacker or you want to learn more about Parrot Virtual Machine, we would greatly appreciate your help!

Real-Time Embedded Parrots

|
I am very proud to announce that Parrot Virtual Machine has been ported to RTEMS, a real-time embedded operating system (RTOS). Here is one of the first runs of Parrot code atop RTEMS, this example being the Fibonacci number benchmark with debug tracing turned on, posted by Chris Johns:

RTEMS SHELL (Ver.1.0-FRC):/dev/console. Oct 21 2009. 'help' to list commands.
fstst [/] # parrot -v p/benchmarks/fib.pir
debug = 0x0
Reading p/benchmarks/fib.pir
using optimization '-O' (0)
Starting parse...
sub _main:
        registers in .pir:       I3, N2, S1, P2
        0 labels, 0 lines deleted, 0 if_branch, 0 branch_branch
        0 branch_cond_loop
        0 used once deleted
        0 invariants_moved
        registers needed:        I3, N2, S1, P3
        registers in .pasm:      I3, N2, S1, P2 - 0
        4 basic_blocks, 4 edges  
sub _fib: 
        registers in .pir:       I5, N0, S0, P0
        0 labels, 0 lines deleted, 0 if_branch, 0 branch_branch
        0 branch_cond_loop
        0 used once deleted
        0 invariants_moved
        registers needed:        I5, N0, S0, P2
        registers in .pasm:      I5, N0, S0, P2 - 0
        5 basic_blocks, 5 edges  
1 lines compiled.
Running...
fib(28) = 317811 71.1315400600433s

Many many thanks to all who have helped. -- kiwichris

I met Chris Johns a.k.a. 'kiwichris', a core RTEMS developer, at the Google Summer of Code Mentor Summit 2009, and we have been attempting to get Parrot working on RTEMS ever since. With his extreme hard work and hand-tuned Makefile kung-fu, he cross-compiled Parrot, totally bypassing our configure/build system, which does not support cross-compiling!

Getting RTEMS working on Parrot definitely pointed out a few rough edges on Parrot's part, such as using powl() from math.h without checking for it, even though it is an *optional* POSIX extension. RTEMS does not currently have powl() so our string subsystem blew up. Thankfully, just replacing it with pow() works just fine, for now. I've added a ticket to add support for checking at Configure-time whether powl() exists, and to only use it then, which is the correct behavior.

Also, the Parrot exit handler calls the system exit() after all other exit handlers currently, which reboots a real-time OS like RTEMS. So we are going to have to provide some interface to allow alternate final exit handlers.

But these issues are small, most of the hard work has already been done by Chris. At this point, we will start fixing our configure/build system so that a RTEMS cross-compile can be done with a Configure.pl flag, such as

perl Configure.pl --cross=rtems-i386

Thanks again to Chris Johns and all RTEMS/Parrot core developers that helped make this possible.

Stay tuned to the intertubes for more updates about Parrot on RTEMS!

Learning Parrot with the Parrot Shell

|
I recently have been working a lot on Parrot (I am the release manager for 1.7) and learning by the seat of my pants. The first thing that I yearned for was something REPL-like, for rapid prototyping and just hacking out code with immediate feedback. Enter the Parrot Shell. To start up the shell, go to the root of the Parrot source code and type:

perl tools/dev/parrot_shell.pl

This should be after you have configured and compiled Parrot with

perl Configure.pl && make

You should see something like:

Welcome to the Parrot Shell, it's experimental!
Type h or help for some basic help
Type q or quit to flee the madness

parrot_shell 0>


The Parrot Shell reads input until it sees a single period (".") on a line (ignoring whitespace), then executes the code that you gave it. That is it. As a simple first example, let's print something:

parrot_shell 0> say "This Parrot talks!"
.
Time: 0.062809 wallclock secs
Output:
This Parrot talks!

But you may ask, there is no ".sub main" followed by a ".end" at the end of the code, so that is not valid PIR. You would be correct. The Parrot Shell wraps the commands you give it in ".sub main" and ".end" if the first line of your input does not start with ".sub". So it allows one to be lazy for short pieces of code, but allows specifying the name of the main subroutine for when that is necessary. This shows how to do that:

parrot_shell 2> .sub main
    foo()
.end
.sub foo
        say "foo"  
.end
.
Time: 0.0438812 wallclock secs
Output:
foo

Some other things to know about the Parrot Shell are that you can type "h" or "help" to get a quick reminder of how Parrot Shell works and that each numbered Parrot Shell session is run in its own interpreter, so no registers or variables are shared/leaked between them. To leave the Parrot Shell, just type "q" or "quit" . 

parrot_shell 1> quit
Thanks for visiting the Parrot Shell, come back soon
!

Here is one more example which shows basic interaction with Parrot registers via the Parrot Shell

parrot_shell 0>
$I0 = 42
$N1 = sqrt $I0
say $N1
 .
Time: 0.0245831 wallclock secs
Output:
6.48074069840786


The Parrot Shell is great for pasting code snippets from Trac tickets or from people on IRC asking "does this code do X on your system?" I also find it is very helpful to have a Parrot Shell open while reading Parrot documentation (like the PIR book), so that you can test things as you learn them. In addition to all the docs that Parrot comes with, there are some really good online guides popping up, like Brian Wisti's Parrot Baby Steps.

If you are trying to write tests for a certain Parrot feature, the Parrot Shell can often help you figure out how to write it in the least amount of time.

I hope that the Parrot Shell gives you an easy on-ramp to start playing with Parrot, please let me know how you use it and if you have any suggestions for improvement. As always, patches welcome!

Blizkost is passing 102 tests!

|
Blizkost, Perl 5 on Parrot, is really starting to pick up speed. I just imported the base Perl 5 tests (the ones that live in t/base) and Blizkost passes every file except lex.t and one TODO test about implementing the -l command line argument. If you can make lex.t work, you get super-cool ninja points. It is probably the use of the package keyword in lex.t that borks the test. If you feel like porting other tests from the Perl 5 Test Suite, they live in t/spec in the Blizkost repo.

Currently, Blizkost cannot load XS modules, such as Data::Dumper. You get an error like:

Can't load '/usr/lib/perl/5.10/auto/Data/Dumper/Dumper.so' for module Data::Dumper: /usr/lib/perl/5.10/auto/Data/Dumper/Dumper.so: undefined symbol: Perl_sv_cmp at /usr/lib/perl/5.10/XSLoader.pm line 64.
 at /usr/lib/perl/5.10/Data/Dumper.pm line 36


If you can help get enough XS-loading machinery to work so that Data::Dumper works, that would really help development and testing. If you can try out Blizkost on your system and make sure that it compiles and the test suite passes, we would greatly appreciate it. Feel free to open an issue on github if you find a bug.

Parrot Hacktivity Report

|

  • Applied many patches from flh++ and darbelo++
  • Translated lots of tests written in Perl to PIR. This makes the test suite faster! It also makes the code easier to debug. It has also uncovered lots of bugs. Fun!
  • Implemented throws_like() in test_more.pir, which allows for easily testing if a bit of PIR code throws an exception that matches a PGE pattern. This is inspired from warnings_like() in Test::Warn and uses the like() from test_more.pir under the hood, so it was reasonably simple to implement. Added lots of tests that use throws_like() to verify that proper errors are thrown.
  • Fixed a bug in FixedPMCArray where it would not check for negative lengths and core dump Parrot. Oops!
  • Wrote an interactive Parrot Shell for REPL/rapid prototyping. Type perl tools/dev/parrot_shell.pl in the Parrot source code to start it up. More about this soon!
  • Wrote many tests for the proper handling of Inf/NaN, which exposed some bugs in fdiv, cmod and mod when using PMCs .
  • Updated editor documentation regarding generating tags
  • Updated documentation about math functions in Ch04 of The PIR Book
  • I will be the release manager of Parrot 1.7, which is getting released October 20th. Exciting!

Parrot Hacktivity Report

|
What I've been hackin' on in Parrot-land, gonzo-style:

Note: PMC's are PolyMorphic Classes or as the Parrot hackers like to call them, Parrot Magic Cookies.

  • Wrote a bunch of tests, fixed some bugs and updated documentation for the parrot_debugger
  • Added the ability to assign to most registers in the parrot_debugger, since eval is currently broke :( . Assigning to PMC registers still needs to be implemented.
  • Added the rand and srand dynops (dynamically loadable opcodes). Parrot does not need to be recompiled to change or modify dynops.
  • Wrote tests for the other dynops, slyly named obscure
  • Got rid of the deprecated Random PMC as well as updating all tests and documentation which referred to it. This involved fiddling with fun stuff like the befunge interpreter written in PIR and the OpenGL examples (which evidently don't work on the latest OS X)
  • Rakudo was using the Random PMC internally and it's test suite, so my commit to Parrot broke Rakudo from even compiling on the latest Parrot. I attempted to switch Rakudo over to use the new rand/srand dynops, but they were not being installed. I added the files to Parrot's MANIFEST.generated (thanks to jnthn++ for the hint), which stores where all installable files live. Rakudo was then able to find them and I had a once again compiling Rakudo with a passing "make test". With help from moritz++, my patch was accepted and Rakudo was once again passing "make spectest." The turn-around time on this was amazing and I would like to thank everyone in #parrot and #perl6 for making it happen.
  • Added more tests for correct handling of NaN/Inf
  • Added the test directory t/pir for compiler-agnostic tests for PIR, after talking to bacek++. I attempted to test the broken .macro_local feature, but it seems that it is not currently possible to write TODO tests in PIR for PIR that does not parse. Punted on that and just wrote some simple tests for basic macro expansion.
  • Wrote tests for pbc_disassemble and pbc_dump
  • I have a mirror of the Parrot subversion repo on github, with all of the currently active Parrot branches mirrored as well. The github network mode lets you see the relative states of each branch (flash required).
Clicky Web Analytics 42