BOINC: Failed WorkUnits
When work units fail in BOINC, it poses a question of how to handle the remaining work units still being processed. I’ve added code to minimally handle failures so that manual user intervention isn’t required, however there is still a burden on the developer to understand this situation and decide how to recover from it.
This is really no different from handling exceptions in non-distributed code, but I know all to well how exceptions are normally handled (hint: they aren’t). If you are interested, you can read my full post on the PyMW blog.
PyMW Site Finished, for Now
I’ve finished converting the PyMW website over to WordPress and implemented the new design and logo. It still needs more work, but I’m going to switch back into core BOINC interface mode again for a while.
The interface is working well now, but BOINC work unit failures still require manual user intervention. I would like to automate as much failure recovery as possible, so I will be focusing on this for the next few days.
PyMW Distributed Pacman Server
To create a real application for PyMW, I think I am going to create a distributed Pacman server.
Last semester I took an artificial intelligence class that used Pacman as a teaching tool. At the end of the semester, there was a tournament where each team could pit their Pacman AI client against each other in a game of Pacman-style capture the flag. We submitted our clients to a server and then waited 24 hours or so for the results to appear. If your client crashed, you had to wait another 24 hours to see your standings.
My idea is this:
- Create a PyMW application that runs Pacman tournaments
- Each job will be 3 matches between two clients
- An animated GIF will be created for one of the 3 matches (one that agrees with the outcome)
- The BOINC interface will be used so students can contribute compute time
- The output of the PyMW application will be records in a MySQL database
- Create a website for statistics
To test the tournament server, I am going to get the AI client code from last semester’s teams and then run it on the BOINC Alpha group. This should provide a solid test of the PyMW BOINC interface and my tournament server.
Tagged Tags: AI, BOINC, GSoC, PyMW, Python on June 29, 2009 at 12:17 pm
WordPress Complete
The transition to WordPress is complete!
The one straggler is the sketches page. I think I am going to change the sketches page from sketches to design. I will keep the Processing sketches add drawings, graphic design and web design, but display everything as tiles.
Hooray RSS feed!
Tagged Tags: VisualCore on June 29, 2009 at 12:13 pm
Wordpress, maybe
I have had enough fun writing my own blogging web app; I think it’s time to switch over to something more industrial. I downloaded Wordpress today and gave their “famous” 5-minute install process a go, which worked like a charm.
I haven’t fully committed to the switch yet, but so far I like what I see.
Tagged Tags: VisualCore on June 14, 2009 at 9:11 pm
PyMW: Logo and Layout
Created a new logo and web layout/design for PyMW. This isn’t really part of the summer of code gig, but I was feeling inspired so I threw this together. It’s still a work in progress, but you can get a feel for the design.
I’m not much of a graphic artist when it come to identity, but I tried to represent idea of one framework joining many disparate models of computation as well as the general idea of master-worker computing.
Tagged Tags: Design, GSoC, PyMW on June 10, 2009 at 12:08 am
.NET 4.0: Parallel Programming
.NET 4.0 Beta 1 contains some interesting parallel programming constructs. Sounds like a great idea, I’m really interested to see where this is going.
Tagged Tags: C#, Parallel, VB.NET on June 5, 2009 at 11:35 pm
PyMW: Week one
Today is the official end of my 7th day of working on the PyMW interface for BOINC for Google Summer of Code.
It took me three days to get the my first PyMW app to run (monte_pi.py), which ran on 4 virtual nodes (4 tasks). More than four tasks was causing problems, it turned out that there was a bug in the PyMW BOINC interface. Now that that’s fixed, I ran with 200 nodes yesterday and 800 today.
This morning, I tried running with 2 physical nodes: my laptop and my Ubuntu VM, which failed at the end of computation. Somehow the canonical results are not being recognized which causes the BOINC interface to get lost in limbo and hang forever.
This week, I created a pure-Python assimilator for PyMW, which works pretty well, but is perhaps causing the error above.
I also rewrote a big swath of the BOINC interface to stop it from using a new thread for each task during task reclamation (getting data back from BOINC). Since it was using one thread per task, it was reaching the maximum number of scheduler threads. This in turn caused the execution thread to hang until some of the tasks completed. Now it reclaims tasks in a single thread and is able to queue all tasks in a single shot, greatly improving the throughput from PyMW -> BOINC.
Overall, it’s been really fun so far. The first few days were trying, since there was little documentation of how to get PyMW to play nice with BOINC. But seeing the first application run was great 
Tagged Tags: BOINC, GSoC, Parallel, PyMW, Python on June 5, 2009 at 12:01 pm
Glaive, XNA Game
This semester I took a software engineering class and I had the luck of joining a game development group. Our team of 8 created an old school, 2D game kind of like Gauntlet. We mashed-up of lots of old 2D graphics and music from games like Super Mario Bros., Sonic the Hedgehog, Chronotrigger and others.
If you have XNA GameStudio 3.0 installed, you can run the game now. If not, you will have to wait until we get the installer working (hopefully soon!). Until then, you can check the videos out.
Also, it’s open source (GPL), so feel free to check out the code and use it in your project!
Tagged Tags: C#, UCB, Video, XNA on June 1, 2009 at 11:37 am
Google Summer of Code!
My proposal has been accepted by the Python Software Foundation for Google Summer of Code 2009!
I will be working on the Python Master-Worker computing project (PyMW), a Python API that provides access to various distributed and parallel computing frameworks. In particular, I will be working on the Berkeley Open Infrastructure for Network Computing (BOINC) integration. BOINC is best known as the underlying system that enabled SETI@Home and I’m really excited to peek under the hood!
My goal is to make it easier for PyMW users to create BOINC applications by simplifying the setup process as well as adding new support for some important BOINC features. This will require some changes to PyMW as well as some changes to the BOINC server. If you are interested in seeing all the details, a public copy of my proposal is posted at the GSoC website.
Tagged Tags: BOINC, GSoC, News, Parallel, PyMW, Python on April 23, 2009 at 12:33 am
