Tuesday, December 15, 2009
During the SciPy 2009 conference in August, Prabhu Ramachandran spoke about the Free and Open source Software for Science and Engineering Education (FOSSEE) project he was running at IIT Bombay. The FOSSEE project is an ambitious project to promote the use of Python in numerical computing in college curriculum. Prabhu has an interesting post on the contributions the scientific Python community has made to the larger Python community. FOSSEE is actually just one part of an even more ambitious $1 billion dollar (US) government program called the National Mission on Education through Information and Communication Technology.
Starting at the end of May 2009, Prabhu very quickly gathered together an amazing team that immediately created a significant amount of documentation and training materials including tutorials, audio/video demonstrations, written material, and lectures. They've created a great two-day hands-on introductory tutorial to scientific programming with Python and have all ready conducted several of these tutorials all across India. Now they are working on creating a couple of semester long college courses and will be offering the first one next semester at IIT Bombay.
At the end of the SciPy 2009 conference in August, Prabhu proposed that we put together a SciPy conference in India and I immediately agreed. Not wanting to delay, we decided to have the conference before the end of the year. After all putting together an international scientific conference in less than four months was keeping with the overall ambition of the FOSSEE project. As soon as Prabhu returned to Mumbai, he contacted Vimal Josef at SPACE Kerala about hosting the conference in Thiruvananthapuram. Shortly after that we announced the first international on Scientific Computing with Python (Scipy.in 09) from December 12th to the 17th at Technopark, Thiruvananthapuram sponsored by FOSSEE, IIT Bombay and SPACE Kerala.
Once we finalized the dates for the conference, I called Travis Oliphant, the president of Enthought, and asked him to deliver the keynote address, which he quickly agreed to do. Among his many accomplishments, Travis is one of the original authors of SciPy and the primary developer of NumPy. David Cournapeau (one of the core NumPy and SciPy developers) and Chris Burns (one of the core developers of the neuroimaging in Python project) also agreed to deliver invited talks.
The FOSSEE and SPACE teams were invaluable in organizing the conference. In particular, Madhusudan.C.S from the FOSSEE team worked very closely with me on the conference website and putting together the conference program. I will write another blog post in the next day or so with a description of the actual conference. For now, you can read a short write-up from one of the local newspapers.
Sunday, November 29, 2009
And the view from my window:
Once I arrived I took a short nap and then went out with Gael to hunt and gather in the market above the passage he lives on. Here are a two pictures I took at the local market (they had all kinds of things, but cheese and meat are, of course, the things that attracted me most):
We also went into a store called frozen food store called Picard:
After lunch, Gael and I headed to Paris. Over the last five years, I've tried to visit the catacombs numerous times. Unfortunately, every time I've visited, they've been closed. This time turned out to be no different. The picture on the left is Gael standing in front of the entrance to the catacombs (you can see a sign on the door, which states that they will be closed for the next month) and the picture on the right is of the road behind me:
Since I couldn't visit the catacombs, we decided to head to the Montmarte district to walk around for the afternoon. Here are a couple of pictures of the Sacre Coeur at the summit of Montmarte:
I forgot to take pictures for the rest of the day, but after Montmarte we headed back to the Latin Quarter to spend a couple hours talking about the SciPy proceedings (which we hope to finish today) in a cafe with wireless internet. And we grabbed an early dinner at 8pm with Emmanuelle and a couple of colleagues from Neurospin.
Wednesday, November 18, 2009
Our web-based documentation editing system continues to be a great asset. In just over a year, this system has helped us to vastly expand and improve our documentation. For instance, our reference guide has gone from under 10,000 words to over 110,000. When Guido came to visit Berkeley a few weeks ago, Fernando Pérez showed him the web-based documentation editor and he was very impressed and even commented that it would be nice for the standard library to use a system like this.
David Cournapeau is again serving as release manager and he is also responsible for much of the code improvements for this release. I just noticed the other day, that according to ohloh, David is quickly approaching Travis Oliphant's number of commits. While it is hard to attach any specific meaning to this statistic, it is clear that at this point David is one the most significant contributors to NumPy. Among his many contributions to this release, he reduced numpy's import time by 20%-30% by adding a small numpy.lib.inspect module and using it instead of the upstream inspect module. Another very useful improvement by David is that you can now link against the core math library in an extension.
In addition to all the work David's done for the 1.4 release, some of his recent work won't be included until the 1.5 release. Once David branches for 1.4, he has all ready promised to merge his Python 3 support for numpy.distutils into the trunk. While we are just beginning to plan migrating to Python 3, this is an important early step.
Unfortunately I am not sure the new datetime dtype support for dealing with dates in arrays will be included in the 1.4 release. This useful functionality was developed over the summer by Travis Oliphant and Marty Fuhry. Marty was my Google Summer of Code student; although, I was pretty busy so Pierre Gerard-Marchant did most of the day-to-day mentoring. Despite the fact that this code was merged with the trunk at the end of the summer, there is a reasonable chance that it will be pulled before the 1.4 release due to the lack of documentation for the public C API.
I've only touched on a few of the many improvements you can expect to see with NumPy 1.4. For more details about the upcoming release, please see the release notes. Thanks to everyone who worked on this release and to David in particular.
Friday, November 13, 2009
Tutorials Monday (6/28) - Tuesday (6/29)
Conference Wednesday (6/30) - Thursday (7/1)
Sprints Friday (7/2) - Sunday (7/4)
From 2001 to 2009, the SciPy conference was held at Caltech with the generous support of the Center for Advanced Computing Research. This year we've decided start holding the conference at rotating locations to ensure more people will get the opportunity to attend the conference. Since the initial conference in 2001, the scientific computing in Python community has rapidly grown. In addition to the main SciPy conference, we now have dedicated SciPy conferences in both Europe and India. Moreover, there are an increasing number of non-SciPy conferences with dedicated sessions on scientific computing with Python. For example, the SIAM Conference on Computational Science and Engineering (CSE09) had a three-part minisymposium on Python for Scientific Computing.
This year we will also hold the conference earlier in the summer than in previous years--moving from the end of August to the end of June. Holding the conference at the end of August meant that the conference coincided with the start of the spring semester for many attendees, which caused some scheduling conflicts. Having the conference in late August also made it difficult to get a quick turn around on publishing our post-conference proceedings since most of the authors and reviewers had to focus on the beginning of the academic year.
We are also extending the post-conference sprint from two days to three. Our post-conference sprints have been increasingly successful and we hope that by extending the sprint we will be able to get a lot more done. We are also hoping that by moving the conference to earlier in the summer will give us more time to finish work started during the sprint before the fall semester. For example, this year David Warde-Farley and I worked on migrating the SciPy homepage from a MoinMoin wiki to a Sphinx site. During the sprint we made significant progress on the new site, but stopped short of being able to actually deploy it. Both David and I haven't had time to finish our work due to the academic semester starting immediately after the conference. The changes to the conference dates will hopefully increase our success in getting bigger projects, which are started during the sprint, finalized before the start of semester. (Work is slowly progressing on the new site and I hope to switch to the new site in December, during the SciPy India 2009 sprint.)
And if you stay for the entire sprint, you will get to be part of a Texas-sized celebration of the fourth of July. I was able to witness the festivities myself two years ago during the 2008 Mayavi sprint. While some people may not be able to attend the sprint because of its overlap with the fourth of July celebration, these dates were our best options given all our constraints. This summer is going to be a very busy one for the SciPy community. There are at least two other SciPy-related events going on during the same time as our conference. Sage Days 22 on elliptic curves will be held in Berkeley from June 21st to July 2nd, which unfortunately means that few Sage developers will be able to attend SciPy 2010. And the 2nd European Seminar of Coupled Problems (ESCO) will be held June 28th to July 2nd in the Czech Republic. It will feature a track on next generation scientific computing with a focus on Python. While it is great to see scientific computing with Python being presented in more and more venues, this particular conference means that Gaël Varoquaux will most likely not be attending the 2010 SciPy conference. As the program chair for both SciPy 2008 and SciPy 2009, Gaël has been essential to the success of the last two SciPy conferences.
Austin is called the "Silicon Hills" due to the large number of technology corporations located there and it advertises itself as the live music capital of the world. Almost every establishment provides live music every evening. Restaurants offer a wide variety of interesting fare with a focus on BBQ and Mexican Food. If you need some exercise, the river that runs through the downtown area has a very well-used running path, and the hill country surrounding Austin has many hiking and biking trails. And most importantly, Austin is home to Enthought. Enthought is the main sponsor of both the SciPy project and the conference.
I am looking forward to the SciPy 2010 conference and hope to see many of you there. If you are interested in helping out with the program committee, please send me an email. I would also like to continue and expand the student travel funding, so if you are interested in sponsoring students to attend the conference, please contact me. I will be posting more information regarding the venue and timeline in January (after I finish organizing the SciPy India conference).
Saturday, November 7, 2009
Thursday, November 5, 2009
We look forward to hearing your how you are using Python! For more information, please read the full call for presentations.
About the SciPy India 2009 Conference
The first SciPy India Conference will be held from December 12th to 17th, 2009 at the Technopark in Trivandrum, Kerala, India.
The theme of the conference is "Scientific Python in Action" with respect to application and teaching. We are pleased to have Travis Oliphant, the creator and lead developer of numpy as the keynote speaker.
Please register here.
- Friday, Nov. 20: Abstracts Due
- Friday, Nov. 27: Announce accepted talks, post schedule
- Saturday-Sunday, Dec. 12-13 Conference
- Monday-Tuesday, Dec. 14-15 Tutorials
- Wednesday-Thursday, Dec. 16-17 Sprints
- Jarrod Millman, Helen Wills Neuroscience Institute, University of California, Berkeley, USA (Conference Co-Chair)
- Prabhu Ramachandran, Department of Aerospace Engineering, Indian Institute of Technology, Bombay, India (Conference Co-Chair)
- FOSSEE Team
Wednesday, November 4, 2009
After Fernando finished his introduction, we heard nine short 4-minute lightning talks in rapid succession. This format really underscored how important Python is for so many different scientific disciplines. In addition to several presentations by faculty, staff, and students from UC Berkeley and LBL, we were fortunate to have two other visitors. William Stein spoke about Sage before running upstairs to present at the number theory seminar. And Ondřej Čertík spoke about sympy.
Following the formal presentations, Fernando facilitated an open discussion with Guido where we talked about everything from the transition to Python 3000 to the unladen swallow project. Hopefully, Fernando will post more details including links to the slides on his blog soon.
Saturday, July 11, 2009
About the conference
SciPy 2009, the 8th Python in Science conference, will be held from August 18-23, 2009 at Caltech in Pasadena, CA, USA. The conference starts with two days of tutorials to the scientific Python tools. There will be two tracks, one for introduction of the basic tools to beginners, and one for more advanced tools. The tutorials will be followed by two days of talks. Both days of talks will begin with a keynote address. The first day’s keynote will be given by Peter Norvig, the Director of Research at Google; while, the second keynote will be delivered by Jon Guyer, a Materials Scientist in the Thermodynamics and Kinetics Group at NIST. The program committee will select the remaining talks from submissions to our call for papers. All selected talks will be included in our conference proceedings edited by the program committee. After the talks each day we will provide several rooms for impromptu birds of a feather discussions.
Finally, the last two days of the conference will be used for a number of coding sprints on the major software projects in our community.
For the 8th consecutive year, the conference will bring together the developers and users of the open source software stack for scientific computing with Python. Attendees have the opportunity to review the available tools and how they apply to specific problems. By providing a forum for developers to share their Python expertise with the wider commercial, academic, and research communities, this conference fosters collaboration and facilitates the sharing of software components, techniques, and a vision for high level language use in scientific computing.
For further information, please visit the conference homepage:
Friday, July 3: Abstracts Due
Wednesday, July 15: Announce accepted talks, post schedule
Friday, July 17: Early Registration ends
Tuesday-Wednesday, August 18-19: Tutorials
Thursday-Friday, August 20-21: Conference
Saturday-Sunday, August 22-23: Sprints
Friday, September 4: Papers for proceedings due
Jarrod Millman, UC Berkeley, USA (Conference Chair)
Gaël Varoquaux, INRIA Saclay, France (Program Co-Chair)
Stéfan van der Walt, University of Stellenbosch, South Africa (Program Co-Chair)
Fernando Pérez, UC Berkeley, USA (Tutorial Chair)
Saturday, March 28, 2009
Friday, March 27, 2009
- Tutorials (Tuesday, August 18 - Wednesday, August 19)
- Conference (Thursday, August 20 - Friday, August 21)
- Sprints (Saturday, August 22 - Sunday, August 23)
Wednesday, March 25, 2009
To see a list of possible project ideas please see the PSF's SoC wiki. A number of students have all ready expressed interest in working on a SciPy-related project this year. If you are interested, the best place to discuss ideas is on the NumPy and SciPy developer mailing lists.
Wednesday, February 11, 2009
This release comes sixteen months after the 0.6.0 release and contains many new features, numerous bug-fixes, improved test coverage, and better documentation. Please note that SciPy 0.7.0 requires Python 2.4 or greater (but not Python 3) and NumPy 1.2.0 or greater.
For information, please see the release notes or my previous post. You can download the release from here. Thanks to everybody who contributed to this release.
Wednesday, February 4, 2009
The Python developers have attempted to make this transition as painless as possible. For instance, Python 2.6 helps simplify the migration path from the 2.x to the 3.x release series. Python 2.6 incorporates everything from 3.0 that doesn't introduce incompatibilities with the 2.x series. It also can be run with a -3 switch to warn about what will no longer work in Python 3. The developers also provide a Python program, called 2to3, to automatically convert Python 2.x source code to valid 3.x code.
The suggested strategy for migrating to Python 3 is essentially:
- port to 2.6
- fix all the warnings raised by the -3 switch
- run 2to3
- fix any remaining issues
While the above procedure of using the 2to3 tool works relatively for pure Python code, there is no automatic conversion tool for extension code. NumPy is mostly written in C and makes extensive use of the Python C-API. So converting NumPy will require much more than running the 2to3 tool. Once NumPy has been successfully ported, we will port SciPy to Python 3. Porting SciPy should be considerably easier. Regardless before porting either project to Python 3, we will need to ensure that both projects fully support Python 2.6.
Porting NumPy/SciPy to Python 2.6
Porting to Python 2.6 is a very pressing issue as at least one Linux distribution (openSUSE 11.1) has all ready moved to Python 2.6. Fedora 11 (scheduled to be released on 5/26/2009) will be based on Python 2.6.
The main issue with 2.6 support is NumPy. Over the last month or so, there has been a significant focus on making both NumPy and SciPy compatible with Python 2.6 largely thanks to the efforts of David Cournapeau. For example, the upcoming SciPy 0.7.0 release has replaced md5 and popen4 with hashlib and subprocess respectively (unfortunately, it appears that subprocess has a potential race condition). On UNIX (including Mac OS X), NumPy 1.2.1 mostly works under Python 2.6. On Windows, however, 1.2.1 has a number of problems related to the compilation process. Fixing these compilation issues required some fairly extensive changes and, thus, will not be included in a 1.2.x bug-fix release. However, these issues have mostly been addressed on the development trunk and will be included in the upcoming NumPy 1.3 release.
Hopefully, we will have a beta release of NumPy 1.3 out in a few weeks. And we should have a release candidate out shortly after. If all goes well, both NumPy and SciPy will be Python 2.6 compatible for all platforms by March.
Porting NumPy/SciPy to Python 3
Once we finish porting to Python 2.6 and remove all the warnings raised by the -3 switch, we will be ready to start seriously planning to port NumPy to Python 3. Supporting Python 3 will require significant effort, since a lot of C code has to be ported. We have taken a preliminary look at what this port will entail and have identified at least the following issues to be addressed:
- PyNumberMethods has changed: nb_divide, nb_coerce, nb_oct, nb_hex, and nb_inplace_divide have been removed
- PyObject_VAR_HEAD has changed to conform to standard C
- PyString_* is gone, all occurrences will need to be replaced with PyUnicode_* or PyBytes_*
- PyInt_* is gone, all occurrences will need to be replaced with PyLong_*
- Buffer interface has changed; this is a fairly big change and will require the most work
Before porting to Python 3, we will be paying close attention to how the major Linux distributions will be handling the transition to Python 3. Fedora developers have started a lively discussion about how they will handle the transition to Python 3. The discussion indicates that there is a reluctance on their part to support both Python 2 and 3 in the same release.
With the last release of NumPy (1.2) and the upcoming SciPy release (0.7) we dropped support for Python 2.3. Since many scientists may not be able to quickly upgrade to Python 3, we will be reluctant to drop support for Python 2 for some time. Given our desire to provide the newest releases of NumPy and SciPy to as many users as possible, we will be closely listening to them to determine how quickly we can move to Python 3 and drop support for Python 2. And we will use this time to continue adding features, fixing bugs, improving documentation, and perhaps most importantly extending our test coverage.
It is likely that NumPy 1.4 and SciPy 0.8 (I am hoping that we will be able to release both by the end of 2009 or early 2010) will be based on Python 2. The following releases, NumPy 1.5 and SciPy 0.9, would be the earliest point that I can see us switching to Python 3.
Over the last year there has been a lot of discussion about the transition to Python 3 on the mailing lists, at our annual conference, during coding sprints and planning meetings, as well as private conversations. One topic that has come up is whether this transition would be a good opportunity to simultaneously do a major redesign of NumPy.
While there is a temptation to take advantage of this opportunity for a major release, we quickly realized that doing so would be a huge mistake. First, it would make it difficult for scientists and researchers to isolate the root cause of errors in their code when porting to the new release. Is the problem with the Python 3 port or the port to the new NumPy release? Second, it would be extremely poor community behavior. If other major packages succumb to this temptation as well, switching to Python 3 will become an increasingly daunting task for all the code out there, which use these packages. So when we port NumPy to Python 3, we will do so in a release that includes no API or ABI changes not strictly related to Python 3.
Friday, January 23, 2009
For more information, please see the release notes and my previous blog post. You can download the release from here.
Sunday, January 18, 2009
Unfortunately, the Matrix SIG archive (1995-2000) was missing. And had been missing since at least 2005 when Robert Kern noticed that it had disappeared. Thanks to Skip Montanaro, Brad Knowles, and Barry Warsaw the archive has been restored.
Monday, January 12, 2009
After 16 months of hard work the next stable release of scipy is almost ready to be tagged. This is the most significant scipy release in several years. It contains many new features, numerous bug-fixes, improved test coverage, and better documentation.
The scipy developer community started porting scipy to numpy at the end of 2005. Most of the work during 2006-2007 was focused on porting and bug-fixes and culminated in scipy 0.6. However, since the 0.6 release, the development effort has shifted from porting and maintenance to a much greater focus on infrastructure, architecture, and functionality.
One of the most important developments during the last year has been the extensive work on the testing and documentation framework. The improvements to testing and documentation first appeared in the numpy 1.2 release.
Our new testing infrastructure is based on the nose testing framework. This testing framework makes writing tests much easier than previously. The simplified testing framework as well as an increasing recognition of the importance of unit testing among the scipy developers has led to a doubling of the number of tests since the last stable release.
This release also brings immense documentation improvements. You can now view the scipy reference manual online or download it as a PDF file. The new reference guide was built using the popular Sphinx tool. We have also updated the scipy tutorial, which hadn't been touched for several years. Both the reference manual and the tutorial are easily editable using our web-based documentation editor. If you find want to improve the documentation, please register a user name in our web-based editor and correct the issues.
In addition to huge improvements in the testing and documentation infrastructure, this release cleans up a number of annoyances and removes old cruft. There have been a number of deprecations and well-documented API changes in this release. This is also the first stable release since the sandbox was removed.
We also did a fairly extensive review of the code to make sure that all the code is correctly licensed. About a year ago, I noticed that there was code not licensed under the revised BSD license. I was trying to "correct" the license information for the scipy Fedora package, but was told that the Fedora packaging policy required including all licenses found in a package. After doing some grepping and looking at svn commit logs, it was easy to figure out what code needed to be relicensed and who had committed the code. Fortunately I was able to track everyone down and everyone kindly agreed to relicense their code. There was also some code included in scipy derived from "Numerical Recipes in C" code. Unfortunately, the "Numerical Recipes in C" code doesn't permit redistribution. The scipy developers quickly reimplemented the offending code from scratch. With the 0.7 release, scipy only includes code licensed under the revised BSD.
The 0.7 release should be out by the end of the month and we have all ready started working on the next feature release. During the development of the 0.7 release, there has been a rapid increase in community involvement and numerous infrastructure improvements to lower the barrier to contributions (e.g., more explicit coding standards, improved testing infrastructure, better documentation tools). I look forward to seeing this trend continue and invite everyone to become more involved. To learn more about the changes in the 0.7 release, please see the release notes.