Article: Open Source Econometric Software better performance, accuracy and bug fixing than commercial software

Yalta and Yalta 2010 ("Should Economists Use Open Source Software for Doing Research?") examine the reliability, accuracy and bug fixing time for an Open Source econometric software package and five commercial econometric software packages. They find that after 5 years many of the bugs in the commercial software have not been fixed, whereas similar bugs in the Open Source software are fixed and released within a week of the discovery of the bugs.

Building on the work done by McCullough in 2004 applying a set of tests called Wilkinson's tests to the five commercial software packages, they re-apply these tests to the new versions of the commercial software, and apply the tests to the Open Source econometrics software Gretl.


The idea behind the Yalta and Yalta paper is to evaluate the bug fix time of the Open Source software, and compare this to the fixes -- if any -- and their times, that have been applied to the commercial software, since the 2004 McCullough paper. More specifically, to examine the bugs found by McCullough and see if they have been fixed.


The five packages examined here Yalta and Yalta, and earlier by McCullough are:

Bugs and bug fixing times for Gretl and commercial software

Commercial Packages
PackageBugTime to fix bug
and release new version
Gretl Reading files 3 days
Gretl Rounding error 4 days
Gretl Standard deviation 1 day
Gretl Spearman value 1 day
E_Views Correlations coefficients unit bug <5yrs
LIMDEP All DNAa
RATS ZERO Correlations >5yrs (not fixed)
RATS Singularity correlation estimates >5yrs (not fixed)
SHAZAM Missing values <5yrs
SHAZAM X,BIG,LITTLE,MISS <5yrs
SHAZAM Correlations, Spearman correlation >5yrs (not fixed)
SHAZAM MISS correlation >5yrs (not fixed)
SHAZAM ZERO correlation >5yrs (not fixed)
TSP ZERO correlation <5yrs
TSP Test IIB Failed

a We could not apply the tests on Package2 [LIMDEP] because, unlike the other packages, the demo version offered by the vendor only allows using several built-in data sets. As a result, it was not possible without payment to know whether or not they have fixed the flaws in their product. - Yalta and Yalta 2010

Conclusions

The authors make a number of important points in their conclusions:

  • On the other hand, studies in the last 15 years show that commercial software vendors can also introduce various difficulties to the research process by not correcting the known errors, avoiding to give details on the algorithms, or providing false information regarding their programs. Closed source software can hurt the reliability of computational results by making it impossible to study and verify the programming code performing the myriad functions expected from today's typical econometric package. It also complicates the process of research replication, which is already an exception and not a rule in the field of economics.

  • The open source movement,which has started to pick momentum after 1998, is now resulting in scientific software reaching and in some cases surpassing in terms of features and usability some of the proprietary alternatives. This new paradigm also brings its own set of inefficiencies such as an over-supply or under-supply of certain types of software,a surplus of licenses as well as the potential for wasted effort due to 'hijacking,' 'forking,' and 'abandoning.' When it comes to reliability and accountability, however, FLOSS helps avoid some of the difficulties associated with proprietary programs. Open source development is a transparent and merit based process similar in some ways to academics. The availability of the source code enables its verification by a large number of people with in the economics profession. Because it is free,everyone has access to it.It is flexible and future proof. These not only result in software of a high standard, but also facilitate peer review and help advance research replication.

  • In an attempt to assess reliability and accountability, we applied an entry level test suite of accuracy on the gretl econometric package and discovered a number of software defects. However, because gretl is open source, our experience was considerably different in comparison to earlier studies assessing various proprietary packages...unlike the other studies, all of the errors were corrected within a week of our reporting. Moreover, each time there was a revision to one of the source files, the updated version of the program was immediately available for download and inspection...When we applied the same tests on four widely-used proprietary econometric programs, we found that the various flaws uncovered and reported in an earlier study were not necessarily corrected. Despite the 5 years passing, only two of the software vendors have fixed all of the reported errors and still there were problems in all of the packages that we were able to test.



The authors also list what they consider significant Open Source software in the economic and econometrics space:











ProjectCategoryYear DevelopersSLOC Effort
GNU Octave Numerical analysis 1988 74 853,439 238
Gnumeric Spreadsheet 2001 9 384,341 100
Gnuplot Scientific plotting 1986 6 95,380 24
Gretl Econometrics 2000a 10 361,393 94
Maxima Algebra
1998a		 17 616,576 167
PSPP Statistics 1998 3 152,593 39
R Statistics 1997 13 549,780b 151
Sage Mathematics 2005 142 195,602 51
Scilab Numerical analysis 1994 35 1,234,895 341
SciPy Mathematical library 2001 31 455,903 124
Source: Ohloh.net.
a Shows the year the program became Open Source
b Base system only. The more than 1700 contributed R extension packages are not included


New York Times article: Data Analysts Captivated by R’s Power

References and related references

Comments

Post a Comment

Popular posts from this blog

Java, MySql increased performance with Huge Pages

[Resources updates: 2010.07.07 , 2009.11.03 , 2009.05.27 ] Long running, large memory, high performant applications often have special needs with respect to their memory management. On Linux, Solaris and other modern OSes, the translation look-aside buffer ( TLB ) - whose page size of 4k for many CPUs/OSes - becomes a scalability issue in these extreme conditions. In order to get around TLB scalability issues, huge page sizes are used to reduce the impact on performance. This can be of use to installations with large scale Java, MySql and other large memory applications. 300% improvement : " Well, in my case, I was able to achieve an over 3x improvement in my EJB 3 application, of which fully 60 to 70% of that was due to using large page memory with a 3.5GB heap. Now, a 3.5GB heap without the large memory pages didn't provide any benefit over smaller heaps without large pages. Besides the throughput improvements, I also noticed that GC frequency was cut down by two-third
Read more

IBM on Linux: "Lean, clean, and green"

IBM developerWorks has an article ( Linux: Lean, clean, and green: How GNU/Linux is becoming more eco-friendly - 26 May 2009) which examines some of the Green benefits of the Linux operating system. It focuses primarily on the low resource demands Linux has on systems (as well as its support for older systems), thus extending the life of machines that would otherwise be junked. Also discussed is virtualization and aspects of the Linux OS that reduce power consumption in servers. Additional Green Linux and Open Source resource: LessWatts.org - Saving Power on Intel systems with Linux Ten ways Linux can turn you green . 2009 Green Computing With Open Source Software . 2009 Open Source is Already Naturally Green: Fewer Lawyers, Fewer Showers, More Real People Add Up to Big Green Wins . 2009 Is Linux the Greenest Operating System . 2009 Go Green, Save Green with Linux . 2008 Linux captures the 'green' flag, beats Windows 2008 power-saving measures 2008 Canadian Green Party
Read more

Mars Inc. Cacao Genome Database claims Open Access, public domain: falls short

This initially looked very promising: Mars , along with a number of collaborators (USDA, IBM, Clemson University Genomics Institute; Public Intellectual Property Resource for Agriculture at the University of California-Davis; National Center for Genome Resources; Center for Genomics and Bioinformatics at Indiana University; HudsonAlpha Institute for Biotechnology; and Washington State University), have sequenced the cacao genome and released it "Open Access" and "public domain" for the benefit of all, at a site called the Cacao Genome Project : McLean, VA –Today, Mars, Incorporated, the U.S. Department of Agriculture-Agricultural Research Service (USDA-ARS), and IBM released the preliminary findings of their breakthrough cacao genome sequence and made it available in the public domain . - From the Mars Inc. press release 15 September 2010 A quote from the Independent article on the release ( First rice, then wheat – now cocoa genome unravelled 15 Sep 2010) f
Read more