New work: The Fourth Paradigm: Data-Intensive Scientific Discovery

Microsoft Research has put together a quite amazing collection looking at the revolution that is data intensive research, calling it the fourth paradigm:
The Fourth Paradigm: Data-Intensive Scientific Discovery
Edited by Tony Hey, Stewart Tansley, and Kristin Tolle


Comments

Post a Comment

Popular posts from this blog

Java, MySql increased performance with Huge Pages

[Resources updates: 2010.07.07 , 2009.11.03 , 2009.05.27 ] Long running, large memory, high performant applications often have special needs with respect to their memory management. On Linux, Solaris and other modern OSes, the translation look-aside buffer ( TLB ) - whose page size of 4k for many CPUs/OSes - becomes a scalability issue in these extreme conditions. In order to get around TLB scalability issues, huge page sizes are used to reduce the impact on performance. This can be of use to installations with large scale Java, MySql and other large memory applications. 300% improvement : " Well, in my case, I was able to achieve an over 3x improvement in my EJB 3 application, of which fully 60 to 70% of that was due to using large page memory with a 3.5GB heap. Now, a 3.5GB heap without the large memory pages didn't provide any benefit over smaller heaps without large pages. Besides the throughput improvements, I also noticed that GC frequency was cut down by two-third...
Read more

Lucene concurrent search performance with 1,2,4,8 IndexReaders

Image
My last Lucene evaluation ( Simultaneous (Threaded) Query Lucene Performance ) from a couple of days ago was looking at concurrent (multithreaded) queries using a single IndexReader across all threads. Due to suggestions/demand from the Lucene User mailing list, I have expanded the evaluation to include multiple IndexReaders . It is known that a single IndexReader is a limiting factor in a multithreaded environment. So I decided to run the same tests with 1,2,4 and 8 IndexReaders (actually I create IndexReaders and then create an IndexSearcher from each of these and share the IndexSearchers). Below are the results. All of the test environment are the same as my previous evaluation, except: It goes up to 8192 threads instead of the original 4096 threads I had to pass in to the Java VM: -Xmx4000m because the VM was running out of heap for 8 readers I've made the graph larger (Click on graphic to see results) As you can see, 2,4 and 8 readers significantly improve query rate over a...
Read more

Lucene indexing performance benchmarks for journal article metadata and full-text

I posted these journal article metadata & full-text Lucene indexing benchmarks to the lucene user mailing list using the suggested XML format , but it seems like that was not the proper thing to do. One of the list members (Cass Costello) converted it to HTML (thanks :-) ). I've decided to give it a permanent home here. If you have any questions, just let me know. I have some other benchmarks I will be posting with more records (~25 million) but only article metadata, not full-text. The loader that does all of this was developed as part of my Ungava project. Hardware Environment Dedicated machine for indexing : yes CPU : Dual processor dual core Xeon CPU 3.00GHz; hyperthreading ON for 8 virtual cores RAM : 8GB Drive configuration : Dell EMC AX150 storage array fibre channel Software environment Lucene Version : 2.3.1 Java Version : Java(TM) SE Runtime Environment (build 1.6.0_02-b05) Java VM : Java HotSpot(TM) 64-Bit Server VM (build 1.6.0_02-b05, mixed mode) OS Version : ...
Read more