Extremely Large Databases

The First Workshop on Extremely Large Databases was held at the Stanford Linear Accelerator Center, October 2007. Many of the heavy hitters were there (Google, Yahoo, Microsoft, IBM, Oracle, Terrasoft, SLAC, NCSA, eBay, AT&T, etc) from industry, academia and science (? their classification).

A report is available and I thought I'd touch on some of the more interesting things I found in it:

Scale:
  • Most have systems with > 100TB of data, with 20% of scientific databases > 1PB of data;
  • All from industry reps had >100PB of data, with all having at least one system with >1PB
  • Industry had single tables with > 1 trillion rows; science ~100 times smaller.
  • Need for multi-trillion-row tables in <10 years
  • Peak ingest: 1B rows per hour; 1B rows per day common
  • "All users said that even though their databases were already growing rapidly, they would store even more data in databases if it were affordable. Estimates of the potential ranged from ten to one hundred times current usage. The participants unanimously agreed that "no vendor meets our database needs"."
Usage:
  • The most surprising observation: "...highly unpredictable query loads, with up to 90% of queries being new." Wow: not good for modern learning query optimizers.

Open Source Software
  • "Both groups often use free and/or open source software such as Linux, MySQL, and PostgreSQL extensively to reduce costs."
MapReduce for some operations
  • "The map/reduce paradigm has built substantial mind-share thanks to its relatively simple processing model, easy scalability, and fault tolerance. It fits well with the aforementioned need for full table scans. It was pointed out that the join capabilities of this model are limited, with sort/merge being the primary large-scale method being used today". See previous entry on MapReduce/Hadoop.

Important Science Differences:
  • "The longevity of large scientific projects, typically measured in decades, forces scientists to introduce extra layers in order to isolate different components and ease often unavoidable migrations, adding to system complexity. Unfortunately, those layers aretypically used only to abstract the storage model and not the processing model."

Comments

Post a Comment

Popular posts from this blog

Java, MySql increased performance with Huge Pages

[Resources updates: 2010.07.07 , 2009.11.03 , 2009.05.27 ] Long running, large memory, high performant applications often have special needs with respect to their memory management. On Linux, Solaris and other modern OSes, the translation look-aside buffer ( TLB ) - whose page size of 4k for many CPUs/OSes - becomes a scalability issue in these extreme conditions. In order to get around TLB scalability issues, huge page sizes are used to reduce the impact on performance. This can be of use to installations with large scale Java, MySql and other large memory applications. 300% improvement : " Well, in my case, I was able to achieve an over 3x improvement in my EJB 3 application, of which fully 60 to 70% of that was due to using large page memory with a 3.5GB heap. Now, a 3.5GB heap without the large memory pages didn't provide any benefit over smaller heaps without large pages. Besides the throughput improvements, I also noticed that GC frequency was cut down by two-third
Read more

IBM on Linux: "Lean, clean, and green"

IBM developerWorks has an article ( Linux: Lean, clean, and green: How GNU/Linux is becoming more eco-friendly - 26 May 2009) which examines some of the Green benefits of the Linux operating system. It focuses primarily on the low resource demands Linux has on systems (as well as its support for older systems), thus extending the life of machines that would otherwise be junked. Also discussed is virtualization and aspects of the Linux OS that reduce power consumption in servers. Additional Green Linux and Open Source resource: LessWatts.org - Saving Power on Intel systems with Linux Ten ways Linux can turn you green . 2009 Green Computing With Open Source Software . 2009 Open Source is Already Naturally Green: Fewer Lawyers, Fewer Showers, More Real People Add Up to Big Green Wins . 2009 Is Linux the Greenest Operating System . 2009 Go Green, Save Green with Linux . 2008 Linux captures the 'green' flag, beats Windows 2008 power-saving measures 2008 Canadian Green Party
Read more

Postscript coding resources

Image
Postscript is one of my favourite programming languages. Resources for postscript programming are not too common, and are always popping up and then disappearing, as things tend to do on the Web. Here is a snapshot of links that work: I will do my best to keep them up-to-date.
Read more