Java, MySql increased performance with Huge Pages

[Resources updates: 2010.07.07, 2009.11.03, 2009.05.27]

Long running, large memory, high performant applications often have special needs with respect to their memory management. On Linux, Solaris and other modern OSes, the translation look-aside buffer (TLB) - whose page size of 4k for many CPUs/OSes - becomes a scalability issue in these extreme conditions. In order to get around TLB scalability issues, huge page sizes are used to reduce the impact on performance. This can be of use to installations with large scale Java, MySql and other large memory applications.
300% improvement: "Well, in my case, I was able to achieve an over 3x improvement in my EJB 3 application, of which fully 60 to 70% of that was due to using large page memory with a 3.5GB heap. Now, a 3.5GB heap without the large memory pages didn't provide any benefit over smaller heaps without large pages. Besides the throughput improvements, I also noticed that GC frequency was cut down by two-thirds, and GC time was also cut down by a similar percentage (each individual GC event was much shorter in duration). Of course, your mileage will vary, but this one optimization is worth looking at for any high throughput application." [20]

25% - 300% improvement: "These memory accesses are then frequently cache misses which introduces a high latency to the memory request. Increasing page sizes from 4K to 16M significantly reduces this problem as the number of tlb misses drops. Typically it will reduce runtimes by 25-30% but in an extreme case I've seen an SPH code run 3x faster simply by enabling large pages." [8]

In this article[11], the too common problem of an intermittant, ephemeral but huge reduction in performance was solved by recognizing that 5GB of memory was taken-up in the page tables, made up of 4k pages. Solution: use Linux huge page size support and make pages larger, reducing the page table size to 200MB.

"Many Java applications, especially those using large heaps, can benefit from what the operating systems call large or huge pages."[10]

"17.26 times faster" (Linux, Java, 64bit) [19]

"While 16 GB pages are intended to only be used in very high-performance environments, 64 KB pages are general-purpose, and most workloads are likely to see a benefit by using 64 KB pages rather than 4 KB pages."[7]

Java on an OS that supports large page sizes has better performance for many application. Which applications? Use tools[43] to see the TLB cache hit rate when these applications run. Different CPUs support different page sizes:
  • i386: 4K and 4M (2M in PAE mode)
  • ia64: 4K, 8K, 64K, 256K, 1M, 4M, 16M, 256M
  • PPC64: 4K and 16M
  • POWER5+: 4K, 64K, 16MB, 16GB (!!)
  • UltraSparc III: 8K, 64K, 512K, 4M
  • UltraSparc T2: 8K, 64K, 4M, 256M
Just add -XX:+UseLargePages to the JVM command line to use after setting up. For CPUs that support multiple sizes, use -XX:LargePageSizeInBytes=2m to define the page size you want to use (i.e. for 2MB pages, etc.).

Additional Huge Page resources


    Comments

    Dexter said…
    Wonderful resource..

    I need your assistance on Java code for restoring the backup file into MySQL. If you can provide any kind of assistance that will be great.

    Thanks
    16 April, 2009 18:26
    Anonymous said…
    Minor correction:

    "5GB of memory was taken-up in the TLB" should be "5GB of memory was taken-up in the page tables".
    08 July, 2010 13:35
    Glen Newton said…
    Correction made. Thanks! :-)
    08 July, 2010 18:52
    Gavin Carr said…
    Great resource - thanks Glen.

    FYI, link [54] is broken. Seems to be http://events.linuxfoundation.org/slides/lfcs09_hansen1.pdf now.
    04 March, 2011 12:13
    Post a Comment

    Popular posts from this blog

    IBM on Linux: "Lean, clean, and green"

    IBM developerWorks has an article ( Linux: Lean, clean, and green: How GNU/Linux is becoming more eco-friendly - 26 May 2009) which examines some of the Green benefits of the Linux operating system. It focuses primarily on the low resource demands Linux has on systems (as well as its support for older systems), thus extending the life of machines that would otherwise be junked. Also discussed is virtualization and aspects of the Linux OS that reduce power consumption in servers. Additional Green Linux and Open Source resource: LessWatts.org - Saving Power on Intel systems with Linux Ten ways Linux can turn you green . 2009 Green Computing With Open Source Software . 2009 Open Source is Already Naturally Green: Fewer Lawyers, Fewer Showers, More Real People Add Up to Big Green Wins . 2009 Is Linux the Greenest Operating System . 2009 Go Green, Save Green with Linux . 2008 Linux captures the 'green' flag, beats Windows 2008 power-saving measures 2008 Canadian Green Party...
    Read more

    Lucene indexing performance benchmarks for journal article metadata and full-text

    I posted these journal article metadata & full-text Lucene indexing benchmarks to the lucene user mailing list using the suggested XML format , but it seems like that was not the proper thing to do. One of the list members (Cass Costello) converted it to HTML (thanks :-) ). I've decided to give it a permanent home here. If you have any questions, just let me know. I have some other benchmarks I will be posting with more records (~25 million) but only article metadata, not full-text. The loader that does all of this was developed as part of my Ungava project. Hardware Environment Dedicated machine for indexing : yes CPU : Dual processor dual core Xeon CPU 3.00GHz; hyperthreading ON for 8 virtual cores RAM : 8GB Drive configuration : Dell EMC AX150 storage array fibre channel Software environment Lucene Version : 2.3.1 Java Version : Java(TM) SE Runtime Environment (build 1.6.0_02-b05) Java VM : Java HotSpot(TM) 64-Bit Server VM (build 1.6.0_02-b05, mixed mode) OS Version : ...
    Read more