Zzzoot

Tomorrow is (sadly) my last official day * as a university visitor at the Australian National University (ANU), Canberra . I've been here since late June, invited by ANU adjunct and Funnelback chief scientist (and ex-CSIRO ) David Hawking , to visit the Algorithms and Data Research Group , School of Computer Science , College of Engineering and Computer Science . I was installed in a lovely office looking out into the campus, where I've been working on large scale journal visualization, a continuation of the Torngat project . I've been working on a couple of things, including applying Mulan to the multi-label problem of the corpus I am working with, so I can get precision and recall to evaluate this method empirically. My productivity has been hampered by a recurring stomach problem (which appears to be gone this last week: yay!), so I've not progressed as much as I would have wanted to.... :-( At the end of last week I gave a presentation at CSIRO (in the same bui...

Using Open Source Tools for Visualization and Semantic Mapping in a Large Scale Article Digital Library [slideshare] - Questions: Anyone working in the cloud? Converting 4TB of TIFFS to PDF in 24hrs: Hadoop + EC2 + S3 = Super alternatives for researchers (& real people too!)

Project Torngat is a research project here at NRC - CISTI   [ Note that I am no longer at CISTI and that I am now continuing this work at Carleton University - GN 2010 04 07 ] that looks to use the full-text of journal articles to construct semantic journal maps for use in -- among other things -- projecting article search results onto the map to visualize the results and support interactive exploration and discovery of related articles, term and journals. Starting with 5.7 million full-text articles from 2200+ journals (mostly science, technology and medical (STM)), and using LuSql , Lucene , Semantic Vectors , R , and processing , a two dimensional mapping of a 512 dimension semantic space was created which revealed an excellent correspondence with the 23 human-created journal categories: Semantic Journal Space of 2231 Journals Scaled to Two Dimensions This initial work was initiated to find a technique that would scale, and follow-up work is looking at integrati...

Gave my LuSql talk today at code4lib2009 and didn't get cut down by any Solr/Lucene dudes! Met Erik Hatcher of Lucene/Solr fame (and now of Lucid Imagination fame) & hopefully we can collaborate on some Lucene/indexing Solr stuff in the future. I also spoke with Tom Burton-West of UMich about Lucene indexing and search performance for their 1M+ Google Books index (they use Solr). These are documents that are a lot longer than the STM articles I work with. They have 220GB sized indexes and - as they have to keep stops words for their Humanities for phrase searching - suffer from poor query performance (despite 32GB RAM). I pointed to some of my previous work on high performance indexing and searching [ 1 , 2 , 3 ]. I'd like to get at their data to examine some performance issues in Lucene, both on the indexing and searching side. I was wondering if Solr is configurable for the initial/max number of IndexSearchers. I couldn't find this in the Solr wiki, but did see in...

I have been doing some indexing performance tests with LuSql , and have some numbers comparing Lucene 2.3.1 with 2.4. Despite some discussion about 2.4 having poorer indexing performance, my tests with LuSql 0.9 suggest otherwise: Lucene 2.3.1 Number of records added= 2000000 Optimizing index Closing index Optimizing index time: 311 seconds Closing JDBC: result set Closing JDBC: statement Closing JDBC: connection *********** Elapsed time: 854 seconds 15m 18s Lucene 2.4 Number of records added= 2000000 Optimizing index Closing index Optimizing index time: 322 seconds Closing JDBC: result set Closing JDBC: statement Closing JDBC: connection *********** Elapsed time: 759 seconds 12m 39s Index size: 3.7GB. It is interesting that the overall indexing time is significantly less, but the optimizing time is slightly higher. Data, hardware and system configuration: as per my previous Lucene benchmarking . Note that this is a simple benchmark, so YMWV. This benchmark was done with the LuSql de...

LuSql is a simple but powerful tool for building Lucene indexes from relational databases. It is a command-line Java application for the construction of a Lucene index from an arbitrary SQL query of a JDBC -accessible SQL database. It allows a user to control a number of parameters, including the SQL query to use, individual indexing/storage/term-vector nature of fields, analyzer, stop word list, and other tuning parameters. In its default mode it uses threading to take advantage of multiple cores. LuSql can handle complex queries, allows for additional per record sub-queries, and has a plug-in architecture for arbitrary Lucene document manipulation. Its only dependencies are three Apache Commons libraries, the Lucene core itself, and a JDBC driver. LuSql has been extensively tested, including a large 6+ million full-text & article metadata document collection, producing an 86GB Lucene index. I am the author of the LuSql software. LuSql at CISTI Lab LuSql at freshmeat Upda...

I am excited at the release of Katta , a technology built on Lucene , Zookeeper and Hadoop allowing for Lucene indexes to be distributed across a number of nodes for distributed & fault tolerant search. Note that it does not create the indexes, simply deploys existing indexes onto this infrastructure.

My last Lucene evaluation ( Simultaneous (Threaded) Query Lucene Performance ) from a couple of days ago was looking at concurrent (multithreaded) queries using a single IndexReader across all threads. Due to suggestions/demand from the Lucene User mailing list, I have expanded the evaluation to include multiple IndexReaders . It is known that a single IndexReader is a limiting factor in a multithreaded environment. So I decided to run the same tests with 1,2,4 and 8 IndexReaders (actually I create IndexReaders and then create an IndexSearcher from each of these and share the IndexSearchers). Below are the results. All of the test environment are the same as my previous evaluation, except: It goes up to 8192 threads instead of the original 4096 threads I had to pass in to the Java VM: -Xmx4000m because the VM was running out of heap for 8 readers I've made the graph larger (Click on graphic to see results) As you can see, 2,4 and 8 readers significantly improve query rate over a...

I've recently had to do some performance query tests on Lucene (v2.3.1) under concurrent request load. These benchmarks are on the same machine, VM, OS described in my earlier Lucene indexing performance benchmarks , however the index is a little different: it is an index of title, author, journal name, keywords, etc metadata (no full-text) for 25.6M journal articles. The index size is 19GB and -- using the same framework as the previous benchmarks, above -- indexing time of 4.25 hours. YMMV. Using a set of 2900 user queries (ranging from single word queries to queries with >600 characters and using multiple fields and operators; no range queries), Lucene was pre-warmed with 2000 (different) queries. Ten runs were performed and averaged. Below are the results plotting #requests per second handled vs. #threads making requests . This was all run in the same VM, using an instance of java.util.concurrent.ThreadPoolExecutor to parallelize things: The best results were for 6 or 7 t...

I am excited to learn from The Search Guy about the recently released research-oriented full-text engine from Sun Labs called ' Minion '. Minion the Open Source Search: One of the secret weapons underlying the Search Inside the Music project and Project Aura is a high quality search engine called Minion. Minion handles everything that has to do with Text for these projects. In addition to traditional search, we use Minion for document similarity (the core technique used for Tagomendations), item clustering, sense disambiguation, classification and autotagging. Minion is a research-oriented search engine - meaning that it is designed to allow for all sorts of variations. It is ultra-configurable and has a simple API. The big news is that the process to open source the Minion engine is underway. Steve Green (aka the search guy) has created a Minion project on Java.net - and soon, the Minion search engine will be available for all. " Right now there is limited real info on ...

I posted these journal article metadata & full-text Lucene indexing benchmarks to the lucene user mailing list using the suggested XML format , but it seems like that was not the proper thing to do. One of the list members (Cass Costello) converted it to HTML (thanks :-) ). I've decided to give it a permanent home here. If you have any questions, just let me know. I have some other benchmarks I will be posting with more records (~25 million) but only article metadata, not full-text. The loader that does all of this was developed as part of my Ungava project. Hardware Environment Dedicated machine for indexing : yes CPU : Dual processor dual core Xeon CPU 3.00GHz; hyperthreading ON for 8 virtual cores RAM : 8GB Drive configuration : Dell EMC AX150 storage array fibre channel Software environment Lucene Version : 2.3.1 Java Version : Java(TM) SE Runtime Environment (build 1.6.0_02-b05) Java VM : Java HotSpot(TM) 64-Bit Server VM (build 1.6.0_02-b05, mixed mode) OS Version : ...

I am in Denver today, giving the afternoon plenary at today's Next Gen Library Interface Conference , put on by the Colorado Alliance of Research Libraries . I'd like to thank George Machovec , associate director of "the Alliance", for inviting me down. The meeting is a mix of vendor presentations with two plenaries: Morning Keynote: Marshall Breeding , Director for Innovative Technologies and Research, Vanderbilt University Library Vendor presentation: Encore (III) break Vendor presentation: AquaBrowser Vendor presentation: Primo (Ex Libris) lunch Afternoon Keynote: Glen Newton , Researcher, Information Science, NRC-CISTI, National Research Council Canada: Open source tools and case study . break Vendor presentation: WorldCat Local (OCLC) Vendor presentation: Indigo (TLC) Here is my presentation . PowerPoint created in Open Office. Related post (Marshal Breeding): 2008 Feb 23: Openness in the library (technology) Update March 28: Link to Colorado Alliance of Resear...

Extensible Text Framework (XTF) : FLOSS platform for access to digital content XTF is the California Digital Library 's amazing access platform for digital content. It is based on Lucene , a tool that is well known as a scalable and stable full-text engine. But XTF is more than Lucene, and is a full end-to-end system, offering ü ber configurable indexing, quering and display. Java-based, completely XSLT-driven presentation-layer, extensible to things like Shibboleth , and has some very nice additioanl features like OAI-PMH provider and SRU . From what I can tell it does not have an SOA architecture, but offers a high degree of modularity which could easily be wrapped in Web services, etc