SOLR

An Open Source Enterprise Search Server

Siegfried Goeschl

Needing a Fulltext Search Engine?!

• You have a lot of unstructured textual data
  • You are counting in gigabytes rather than megabytes
• You are considering a few advanced features
  • Field search
  • Boolean operators
  • Synonym Injection
  • Stemming
  • Phonetic Search
  • Faceted Search
  • Document and field boosting
  • CJK (Chinese, Japanese and Korean) support

Needing a Fulltext Search Engine?!

Search	Result
'rem'	should match 'R.E.M'
'6 feet under'	should match 'Six Feet Under'
'Pink'	should match 'P!nk'
nick cave	'Nick Cave and The Bad Seeds' should score higher than 'A Man for the (Stone) Ages - Nick the Caveman'
'Seimon Garfankl'	could match 'Simon & Garfunkel' and 'Paul Simon and Art Garfunkel'
'Otzi'	should match 'DJ Ötzi'
'Connecting'	should match 'Stereo MCs - Connected'

You Make The Decision

• Roll your own solution
  • You are a Perl hacker and have already started vi to do some coding ...
• Relying on your favourite database
  • MySQL fulltext search does not scale well
  • Scalablity is expensive in Oracle land
  • Difficult when supporting multiple database vendors
• Using an off-the-shelf component
  • Lucene
  • Sphinx
  • Egothor
  • Namazu

A Case For Lucene

• Written in Java
• Large and active OSS community
• Business friendly Apache 2.0 licence
• Good documentation, e.g. "Lucene in Action"
• Small enough for desktop search
• Big enough for Wikipedia and Yahoo
• Updatable index for data changes

Hitting The Lucene Wall

• Lucene is only a fulltext search library
  • No backup/restore mechanism
  • No remote query interface
  • No out-of-the-box data ingestion
  • No administration user interface for trouble shooting
  • No ${you name it} ...
• A lot of Lucene consultants are out there writing infrastructure code
  • Looks like reinventing wheels
  • So an open source wheel was invented
  • And it was called SOLR

SOLR - A Lucene Search Server

• The SOLR webapp runs in any servlet container
• XML and CSV data import
• RESTful HTTP interface
• Configurable response formats (e.g XML, JSON)
• HTML Administration GUI
• Caches out of the box (filter, query result, document)
• Replication support
• Extensibility by design (request handling, response handling, analyzers, caches...)

SOLR Architecture

iTunes Sample - Overview

• Export an iTunes library into XML
• Transform this XML into Lucene documents
• Import the Lucene documents into SOLR
• Run a few Lucene queries a display the results
• Update Lucene documents to fix wrong track names
• Delete Lucene documents to delete an album

iTunes Sample - Library Export

iTunes Sample - PList Document

iTunes Sample - Lucene Document

iTunes Sample - Solr Schema

iTunes Sample - SOLR Queries

• Search for all tracks sounding like "mozilla" => type:track track:mozilla~

• Get all albums from 'Nick Cave' sorted by release year => type:album artist:(nick cave); year asc

• Get all tracks containing/starting 'rose' from 'Nick Cave' sorted by album name => type:track track:(rose*) artist:(nick cave); album asc

• Get all tracks from Nick Cave's album 'The Best Of' sorted by track nr => type:track artist:(nick cave) album:(the best of); tracknr asc

XML Commands

• Add/update documents using <add>
• Delete documents by id or query using <delete>
• Commit your changes using <commit/>
• Optimize the Lucene index using <optimize/>

XML Update Command

• Something went wrong for 'The Queens Of The Stone Age' album => type:track artist:(queen of the stone age)
• Need to run an <update> command
• itunes> sh ./bin/post.sh ./xml/update.xml
• And now it is fixed => type:track album:(queen of the stone age)

XML Delete Command

• Oops, what are the "Zehn kleine Krabbelfinger" => type:track album:(kleine krabbelfinger)
• Need to run an <delete> command
• itunes> sh ./bin/post.sh ./xml/delete.xml
• And now it is fixed => type:track album:(kleine krabbelfinger)

SOLR Admin

Extending SOLR @ Verisign

• Custom SOLR Request Handler
  • Providing application specific query language
  • operator:at_mobilkom affiliate:demo itemtype:asset assettype:fullaudio hsfilter:true handset:nokia_6680_emea name:sinead
• Custom SOLR Response Handler
  • Generate a CSV response
  • Doing some really ugly post filtering there
• Custom Data Ingestion
  • Quartz scheduler triggers database resync
  • Database integration using Apache iBatis
  • Document ingestion with SOLR UpdateHandler

Common Pitfalls

• You can't update a single field
• Lucene is not a database
• There is only one document schema
• Escape user search input properly
• Don't skip testing

Conclusion

• SOLR is an easy to use Lucene search server
• XML data ingestion gets you up and running quickly
• HTTP protocol together with various response formats gives you a lot of integration points
• "Configuration over Coding" - delivers a lot of Lucene's horse power without coding
• Provides a couple of extension points to customize the functionality
• Replication support is a bonus just in case you need it

Reference

• The SOLR homepage: http://lucene.apache.org/solr/
• Grant Ingersoll: Search smarter with Apache Solr, Part1
• Grant Ingersoll: Search smarter with Apache Solr, Part2
• Erik Hatcher and Otis Gospodnetic: Lucene in Action, Manning Publications, 2004
• Peter Zaitsev, Vadim Tkachenko: High Performance Full Text Search for Database Content, EuroOSCON 2006
• Namazu, a Perl Full-Text Search Engine: http://www.namazu.org

What is Stemming Algorithm

A stemming algorithm is a process of linguistic normalisation, in which the variant forms of a word are reduced to a common form, for example 'connect'

• connection
• connections
• connective
• connected
• connecting