Lucene nightly benchmarks
Each night, an automated Python tool checks out the Lucene/Solr trunk source code and runs multiple benchmarks: indexing the entire Wikipedia English export three times (with different settings / document sizes); running a near-real-time latency test; running a set of "hardish" auto-generated queries and tasks. The tests take around 2.5 hours to run, and the results are verified against the previous run and then added to the graphs linked below.The goal is to spot any long-term regressions (or, gains!) in Lucene's performance that might otherwise accidentally slip past the committers, hopefully avoiding the fate of the boiling frog.
See more details in this blog post.
Results:
Indexing throughput
Near-real-time latency
BooleanQuery (AND, high freq, high freq term)
BooleanQuery (AND, high freq, medium freq term)
BooleanQuery (OR, high freq, high freq term)
BooleanQuery (OR, high freq, medium freq term)
FuzzyQuery (edit distance 1)
FuzzyQuery (edit distance 2)
NumericRangeQuery (int)
PhraseQuery (exact)
PhraseQuery (sloppy)
PrefixQuery (3 characters)
Primary Key Lookup
Respell
SpanNearQuery
Term (bgroup)
Term (bgroup, 1pass)
TermDateFacets
TermGroup100
TermGroup10K
TermGroup1M
TermQuery
TermQuery (date/time sort)
TermQuery (title sort)
WildcardQuery
Details by day:
Sun 04/03/2011
Mon 04/04/2011
Wed 04/06/2011
Thu 04/07/2011
Fri 04/08/2011
Sat 04/09/2011
Sun 04/10/2011
Mon 04/11/2011
Tue 04/12/2011
Wed 04/13/2011
Thu 04/14/2011
Fri 04/15/2011
Sat 04/16/2011
Sun 04/17/2011
Mon 04/18/2011
Tue 04/19/2011
Wed 04/20/2011
Thu 04/21/2011
Fri 04/22/2011
Sat 04/23/2011
Sun 04/24/2011
Mon 04/25/2011
Tue 04/26/2011
Wed 04/27/2011
Thu 04/28/2011
Fri 04/29/2011
Sat 04/30/2011
Mon 05/02/2011
Tue 05/03/2011
Wed 05/04/2011
Thu 05/05/2011
Fri 05/06/2011
Sat 05/07/2011
Sun 05/08/2011
Mon 05/09/2011
Tue 05/10/2011
Wed 05/11/2011
Thu 05/12/2011
Fri 05/13/2011
Sat 05/14/2011
Sun 05/15/2011
Mon 05/16/2011
Tue 05/17/2011
Wed 05/18/2011
Thu 05/19/2011
Fri 05/20/2011
Sat 05/21/2011
Sun 05/22/2011
Tue 05/24/2011
Wed 05/25/2011
Thu 05/26/2011
Fri 05/27/2011
Sat 05/28/2011
Sun 05/29/2011
Mon 05/30/2011
Tue 05/31/2011
Wed 06/01/2011
Fri 06/03/2011
Sat 06/04/2011
Sun 06/05/2011
Mon 06/06/2011
Wed 06/08/2011
Thu 06/09/2011
Fri 06/10/2011
Sat 06/11/2011
Sun 06/12/2011
Mon 06/13/2011
Tue 06/14/2011
Wed 06/15/2011
Thu 06/16/2011
Fri 06/17/2011
Sat 06/18/2011
Sun 06/19/2011
Mon 06/20/2011
Wed 06/22/2011
Thu 06/23/2011
Fri 06/24/2011
Sat 06/25/2011
Sun 06/26/2011
Mon 06/27/2011
Tue 06/28/2011
Wed 06/29/2011
Thu 06/30/2011
Fri 07/01/2011
Sat 07/02/2011
Sun 07/03/2011
Mon 07/04/2011
Tue 07/05/2011
Wed 07/06/2011
Thu 07/07/2011
Fri 07/08/2011
Sat 07/09/2011
Sun 07/10/2011
Mon 07/11/2011
Tue 07/12/2011
Wed 07/13/2011
Thu 07/14/2011
Fri 07/15/2011
Sat 07/16/2011
Sun 07/17/2011
Mon 07/18/2011
Tue 07/19/2011
Wed 07/20/2011
Thu 07/21/2011
Fri 07/22/2011
Sat 07/23/2011
Sun 07/24/2011
Mon 07/25/2011
Tue 07/26/2011
Wed 07/27/2011
Thu 07/28/2011
Fri 07/29/2011
Sat 07/30/2011
Sun 07/31/2011
Mon 08/01/2011
Tue 08/02/2011
Wed 08/03/2011
Thu 08/04/2011
Fri 08/05/2011
Sat 08/06/2011
Sun 08/07/2011
Mon 08/08/2011
Tue 08/09/2011
Wed 08/10/2011
Thu 08/11/2011
Fri 08/12/2011
Sat 08/13/2011
Sun 08/14/2011
Mon 08/15/2011
Tue 08/16/2011
Wed 08/17/2011
Thu 08/18/2011
Fri 08/19/2011
Sat 08/20/2011
Sun 08/21/2011
Mon 08/22/2011
Tue 08/23/2011
Wed 08/24/2011
Thu 08/25/2011
Fri 08/26/2011
Sat 08/27/2011
Mon 08/29/2011
Tue 08/30/2011
Wed 08/31/2011
Thu 09/01/2011
Fri 09/02/2011
Sat 09/03/2011
Sun 09/04/2011
Mon 09/05/2011
Tue 09/06/2011
Wed 09/07/2011
Thu 09/08/2011
Fri 09/09/2011
Sat 09/10/2011
Sun 09/11/2011
Mon 09/12/2011
Tue 09/13/2011
Wed 09/14/2011
Thu 09/15/2011
Fri 09/16/2011
Sat 09/17/2011
Sun 09/18/2011
Mon 09/19/2011
Tue 09/20/2011
Wed 09/21/2011
Thu 09/22/2011
Fri 09/23/2011
Sat 09/24/2011
Sun 09/25/2011
Mon 09/26/2011
Tue 09/27/2011
Wed 09/28/2011
Thu 09/29/2011
Fri 09/30/2011
Sat 10/01/2011
Sun 10/02/2011
Mon 10/03/2011
Tue 10/04/2011
Wed 10/05/2011
Mon 10/10/2011
Tue 10/11/2011
Wed 10/12/2011
Thu 10/13/2011
Fri 10/14/2011
Sat 10/15/2011
Sun 10/16/2011
Mon 10/17/2011
Tue 10/18/2011
Wed 10/19/2011
Thu 10/20/2011
Fri 10/21/2011
Sat 10/22/2011
Sun 10/23/2011
Mon 10/24/2011
Tue 10/25/2011
Wed 10/26/2011
Thu 10/27/2011
Fri 10/28/2011
Sun 10/30/2011
Mon 10/31/2011
Tue 11/01/2011
Wed 11/02/2011
Thu 11/03/2011
Fri 11/04/2011
Sat 11/05/2011
Sun 11/06/2011
Mon 11/07/2011
Tue 11/08/2011
Wed 11/09/2011
Thu 11/10/2011
Fri 11/11/2011
Sat 11/12/2011
Sun 11/13/2011
Mon 11/14/2011
Tue 11/15/2011
Wed 11/16/2011
Thu 11/17/2011
Fri 11/18/2011
Sat 11/19/2011
Sun 11/20/2011
Mon 11/21/2011
Tue 11/22/2011
Wed 11/23/2011
Thu 11/24/2011
Fri 11/25/2011
Sat 11/26/2011
Sun 11/27/2011
Mon 11/28/2011
Tue 11/29/2011
Wed 11/30/2011
Thu 12/01/2011
Fri 12/02/2011
Sat 12/03/2011
Sun 12/04/2011
Mon 12/05/2011
Tue 12/06/2011
Wed 12/07/2011
Thu 12/08/2011
Fri 12/09/2011
Sat 12/10/2011
Sun 12/11/2011
Mon 12/12/2011
Tue 12/13/2011
Wed 12/14/2011
Thu 12/15/2011
Fri 12/16/2011
Sat 12/17/2011
Sun 12/18/2011
Mon 12/19/2011
Tue 12/20/2011
Wed 12/21/2011
Thu 12/22/2011
Fri 12/23/2011
Sat 12/24/2011
Sun 12/25/2011
Mon 12/26/2011
Tue 12/27/2011
Wed 12/28/2011
Thu 12/29/2011
Fri 12/30/2011
Sat 12/31/2011
Sun 01/01/2012
Mon 01/02/2012
Tue 01/03/2012
Wed 01/04/2012
Thu 01/05/2012
Fri 01/06/2012
Sat 01/07/2012
Sun 01/08/2012
Mon 01/09/2012
Tue 01/10/2012
Wed 01/11/2012
Thu 01/12/2012
Fri 01/13/2012
Sat 01/14/2012
Sun 01/15/2012
Mon 01/16/2012
Tue 01/17/2012
Wed 01/18/2012
Thu 01/19/2012
Fri 01/20/2012
Sat 01/21/2012
Sun 01/22/2012
Mon 01/23/2012
Tue 01/24/2012
Wed 01/25/2012
Thu 01/26/2012
Sat 01/28/2012
Sun 01/29/2012
Mon 01/30/2012
Tue 01/31/2012
Wed 02/01/2012
Thu 02/02/2012
Fri 02/03/2012
Sat 02/04/2012
Sun 02/05/2012
Wed 02/08/2012
Thu 02/09/2012
Fri 02/10/2012
Sat 02/11/2012
Sun 02/12/2012
Mon 02/13/2012
Tue 02/14/2012
Wed 02/15/2012
Thu 02/16/2012
Fri 02/17/2012
Sun 02/19/2012
Mon 02/20/2012
Tue 02/21/2012
Wed 02/22/2012
Thu 02/23/2012
Fri 02/24/2012
Sat 02/25/2012
Sun 02/26/2012
Mon 02/27/2012
Tue 02/28/2012
Wed 02/29/2012
Thu 03/01/2012
Fri 03/02/2012
Sat 03/03/2012
Sun 03/04/2012
Mon 03/05/2012
Tue 03/06/2012
Wed 03/07/2012
Thu 03/08/2012
Fri 03/09/2012
Sun 03/11/2012
Mon 03/12/2012
Tue 03/13/2012
Wed 03/14/2012
Thu 03/15/2012
Fri 03/16/2012
Sat 03/17/2012
Sun 03/18/2012
Mon 03/19/2012
Tue 03/20/2012
Wed 03/21/2012
Thu 03/22/2012
Fri 03/23/2012
Sat 03/24/2012
Sun 03/25/2012
Mon 03/26/2012
Tue 03/27/2012
Wed 03/28/2012
Thu 03/29/2012
Fri 03/30/2012
Sat 03/31/2012
Sun 04/01/2012
Mon 04/02/2012
Tue 04/03/2012
Wed 04/04/2012
Thu 04/05/2012
Fri 04/06/2012
Sat 04/07/2012
Sun 04/08/2012
Mon 04/09/2012
Tue 04/10/2012
Wed 04/11/2012
Thu 04/12/2012
Fri 04/13/2012
Sat 04/14/2012
Sun 04/15/2012
Mon 04/16/2012
Wed 04/18/2012
Thu 04/19/2012
Fri 04/20/2012
Sat 04/21/2012
Sun 04/22/2012
Mon 04/23/2012
Tue 04/24/2012
Wed 04/25/2012
Thu 04/26/2012
Fri 04/27/2012
Sat 04/28/2012
Sun 04/29/2012
Mon 04/30/2012
Tue 05/01/2012
Wed 05/02/2012
Thu 05/03/2012
Fri 05/04/2012
Sat 05/05/2012
Sun 05/06/2012
Mon 05/07/2012
Wed 05/09/2012
Thu 05/10/2012
Fri 05/11/2012
Sat 05/12/2012
Sun 05/13/2012
Mon 05/14/2012
Tue 05/15/2012
Wed 05/16/2012
Thu 05/17/2012
Fri 05/18/2012
Sat 05/19/2012
Sun 05/20/2012
Mon 05/21/2012
Tue 05/22/2012
Wed 05/23/2012
Thu 05/24/2012
Fri 05/25/2012
Sat 05/26/2012
Sun 05/27/2012
Mon 05/28/2012
Tue 05/29/2012
Thu 05/31/2012
Fri 06/01/2012
Sat 06/02/2012
Sun 06/03/2012
Mon 06/04/2012
Tue 06/05/2012
Wed 06/06/2012
Thu 06/07/2012
Fri 06/08/2012
Sat 06/09/2012
Sun 06/10/2012
Mon 06/11/2012
Tue 06/12/2012
Wed 06/13/2012
Thu 06/14/2012
Fri 06/15/2012
Sat 06/16/2012
Sun 06/17/2012
Mon 06/18/2012
Thu 06/21/2012
Fri 06/22/2012
Sat 06/23/2012
Sun 06/24/2012
Mon 06/25/2012
Tue 06/26/2012
Wed 06/27/2012
Thu 06/28/2012
Fri 06/29/2012
Sun 07/01/2012
Mon 07/02/2012
Tue 07/03/2012
Wed 07/04/2012
Thu 07/05/2012
Fri 07/06/2012
Sat 07/07/2012
Sun 07/08/2012
Mon 07/09/2012
Tue 07/10/2012
Wed 07/11/2012
Thu 07/12/2012
Fri 07/13/2012
Sat 07/14/2012
Sun 07/15/2012
Mon 07/16/2012
Tue 07/17/2012
Wed 07/18/2012
Thu 07/19/2012
Fri 07/20/2012
Sat 07/21/2012
Sun 07/22/2012
Mon 07/23/2012
Tue 07/24/2012
Wed 07/25/2012
Thu 07/26/2012
Fri 07/27/2012
Sat 07/28/2012
Sun 07/29/2012
Mon 07/30/2012
Tue 07/31/2012
Wed 08/01/2012
Thu 08/02/2012
Sat 08/04/2012
Sun 08/05/2012
Mon 08/06/2012
Tue 08/07/2012
Wed 08/08/2012
Thu 08/09/2012
Fri 08/10/2012
Sat 08/11/2012
Mon 08/13/2012
Tue 08/14/2012
Wed 08/15/2012
Thu 08/16/2012
Fri 08/17/2012
Sat 08/18/2012
Sun 08/19/2012
Mon 08/20/2012
Tue 08/21/2012
Wed 08/22/2012
Thu 08/23/2012
Fri 08/24/2012
Sat 08/25/2012
Sun 08/26/2012
Mon 08/27/2012
Tue 08/28/2012
Wed 08/29/2012
Thu 08/30/2012
Fri 08/31/2012
Sat 09/01/2012
Sun 09/02/2012
Mon 09/03/2012
Tue 09/04/2012
Thu 09/06/2012
Fri 09/07/2012
Sat 09/08/2012
Sun 09/09/2012
Mon 09/10/2012
Tue 09/11/2012
Wed 09/12/2012
Thu 09/13/2012
Fri 09/14/2012
Sat 09/15/2012
Sun 09/16/2012
Mon 09/17/2012
Tue 09/18/2012
Wed 09/19/2012
Thu 09/20/2012
Fri 09/21/2012
Sat 09/22/2012
Sun 09/23/2012
Mon 09/24/2012
Tue 09/25/2012
Wed 09/26/2012
Thu 09/27/2012
Fri 09/28/2012
Sat 09/29/2012
Sun 09/30/2012
Mon 10/01/2012
Tue 10/02/2012
Wed 10/03/2012
Thu 10/04/2012
Fri 10/05/2012
Sat 10/06/2012
Sun 10/07/2012
Mon 10/08/2012
Tue 10/09/2012
Wed 10/10/2012
Thu 10/11/2012
Mon 10/15/2012
Tue 10/16/2012
Wed 10/17/2012
Thu 10/18/2012
Fri 10/19/2012
Sat 10/20/2012
Sun 10/21/2012
Mon 10/22/2012
Tue 10/23/2012
Wed 10/24/2012
Thu 10/25/2012
Fri 10/26/2012
Sat 10/27/2012
Sun 10/28/2012
Wed 10/31/2012
Thu 11/01/2012
Fri 11/02/2012
Sat 11/03/2012
Sun 11/04/2012
Mon 11/05/2012
Tue 11/06/2012
Wed 11/07/2012
Thu 11/08/2012
Fri 11/09/2012
Sat 11/10/2012
Sun 11/11/2012
Mon 11/12/2012
Tue 11/13/2012
Wed 11/14/2012
Thu 11/15/2012
Fri 11/16/2012
Sat 11/17/2012
Sun 11/18/2012
Mon 11/19/2012
Tue 11/20/2012
Wed 11/21/2012
Thu 11/22/2012
Fri 11/23/2012
Sat 11/24/2012
Sun 11/25/2012
Mon 11/26/2012
Tue 11/27/2012
Wed 11/28/2012
Thu 11/29/2012
Fri 11/30/2012
Sat 12/01/2012
Sun 12/02/2012
Mon 12/03/2012
Tue 12/04/2012
Tue 12/04/2012
Thu 12/06/2012
Thu 12/06/2012
Fri 12/07/2012
Sat 12/08/2012
Sun 12/09/2012
Mon 12/10/2012
Tue 12/11/2012
Wed 12/12/2012
Fri 12/14/2012
Sat 12/15/2012
Sun 12/16/2012
Mon 12/17/2012
Tue 12/18/2012
Wed 12/19/2012
Thu 12/20/2012
Fri 12/21/2012
Sat 12/22/2012
Sun 12/23/2012
Mon 12/24/2012
Tue 12/25/2012
Wed 12/26/2012
Thu 12/27/2012
Fri 12/28/2012
Sat 12/29/2012
Tue 01/01/2013
Thu 01/03/2013
Fri 01/04/2013
Sat 01/05/2013
Sun 01/06/2013
Mon 01/07/2013
Tue 01/08/2013
Wed 01/09/2013
Thu 01/10/2013
Fri 01/11/2013
Sat 01/12/2013
Sun 01/13/2013
Mon 01/14/2013
Tue 01/15/2013
Thu 01/17/2013
Thu 01/17/2013
Fri 01/18/2013
Sat 01/19/2013
Sun 01/20/2013
Mon 01/21/2013
Mon 01/21/2013
Tue 01/22/2013
Wed 01/23/2013
Thu 01/24/2013
Fri 01/25/2013
Sat 01/26/2013
Sun 01/27/2013
Mon 01/28/2013
Tue 01/29/2013
Wed 01/30/2013
Thu 01/31/2013
Sat 02/02/2013
Sun 02/03/2013
Mon 02/04/2013
Tue 02/05/2013
Wed 02/06/2013
Thu 02/07/2013
Fri 02/08/2013
Sat 02/09/2013
Sun 02/10/2013
Mon 02/11/2013
Tue 02/12/2013
Tue 02/12/2013
Wed 02/13/2013
Thu 02/14/2013
Fri 02/15/2013
Sat 02/16/2013
Sun 02/17/2013
Mon 02/18/2013
Tue 02/19/2013
Wed 02/20/2013
Thu 02/21/2013
Fri 02/22/2013
Sat 02/23/2013
Sun 02/24/2013
Mon 02/25/2013
Tue 02/26/2013
Wed 02/27/2013
Thu 02/28/2013
Fri 03/01/2013
Sat 03/02/2013
Sun 03/03/2013
Mon 03/04/2013
Tue 03/05/2013
Wed 03/06/2013
Thu 03/07/2013
Fri 03/08/2013
Sat 03/09/2013
Sun 03/10/2013
Mon 03/11/2013
Tue 03/12/2013
Wed 03/13/2013
Thu 03/14/2013
Fri 03/15/2013
Sat 03/16/2013
Sun 03/17/2013
Mon 03/18/2013
Tue 03/19/2013
Wed 03/20/2013
Thu 03/21/2013
Fri 03/22/2013
Sat 03/23/2013
Sun 03/24/2013
Mon 03/25/2013
Tue 03/26/2013
Wed 03/27/2013
Thu 03/28/2013
Sat 03/30/2013
Sun 03/31/2013
Mon 04/01/2013
Tue 04/02/2013
Wed 04/03/2013
Thu 04/04/2013
Fri 04/05/2013
Sat 04/06/2013
Sun 04/07/2013
Mon 04/08/2013
Tue 04/09/2013
Wed 04/10/2013
Thu 04/11/2013
Fri 04/12/2013
Sat 04/13/2013
Sun 04/14/2013
Mon 04/15/2013
Tue 04/16/2013
Wed 04/17/2013
Thu 04/18/2013
Fri 04/19/2013
Sat 04/20/2013
Sun 04/21/2013
Mon 04/22/2013
Tue 04/23/2013
Wed 04/24/2013
Thu 04/25/2013
Fri 04/26/2013
Sat 04/27/2013
Sun 04/28/2013
Mon 04/29/2013
Tue 04/30/2013
Wed 05/01/2013
Thu 05/02/2013
Fri 05/03/2013
Sat 05/04/2013
Sun 05/05/2013
Mon 05/06/2013
Tue 05/07/2013
Wed 05/08/2013
Thu 05/09/2013
Fri 05/10/2013
Sat 05/11/2013
Sun 05/12/2013
Mon 05/13/2013
Tue 05/14/2013
Wed 05/15/2013
Thu 05/16/2013
Fri 05/17/2013
Sat 05/18/2013
Sun 05/19/2013
Mon 05/20/2013
Tue 05/21/2013
Wed 05/22/2013
[last updated: 2013-05-23 02:21:08.098383; send questions to Mike McCandless]