A (2011-04-25): Switched from a traditional spinning-magnets hard drive (Western Digital Caviar Green, 1TB) to a 240 GB OCZ Vertex III SSD; this change gave a small increase in indexing rate, drastically reduced variance on the NRT reopen time (NRT is IO intensive), and didn't affect query performance (which is expected since the postings are small enough to fit into the OS's IO cache.
B (2011-05-02): Concurrent flushing, a major improvement to Lucene, was committed. Before this change, flushing a segment in IndexWriter was single-threaded and blocked all other indexing threads; after this change, each indexing thread flushes its own segment without blocking indexing of other threads. On highly concurrent hardware (the machine running these tests has 24 cores) this can result in a tremendous increase in Lucene's indexing throughput. See this post for details.
Some queries did get slower, because the index now has more segments. Unfortunately, the index produced by concurrent flushing will vary, night to night, in how many segments it contains, so this is a further source of noise in the search results.
C (2011-05-06): Changed how I build the index used for searching, to only use one thread. This results in exactly the same index structure (same segments, same docs per segment) from night to night, to avoid the added noise from change B.
D (2011-05-07): Increased number of indexing threads from 6 to 20 and dropped the IndexWriter RAM buffer from 512 MB to 350 MB. See this post for details.
E (2011-05-11): Added TermQuery, sorting by date/time and title fields.
F (2011-05-14): Added TermQuery, grouping by fields with 100, 10K, 1M unique values.
G (2011-06-03): Added Term (bgroup) and Term (bgroup, 1pass) using the BlockGroupingCollector for grouping into 1M unique groups.
H (2011-06-26): Switched to MemoryCodec for the primary-key 'id' field so that lookups (either for PKLookup test or for deletions during reopen in the NRT test) are fast, with no IO. Also switched to NRTCachingDirectory for the NRT test, so that small new segments are written only in RAM.
I (2011-07-04): Switched from Java 1.6.0_21 to 1.6.0_26
J (2011-07-11): LUCENE-3233: fast SynonymFilter using an FST, including an optimization to the FST representation allowing array arcs even when some arcs have large outputs; this resulted in a good speedup for MemoryCodec, which also speeds up the primary key lookup performance.
K (2011-07-22): LUCENE-3328: If all clauses of a BooleanQuery are MUST and are TermQuery then create a specialized scorer for scoring this common case.
L (2011-07-30): Switched back to Java 1.6.0_21 from 1.6.0_26 because _26 would sometimes deadlock threads.
M (2011-08-20): LUCENE-3030: cutover to more efficient BlockTree terms dict.
N (2011-09-22): LUCENE-3215: more efficient scoring for sloppy PhraseQuery.
O (2011-11-30): LUCENE-3584: make postings bulk API codec-private
P (2011-12-07): Switched to Java 1.7.0_01
Q (2011-12-16): LUCENE-3648: JIT optimizations to Lucene40 DocsEnum
R (2012-01-30): LUCENE-2858: Split IndexReader in AtomicReader and CompositeReader
S (2012-03-18): LUCENE-3738: Be consistent about negative vInt/vLong
T (2012-05-06): LUCENE-4024: FuzzyQuery never does edit distance > 2
V (2012-05-25): LUCENE-4062: new aligned packed-bits implementations
W (2012-05-28): Disable Java's compressed OOPS (-XX:-UseCompressedOops), and LUCENE-4055: refactor SegmentInfos/FieldInfos
X (2012-06-02): Re-enable Java's compressed OOPS
Y (2012-06-06): Switched to Java 1.7.0_04
Z (2012-06-26): Fixed silly performance bug in PKLookupTask.java
AA (2012-10-06): Stopped overclocking the computer running benchmarks.
AB (2012-10-15): LUCENE-4446: switch to BlockPostingsFormat
AC (2012-12-10): LUCENE-4598: small optimizations to facet aggregation
AD (2013-01-11): LUCENE-4620: IntEncoder/Decoder bulk API
AE (2013-01-17): Facet performance improvements: LUCENE-4686, LUCENE-4620, LUCENE-4602
AF (2013-01-21): Facet performance improvements: LUCENE-4600
AG (2013-01-24): Switched to NO_PARENTS faceting
AI (2013-02-12): LUCENE-4764: new Facet42DocValuesFormat for faster but more RAM-consuming DocValues
AJ (2013-02-22): LUCENE-4791: optimize ConjunctionTermScorer to use skipping on first term
AK (2013-03-14): LUCENE-4607: add DISI/Spans.cost
AL (2013-05-03): LUCENE-4946: SorterTemplate
AM (2013-06-20): LUCENE-5063: compress int and long FieldCache entries
AN (2013-07-31): LUCENE-5140: recover slowdown in span queries and exact phrase query
AO (2013-09-10): Switched to Java 1.7.0_40
AP (2013-11-09): Switched to DirectDocValuesFormat for the Date facets field.
AQ (2014-02-06): LUCENE-5425: performance improvement for FixedBitSet.iterator
AR (2014-03-11): LUCENE-5487: add BulkScorer
AS (2014-04-05): LUCENE-527: LeafCollector (made CachingCollector slower)
AT (2014-04-25): Upgraded to Ubuntu 14.04 LTS (kernel 3.13.0-32-generic #57)
AU (2014-06-10): Switched from DirectDVFormat to Lucene's default for Date facet field
AV (2014-07-25): Disabled transparent huge pages
AW (2014-08-30): Re-enabled transparent huge pages
AX (2014-11-01): LUCENE-6030: norms compression
AY (2014-11-22): Upgrade from java 1.7.0_55-b13 to java 1.8.0_20-ea-b05
AZ (2015-01-15): LUCENE-6179: remove out-of-order scoring
BA (2015-01-19): LUCENE-6184: BooleanScorer better deals with sparse clauses
BB (2015-02-13): LUCENE-6198: Two phase intersection (approximations are not needed by any query in this benchmark, but the change refactored ConjunctionScorer a bit)
BC (2015-03-02): LUCENE-6320: Speed up CheckIndex
BD (2015-03-06): Upgrade JDK from 1.8.0_25-b17 to 1.8.0_40-b25
BE (2015-04-02): LUCENE-6308: span queries support two-phased iteration
BF (2015-04-04): LUCENE-5879: add auto-prefix terms
BG (2015-06-24): LUCENE-6548: some optimizations to block tree intersect
BH (2015-09-15): LUCENE-6789 switch to BM25 scoring by default
BI (2015-10-05): Randomize what time of day benchmark runs
BJ (2015-12-02): Upgrade to beast2 (72 cores, 256 GB RAM)
BN (2016-05-25): Fix another benchmark bottleneck for 1 KB docs
BO (2016-06-13): LUCENE-7330: Speed up conjunctions
BP (2016-07-16): Upgrade beast2 kernel from 4.4.x to 4.6.x