Aggregator

This applies for Camel version 2.3 or newer. If you use an older version then use this Aggregator link instead.

The Aggregator from the EIP patterns allows you to combine a number of messages together into a single message.

A correlation Expression is used to determine the messages which should be aggregated together. If you want to aggregate all messages into a single message, just use a constant expression. An AggregationStrategy is used to combine all the message exchanges for a single correlation key into a single message exchange.

Aggregator options

The aggregator supports the following options:

Option Default Description
correlationExpression   Mandatory Expression which evaluates the correlation key to use for aggregation. The Exchange which has the same correlation key is aggregated together. If the correlation key could not be evaluated an Exception is thrown. You can disable this by using the ignoreBadCorrelationKeys option.
aggregationStrategy   Mandatory AggregationStrategy which is used to merge the incoming Exchange with the existing already merged exchanges. At first call the oldExchange parameter is null. On subsequent invocations the oldExchange contains the merged exchanges and newExchange is of course the new incoming Exchange.
strategyRef   A reference to lookup the AggregationStrategy in the Registry.
completionSize   Number of messages aggregated before the aggregation is complete. This option can be set as either a fixed value or using an Expression which allows you to evaluate a size dynamically - will use Integer as result. If both are set Camel will fallback to use the fixed value if the Expression result was null or 0.
completionTimeout   Time in millis that an aggregated exchange should be inactive before its complete. This option can be set as either a fixed value or using an Expression which allows you to evaluate a timeout dynamically - will use Long as result. If both are set Camel will fallback to use the fixed value if the Expression result was null or 0. You cannot use this option together with completionInterval, only one of the two can be used.
completionInterval   A repeating period in millis by which the aggregator will complete all current aggregated exchanges. Camel has a background task which is triggered every period. You cannot use this option together with completionTimeout, only one of them can be used.
completionPredicate   A Predicate to indicate when an aggregated exchange is complete.
completionFromBatchConsumer false This option is if the exchanges are coming from a Batch Consumer. Then when enabled the Aggregator2 will use the batch size determined by the Batch Consumer in the message header CamelBatchSize. See more details at Batch Consumer. This can be used to aggregate all files consumed from a File endpoint in that given poll.
forceCompletionOnStop false Camel 2.9 Indicates to complete all current aggregated exchanges when the context is stopped
eagerCheckCompletion false Whether or not to eager check for completion when a new incoming Exchange has been received. This option influences the behavior of the completionPredicate option as the Exchange being passed in changes accordingly. When false the Exchange passed in the Predicate is the aggregated Exchange which means any information you may store on the aggregated Exchange from the AggregationStrategy is available for the Predicate. When true the Exchange passed in the Predicate is the incoming Exchange, which means you can access data from the incoming Exchange.
groupExchanges false If enabled then Camel will group all aggregated Exchanges into a single combined org.apache.camel.impl.GroupedExchange holder class that holds all the aggregated Exchanges. And as a result only one Exchange is being sent out from the aggregator. Can be used to combine many incoming Exchanges into a single output Exchange without coding a custom AggregationStrategy yourself. Important: This option does not support persistant repository with the aggregator.
ignoreInvalidCorrelationKeys false Whether or not to ignore correlation keys which could not be evaluated to a value. By default Camel will throw an Exception, but you can enable this option and ignore the situation instead.
closeCorrelationKeyOnCompletion   Whether or not too late Exchanges should be accepted or not. You can enable this to indicate that if a correlation key has already been completed, then any new exchanges with the same correlation key be denied. Camel will then throw a closedCorrelationKeyException exception. When using this option you pass in a integer which is a number for a LRUCache which keeps that last X number of closed correlation keys. You can pass in 0 or a negative value to indicate a unbounded cache. By passing in a number you are ensured that cache won't grow too big if you use a log of different correlation keys.
discardOnCompletionTimeout false Camel 2.5: Whether or not exchanges which complete due to a timeout should be discarded. If enabled then when a timeout occurs the aggregated message will not be sent out but dropped (discarded).
aggregationRepository   Allows you to plugin you own implementation of org.apache.camel.spi.AggregationRepository which keeps track of the current inflight aggregated exchanges. Camel uses by default a memory based implementation.
aggregationRepositoryRef   Reference to lookup a aggregationRepository in the Registry.
parallelProcessing false When aggregated are completed they are being send out of the aggregator. This option indicates whether or not Camel should use a thread pool with multiple threads for concurrency. If no custom thread pool has been specified then Camel creates a default pool with 10 concurrent threads.
executorService   If using parallelProcessing you can specify a custom thread pool to be used. In fact also if you are not using parallelProcessing this custom thread pool is used to send out aggregated exchanges as well.
executorServiceRef   Reference to lookup a executorService in the Registry
timeoutCheckerExecutorService   Camel 2.9: If using either of the completionTimeout, completionTimeoutExpression, or completionInterval options a background thread is created to check for the completion for every aggregator. Set this option to provide a custom thread pool to be used rather than creating a new thread for every aggregator.
timeoutCheckerExecutorServiceRef   Camel 2.9: Reference to lookup a timeoutCheckerExecutorService in the Registry

Exchange Properties

The following properties are set on each aggregated Exchange:

header type description
CamelAggregatedSize int The total number of Exchanges aggregated into this combined Exchange.
CamelAggregatedCompletedBy String Indicator how the aggregation was completed as a value of either: predicate, size, consumer, timeout or interval.

About AggregationStrategy

The AggregationStrategy is used for aggregating the old (lookup by its correlation id) and the new exchanges together into a single exchange. Possible implementations include performing some kind of combining or delta processing, such as adding line items together into an invoice or just using the newest exchange and removing old exchanges such as for state tracking or market data prices; where old values are of little use.

Notice the aggregation strategy is a mandatory option and must be provided to the aggregator.

Here are a few example AggregationStrategy implementations that should help you create your own custom strategy.

//simply combines Exchange String body values using '+' as a delimiter
class StringAggregationStrategy implements AggregationStrategy {

    public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
        if (oldExchange == null) {
            return newExchange;
        }

        String oldBody = oldExchange.getIn().getBody(String.class);
        String newBody = newExchange.getIn().getBody(String.class);
        oldExchange.getIn().setBody(oldBody + "+" + newBody);
        return oldExchange;
    }
}

//simply combines Exchange body values into an ArrayList<Object>
class ArrayListAggregationStrategy implements AggregationStrategy {

    public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
	Object newBody = newExchange.getIn().getBody();
	ArrayList<Object> list = null;
        if (oldExchange == null) {
		list = new ArrayList<Object>();
		list.add(newBody);
		newExchange.getIn().setBody(list);
		return newExchange;
        } else {
	        list = oldExchange.getIn().getBody(ArrayList.class);
		list.add(newBody);
		return oldExchange;
	}
    }
}

About completion

When aggregation Exchanges at some point you need to indicate that the aggregated exchanges is complete, so they can be send out of the aggregator. Camel allows you to indicate completion in various ways as follows:

  • completionTimeout - Is an inactivity timeout in which is triggered if no new exchanges have been aggregated for that particular correlation key within the period.
  • completionInterval - Once every X period all the current aggregated exchanges are completed.
  • completionSize - Is a number indicating that after X aggregated exchanges it's complete.
  • completionPredicate - Runs a Predicate when a new exchange is aggregated to determine if we are complete or not
  • completionFromBatchConsumer - Special option for Batch Consumer which allows you to complete when all the messages from the batch has been aggregated.
  • forceCompletionOnStop - Camel 2.9 Indicates to complete all current aggregated exchanges when the context is stopped

Notice that all the completion ways are per correlation key. And you can combine them in any way you like. It's basically the first which triggers that wins. So you can use a completion size together with a completion timeout. Only completionTimeout and completionInterval cannot be used at the same time.

Notice the completion is a mandatory option and must be provided to the aggregator. If not provided Camel will thrown an Exception on startup.

Persistent AggregationRepository

The aggregator provides a pluggable repository which you can implement your own org.apache.camel.spi.AggregationRepository.
If you need persistent repository then you can use either Camel HawtDB or SQL Component components.

Examples

See some examples from the old Aggregator which is somewhat similar to this new aggregator.

Setting options in Spring XML
Many of the options are configurable as attributes on the <aggregate> tag when using Spring XML.

Using completionTimeout

In this example we want to aggregate all incoming messages and after 3 seconds of inactivity we want the aggregation to complete. This is done using the completionTimeout option as shown:

from("direct:start")
    // aggregate all exchanges correlated by the id header.
    // Aggregate them using the BodyInAggregatingStrategy strategy which
    // and after 3 seconds of inactivity them timeout and complete the aggregation
    // and send it to mock:aggregated
    .aggregate(header("id"), new BodyInAggregatingStrategy()).completionTimeout(3000)
        .to("mock:aggregated");

And the same example using Spring XML:

<camelContext xmlns="http://camel.apache.org/schema/spring">
    <route>
        <from uri="direct:start"/>
        <aggregate strategyRef="aggregatorStrategy" completionTimeout="3000">
            <correlationExpression>
                <simple>header.id</simple>
            </correlationExpression>
            <to uri="mock:aggregated"/>
        </aggregate>
    </route>
</camelContext>

<bean id="aggregatorStrategy" class="org.apache.camel.processor.BodyInAggregatingStrategy"/>

Using completionSize

In this example we want to aggregate all incoming messages and when we have 3 messages aggregated (in the same correlation group) we want the aggregation to complete. This is done using the completionSize option as shown:

from("direct:start")
    // aggregate all exchanges correlated by the id header.
    // Aggregate them using the BodyInAggregatingStrategy strategy which
    // and after 3 messages has been aggregated then complete the aggregation
    // and send it to mock:aggregated
    .aggregate(header("id"), new BodyInAggregatingStrategy()).completionSize(3)
        .to("mock:aggregated");

And the same example using Spring XML:

<camelContext xmlns="http://camel.apache.org/schema/spring">
    <route>
        <from uri="direct:start"/>
        <aggregate strategyRef="aggregatorStrategy" completionSize="3">
            <correlationExpression>
                <simple>header.id</simple>
            </correlationExpression>
            <to uri="mock:aggregated"/>
        </aggregate>
    </route>
</camelContext>

<bean id="aggregatorStrategy" class="org.apache.camel.processor.BodyInAggregatingStrategy"/>

Using completionPredicate

In this example we want to aggregate all incoming messages and use a Predicate to determine when we are complete. The Predicate can be evaluated using either the aggregated exchange (default) or the incoming exchange. We will so both situations as examples. We start with the default situation as shown:

from("direct:start")
    // aggregate all exchanges correlated by the id header.
    // Aggregate them using the BodyInAggregatingStrategy strategy which
    // and when the aggregated body contains A+B+C then complete the aggregation
    // and send it to mock:aggregated
    .aggregate(header("id"), new BodyInAggregatingStrategy()).completionPredicate(body().contains("A+B+C"))
        .to("mock:aggregated");

And the same example using Spring XML:

<camelContext xmlns="http://camel.apache.org/schema/spring">
    <route>
        <from uri="direct:start"/>
        <aggregate strategyRef="aggregatorStrategy">
            <correlationExpression>
                <simple>header.id</simple>
            </correlationExpression>
            <completionPredicate>
                <simple>${body} contains 'A+B+C'</simple>
            </completionPredicate>
            <to uri="mock:aggregated"/>
        </aggregate>
    </route>
</camelContext>

<bean id="aggregatorStrategy" class="org.apache.camel.processor.BodyInAggregatingStrategy"/>

And the other situation where we use the eagerCheckCompletion option to tell Camel to use the incoming Exchange. Notice how we can just test in the completion predicate that the incoming message is the END message:

from("direct:start")
    // aggregate all exchanges correlated by the id header.
    // Aggregate them using the BodyInAggregatingStrategy strategy
    // do eager checking which means the completion predicate will use the incoming exchange
    // which allows us to trigger completion when a certain exchange arrived which is the
    // END message
    .aggregate(header("id"), new BodyInAggregatingStrategy())
        .eagerCheckCompletion().completionPredicate(body().isEqualTo("END"))
        .to("mock:aggregated");

And the same example using Spring XML:

<camelContext xmlns="http://camel.apache.org/schema/spring">
    <route>
        <from uri="direct:start"/>
        <aggregate strategyRef="aggregatorStrategy" eagerCheckCompletion="true">
            <correlationExpression>
                <simple>header.id</simple>
            </correlationExpression>
            <completionPredicate>
                <simple>${body} == 'END'</simple>
            </completionPredicate>
            <to uri="mock:aggregated"/>
        </aggregate>
    </route>
</camelContext>

<bean id="aggregatorStrategy" class="org.apache.camel.processor.BodyInAggregatingStrategy"/>

Using dynamic completionTimeout

In this example we want to aggregate all incoming messages and after a period of inactivity we want the aggregation to complete. The period should be computed at runtime based on the timeout header in the incoming messages. This is done using the completionTimeout option as shown:

from("direct:start")
    // aggregate all exchanges correlated by the id header.
    // Aggregate them using the BodyInAggregatingStrategy strategy which
    // and the timeout header contains the timeout in millis of inactivity them timeout and complete the aggregation
    // and send it to mock:aggregated
    .aggregate(header("id"), new BodyInAggregatingStrategy()).completionTimeout(header("timeout"))
        .to("mock:aggregated");

And the same example using Spring XML:

<camelContext xmlns="http://camel.apache.org/schema/spring">
    <route>
        <from uri="direct:start"/>
        <aggregate strategyRef="aggregatorStrategy">
            <correlationExpression>
                <simple>header.id</simple>
            </correlationExpression>
            <completionTimeout>
                <header>timeout</header>
            </completionTimeout>
            <to uri="mock:aggregated"/>
        </aggregate>
    </route>
</camelContext>

<bean id="aggregatorStrategy" class="org.apache.camel.processor.BodyInAggregatingStrategy"/>

Note: You can also add a fixed timeout value and Camel will fallback to use this value if the dynamic value was null or 0.

Using dynamic completionSize

In this example we want to aggregate all incoming messages based on a dynamic size per correlation key. The size is computed at runtime based on the mySize header in the incoming messages. This is done using the completionSize option as shown:

from("direct:start")
    // aggregate all exchanges correlated by the id header.
    // Aggregate them using the BodyInAggregatingStrategy strategy which
    // and the header mySize determines the number of aggregated messages should trigger the completion
    // and send it to mock:aggregated
    .aggregate(header("id"), new BodyInAggregatingStrategy()).completionSize(header("mySize"))
        .to("mock:aggregated");

And the same example using Spring XML:

<camelContext xmlns="http://camel.apache.org/schema/spring">
    <route>
        <from uri="direct:start"/>
        <aggregate strategyRef="aggregatorStrategy">
            <correlationExpression>
                <simple>header.id</simple>
            </correlationExpression>
            <completionSize>
                <header>mySize</header>
            </completionSize>
            <to uri="mock:aggregated"/>
        </aggregate>
    </route>
</camelContext>

<bean id="aggregatorStrategy" class="org.apache.camel.processor.BodyInAggregatingStrategy"/>

Note: You can also add a fixed size value and Camel will fallback to use this value if the dynamic value was null or 0.

Using This Pattern

If you would like to use this EIP Pattern then please read the Getting Started, you may also find the Architecture useful particularly the description of Endpoint and URIs. Then you could try out some of the Examples first before trying this pattern out.

Manually Force the Completion of All Aggregated Exchanges Immediately

Available as of Camel 2.9
You can manually complete all current aggregated exchanges by sending in a message containing the header Exchange.AGGREGATION_COMPLETE_ALL_GROUPS set to true. The message is considered a signal message only, the message headers/contents will not be processed otherwise.

See also

© 2004-2011 The Apache Software Foundation.
Apache Camel, Camel, Apache, the Apache feather logo, and the Apache Camel project logo are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.
Graphic Design By Hiram