org.apache.hadoop.chukwa.datacollection.sender
Class ChukwaHttpSender

java.lang.Object
  extended by org.apache.hadoop.chukwa.datacollection.sender.ChukwaHttpSender
All Implemented Interfaces:
ChukwaSender
Direct Known Subclasses:
AsyncAckSender

public class ChukwaHttpSender
extends Object
implements ChukwaSender

Encapsulates all of the http setup and connection details needed for chunks to be delivered to a collector. This class should encapsulate the details of the low level data formatting. The Connector is responsible for picking what to send and to whom; retry policy is encoded in the collectors iterator. This class is not thread safe. Synchronization is the caller's responsibility.

On error, tries the list of available collectors, pauses for a minute, and then repeats.

Will wait forever for collectors to come up.


Nested Class Summary
static class ChukwaHttpSender.CommitListEntry
           
 
Field Summary
static String COLLECTOR_TIMEOUT_OPT
           
protected  Iterator<String> collectors
           
 
Constructor Summary
ChukwaHttpSender(org.apache.hadoop.conf.Configuration c)
           
 
Method Summary
protected  List<String> doRequest(org.apache.commons.httpclient.HttpMethodBase method, String dest)
          Responsible for performing a single operation to a specified collector URL.
protected  boolean failedCollector(String downCollector)
          A hook for taking action when a collector is declared failed.
 List<ChukwaHttpSender.CommitListEntry> postAndParseResponse(org.apache.commons.httpclient.methods.PostMethod method, List<ChukwaHttpSender.CommitListEntry> expectedCommitResults)
           
protected  List<String> reliablySend(org.apache.commons.httpclient.HttpMethodBase method, String pathSuffix)
          Responsible for executing the supplied method on at least one collector
 List<ChukwaHttpSender.CommitListEntry> send(List<Chunk> toSend)
          grab all of the chunks currently in the chunkQueue, stores a copy of them locally, calculates their size, sets them up
 void setCollectors(Iterator<String> collectors)
          Set up a list of connectors for this client to send Chunks to
 void stop()
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

COLLECTOR_TIMEOUT_OPT

public static final String COLLECTOR_TIMEOUT_OPT
See Also:
Constant Field Values

collectors

protected Iterator<String> collectors
Constructor Detail

ChukwaHttpSender

public ChukwaHttpSender(org.apache.hadoop.conf.Configuration c)
Method Detail

setCollectors

public void setCollectors(Iterator<String> collectors)
Set up a list of connectors for this client to send Chunks to

Specified by:
setCollectors in interface ChukwaSender
Parameters:
collectors -

send

public List<ChukwaHttpSender.CommitListEntry> send(List<Chunk> toSend)
                                            throws InterruptedException,
                                                   IOException
grab all of the chunks currently in the chunkQueue, stores a copy of them locally, calculates their size, sets them up

Specified by:
send in interface ChukwaSender
Parameters:
toSend - a list of chunks to commit
Returns:
array of chunk id's which were ACKed by collector
Throws:
InterruptedException - if interrupted while trying to send
IOException

postAndParseResponse

public List<ChukwaHttpSender.CommitListEntry> postAndParseResponse(org.apache.commons.httpclient.methods.PostMethod method,
                                                                   List<ChukwaHttpSender.CommitListEntry> expectedCommitResults)
                                                            throws IOException,
                                                                   InterruptedException
Parameters:
method - the data to push
expectedCommitResults - the list
Returns:
the list of committed chunks
Throws:
IOException
InterruptedException

reliablySend

protected List<String> reliablySend(org.apache.commons.httpclient.HttpMethodBase method,
                                    String pathSuffix)
                             throws InterruptedException,
                                    IOException
Responsible for executing the supplied method on at least one collector

Parameters:
method -
Returns:
Throws:
InterruptedException
IOException - if no collector responds with an OK

failedCollector

protected boolean failedCollector(String downCollector)
A hook for taking action when a collector is declared failed. Returns whether to retry current post, or junk it

Parameters:
downCollector -

doRequest

protected List<String> doRequest(org.apache.commons.httpclient.HttpMethodBase method,
                                 String dest)
                          throws IOException,
                                 org.apache.commons.httpclient.HttpException
Responsible for performing a single operation to a specified collector URL.

Parameters:
dest - the URL being requested. (Including hostname)
Throws:
IOException
org.apache.commons.httpclient.HttpException

stop

public void stop()
Specified by:
stop in interface ChukwaSender


Copyright © 2011 Apache Software Foundation. All Rights Reserved.