Distributed Processing (Aspire 2)

From wiki.searchtechnologies.com
Jump to: navigation, search

For Information on Aspire 3.1 Click Here

What is Aspire Distributed Processing?

This feature allows jobs to be sent to remote Aspire Distributions (called remote nodes). This can increase performance when there are high resource consuming pipelines, by load balancing the work across all available remote nodes.

Communications between remote nodes will be tightly coupled, meaning that they must be on the same intranet, geographically on the same place and (recommended) with no firewalls and other security mechanisms between nodes on the cluster.

Discovery Managers

A Discovery Manager is a component that handles different methods of discovery for remote nodes. There are three different Discovery Managers: Default, Amazon EC2 and ZooKeeper.

Default Discovery Manager

This Discovery Manager has the basic functionality for remote nodes and resource discovery.

Example configuration:

<discoveryManager type="default">

</discoveryManager>

Discovery Methods

One of the following discovery methods must be configured inside the <discoveryManager type="default"> tag.

Static Discovery

Useful for debugging or in well known cluster setups (with static IP addresses configurations). Reads a list of remote nodes and registers them for remote branching.

Each node is identified by its IP Address and its distributed communications port.

Available options are:

Element Type Default Description
checkTimeout long 15000 (optional) Time in milliseconds to wait between each ping to the configured remote node. After a few retries, if a node is unavailable, it will be blacklisted (remote branching to that node will not be possible).
remoteNodes none Static list of remote nodes to register.

Example configuration:

<discoveryManager type="default">
	<discovery type="static">
		<checkTimeout>45000</checkTimeout>
		<remoteNodes>
			<remoteNode portNumber="51510">10.10.30.122</remoteNode>
			<remoteNode portNumber="51515">10.10.20.139</remoteNode>
		</remoteNodes>
	</discovery>
</discoveryManager>

You must specify each remote node IP Address and port as shown above. Notice that the port number corresponds to the distributed communications port, not the Aspire ordinary HTTP port.

Broadcast Discovery

Intercepts discovery messages sent by other nodes. If the incoming message is from a new node, it is registered. Otherwise, that node information is updated.

You must enable this discovery method if you want a node to broadcast information about itself.

Available options are:

Element Type Default Description
broadcastPort int none (required) Port used to listen for other nodes messages and to broadcast information about the current node.
multicastAddressGroup none Multicast address group used to listen and send broadcast messages. This group must be the same on all nodes on the same cluster.

Example configuration:

<discoveryManager type="default">
	<discovery type="broadcast">
		<broadcastPort>10324</broadcastPort>
		<multicastAddressGroup>230.0.0.1</multicastAddressGroup>
	</discovery>
</discoveryManager>

Complete example


<distributedCommunications enabled="true">
	<checkpointJobRequests>true</checkpointJobRequests>
	<connectionIdleTimeout>120000</connectionIdleTimeout>
	<port>51510</port>
	<pollTimeout>100</pollTimeout>
	<tcp>
		<keepAlive>false</keepAlive>
		<trafficClass>2</trafficClass>
		<reuseAddress>false</reuseAddress>
		<readTimeout>10000</readTimeout>
		<tcpNoDelay>false</tcpNoDelay>
	</tcp>
	<discoveryManager type="default">
		<discovery type="static">
			<checkTimeout>45000</checkTimeout>
			<remoteNodes>
				<remoteNode portNumber="51515">192.168.0.122</remoteNode>
				<remoteNode portNumber="51515">10.10.20.139</remoteNode>
			</remoteNodes>
		</discovery>
	
		<discovery type="broadcast">
			<broadcastPort>10324</broadcastPort>
			<multicastAddressGroup>230.0.0.1</multicastAddressGroup>
		</discovery>
	</discoveryManager>
</distributedCommunications>

Amazon EC2 Discovery Manager

Since you cannot use the broadcast discovery method at Amazon Elastic Compute Cloud because of network restrictions, you can use the Amazon EC2 Discovery Manager for dynamic discovering of remote nodes.

Element Type Default Description
implementation string none (required) Maven coordinates of the jar file that contains the implementation for the Discovery Manager.

Example configuration:

<discoveryManager type="amazonec2">
      <implementation>com.searchtechnologies.aspire:aspire-amazonec2-dm</implementation>

</discoveryManager>

Amazon EC2 Discovery Method

Element Type Default Description
accessKey string none (required) Encrypted access key provided by Amazon for your AWS Account.
secretKey string none (required) Encrypted secret key provided by Amazon for your AWS Account.
usePublicIP boolean false (Optional) Specifies whether or not the remote nodes should be accessed by its public IP Address.
securityGroup string none (Optional) Specifies the name of the security group of the ec2 instances that should be discovered.
pollFrequency int 1000 (Optional) Specifies the frequency for polling the information from Amazon EC2.

Example configuration:

<discoveryManager type="amazonec2">
	<implementation>com.searchtechnologies.aspire:aspire-amazonec2-dm</implementation>

	<discovery type="amazonec2">
		<accessKey>encrypted:ENCRYPTED_ACCESS_KEY</accessKey>
		<secretKey>encrypted:ENCRYPTED_SECRET_KEY</secretKey>
	</discovery>

</discoveryManager>

Advanced configuration:

<discoveryManager type="amazonec2">
	<implementation>com.searchtechnologies.aspire:aspire-amazonec2-dm</implementation>

	<discovery type="amazonec2">
		<accessKey>encrypted:ENCRYPTED_ACCESS_KEY</accessKey>
		<secretKey>encrypted:ENCRYPTED_SECRET_KEY</secretKey>
		<usePublicIP>false</usePublicIP>
		<securityGroup>MySecurityGroup</securityGroup>
		<pollFrequency>1000</pollFrequency>
	</discovery>

</discoveryManager>

Zookeeper Discovery Manager

This discovery manager uses zookeeper as a centralized site for discovering remote nodes and their resources as well. Click here for details about Zookeeper installation and configuration.

Element Type Default Description
implementation string none (required) Maven coordinates of the jar file that contains the implementations for the Discovery Manager.
zookeeperConnection string none (required) Comma separated list of zookeeper servers.
zookeeperPath string /aspire/nodes (Optional) Znode root path where the remote nodes are registered.
zookeeperTimeout int 3000 (Optional) Timeout (milliseconds) for the zookeeper server connection. After this time the connection will be considered as disconnected until the zookeeper server comes back online or connection is established to another server.
resourceUpdateTime int 2000 (Optional) Frequency (milliseconds) for checking for new local resources to register in zookeeper.

Example configuration:

<discoveryManager type="zookeeper">
	<implementation>com.searchtechnologies.aspire:aspire-zk-dm</implementation>

	<zookeeperConnection>127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2181</zookeeperConnection>

</discoveryManager>

Advanced configuration:

<discoveryManager type="zookeeper">
	<implementation>com.searchtechnologies.aspire:aspire-zk-dm</implementation>

	<zookeeperConnection>127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2181</zookeeperConnection>
	<zookeeperPath>/aspire/nodes</zookeeperPath>
	<zookeeperTimeout>3000</zookeeperTimeout>
	<resourceUpdateTime>2000</resourceUpdateTime>

</discoveryManager>

Known Issues

  • In Aspire 2.X, the ZooKeeper Discovery Manager does not work in any version previous to  (2.0.3 Release)  .
  • The port settings field is required to be specified in any 2.0.X including 2.0.