org.apache.ignite.hadoop.mapreduce

Class IgniteHadoopWeightedMapReducePlanner

java.lang.Object

org.apache.ignite.hadoop.planner.HadoopAbstractMapReducePlanner

org.apache.ignite.hadoop.mapreduce.IgniteHadoopWeightedMapReducePlanner

All Implemented Interfaces:

HadoopMapReducePlanner

public class IgniteHadoopWeightedMapReducePlanner
extends HadoopAbstractMapReducePlanner

Map-reduce planner which assigns mappers and reducers based on their "weights". Weight describes how much resources are required to execute particular map or reduce task.

Plan creation consists of two steps: assigning mappers and assigning reducers.

Mappers are assigned based on input split data location. For each input split we search for nodes where its data is stored. Planner tries to assign mappers to their affinity nodes first. This process is governed by two properties:

• localMapperWeight - weight of a map task when it is executed on an affinity node;
• remoteMapperWeight - weight of a map task when it is executed on a non-affinity node.

Planning algorithm assign mappers so that total resulting weight on all nodes is minimum possible.

Reducers are assigned differently. First we try to distribute reducers across nodes with mappers. This approach could minimize expensive data transfer over network. Reducer assigned to a node with mapper is considered local. Otherwise it is considered remote. This process continue until certain weight threshold is reached what means that current node is already too busy and it should not have higher priority over other nodes any more. Threshold can be configured using preferLocalReducerThresholdWeight property.

When local reducer threshold is reached on all nodes, we distribute remaining reducers based on their local and remote weights in the same way as it is done for mappers. This process is governed by two properties:

• localReducerWeight - weight of a reduce task when it is executed on a node with mappers;
• remoteReducerWeight - weight of a map task when it is executed on a node without mappers.

Field Summary

Fields

Modifier and Type	Field and Description
static int	DFLT_LOC_MAPPER_WEIGHT Default local mapper weight.
static int	DFLT_LOC_REDUCER_WEIGHT Default local reducer weight.
static int	DFLT_PREFER_LOCAL_REDUCER_THRESHOLD_WEIGHT Default reducer migration threshold weight.
static int	DFLT_RMT_MAPPER_WEIGHT Default remote mapper weight.
static int	DFLT_RMT_REDUCER_WEIGHT Default remote reducer weight.

Fields inherited from class org.apache.ignite.hadoop.planner.HadoopAbstractMapReducePlanner

ignite, log

Constructor Summary

Constructors

Constructor and Description
IgniteHadoopWeightedMapReducePlanner()

Method Summary

Modifier and Type	Method and Description
int	getLocalMapperWeight() Get local mapper weight.
int	getLocalReducerWeight() Get local reducer weight.
int	getPreferLocalReducerThresholdWeight() Get reducer migration threshold weight.
int	getRemoteMapperWeight() Get remote mapper weight.
int	getRemoteReducerWeight() Get remote reducer weight.
HadoopMapReducePlan	preparePlan(HadoopJob job, Collection<ClusterNode> nodes, @Nullable HadoopMapReducePlan oldPlan) Prepares map-reduce execution plan for the given job and topology.
IgniteHadoopWeightedMapReducePlanner	setLocalMapperWeight(int locMapperWeight) Set local mapper weight.
IgniteHadoopWeightedMapReducePlanner	setLocalReducerWeight(int locReducerWeight) Set local reducer weight.
IgniteHadoopWeightedMapReducePlanner	setPreferLocalReducerThresholdWeight(int reducerMigrationThresholdWeight) Set reducer migration threshold weight.
IgniteHadoopWeightedMapReducePlanner	setRemoteMapperWeight(int rmtMapperWeight) Set remote mapper weight.
IgniteHadoopWeightedMapReducePlanner	setRemoteReducerWeight(int rmtReducerWeight) Set remote reducer weight.
String	toString()

Methods inherited from class org.apache.ignite.hadoop.planner.HadoopAbstractMapReducePlanner

groupByHost, topology

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DFLT_LOC_MAPPER_WEIGHT

public static final int DFLT_LOC_MAPPER_WEIGHT

Default local mapper weight.

See Also:

Constant Field Values

DFLT_RMT_MAPPER_WEIGHT

public static final int DFLT_RMT_MAPPER_WEIGHT

Default remote mapper weight.

See Also:

Constant Field Values

DFLT_LOC_REDUCER_WEIGHT

public static final int DFLT_LOC_REDUCER_WEIGHT

Default local reducer weight.

See Also:

Constant Field Values

DFLT_RMT_REDUCER_WEIGHT

public static final int DFLT_RMT_REDUCER_WEIGHT

Default remote reducer weight.

See Also:

Constant Field Values

DFLT_PREFER_LOCAL_REDUCER_THRESHOLD_WEIGHT

public static final int DFLT_PREFER_LOCAL_REDUCER_THRESHOLD_WEIGHT

Default reducer migration threshold weight.

See Also:

Constant Field Values

Constructor Detail

IgniteHadoopWeightedMapReducePlanner

public IgniteHadoopWeightedMapReducePlanner()

Method Detail

preparePlan

public HadoopMapReducePlan preparePlan(HadoopJob job,
                                       Collection<ClusterNode> nodes,
                                       @Nullable
                                       @Nullable HadoopMapReducePlan oldPlan)
                                throws IgniteCheckedException

Prepares map-reduce execution plan for the given job and topology.

Parameters:

job - Job.

nodes - Topology.

oldPlan - Old plan in case of partial failure.

Returns:

Map reduce plan.

Throws:

IgniteCheckedException - If an error occurs.

getLocalMapperWeight

public int getLocalMapperWeight()

Get local mapper weight. This weight is added to a node when a mapper is assigned and it's input split data is located on this node (at least partially).

Defaults to DFLT_LOC_MAPPER_WEIGHT.

Returns:

Remote mapper weight.

setLocalMapperWeight

public IgniteHadoopWeightedMapReducePlanner setLocalMapperWeight(int locMapperWeight)

Set local mapper weight. See getLocalMapperWeight() for more information.

Parameters:

locMapperWeight - Local mapper weight.

Returns:

this for chaining.

getRemoteMapperWeight

public int getRemoteMapperWeight()

Get remote mapper weight. This weight is added to a node when a mapper is assigned, but it's input split data is not located on this node.

Defaults to DFLT_RMT_MAPPER_WEIGHT.

Returns:

Remote mapper weight.

setRemoteMapperWeight

public IgniteHadoopWeightedMapReducePlanner setRemoteMapperWeight(int rmtMapperWeight)

Set remote mapper weight. See getRemoteMapperWeight() for more information.

Parameters:

rmtMapperWeight - Remote mapper weight.

Returns:

this for chaining.

getLocalReducerWeight

public int getLocalReducerWeight()

Get local reducer weight. This weight is added to a node when a reducer is assigned and the node have at least one assigned mapper.

Defaults to DFLT_LOC_REDUCER_WEIGHT.

Returns:

Local reducer weight.

setLocalReducerWeight

public IgniteHadoopWeightedMapReducePlanner setLocalReducerWeight(int locReducerWeight)

Set local reducer weight. See getLocalReducerWeight() for more information.

Parameters:

locReducerWeight - Local reducer weight.

Returns:

this for chaining.

getRemoteReducerWeight

public int getRemoteReducerWeight()

Get remote reducer weight. This weight is added to a node when a reducer is assigned, but the node doesn't have any assigned mappers.

Defaults to DFLT_RMT_REDUCER_WEIGHT.

Returns:

Remote reducer weight.

setRemoteReducerWeight

public IgniteHadoopWeightedMapReducePlanner setRemoteReducerWeight(int rmtReducerWeight)

Set remote reducer weight. See getRemoteReducerWeight() for more information.

Parameters:

rmtReducerWeight - Remote reducer weight.

Returns:

this for chaining.

getPreferLocalReducerThresholdWeight

public int getPreferLocalReducerThresholdWeight()

Get reducer migration threshold weight. When threshold is reached, a node with mappers is no longer considered as preferred for further reducer assignments.

Defaults to DFLT_PREFER_LOCAL_REDUCER_THRESHOLD_WEIGHT.

Returns:

Reducer migration threshold weight.

setPreferLocalReducerThresholdWeight

public IgniteHadoopWeightedMapReducePlanner setPreferLocalReducerThresholdWeight(int reducerMigrationThresholdWeight)

Set reducer migration threshold weight. See getPreferLocalReducerThresholdWeight() for more information.

Parameters:

reducerMigrationThresholdWeight - Reducer migration threshold weight.

Returns:

this for chaining.

toString

public String toString()

Overrides:

toString in class Object