org.apache.ignite.igfs

Class IgfsGroupDataBlocksKeyMapper

java.lang.Object

org.apache.ignite.internal.processors.cache.GridCacheDefaultAffinityKeyMapper

org.apache.ignite.igfs.IgfsGroupDataBlocksKeyMapper

All Implemented Interfaces:

Serializable, AffinityKeyMapper

public class IgfsGroupDataBlocksKeyMapper
extends org.apache.ignite.internal.processors.cache.GridCacheDefaultAffinityKeyMapper

IGFS class providing ability to group file's data blocks together on one node. All blocks within the same group are guaranteed to be cached together on the same node. Group size parameter controls how many sequential blocks will be cached together on the same node.

For example, if block size is 64kb and group size is 256, then each group will contain 64kb * 256 = 16Mb. Larger group sizes would reduce number of splits required to run map-reduce tasks, but will increase inequality of data size being stored on different nodes.

Note that getGroupSize() parameter must correlate to Hadoop split size parameter defined in Hadoop via mapred.max.split.size property. Ideally you want all blocks accessed within one split to be mapped to 1 group, so they can be located on the same grid node. For example, default Hadoop split size is 64mb and default IGFS block size is 64kb. This means that to make sure that each split goes only through blocks on the same node (without hopping between nodes over network), we have to make the getGroupSize() value be equal to 64mb / 64kb = 1024.

It is required for IGFS data cache to be configured with this mapper. Here is an example of how it can be specified in XML configuration:

 <bean id="cacheCfgBase" class="org.apache.ignite.cache.CacheConfiguration" abstract="true">
     ...
     <property name="affinityMapper">
         <bean class="org.apache.ignite.igfs.IgfsGroupDataBlocksKeyMapper">
             <!-- How many sequential blocks will be stored on the same node. -->
             <property name="groupSize" value="512"/>
         </bean>
     </property>
     ...
 </bean>
 

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static int	DFLT_GRP_SIZE Default group size.

Fields inherited from class org.apache.ignite.internal.processors.cache.GridCacheDefaultAffinityKeyMapper

ignite, log

Constructor Summary

Constructors

Constructor and Description
IgfsGroupDataBlocksKeyMapper() Default constructor.
IgfsGroupDataBlocksKeyMapper(int grpSize) Constructs affinity mapper to group several data blocks with the same key.

Method Summary

Modifier and Type	Method and Description
Object	affinityKey(Object key) If key class has annotation AffinityKeyMapped, then the value of annotated method or field will be used to get affinity value instead of the key itself.
int	getGroupSize() Get group size.
IgfsGroupDataBlocksKeyMapper	setGroupSize(int grpSize) Set group size.
String	toString()

Methods inherited from class org.apache.ignite.internal.processors.cache.GridCacheDefaultAffinityKeyMapper

affinityKeyPropertyName, ignite, reset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DFLT_GRP_SIZE

public static final int DFLT_GRP_SIZE

Default group size.

See Also:

Constant Field Values

Constructor Detail

IgfsGroupDataBlocksKeyMapper

public IgfsGroupDataBlocksKeyMapper()

Default constructor.

IgfsGroupDataBlocksKeyMapper

public IgfsGroupDataBlocksKeyMapper(int grpSize)

Constructs affinity mapper to group several data blocks with the same key.

Parameters:

grpSize - Size of the group in blocks.

Method Detail

affinityKey

public Object affinityKey(Object key)

If key class has annotation AffinityKeyMapped, then the value of annotated method or field will be used to get affinity value instead of the key itself. If there is no annotation, then the key is returned as is.

Specified by:

affinityKey in interface AffinityKeyMapper

Overrides:

affinityKey in class org.apache.ignite.internal.processors.cache.GridCacheDefaultAffinityKeyMapper

Parameters:

key - Key to get affinity key for.

Returns:

Affinity key for given key.

getGroupSize

public int getGroupSize()

Get group size.

Group size defines how many sequential file blocks will reside on the same node. This parameter must correlate to Hadoop split size parameter defined in Hadoop via mapred.max.split.size property. Ideally you want all blocks accessed within one split to be mapped to 1 group, so they can be located on the same grid node. For example, default Hadoop split size is 64mb and default IGFS block size is 64kb. This means that to make sure that each split goes only through blocks on the same node (without hopping between nodes over network), we have to make the group size be equal to 64mb / 64kb = 1024.

Defaults to DFLT_GRP_SIZE.

Returns:

Group size.

setGroupSize

public IgfsGroupDataBlocksKeyMapper setGroupSize(int grpSize)

Set group size. See getGroupSize() for more information.

Parameters:

grpSize - Group size.

Returns:

this for chaining.

toString

public String toString()

Overrides:

toString in class Object