org.apache.ignite.ml.dataset.impl.cache.util

Class ComputeUtils

java.lang.Object

org.apache.ignite.ml.dataset.impl.cache.util.ComputeUtils

public class ComputeUtils
extends Object

Util class that provides common methods to perform computations on top of the Ignite Compute Grid.

Constructor Summary

Constructors

Constructor and Description
ComputeUtils()

Method Summary

Modifier and Type	Method and Description
static <R> Collection<R>	affinityCallWithRetries(Ignite ignite, Collection<String> cacheNames, IgniteFunction<Integer,R> fun, int retries, DeployingContext deployingContext) Calls the specified fun function on all partitions so that is't guaranteed that partitions with the same index of all specified caches will be placed on the same node and will not be moved before computation is finished.
static <R> Collection<R>	affinityCallWithRetries(Ignite ignite, Collection<String> cacheNames, IgniteFunction<Integer,R> fun, int retries, int interval, DeployingContext deployingContext) Calls the specified fun function on all partitions so that is't guaranteed that partitions with the same index of all specified caches will be placed on the same node and will not be moved before computation is finished.
static <C extends Serializable> C	getContext(Ignite ignite, String datasetCacheName, int part) Extracts partition context from the Ignite Cache.
static <K,V,C extends Serializable,D extends AutoCloseable> D	getData(Ignite ignite, String upstreamCacheName, IgniteBiPredicate<K,V> filter, UpstreamTransformerBuilder transformerBuilder, String datasetCacheName, UUID datasetId, PartitionDataBuilder<K,V,C,D> partDataBuilder, LearningEnvironment env, boolean isKeepBinary) Extracts partition data from the local storage, if it's not found in local storage recovers this data from a partition upstream and context.
static LearningEnvironment	getLearningEnvironment(Ignite ignite, UUID datasetId, int part, LearningEnvironmentBuilder envBuilder) Gets learning environment for given partition.
static <K,V,C extends Serializable> void	initContext(Ignite ignite, String upstreamCacheName, UpstreamTransformerBuilder transformerBuilder, IgniteBiPredicate<K,V> filter, String datasetCacheName, PartitionContextBuilder<K,V,C> ctxBuilder, LearningEnvironmentBuilder envBuilder, int retries, int interval, boolean isKeepBinary, DeployingContext deployingContext) Initializes partition context by loading it from a partition upstream.
static void	removeData(Ignite ignite, UUID datasetId) Remove data from local cache by Dataset ID.
static void	removeLearningEnv(Ignite ignite, UUID datasetId) Remove learning environment from local cache by Dataset ID.
static <C extends Serializable> void	saveContext(Ignite ignite, String datasetCacheName, int part, C ctx) Saves the specified partition context into the Ignite Cache.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ComputeUtils

public ComputeUtils()

Method Detail

affinityCallWithRetries

public static <R> Collection<R> affinityCallWithRetries(Ignite ignite,
                                                        Collection<String> cacheNames,
                                                        IgniteFunction<Integer,R> fun,
                                                        int retries,
                                                        int interval,
                                                        DeployingContext deployingContext)

Calls the specified fun function on all partitions so that is't guaranteed that partitions with the same index of all specified caches will be placed on the same node and will not be moved before computation is finished. If partitions are placed on different nodes then call will be retried, but not more than retries times with interval interval specified in milliseconds.

Type Parameters:

R - Type of a result.

Parameters:

ignite - Ignite instance.

cacheNames - Collection of cache names.

fun - Function to be applied on all partitions.

retries - Number of retries for the case when one of partitions not found on the node.

interval - Interval of retries for the case when one of partitions not found on the node.

deployingContext - Deploy context of user-defined classes for peer class loading.

Returns:

Collection of results.

affinityCallWithRetries

public static <R> Collection<R> affinityCallWithRetries(Ignite ignite,
                                                        Collection<String> cacheNames,
                                                        IgniteFunction<Integer,R> fun,
                                                        int retries,
                                                        DeployingContext deployingContext)

Calls the specified fun function on all partitions so that is't guaranteed that partitions with the same index of all specified caches will be placed on the same node and will not be moved before computation is finished. If partitions are placed on different nodes then call will be retried, but not more than retries times.

Type Parameters:

R - Type of a result.

Parameters:

ignite - Ignite instance.

cacheNames - Collection of cache names.

fun - Function to be applied on all partitions.

retries - Number of retries for the case when one of partitions not found on the node.

deployingContext - Deploy context.

Returns:

Collection of results.

getLearningEnvironment

public static LearningEnvironment getLearningEnvironment(Ignite ignite,
                                                         UUID datasetId,
                                                         int part,
                                                         LearningEnvironmentBuilder envBuilder)

Gets learning environment for given partition. If learning environment is not found in local node map, it will be created with specified LearningEnvironmentBuilder.

Parameters:

ignite - Ignite instance.

datasetId - Dataset id.

part - Partition index.

envBuilder - LearningEnvironmentBuilder.

Returns:

Learning environment for given partition.

getData

public static <K,V,C extends Serializable,D extends AutoCloseable> D getData(Ignite ignite,
                                                                             String upstreamCacheName,
                                                                             IgniteBiPredicate<K,V> filter,
                                                                             UpstreamTransformerBuilder transformerBuilder,
                                                                             String datasetCacheName,
                                                                             UUID datasetId,
                                                                             PartitionDataBuilder<K,V,C,D> partDataBuilder,
                                                                             LearningEnvironment env,
                                                                             boolean isKeepBinary)

Extracts partition data from the local storage, if it's not found in local storage recovers this data from a partition upstream and context. Be aware that this method should be called from the node where partition is placed.

Type Parameters:

K - Type of a key in upstream data.

V - Type of a value in upstream data.

C - Type of a partition context.

D - Type of a partition data.

Parameters:

ignite - Ignite instance.

upstreamCacheName - Name of an upstream cache.

filter - Filter for upstream data.

transformerBuilder - Builder of upstream transformers.

datasetCacheName - Name of a partition context cache.

datasetId - Dataset ID.

partDataBuilder - Partition data builder.

env - Learning environment.

Returns:

Partition data.

removeData

public static void removeData(Ignite ignite,
                              UUID datasetId)

Remove data from local cache by Dataset ID.

Parameters:

ignite - Ignite instance.

datasetId - Dataset ID.

removeLearningEnv

public static void removeLearningEnv(Ignite ignite,
                                     UUID datasetId)

Remove learning environment from local cache by Dataset ID.

Parameters:

ignite - Ingnite instance.

datasetId - Dataset ID.

initContext

public static <K,V,C extends Serializable> void initContext(Ignite ignite,
                                                            String upstreamCacheName,
                                                            UpstreamTransformerBuilder transformerBuilder,
                                                            IgniteBiPredicate<K,V> filter,
                                                            String datasetCacheName,
                                                            PartitionContextBuilder<K,V,C> ctxBuilder,
                                                            LearningEnvironmentBuilder envBuilder,
                                                            int retries,
                                                            int interval,
                                                            boolean isKeepBinary,
                                                            DeployingContext deployingContext)

Initializes partition context by loading it from a partition upstream.

Type Parameters:

K - Type of a key in upstream data.

V - Type of a value in upstream data.

C - Type of a partition context.

Parameters:

ignite - Ignite instance.

upstreamCacheName - Name of an upstream cache.

filter - Filter for upstream data.

transformerBuilder - Upstream transformer builder.

ctxBuilder - Partition context builder.

envBuilder - Environment builder.

isKeepBinary - Support of binary objects.

deployingContext - Deploy context.

getContext

public static <C extends Serializable> C getContext(Ignite ignite,
                                                    String datasetCacheName,
                                                    int part)

Extracts partition context from the Ignite Cache.

Type Parameters:

C - Type of a partition context.

Parameters:

ignite - Ignite instance.

datasetCacheName - Dataset cache names.

part - Partition index.

Returns:

Partition context.

saveContext

public static <C extends Serializable> void saveContext(Ignite ignite,
                                                        String datasetCacheName,
                                                        int part,
                                                        C ctx)

Saves the specified partition context into the Ignite Cache.

Type Parameters:

C - Type of a partition context.

Parameters:

ignite - Ignite instance.

datasetCacheName - Dataset cache name.

part - Partition index.