org.apache.ignite

Interface IgniteFileSystem

All Superinterfaces:

IgniteAsyncSupport

public interface IgniteFileSystem
extends IgniteAsyncSupport

IGnite File System API. It provides a typical file system "view" on a particular cache:

• list directories or get information for a single path
• create/move/delete files or directories
• write/read data streams into/from files

The data of each file is split on separate data blocks and stored in the cache. You can access file's data with a standard Java streaming API. Moreover, for each part of the file you can calculate an affinity and process file's content on corresponding nodes to escape unnecessary networking.

This API is fully thread-safe and you can use it from several threads.

IGFS Configuration

The simplest way to run a Ignite node with configured file system is to pass special configuration file included in Ignite distribution to ignite.sh or ignite.bat scripts, like this: ignite.sh config/hadoop/default-config.xml

IGFS can be started as a data node or as a client node. Data node is responsible for caching data, while client node is responsible for basic file system operations and accessing data nodes remotely. When used as Hadoop file system, clients nodes usually started together with job-submitter or job-scheduler processes, while data nodes are usually started together with Hadoop task-tracker processes.

Integration With Hadoop

In addition to direct file system API, IGFS can be integrated with Hadoop by plugging in as Hadoop FileSystem. Refer to org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem or org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem for more information.

NOTE: integration with Hadoop is available only in In-Memory Accelerator For Hadoop edition.

Field Summary

Fields

Modifier and Type	Field and Description
static String	IGFS_SCHEME IGFS scheme name.

Method Summary

Modifier and Type	Method and Description
Collection<IgfsBlockLocation>	affinity(IgfsPath path, long start, long len) Gets affinity block locations for data blocks of the file, i.e. the nodes, on which the blocks are stored.
Collection<IgfsBlockLocation>	affinity(IgfsPath path, long start, long len, long maxLen) Get affinity block locations for data blocks of the file.
IgfsOutputStream	append(IgfsPath path, boolean create) Opens an output stream to an existing file for appending data.
IgfsOutputStream	append(IgfsPath path, int bufSize, boolean create, @Nullable Map<String,String> props) Opens an output stream to an existing file for appending data.
void	clear() Formats the file system removing all existing entries from it, but not removing anything in secondary file system (if any).
IgniteFuture<Void>	clearAsync() Formats the file system removing all existing entries from it, but not removing anything in secondary file system (if any).
FileSystemConfiguration	configuration() Gets IGFS configuration.
IgfsOutputStream	create(IgfsPath path, boolean overwrite) Creates a file and opens it for writing.
IgfsOutputStream	create(IgfsPath path, int bufSize, boolean overwrite, @Nullable IgniteUuid affKey, int replication, long blockSize, @Nullable Map<String,String> props) Creates a file and opens it for writing.
IgfsOutputStream	create(IgfsPath path, int bufSize, boolean overwrite, int replication, long blockSize, @Nullable Map<String,String> props) Creates a file and opens it for writing.
boolean	delete(IgfsPath path, boolean recursive) Deletes file.
<T,R> R	execute(Class<? extends IgfsTask<T,R>> taskCls, @Nullable IgfsRecordResolver rslvr, Collection<IgfsPath> paths, boolean skipNonExistentFiles, long maxRangeLen, T arg) Executes IGFS task with overridden maximum range length (see FileSystemConfiguration.getMaximumTaskRangeLength() for more information).
<T,R> R	execute(Class<? extends IgfsTask<T,R>> taskCls, @Nullable IgfsRecordResolver rslvr, Collection<IgfsPath> paths, T arg) Executes IGFS task.
<T,R> R	execute(IgfsTask<T,R> task, @Nullable IgfsRecordResolver rslvr, Collection<IgfsPath> paths, boolean skipNonExistentFiles, long maxRangeLen, T arg) Executes IGFS task with overridden maximum range length (see FileSystemConfiguration.getMaximumTaskRangeLength() for more information).
<T,R> R	execute(IgfsTask<T,R> task, @Nullable IgfsRecordResolver rslvr, Collection<IgfsPath> paths, T arg) Executes IGFS task.
<T,R> IgniteFuture<R>	executeAsync(Class<? extends IgfsTask<T,R>> taskCls, @Nullable IgfsRecordResolver rslvr, Collection<IgfsPath> paths, boolean skipNonExistentFiles, long maxRangeLen, T arg) Executes IGFS task asynchronously with overridden maximum range length (see FileSystemConfiguration.getMaximumTaskRangeLength() for more information).
<T,R> IgniteFuture<R>	executeAsync(Class<? extends IgfsTask<T,R>> taskCls, @Nullable IgfsRecordResolver rslvr, Collection<IgfsPath> paths, T arg) Executes IGFS task asynchronously.
<T,R> IgniteFuture<R>	executeAsync(IgfsTask<T,R> task, @Nullable IgfsRecordResolver rslvr, Collection<IgfsPath> paths, boolean skipNonExistentFiles, long maxRangeLen, T arg) Executes IGFS task asynchronously with overridden maximum range length (see FileSystemConfiguration.getMaximumTaskRangeLength() for more information).
<T,R> IgniteFuture<R>	executeAsync(IgfsTask<T,R> task, @Nullable IgfsRecordResolver rslvr, Collection<IgfsPath> paths, T arg) Executes IGFS task asynchronously.
boolean	exists(IgfsPath path) Checks if the specified path exists in the file system.
@Nullable IgfsFile	info(IgfsPath path) Gets file information for the specified path.
Collection<IgfsFile>	listFiles(IgfsPath path) Lists files under the specified path.
Collection<IgfsPath>	listPaths(IgfsPath path) Lists file paths under the specified path.
IgfsMetrics	metrics() Gets metrics snapshot for this file system.
void	mkdirs(IgfsPath path) Creates directories under specified path.
void	mkdirs(IgfsPath path, @Nullable Map<String,String> props) Creates directories under specified path with the specified properties.
IgfsMode	mode(IgfsPath path) Get mode for the given path.
String	name() Gets IGFS name.
IgfsInputStream	open(IgfsPath path) Opens a file for reading.
IgfsInputStream	open(IgfsPath path, int bufSize) Opens a file for reading.
IgfsInputStream	open(IgfsPath path, int bufSize, int seqReadsBeforePrefetch) Opens a file for reading.
void	rename(IgfsPath src, IgfsPath dest) Renames/moves a file.
void	resetMetrics() Resets metrics for this file system.
void	setTimes(IgfsPath path, long modificationTime, long accessTime) Sets last access time and last modification time for a given path.
long	size(IgfsPath path) Determines size of the file denoted by provided path.
IgfsPathSummary	summary(IgfsPath path) Gets summary (total number of files, total number of directories and total length) for a given path.
IgfsFile	update(IgfsPath path, Map<String,String> props) Updates file information for the specified path.
long	usedSpaceSize() Gets used space in bytes.
IgniteFileSystem	withAsync() Deprecated.

Methods inherited from interface org.apache.ignite.lang.IgniteAsyncSupport

future, isAsync

Field Detail

IGFS_SCHEME

static final String IGFS_SCHEME

IGFS scheme name.

See Also:

Constant Field Values

Method Detail

name

String name()

Gets IGFS name.

Returns:

IGFS name.

configuration

FileSystemConfiguration configuration()

Gets IGFS configuration.

Returns:

IGFS configuration.

summary

IgfsPathSummary summary(IgfsPath path)
                 throws IgniteException

Gets summary (total number of files, total number of directories and total length) for a given path.

Parameters:

path - Path to get information for.

Returns:

Summary object.

Throws:

IgfsPathNotFoundException - If path is not found.

IgniteException - If failed.

open

IgfsInputStream open(IgfsPath path)
              throws IgniteException

Opens a file for reading.

Parameters:

path - File path to read.

Returns:

File input stream to read data from.

Throws:

IgniteException - In case of error.

IgfsPathNotFoundException - If path doesn't exist.

open

IgfsInputStream open(IgfsPath path,
                     int bufSize)
              throws IgniteException

Opens a file for reading.

Parameters:

path - File path to read.

bufSize - Read buffer size (bytes) or zero to use default value.

Returns:

File input stream to read data from.

Throws:

IgniteException - In case of error.

IgfsPathNotFoundException - If path doesn't exist.

open

IgfsInputStream open(IgfsPath path,
                     int bufSize,
                     int seqReadsBeforePrefetch)
              throws IgniteException

Opens a file for reading.

Parameters:

path - File path to read.

bufSize - Read buffer size (bytes) or zero to use default value.

seqReadsBeforePrefetch - Amount of sequential reads before prefetch is started.

Returns:

File input stream to read data from.

Throws:

IgniteException - In case of error.

IgfsPathNotFoundException - If path doesn't exist.

create

IgfsOutputStream create(IgfsPath path,
                        boolean overwrite)
                 throws IgniteException

Creates a file and opens it for writing.

Parameters:

path - File path to create.

overwrite - Overwrite file if it already exists. Note: you cannot overwrite an existent directory.

Returns:

File output stream to write data to.

Throws:

IgniteException - In case of error.

create

IgfsOutputStream create(IgfsPath path,
                        int bufSize,
                        boolean overwrite,
                        int replication,
                        long blockSize,
                        @Nullable
                        @Nullable Map<String,String> props)
                 throws IgniteException

Creates a file and opens it for writing.

Parameters:

path - File path to create.

bufSize - Write buffer size (bytes) or zero to use default value.

overwrite - Overwrite file if it already exists. Note: you cannot overwrite an existent directory.

replication - Replication factor.

blockSize - Block size.

props - File properties to set.

Returns:

File output stream to write data to.

Throws:

IgniteException - In case of error.

create

IgfsOutputStream create(IgfsPath path,
                        int bufSize,
                        boolean overwrite,
                        @Nullable
                        @Nullable IgniteUuid affKey,
                        int replication,
                        long blockSize,
                        @Nullable
                        @Nullable Map<String,String> props)
                 throws IgniteException

Creates a file and opens it for writing.

Parameters:

path - File path to create.

bufSize - Write buffer size (bytes) or zero to use default value.

overwrite - Overwrite file if it already exists. Note: you cannot overwrite an existent directory.

affKey - Affinity key used to store file blocks. If not null, the whole file will be stored on node where affKey resides.

replication - Replication factor.

blockSize - Block size.

props - File properties to set.

Returns:

File output stream to write data to.

Throws:

IgniteException - In case of error.

append

IgfsOutputStream append(IgfsPath path,
                        boolean create)
                 throws IgniteException

Opens an output stream to an existing file for appending data.

Parameters:

path - File path to append.

create - Create file if it doesn't exist yet.

Returns:

File output stream to append data to.

Throws:

IgniteException - In case of error.

IgfsPathNotFoundException - If path doesn't exist and create flag is false.

append

IgfsOutputStream append(IgfsPath path,
                        int bufSize,
                        boolean create,
                        @Nullable
                        @Nullable Map<String,String> props)
                 throws IgniteException

Opens an output stream to an existing file for appending data.

Parameters:

path - File path to append.

bufSize - Write buffer size (bytes) or zero to use default value.

create - Create file if it doesn't exist yet.

props - File properties to set only in case it file was just created.

Returns:

File output stream to append data to.

Throws:

IgniteException - In case of error.

IgfsPathNotFoundException - If path doesn't exist and create flag is false.

setTimes

void setTimes(IgfsPath path,
              long modificationTime,
              long accessTime)
       throws IgniteException

Sets last access time and last modification time for a given path. If argument is null, corresponding time will not be changed.

Parameters:

path - Path to update.

modificationTime - Optional last modification time to set. Value -1 does not update modification time.

accessTime - Optional last access time to set. Value -1 does not update access time.

Throws:

IgfsPathNotFoundException - If target was not found.

IgniteException - If error occurred.

affinity

Collection<IgfsBlockLocation> affinity(IgfsPath path,
                                       long start,
                                       long len)
                                throws IgniteException

Gets affinity block locations for data blocks of the file, i.e. the nodes, on which the blocks are stored.

Parameters:

path - File path to get affinity for.

start - Position in the file to start affinity resolution from.

len - Size of data in the file to resolve affinity for.

Returns:

Affinity block locations.

Throws:

IgniteException - In case of error.

IgfsPathNotFoundException - If path doesn't exist.

affinity

Collection<IgfsBlockLocation> affinity(IgfsPath path,
                                       long start,
                                       long len,
                                       long maxLen)
                                throws IgniteException

Get affinity block locations for data blocks of the file. In case maxLen parameter is set and particular block location length is greater than this value, block locations will be split into smaller chunks.

Parameters:

path - File path to get affinity for.

start - Position in the file to start affinity resolution from.

len - Size of data in the file to resolve affinity for.

maxLen - Maximum length of a single returned block location length.

Returns:

Affinity block locations.

Throws:

IgniteException - In case of error.

IgfsPathNotFoundException - If path doesn't exist.

metrics

IgfsMetrics metrics()
             throws IgniteException

Gets metrics snapshot for this file system.

Returns:

Metrics.

Throws:

IgniteException - In case of error.

resetMetrics

void resetMetrics()
           throws IgniteException

Resets metrics for this file system.

Throws:

IgniteException - In case of error.

size

long size(IgfsPath path)
   throws IgniteException

Determines size of the file denoted by provided path. In case if path is a directory, then total size of all containing entries will be calculated recursively.

Parameters:

path - File system path.

Returns:

Total size.

Throws:

IgniteException - In case of error.

clear

void clear()
    throws IgniteException

Formats the file system removing all existing entries from it, but not removing anything in secondary file system (if any).

Throws:

IgniteException - In case clear failed.

clearAsync

IgniteFuture<Void> clearAsync()
                       throws IgniteException

Formats the file system removing all existing entries from it, but not removing anything in secondary file system (if any).

Returns:

Future representing pending completion of the clear operation.

Throws:

IgniteException

execute

<T,R> R execute(IgfsTask<T,R> task,
                @Nullable
                @Nullable IgfsRecordResolver rslvr,
                Collection<IgfsPath> paths,
                @Nullable
                T arg)
         throws IgniteException

Executes IGFS task.

Parameters:

task - Task to execute.

rslvr - Optional resolver to control split boundaries.

paths - Collection of paths to be processed within this task.

arg - Optional task argument.

Returns:

Task result.

Throws:

IgniteException - If execution failed.

executeAsync

<T,R> IgniteFuture<R> executeAsync(IgfsTask<T,R> task,
                                   @Nullable
                                   @Nullable IgfsRecordResolver rslvr,
                                   Collection<IgfsPath> paths,
                                   @Nullable
                                   T arg)
                            throws IgniteException

Executes IGFS task asynchronously.

Parameters:

task - Task to execute.

rslvr - Optional resolver to control split boundaries.

paths - Collection of paths to be processed within this task.

arg - Optional task argument.

Returns:

a Future representing pending completion of the task.

Throws:

IgniteException - If execution failed.

execute

<T,R> R execute(IgfsTask<T,R> task,
                @Nullable
                @Nullable IgfsRecordResolver rslvr,
                Collection<IgfsPath> paths,
                boolean skipNonExistentFiles,
                long maxRangeLen,
                @Nullable
                T arg)
         throws IgniteException

Executes IGFS task with overridden maximum range length (see FileSystemConfiguration.getMaximumTaskRangeLength() for more information).

Supports asynchronous execution (see IgniteAsyncSupport).

Parameters:

task - Task to execute.

rslvr - Optional resolver to control split boundaries.

paths - Collection of paths to be processed within this task.

skipNonExistentFiles - Whether to skip non existent files. If set to true non-existent files will be ignored. Otherwise an exception will be thrown.

maxRangeLen - Optional maximum range length. If 0, then by default all consecutive IGFS blocks will be included.

arg - Optional task argument.

Returns:

Task result.

Throws:

IgniteException - If execution failed.

executeAsync

<T,R> IgniteFuture<R> executeAsync(IgfsTask<T,R> task,
                                   @Nullable
                                   @Nullable IgfsRecordResolver rslvr,
                                   Collection<IgfsPath> paths,
                                   boolean skipNonExistentFiles,
                                   long maxRangeLen,
                                   @Nullable
                                   T arg)
                            throws IgniteException

Executes IGFS task asynchronously with overridden maximum range length (see FileSystemConfiguration.getMaximumTaskRangeLength() for more information).

Parameters:

task - Task to execute.

rslvr - Optional resolver to control split boundaries.

paths - Collection of paths to be processed within this task.

skipNonExistentFiles - Whether to skip non existent files. If set to true non-existent files will be ignored. Otherwise an exception will be thrown.

maxRangeLen - Optional maximum range length. If 0, then by default all consecutive IGFS blocks will be included.

arg - Optional task argument.

Returns:

a Future representing pending completion of the task.

Throws:

IgniteException - If execution failed.

execute

<T,R> R execute(Class<? extends IgfsTask<T,R>> taskCls,
                @Nullable
                @Nullable IgfsRecordResolver rslvr,
                Collection<IgfsPath> paths,
                @Nullable
                T arg)
         throws IgniteException

Executes IGFS task.

Supports asynchronous execution (see IgniteAsyncSupport).

Parameters:

taskCls - Task class to execute.

rslvr - Optional resolver to control split boundaries.

paths - Collection of paths to be processed within this task.

arg - Optional task argument.

Returns:

Task result.

Throws:

IgniteException - If execution failed.

executeAsync

<T,R> IgniteFuture<R> executeAsync(Class<? extends IgfsTask<T,R>> taskCls,
                                   @Nullable
                                   @Nullable IgfsRecordResolver rslvr,
                                   Collection<IgfsPath> paths,
                                   @Nullable
                                   T arg)
                            throws IgniteException

Executes IGFS task asynchronously.

Parameters:

taskCls - Task class to execute.

rslvr - Optional resolver to control split boundaries.

paths - Collection of paths to be processed within this task.

arg - Optional task argument.

Returns:

a Future representing pending completion of the task.

Throws:

IgniteException - If execution failed.

execute

<T,R> R execute(Class<? extends IgfsTask<T,R>> taskCls,
                @Nullable
                @Nullable IgfsRecordResolver rslvr,
                Collection<IgfsPath> paths,
                boolean skipNonExistentFiles,
                long maxRangeLen,
                @Nullable
                T arg)
         throws IgniteException

Executes IGFS task with overridden maximum range length (see FileSystemConfiguration.getMaximumTaskRangeLength() for more information).

Supports asynchronous execution (see IgniteAsyncSupport).

Parameters:

taskCls - Task class to execute.

rslvr - Optional resolver to control split boundaries.

paths - Collection of paths to be processed within this task.

skipNonExistentFiles - Whether to skip non existent files. If set to true non-existent files will be ignored. Otherwise an exception will be thrown.

maxRangeLen - Maximum range length.

arg - Optional task argument.

Returns:

Task result.

Throws:

IgniteException - If execution failed.

executeAsync

<T,R> IgniteFuture<R> executeAsync(Class<? extends IgfsTask<T,R>> taskCls,
                                   @Nullable
                                   @Nullable IgfsRecordResolver rslvr,
                                   Collection<IgfsPath> paths,
                                   boolean skipNonExistentFiles,
                                   long maxRangeLen,
                                   @Nullable
                                   T arg)
                            throws IgniteException

Executes IGFS task asynchronously with overridden maximum range length (see FileSystemConfiguration.getMaximumTaskRangeLength() for more information).

Parameters:

taskCls - Task class to execute.

rslvr - Optional resolver to control split boundaries.

paths - Collection of paths to be processed within this task.

skipNonExistentFiles - Whether to skip non existent files. If set to true non-existent files will be ignored. Otherwise an exception will be thrown.

maxRangeLen - Maximum range length.

arg - Optional task argument.

Returns:

a Future representing pending completion of the task.

Throws:

IgniteException - If execution failed.

exists

boolean exists(IgfsPath path)

Checks if the specified path exists in the file system.

Parameters:

path - Path to check for existence in the file system.

Returns:

True if such file exists, otherwise - false.

Throws:

IgniteException - In case of error.

update

IgfsFile update(IgfsPath path,
                Map<String,String> props)
         throws IgniteException

Updates file information for the specified path. Existent properties, not listed in the passed collection, will not be affected. Other properties will be added or overwritten. Passed properties with null values will be removed from the stored properties or ignored if they don't exist in the file info.

When working in DUAL_SYNC or DUAL_ASYNC modes with Hadoop secondary file system only the following properties will be updated:

• usrName - file owner name;
• grpName - file owner group;
• permission - Unix-style string representing file permissions.

Parameters:

path - File path to set properties for.

props - Properties to update.

Returns:

File information for specified path or null if such path does not exist.

Throws:

IgniteException - In case of error.

rename

void rename(IgfsPath src,
            IgfsPath dest)
     throws IgniteException

Renames/moves a file.

You are free to rename/move data files as you wish, but directories can be only renamed. You cannot move the directory between different parent directories.

Examples:

• "/work/file.txt" => "/home/project/Presentation Scenario.txt"
• "/work" => "/work-2012.bkp"
• "/work" => "/backups/work" - such operation is restricted for directories.

Parameters:

src - Source file path to rename.

dest - Destination file path. If destination path is a directory, then source file will be placed into destination directory with original name.

Throws:

IgniteException - In case of error.

IgfsPathNotFoundException - If source file doesn't exist.

delete

boolean delete(IgfsPath path,
               boolean recursive)
        throws IgniteException

Deletes file.

Parameters:

path - File path to delete.

recursive - Delete non-empty directories recursively.

Returns:

True in case of success, false otherwise.

Throws:

IgniteException - In case of error.

mkdirs

void mkdirs(IgfsPath path)
     throws IgniteException

Creates directories under specified path.

Parameters:

path - Path of directories chain to create.

Throws:

IgniteException - In case of error.

mkdirs

void mkdirs(IgfsPath path,
            @Nullable
            @Nullable Map<String,String> props)
     throws IgniteException

Creates directories under specified path with the specified properties. Note that the properties are applied only to created directories, but never updated for existing ones.

Parameters:

path - Path of directories chain to create.

props - Metadata properties to set on created directories.

Throws:

IgniteException - In case of error.

listPaths

Collection<IgfsPath> listPaths(IgfsPath path)
                        throws IgniteException

Lists file paths under the specified path.

Parameters:

path - Path to list files under.

Returns:

List of paths under the specified path.

Throws:

IgniteException - In case of error.

IgfsPathNotFoundException - If path doesn't exist.

listFiles

Collection<IgfsFile> listFiles(IgfsPath path)
                        throws IgniteException

Lists files under the specified path.

Parameters:

path - Path to list files under.

Returns:

List of files under the specified path.

Throws:

IgniteException - In case of error.

IgfsPathNotFoundException - If path doesn't exist.

info

@Nullable
@Nullable IgfsFile info(IgfsPath path)
                           throws IgniteException

Gets file information for the specified path.

Parameters:

path - Path to get information for.

Returns:

File information for specified path or null if such path does not exist.

Throws:

IgniteException - In case of error.

mode

IgfsMode mode(IgfsPath path)

Get mode for the given path.

Parameters:

path - Path.

Returns:

Mode used for this path.

usedSpaceSize

long usedSpaceSize()
            throws IgniteException

Gets used space in bytes.

Returns:

Used space in bytes.

Throws:

IgniteException - In case of error.

withAsync

@Deprecated
IgniteFileSystem withAsync()

Deprecated.

Gets instance of this component with asynchronous mode enabled.

Specified by:

withAsync in interface IgniteAsyncSupport

Returns:

Instance of this component with asynchronous mode enabled.