org.apache.ignite.hadoop.fs.v1

Class IgniteHadoopFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, Configurable

public class IgniteHadoopFileSystem
extends FileSystem

IGFS Hadoop 1.x file system driver over file system API. To use IGFS as Hadoop file system, you should configure this class in Hadoop's core-site.xml as follows:

  <property>
      <name>fs.default.name</name>
      <value>igfs:///</value>
  </property>

  <property>
      <name>fs.igfs.impl</name>
      <value>org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem</value>
  </property>
 

You should also add Ignite JAR and all libraries to Hadoop classpath. To do this, add following lines to conf/hadoop-env.sh script in Hadoop distribution:

 export IGNITE_HOME=/path/to/Ignite/distribution
 export HADOOP_CLASSPATH=$IGNITE_HOME/ignite*.jar

 for f in $IGNITE_HOME/libs/*.jar; do
  export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f;
 done
 

Data vs Clients Nodes

Hadoop needs to use its FileSystem remotely from client nodes as well as directly on data nodes. Client nodes are responsible for basic file system operations as well as accessing data nodes remotely. Usually, client nodes are started together with job-submitter or job-scheduler processes, while data nodes are usually started together with Hadoop task-tracker processes.

For sample client and data node configuration refer to config/hadoop/default-config-client.xml and config/hadoop/default-config.xml configuration files in Ignite installation.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

FileSystem.DirectoryEntries, FileSystem.Statistics

Field Summary

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, LOG, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX

Constructor Summary

Constructors

Constructor and Description
IgniteHadoopFileSystem()

Method Summary

Modifier and Type	Method and Description
FSDataOutputStream	append(Path f, int bufSize, Progressable progress)
protected void	checkPath(Path path)
void	close()
void	colocateFileWrites(boolean colocateFileWrites) Public setter that can be used by direct users of FS or Visor.
FSDataOutputStream	create(Path f, FsPermission perm, boolean overwrite, int bufSize, short replication, long blockSize, Progressable progress)
boolean	delete(Path f)
boolean	delete(Path f, boolean recursive)
protected void	finalize()
ContentSummary	getContentSummary(Path f)
long	getDefaultBlockSize()
short	getDefaultReplication()
BlockLocation[]	getFileBlockLocations(FileStatus status, long start, long len)
FileStatus	getFileStatus(Path f)
static String	getFsHadoopUser() Gets non-null user name as per the Hadoop file system viewpoint.
Path	getHomeDirectory()
org.apache.ignite.internal.processors.igfs.IgfsModeResolver	getModeResolver()
URI	getUri()
Path	getWorkingDirectory()
void	initialize(URI name, Configuration cfg)
FileStatus[]	listStatus(Path f)
boolean	mkdirs(Path f, FsPermission perm)
FSDataInputStream	open(Path f, int bufSize)
boolean	rename(Path src, Path dst)
void	setOwner(Path p, String username, String grpName)
void	setPermission(Path p, FsPermission perm)
void	setTimes(Path p, long mtime, long atime)
void	setWorkingDirectory(Path newPath)
String	toString()
String	user() Returns the user name this File System is created on behalf of.

Methods inherited from class org.apache.hadoop.fs.FileSystem

access, addDelegationTokens, append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createFile, createNewFile, createNonRecursive, createNonRecursive, createNonRecursive, createSnapshot, createSnapshot, createSymlink, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getAllStoragePolicies, getBlockSize, getCanonicalServiceName, getCanonicalUri, getChildFileSystems, getDefaultBlockSize, getDefaultPort, getDefaultReplication, getDefaultUri, getDelegationToken, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getQuotaUsage, getReplication, getScheme, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getStorageStatistics, getTrashRoot, getTrashRoots, getUsed, getUsed, getXAttr, getXAttrs, getXAttrs, globStatus, globStatus, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusBatch, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, setAcl, setDefaultUri, setDefaultUri, setReplication, setStoragePolicy, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

IgniteHadoopFileSystem

public IgniteHadoopFileSystem()

Method Detail

getUri

public URI getUri()

Specified by:

getUri in class FileSystem

getFsHadoopUser

public static String getFsHadoopUser()
                              throws IOException

Gets non-null user name as per the Hadoop file system viewpoint.

Returns:

the user name, never null.

Throws:

IOException - On error.

colocateFileWrites

public void colocateFileWrites(boolean colocateFileWrites)

Public setter that can be used by direct users of FS or Visor.

Parameters:

colocateFileWrites - Whether all ongoing file writes should be colocated.

initialize

public void initialize(URI name,
                       Configuration cfg)
                throws IOException

Overrides:

initialize in class FileSystem

Throws:

IOException

checkPath

protected void checkPath(Path path)

Overrides:

checkPath in class FileSystem

getDefaultReplication

public short getDefaultReplication()

Overrides:

getDefaultReplication in class FileSystem

finalize

protected void finalize()
                 throws Throwable

Overrides:

finalize in class Object

Throws:

Throwable

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class FileSystem

Throws:

IOException

setTimes

public void setTimes(Path p,
                     long mtime,
                     long atime)
              throws IOException

Overrides:

setTimes in class FileSystem

Throws:

IOException

setPermission

public void setPermission(Path p,
                          FsPermission perm)
                   throws IOException

Overrides:

setPermission in class FileSystem

Throws:

IOException

setOwner

public void setOwner(Path p,
                     String username,
                     String grpName)
              throws IOException

Overrides:

setOwner in class FileSystem

Throws:

IOException

open

public FSDataInputStream open(Path f,
                              int bufSize)
                       throws IOException

Specified by:

open in class FileSystem

Throws:

IOException

create

public FSDataOutputStream create(Path f,
                                 FsPermission perm,
                                 boolean overwrite,
                                 int bufSize,
                                 short replication,
                                 long blockSize,
                                 Progressable progress)
                          throws IOException

Specified by:

create in class FileSystem

Throws:

IOException

append

public FSDataOutputStream append(Path f,
                                 int bufSize,
                                 Progressable progress)
                          throws IOException

Specified by:

append in class FileSystem

Throws:

IOException

rename

public boolean rename(Path src,
                      Path dst)
               throws IOException

Specified by:

rename in class FileSystem

Throws:

IOException

delete

public boolean delete(Path f)
               throws IOException

Overrides:

delete in class FileSystem

Throws:

IOException

delete

public boolean delete(Path f,
                      boolean recursive)
               throws IOException

Specified by:

delete in class FileSystem

Throws:

IOException

listStatus

public FileStatus[] listStatus(Path f)
                        throws IOException

Specified by:

listStatus in class FileSystem

Throws:

IOException

getHomeDirectory

public Path getHomeDirectory()

Overrides:

getHomeDirectory in class FileSystem

setWorkingDirectory

public void setWorkingDirectory(Path newPath)

Specified by:

setWorkingDirectory in class FileSystem

getWorkingDirectory

public Path getWorkingDirectory()

Specified by:

getWorkingDirectory in class FileSystem

mkdirs

public boolean mkdirs(Path f,
                      FsPermission perm)
               throws IOException

Specified by:

mkdirs in class FileSystem

Throws:

IOException

getFileStatus

public FileStatus getFileStatus(Path f)
                         throws IOException

Specified by:

getFileStatus in class FileSystem

Throws:

IOException

getContentSummary

public ContentSummary getContentSummary(Path f)
                                 throws IOException

Overrides:

getContentSummary in class FileSystem

Throws:

IOException

getFileBlockLocations

public BlockLocation[] getFileBlockLocations(FileStatus status,
                                             long start,
                                             long len)
                                      throws IOException

Overrides:

getFileBlockLocations in class FileSystem

Throws:

IOException

getDefaultBlockSize

public long getDefaultBlockSize()

Overrides:

getDefaultBlockSize in class FileSystem

getModeResolver

public org.apache.ignite.internal.processors.igfs.IgfsModeResolver getModeResolver()
                                                                            throws IOException

Returns:

Mode resolver.

Throws:

IOException - On error.

toString

public String toString()

Overrides:

toString in class Object

user

public String user()

Returns the user name this File System is created on behalf of.

Returns:

the user name