org.apache.ignite.hadoop.fs.v2

Class IgniteHadoopFileSystem

java.lang.Object

org.apache.hadoop.fs.AbstractFileSystem

org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable

public class IgniteHadoopFileSystem
extends AbstractFileSystem
implements Closeable

IGFS Hadoop 2.x file system driver over file system API. To use IGFS as Hadoop file system, you should configure this class in Hadoop's core-site.xml as follows:

  <property>
      <name>fs.default.name</name>
      <value>igfs://ipc</value>
  </property>

  <property>
      <name>fs.igfs.impl</name>
      <value>org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem</value>
  </property>
 

You should also add Ignite JAR and all libraries to Hadoop classpath. To do this, add following lines to conf/hadoop-env.sh script in Hadoop distribution:

 export IGNITE_HOME=/path/to/Ignite/distribution
 export HADOOP_CLASSPATH=$IGNITE_HOME/ignite*.jar

 for f in $IGNITE_HOME/libs/*.jar; do
  export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f;
 done
 

Data vs Clients Nodes

Hadoop needs to use its FileSystem remotely from client nodes as well as directly on data nodes. Client nodes are responsible for basic file system operations as well as accessing data nodes remotely. Usually, client nodes are started together with job-submitter or job-scheduler processes, while data nodes are usually started together with Hadoop task-tracker processes.

For sample client and data node configuration refer to config/hadoop/default-config-client.xml and config/hadoop/default-config.xml configuration files in Ignite installation.

Field Summary

Fields inherited from class org.apache.hadoop.fs.AbstractFileSystem

statistics

Constructor Summary

Constructors

Constructor and Description
IgniteHadoopFileSystem(URI name, Configuration cfg)

Method Summary

Modifier and Type	Method and Description
void	checkPath(Path path)
void	close()
void	colocateFileWrites(boolean colocateFileWrites) Public setter that can be used by direct users of FS or Visor.
FSDataOutputStream	createInternal(Path f, EnumSet<CreateFlag> flag, FsPermission perm, int bufSize, short replication, long blockSize, Progressable progress, Options.ChecksumOpt checksumOpt, boolean createParent)
boolean	delete(Path f, boolean recursive)
BlockLocation[]	getFileBlockLocations(Path path, long start, long len)
FileChecksum	getFileChecksum(Path f)
FileStatus	getFileStatus(Path f)
FsStatus	getFsStatus()
FsServerDefaults	getServerDefaults()
URI	getUri()
int	getUriDefaultPort()
FileStatus[]	listStatus(Path f)
void	mkdir(Path f, FsPermission perm, boolean createParent)
FSDataInputStream	open(Path f, int bufSize)
void	renameInternal(Path src, Path dst)
void	setOwner(Path p, String usr, String grp)
void	setPermission(Path p, FsPermission perm)
boolean	setReplication(Path f, short replication)
void	setTimes(Path f, long mtime, long atime)
void	setVerifyChecksum(boolean verifyChecksum)
boolean	supportsSymlinks()
String	toString()
String	user() Returns the user name this File System is created on behalf of.

Methods inherited from class org.apache.hadoop.fs.AbstractFileSystem

access, checkScheme, clearStatistics, create, createFileSystem, createSnapshot, createSymlink, deleteSnapshot, equals, get, getAclStatus, getAllStatistics, getAllStoragePolicies, getCanonicalServiceName, getDelegationTokens, getFileLinkStatus, getFsStatus, getHomeDirectory, getInitialWorkingDirectory, getLinkTarget, getServerDefaults, getStatistics, getStatistics, getStoragePolicy, getUriPath, getXAttr, getXAttrs, getXAttrs, hashCode, isValidName, listCorruptFileBlocks, listLocatedStatus, listStatusIterator, listXAttrs, makeQualified, modifyAclEntries, open, printStatistics, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameInternal, renameSnapshot, resolvePath, setAcl, setStoragePolicy, setXAttr, setXAttr, truncate, unsetStoragePolicy

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

IgniteHadoopFileSystem

public IgniteHadoopFileSystem(URI name,
                              Configuration cfg)
                       throws URISyntaxException,
                              IOException

Parameters:

name - URI for file system.

cfg - Configuration.

Throws:

URISyntaxException - if name has invalid syntax.

IOException - If initialization failed.

Method Detail

checkPath

public void checkPath(Path path)

Overrides:

checkPath in class AbstractFileSystem

colocateFileWrites

public void colocateFileWrites(boolean colocateFileWrites)

Public setter that can be used by direct users of FS or Visor.

Parameters:

colocateFileWrites - Whether all ongoing file writes should be colocated.

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

getUri

public URI getUri()

Overrides:

getUri in class AbstractFileSystem

getUriDefaultPort

public int getUriDefaultPort()

Specified by:

getUriDefaultPort in class AbstractFileSystem

getServerDefaults

public FsServerDefaults getServerDefaults()
                                   throws IOException

Specified by:

getServerDefaults in class AbstractFileSystem

Throws:

IOException

setReplication

public boolean setReplication(Path f,
                              short replication)
                       throws IOException

Specified by:

setReplication in class AbstractFileSystem

Throws:

IOException

setTimes

public void setTimes(Path f,
                     long mtime,
                     long atime)
              throws IOException

Specified by:

setTimes in class AbstractFileSystem

Throws:

IOException

getFsStatus

public FsStatus getFsStatus()
                     throws IOException

Specified by:

getFsStatus in class AbstractFileSystem

Throws:

IOException

setPermission

public void setPermission(Path p,
                          FsPermission perm)
                   throws IOException

Specified by:

setPermission in class AbstractFileSystem

Throws:

IOException

setOwner

public void setOwner(Path p,
                     String usr,
                     String grp)
              throws IOException

Specified by:

setOwner in class AbstractFileSystem

Throws:

IOException

open

public FSDataInputStream open(Path f,
                              int bufSize)
                       throws IOException

Specified by:

open in class AbstractFileSystem

Throws:

IOException

createInternal

public FSDataOutputStream createInternal(Path f,
                                         EnumSet<CreateFlag> flag,
                                         FsPermission perm,
                                         int bufSize,
                                         short replication,
                                         long blockSize,
                                         Progressable progress,
                                         Options.ChecksumOpt checksumOpt,
                                         boolean createParent)
                                  throws IOException

Specified by:

createInternal in class AbstractFileSystem

Throws:

IOException

supportsSymlinks

public boolean supportsSymlinks()

Overrides:

supportsSymlinks in class AbstractFileSystem

renameInternal

public void renameInternal(Path src,
                           Path dst)
                    throws IOException

Specified by:

renameInternal in class AbstractFileSystem

Throws:

IOException

delete

public boolean delete(Path f,
                      boolean recursive)
               throws IOException

Specified by:

delete in class AbstractFileSystem

Throws:

IOException

setVerifyChecksum

public void setVerifyChecksum(boolean verifyChecksum)
                       throws IOException

Specified by:

setVerifyChecksum in class AbstractFileSystem

Throws:

IOException

getFileChecksum

public FileChecksum getFileChecksum(Path f)
                             throws IOException

Specified by:

getFileChecksum in class AbstractFileSystem

Throws:

IOException

listStatus

public FileStatus[] listStatus(Path f)
                        throws IOException

Specified by:

listStatus in class AbstractFileSystem

Throws:

IOException

mkdir

public void mkdir(Path f,
                  FsPermission perm,
                  boolean createParent)
           throws IOException

Specified by:

mkdir in class AbstractFileSystem

Throws:

IOException

getFileStatus

public FileStatus getFileStatus(Path f)
                         throws IOException

Specified by:

getFileStatus in class AbstractFileSystem

Throws:

IOException

getFileBlockLocations

public BlockLocation[] getFileBlockLocations(Path path,
                                             long start,
                                             long len)
                                      throws IOException

Specified by:

getFileBlockLocations in class AbstractFileSystem

Throws:

IOException

toString

public String toString()

Overrides:

toString in class Object

user

public String user()

Returns the user name this File System is created on behalf of.

Returns:

the user name