Load Sync Hive

Install GridGain following this Getting Started guide.

Create an account in GridGain Nebula.

Ensure that the following versions are used: Apache Hadoop version 2.x and Apache Hive 2.x with configured transactions.

Run hiveserver2 in order to enable JDBC connections to the Hive cluster.

Go to GridGain Nebula, sign up or sign in if you already have an account.

Download Web Agent from GridGain Nebula:

Configure Web Agent. The extracted package should contain the corresponding Hive JDBC driver in the jdbc-drivers directory. The driver’s dependencies should be included in the java class path of the Web Agent. One possible solution is to place required jars in the Web Agent root directory.

Click Import from Database:

Configure your Hive JDBC connection to allow Web Agent to collect metadata from the target Hive installation:

Select schema(s) to import:

Select the desired tables and configure the cache template:

Configure the results of project generation:

Go to the generated configuration page:

Fill in the configuration fields and click Save and Download. GridGain Nebula will generate a maven project and implement the Hive CacheStore and cluster configurations:

Place the Hive JDBC driver in the application java classpath. Here’s an example maven dependency:

<dependency>
	<groupId>org.apache.hive</groupId>
     <artifactId>hive-jdbc</artifactId>
     <version>2.3.4</version>
     <exclusions>
         <exclusion>
            <groupId>org.eclipse.jetty.aggregate</groupId>
            <artifactId>jetty-all</artifactId>
         </exclusion>
     </exclusions>
</dependency>

try {
    Class.forName("org.apache.hive.jdbc.HiveDriver");
} catch (ClassNotFoundException e) {
    throw new IllegalStateException("No org.apache.hive.jdbc.HiveDriver", e);
}

Edit the generated secret.properties file located in src/main/resources/secret.properties and add the Hive connection details.

Run ServerNodeSpringStartup or ServerNodeCodeStartup (the two classes use different types of configuration).

Add the corresponding GridGain Hadoop Connector dependency to the generated project pom:

 <dependency>
     <groupId>org.gridgain.plugins</groupId>
     <artifactId>gridgain-hive-store</artifactId>
     <version>{version}</version>
 </dependency>

Replace instances of org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory with org.gridgain.cachestore.HiveCacheJdbcPojoStoreFactory:

HiveCacheJdbcPojoStoreFactory cacheStoreFactory = new HiveCacheJdbcPojoStoreFactory();

Set streamerEnabled to true to enable more efficient data loading:

cacheStoreFactory.setStreamerEnabled(true);