Deploying GridGain in Multiple Availability Zones Using AWS

To deploy GridGain in multiple availability zones, perform the steps below:

Create an EKS cluster with auto scaling groups in different availability zones;
Deploy the Cluster AutoScaler to work with given autoscaling groups when scheduling nodes;
Define an affinity backup filter to place backup entries onto availability zones that differ from the primary entries.

Creating EKS Cluster and Deploying Cluster AutoScaler

To create an EKS cluster and to deploy Cluster AutoScaler, perform the steps below:

Follow the EKS Cluster availability workshop to create an EKS cluster, then deploy the cluster autoscaler;

Configure the autoscaler to use the autoscaling groups that you need.

          command:
            - ./cluster-autoscaler
            - --v=4
            - --stderrthreshold=info
            - --cloud-provider=aws
            - --skip-nodes-with-local-storage=false
            - --expander=least-waste
            - --nodes=1:3:autoScale-group-in-availability-region-a
            - --nodes=1:3:autoScale-group-in-availability-region-b

At this point, the autoscaler is directed to create new nodes by using only 2 auto scaling groups. An example of a manual auto scaling group config could be found here.

Adjust your auto scaling groups to have the appropriate capacity.

Configuring Backup Filters

The following sections desribe in detail how to set up a back up filter:

Defining LifeCycle Bean

The Lifecycle bean, which runs before the node is started, issues a REST request for the availability zone of the VM hosting the pod and stores the zone as a user attribute.

Refer to this section for more information.

public class AvailabilityZoneLifecycleBean implements LifecycleBean {

    @IgniteInstanceResource
    Ignite ignite;

    @LoggerResource
    IgniteLogger log;

    @Override
    public void onLifecycleEvent(LifecycleEventType evt) {
        if (evt == LifecycleEventType.BEFORE_NODE_START) {
            try {
                InputStream response = new URL("http://169.254.169.254/latest/meta-data/placement/availability-zone").openConnection().getInputStream();
                Map<String, String> user_attributes = new HashMap<>();
                String availablity_zone = null;
                try (Scanner scanner = new Scanner(response)) {
                    availablity_zone = scanner.useDelimiter("\\A").next();
                }
                user_attributes.put("AVAILABILITY_ZONE", availablity_zone);
                ignite.configuration().setUserAttributes(user_attributes);
            } catch (IOException e) {
                log.error("error setting AVAILABLITY_ZONE", e);
            }
        }
    }
}

Changing Configuaration and Tuning the Backup Filters

This section describes how to change the config to call the lifecycle bean and configure the backup filters to use the availablity zone attribute.

For more information, see the Class ClusterNodeAttributeAffinityBackupFilter section. Note the uses of cache templates in the example below:

<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">

      <property name="lifecycleBeans">
         <list>
            <bean class="AvailabilityZoneLifecycleBean"/>
         </list>
      </property>

       <property name="cacheConfiguration">
       <list>
           <bean id="cache-template-bean" abstract="true" class="org.apache.ignite.configuration.CacheConfiguration">
               <property name="name" value="backupFilterTemplate*"/>

               <property name="affinity">
                   <bean class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">
                       <property name="affinityBackupFilter">
                           <bean class="org.apache.ignite.cache.affinity.rendezvous.ClusterNodeAttributeAffinityBackupFilter">
                               <constructor-arg>
                                   <array value-type="java.lang.String">
                                       <!-- Backups must go to different AZs -->
                                       <value>AVAILABILITY_ZONE</value>
                                   </array>
                               </constructor-arg>
                           </bean>
                       </property>
                   </bean>
               </property>
           </bean>
       </list>
   </property>

Creating a Custom Image Containing the Lifcycle Bean

    FROM gridgain/community
    COPY mylib.jar /opt/gridgain/libs
    docker build .

Deploying Using GridGain Operator

Update the cluster configuration to include the config mentioned above;
Change the cluster_image to the custom one;
Deploy the operator using the custom credentials that are configured in steps 1 and 2.

Deploying GridGain without the Operator

Deploy GridGain per the AWS K8 deployment guide, using the custom image and config mentioned above.
Scale the cluster: You should be able to see that the nodes are allocated only in the availability zones you specified.

As shown in the example below, all gridgain nodes are deployed either to 2b or 2c, but not to 2a:

Verifying the Assigned Availability Zones (Optional)

Log into one of the pods:

kubectl exec -it gridgain-readiness-cluster-0 -n gridgain -- /bin/bash

Launch sql line:

cd /opt/gridgain/bin
./sqlline.sh --verbose=true -u jdbc:ignite:thin://127.0.0.1/

cd /opt/gridgain/bin
./sqlline.bat --verbose=true -u jdbc:ignite:thin://127.0.0.1/

Get the AVAILABLITY_ZONE attribute values

SELECT * FROM IGNITE.NODE_ATTRIBUTES where NAME like 'AVAILABILITY_ZONE';

© 2024 GridGain Systems, Inc. All Rights Reserved. Privacy Policy | Legal Notices. GridGain® is a registered trademark of GridGain Systems, Inc.
Apache, Apache Ignite, the Apache feather and the Apache Ignite logo are either registered trademarks or trademarks of The Apache Software Foundation.

Last updated on Dec 04, 2024