IoT, Streaming Data, and In-Memory Computing

Gartner forecasts that there will be over 8 billion connected devices in use worldwide by the end of 2017. This figure is estimated to reach 20 billion by 2020 and continue to increase dramatically for the foreseeable future. According to the “Cisco Global Cloud Index: Forecast and Methodology, 2015–2020”, the total amount of data created by devices, driven by IoT, will reach 600 ZB per year by 2020, up from 145 ZB per year in 2015. These devices capture vast quantities of data, and it takes a unrivaled performance and scale provided by in-memory computing to turn this ocean of data into valuable insight and action.

IoT, Streaming Data, and In-Memory Computing

Examples of IoT Use Cases

According to 451 Research, 65% of companies are using IoT. 69% of organizations gather data from end points and 94% of those companies use it for business purposes. The highest usage is among Utilities (92%) and Manufacturing (77%).

The IoT data comes from:

  • Datacenter IT Equipment (51%)
  • Cameras and Surveillance Equipment (34%)
  • Smartphones and End Users (30%)
  • Buildings and Other Structures (21%)
  • Environmental Sensors (15%>
  • Factory Equipment (14%)
  • Automobiles/Fleet Equipment (11%)
  • Retail Operations (8%)
  • Medical Devices (7%)

Consider some examples of IoT use cases that demand fast data processing and the resulting real-time insights to drive actions:

  • Healthcare organizations are implementing both hospital and residential-based patient monitoring, as well as environmental monitoring
  • Manufacturers are using sensors to track production, streamline industrial processes, and monitor inventory in industries ranging from food to logistics
  • Self-driving vehicles will produce and consume massive amounts of data while determining optimal routes and minimizing fuel usage
  • Ride-sharing companies  are using apps to broadcast anonymous information such as location and traffic 
  • Connected home products include security devices, thermostats, refrigerators, robots, and garden sensors

In most of these use cases, the processing and analysis of IoT data must occur in real time to trigger appropriate action. If processing and analysis do not take place in a timely fashion, then the benefits of IoT initiative cannot be fully realized.

In-Memory Architectures and Streaming Data Analysis

Many businesses building infrastructures capable of the speed and scale required for IoT rely on hybrid transactional and analytical processing (HTAP) solutions. In-memory HTAP uses a single data store that resides in memory for both transactions and analytics. HTAP can be used to combine analytics of IoT related streaming data with operational data sets to provide critical business insight.

Building an HTAP environment in memory requires a full-featured in-memory computing platform with the following components:

  • An in-memory data grid for distributed data storage and scalability
  • An in-memory compute grid for high performance parallel processing of queries
  • In-memory streaming processing to allow analysis of incoming data in real-time
  • An in-memory SQL grid that provides in-memory database capabilities including support for SQL communications through ODBC/JDBC and may provide geospatial support for location-based IoT use cases

These interconnected components work together to provide the speed and scale needed to gain insight from the non-stop influx of data as it streams from IoT devices. Together, these components create an integrated, scalable, high-performance in-memory computing platform for IoT applications to process large volumes of streaming data in real time.