As technology rapidly progresses, we need increased computational speed and efficiency to drive digital transformation. Achieving this while working with big data requires a powerful and trustworthy solution.
Apache Ignite is a popular distributed database that supports growing speed and scalability demands while maintaining the distributed architecture. It achieves fast processing speeds by leveraging in-memory computing — the quickest way to compute data, as there’s no need to spend time on data transit. Since it’s a distributed database, you can have multiple clusters and nodes running to meet your growing demands.
In this article, let’s go through some real-world use cases that require high-performance computing. We’ll discuss how organizations began leveraging Apache Ignite’s capabilities to overcome and resolve their high-performance computing challenges.
Distributed Data Analysis in Healthcare
In recent years, biomedical data has become increasingly available. With so much data, it’s possible to compare entities — for example, healthy individuals versus those with a specific illness — to identify molecular similarities and differences. Extensive data helps us generate high confidence results, but it also comes with challenges.
Clinical data analytics platform, nference, faced a complex obstacle during development. The nference team was working to develop a computationally fast distributed system capable of accommodating requests for large amounts of data from a wide range of sources. As network data transmission can be slow compared to computation time, they sought a solution that used in-memory computing.
Ignite helped nference in multiple ways. First, Apache Ignite supports out-of-the-box partitioning and replication of data across nodes, making it possible to persist and distribute data across an unlimited number of cluster nodes. Second, Apache Ignite’s collocated processing can achieve distributed computing on the subset of data present in a local node, so no network time is needed. With these features, Apache Ignite helped nference divide both computation and data into N nodes to carry out in-memory computing. This lowered network transmission costs, fulfilling nference’s needs and goals.
Learn more about how nference leveraged Ignite’s capabilities to analyze biomedical data.
Synergy Between Speed and Efficiency in Transportation
Railway transportation requires some of the most complex computing methods — and any mistakes in minor details can be hazardous. For example, signals may be missed or necessary parties may not be informed of schedule changes or maintenance. Resulting problems can be as severe as a train accident or derailment.
These kinds of incidents are what Dutch Railways, a railway network with more than 10,000 train movements daily — spanning 400 stations — wanted to eliminate. Passenger safety was their priority, so they sought a faster, operable solution while keeping infrastructure changes and interoperability with other providers in mind. One of the most challenging goals was to keep the Feedback loop for planners within 10 seconds while performing hundreds of thousands of calculations.
To find a solution for such a complex problem, they turned to Apache Ignite. Apache Ignite offers distributed multi-tiered storage and high-performance computing APIs. This means it can both hold data and perform computations on the data in memory. Multiple nodes interconnect with each other to achieve this, unlike traditional systems where data is stored in a database separate from the servers that perform computation.
This hybrid and distributed behavior helps as it can bring calculations to the data, saving precious time and helping Dutch Railways achieve their 10-second goal. Ignite’s affinity co-location and caching capabilities allow the planner to calculate more than 50,000 movements across 7,000 km of railway track in a single plan. Also, its compute capabilities, along with hash functions as business logic, help find a perfect node to perform the calculations. Ignite’s indexing abilities and in-memory computation accelerate processing, too. And, artificial intelligence support helps to retrieve and predict anomalies to prevent hazardous situations that impact passenger safety.
Learn more about how Ignite helped Dutch Railways improve safety.
Interactive Historical Data Analysis in Manufacturing
As the availability of data increases, so should the capability to utilize it for insights to relevant stakeholders. According to TrendMiner, while data scientists are algorithmic experts, they lack domain-specific expertise. The inverse is also true: Domain experts have expertise in their domain, but they can’t mine meaningful data out of it.
TrendMiner democratizes analytics by trying to bridge a gap between data scientists and domain expertise. TrendMiner reads all industrial data which is mainly time-series data from manufacturing processes and stores them in histograms. Then, it analyzes trends from this data to generate meaningful information for the people with domain expertise. The end goal of TrendMiner is to provide descriptive and predictive analysis to users in visualized form.
The main challenge TrendMiner faced was performing calculations on more than 300 million data points per time series. Unfortunately, the read limit of most fast historian databases isn’t more than 5000 reads per second. And, regardless of these stats, the user expects to get the results in less than 1 second while maintaining higher resolution and advanced, insightful analytics.
TrendMiner 1.0, which wasn’t released using Apache Ignite, couldn’t achieve its goals. This is because file-based storage systems and algorithms could not be made scalable.
But with the release of TrendMiner 2.0, the product leveraged Apache Ignite’s in-memory store and compute capabilities to store fixed slices of data based on a particular time in nodes. Additionally, in their business logic, the TrendMiner team created an affinity model that stores all data related to a given time series in the same node, reducing the network time. Ignite’s computation APIs and ability to support nested multi-level prioritization helped them fulfill their expectations.
Learn more about how TrendMiner and Ignite enabled manufacturers to fully use their historical data.
Data-Driven Drug and Vaccine Discovery
According to Eroom’s Law, the rate of innovation in the drug discovery industry is decreasing, even though costs are massively increasing. But e-therapeutics, a U.K.-based drug discovery group, believes that it can reduce the cost with advanced computation power. The challenge here is that they need a computational grid that can perform thousands of analyses using multiple assumptions.
They used GridGain, an enterprise-level edition of Apache Ignite, which helped them reduce their analysis time to minutes (when it used to take weeks). They leveraged GridGain to perform high-performance transactions 1000 times faster than traditional transactions. GridGain’s API also helped them create a web-based interface for scientists and researchers to work without using the command line interface. It helped e-therapeutics run concurrent projects simultaneously, reducing the time and money spent on the projects.
Learn more about how Ignite and e-therapeutics helped researchers in their quest to further human health.
Exposure and Portfolio Management
When a major FinTech company wanted to create financial exposure management solutions, there were many factors to consider. Their team wanted to create a solution that was secure, distributed, and working in real-time. They needed to ensure it would maintain integrity while running thousands of concurrent transactions.
To find the solution to such a complex problem, they turned to Apache Ignite and GridGain. Apache Ignite’s computational capabilities can significantly improve the performance as no network communication is required. Additionally, Ignite’s developer-friendly SQL experience, with its ability to support indexes and complex JOIN operations, helped them manage exposure in a cost-friendly and effective way.
Learn more about how Ignite aided FinTech innovations.
Conclusion
Ignite’s distributed nature and in-memory computing ability can be a game-changer to companies dealing with big data in analytics, discovery, and exposure management. Learn more about Apache Ignite and GridGain to identify their capabilities and how real-world organizations have leveraged GridGrain to fulfill their needs and surpass their expectations.