we are looking for an opensource in memory database which can help indexes.
In-Memory Databases. An SQLite database is normally stored in a single ordinary disk file. However, in certain circumstances, the database might be stored in memory. The most common way to force an SQLite database to exist purely in memory is to open the database.
The use case is usually that we have got lot of items that are going to develop in a huge method.Each item has a few fields on which we need to question. Currently we shop the information in program's memory. However with growing data, we have to believe about disseminating/sharding the db.
We have looked at a few options
- Redis clustercould end up being used, but it does not have the idea ofindexes or SQL like inquiries.
- Apache Igniteis certainly both in-memory, and dispersed as well as providesSQL questions. However, the issue is that ignite fire all inquiries into all get good at nodes, so that the final result will end up being slower than the slowest of those inquiries. It seems like a problem because a non carrying out/slow node out of a number of nodes can really decrease down the program a lot. More in ignite, scans are accomplished from the masters and slaves are not used, so that it will be challenging to level the questions. Escalating the nodes will possess negative influence as the no of queries will enhance and it will end up being also slower.
- Cassandra- The in-memory choice in cassandra can be used, but itseems that the maximum dimension of a desk per node can be 1 GB. If our desk is more than 1 Gigabyte, we will have got to holiday resort to dividing which will inturn prospect cassandra to create several queries(one per node) and it is a problem(exact same as ignite). Not certain whether scans in cassandra in-memory desk can end up being scaled by growing the number of slaves.
![Open source in-memory database cache Open source in-memory database cache](http://altibase.com/wp-content/uploads/2018/09/Untitled-design-28-1030x539.png)
We are open to additional solutions but wanting to know whether the multi-query will end up being a issue everywhere(such as hazelcast).
The ideal answer for our use situation would become an in-memory database with indexes which could end up being learn scaled by raising the amount of slaves. Making it distributed/sharded will lead to several concerns and we are reluctant because one erring node could decrease the entire system straight down.
TucoTuco26911 yellow metal badge66 sterling silver badges1616 bronze badges
shutas off-topic by meagar♦April 31 '17 at 0:44
This issue appears to become off-topic. The customers who voted to close up gave this specific cause:
- 'Queries wondering us torecommend or discover a book, tool, software program library, short training or other off-site sourceare off-topic for Stack Flood as they tend to bring in opinionated answers and junk e-mail. Instead, describe the problem and what has been carried out so considerably to resolve it.' - meagar
3 Solutions
Hazelcast supports indexes (sorted amp; unsorted) and what will be importantthere can be no Multi-Query issue with Hazelcast.
Hazelcast facilitates a
PartitionPredicate
that restricts the performance of a question to a node that can be a primaryReplica of the essential passed to the constructor of thePartitionPredicate
. Therefore if you know where the data resides you can just concern this node. So no want to repair or put into action anything to support it, you can use it right aside.It'h probably not reasonable to make use of it all the time. Depends on your use-case.For complex questions that check out a great deal of data but come back small outcomes it's better to make use of
tom.bujokmary.bujokItem
inMemoryFormat. You should obtain excellent delivery situations and reduced latencies.1,24411 magic logo1111 metallic badges1919 bronze badges
Disclaimer: I am GridGain worker and Apache Ignite committer.
Several responses on your problems:
1) Sluggish nodes will guide to problems in virtually any clustered environment, therefore I would not really think about this as drawback. This is actuality you should accept and accept. It will be necessary understand why it is certainly gradual and repair/upgrade it.
2) Ignite are usually capable to carry out scans from slaves both for normal cache operations 1 and for SQL queries performed over Duplicated caches. In reality, using REPLICATED cache for referrals data is one of the nearly all important features permitting Ignite to level effortlessly.
3) As you correctly mentioned, presently query is certainly broadcasted to all data nodes. We are heading to improve it. First, we will allow customers to designate dividers to perform the question against 2. 2nd, we are usually heading to enhance our optimizer so that it will consider to compute target information nodes in progress to prevent broadcast 3, 4. Both enhancements will become released really soon.
4) Final, but not least - constant layer will become launched in various a few months 5, signifying that Ignite will turn out to be distributed database with both in-memory and perseverance features.
1 https://ignite.apache.org/produces/mobile/org/apache/ignite/configuration/CacheConfiguration.html#isReadFromBackup
2 https://issues.apache.org/jira/search/IGNITE-4523
3 https://issues.apache.org/jira/browse/IGNITE-4509
4 https://issues.apache.org/jira/browse/IGNITE-4510
5 http://apache-ignite-developers.2346864.n4.nabble.com/GridGain-Donates-Persistent-Distributed-Store-To-ASF-Apache-Ignite-tc16788.html
![Open Open](/uploads/1/2/5/6/125687117/702259683.png)
Vladimir OzerovVladimir Ozerov
I can give opinions on cassandra. Maximum dimension of your desk per node is usually configurable and tunable so it depends on the quantity of the memory that you are prepared to pay out. Partitioning is usually constructed in into cassandra so basically cassandra manages it for you. It'beds relatively easy to do paritioning. Fundamentally first part of the primary key syntax is partitioning key and it decides on which node in the cluster the data life.
But I furthermore speculate you are aware of this since you are usually mentioning several concern per node. I imagine there is usually no great method around it.
Just one slight remark generally there is no get better at slaves in cassandra. Every node will be equal. Basically client requests any node in the bunch, this node after that becomes planner nodes and since it gets partitioning key it knows which node to consult the information for and it provides it after that to the customer.
Other than that I imagine you learn upon cassandra good enough (from what I can discover in your query)
Generally it comes lower to the gain access to design, if you understand how you are heading to access your data after that it's the way to move. But various other databases are also quite decent.
Indexing with cassandra generally hides some possible performance complications. Usually individuals prevent it because in cassandra list provides to be construct for every report there is on entire cluster and it't done per node. This doesn'testosterone levels really scale. Fundamentally you always have to perform query first no matter how ypu put it with cassandra.
Plus the in memory appears to be component of the DSE cassandra. Not the open source or neighborhood a single. You have to take this into account also.
Marko ŠvaljekMarko Švaljek