All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Anyone here want to support OpenSource project?
Hey folks,
I have been working on an OSS project called Marmot. It's a masterless SQLite replication system built on top of NATS. It's still early days (2 months hardly) but we already have a lot of traction from community and people already starting to contribute. I am looking forward to build a site and demo for public to actually see it in action, for which I've started looking around for various host providers. I so far have basic sponsorship from Digital Ocean which I believe will help us do the staging testing, but it won't give me a big enough cluster of nodes (I am looking for at least 5 node cluster, and I can combine it with DO to show cross DC replication). I wanted to check with LET community to see if somebody will be interested to support an OSS project in exchange for putting them in my sponsors list?
P.S. I am already using it in production on a small cluster of 3 but that I can't put up for demo because that thing is not owned by me. So I know it works, it's matter of scaling it out and building a real serious demo out of it.
Comments
Why not at least start with providers who easily dole out $100 credits per sign up for typically 30/60 days?
Vultr, Kupilot as well as Linode supplies this (or close enough). You might need to shell out a bit to get the wheels going but I sure you can get this within your communities fairly easily.
Throw in Karmatera, ServerMania and a few others into the mix and I sure you got a fair bit of instrafacture for insanity cheap if not free for at least 30 days.
Use your real account @Otus9051.
@yoursunny Please share how you know it's Otus. Thanks!
Beat me to it
It's based on deep learning, with 26.9% certainty.
C’mon
Your wonderful sense of humor often makes my day! Thanks!
2x+0.9 where x is Otus’ age and 0.9 is the cross-entropy error factor. Deep.
It sounds like an interesting project, but it does worry me that you consider it "production ready", even though the point of this post is so you can test it on more than 3 machines.
The link to the FAQ doesn't take you to FAQs at all, instead some kind of kanban board, and the link to "For more details and internal workings of marmot" doesn't point to a working page. Hunting around the docs I found a document called internals.md which just has a title and a single sentence.
From this, it's impossible to tell if you've considered any edge cases at all, which would make me incredibly nervous about building anything on this other than insert-only applications. There's a reason most systems are single-master - it's because it's an incredibly hard problem to solve otherwise.
Take the most basic problem - what happens if two different instances have the same row updated to different things "at the same time" (i.e. before they've managed to sync with each other)? You don't seem to even mention this as a possible issue, and yet it's absolutely essential to know the behaviour of the system at this point. Do the databases diverge? Does one update just get discarded? Do the updates get merged if they affect different columns? How would you know if there's a logical relationship between columns in that case? How does a system detect if it's out of sync with the others? How does a system know if data has been lost?
I'd be very interested to see how you'd attempt to answer some of these questions. Because if you can produce a working solution, you might have invented the next paradigm shift. But if you can't, you've just created another way for people's data to silently get lost / corrupted.
PS don't take this negatively - I'm actually very interested in a solution. I'm not particularly keen on litefs, and wanted to work on my own solution in this space too at some point, but for now I planned to evaluate lifefs, because I think that right now, it is actually the best solution out of litefs, rsqlite, dsqlite etc. I'm very interested to see if you can solve some of these problems.
Marmot works by using a pretty basic trick so that each process that's access database can capture changes,
and then Marmot can publish them to rest of the nodes. This is how it works internally:
Each table gets a
__marmot__<table_name>_change_log
that will record a every change via triggers tochange log.
Each table gets
insert
,update
,delete
triggers in that kicks inAFTER
the changes have beencommitted to the table. These triggers record
OLD
orNEW
values into the table.When you are running Marmot process, it's watching for changes on DB file and WAL file. Everytime there is a change
Marmot scans these tables to publish them to other nodes in cluster by:
As soon as change is published to JetStream rest of the nodes replay that log, and row changes are applied via state machine
to local tables of the node. This means every row in database due to RAFT consensus at stream level will have only one
deterministic order of changes getting in cluster in case of race-conditions.
Once the order is determined for a change it's applied in an upsert or delete manner to the table. So it's quite
possible that a row committed locally is overwritten or replaced later because it was not the last one
in order of cluster wide commit order.
You are right about single-master is simpler to implement. And if you look closely with partitioning technique on top of NATS JetStream this is exactly what each partition is. A log of changes ordered via RAFT and applied by nodes in order to replicate changes.
Yes, I am in middle of getting the site up and running and putting all the documents together. That board is my personal kanban board that I am using to track work. Thanks for pointing out I will get it fixed ASAP.
Thanks for interest, the only reason I marked it production ready is because that particular version is just bidirectional live-stream of data and has been in production with just 3 nodes handling a very read heavy traffic (and moderate mixed ~131 writes/second), and has been operating for well over a month now, multiple restarts and nodes catching up (there is manual snapshotting RN which I am converting to automated snapshotting in
0.6.x
). Once I've done that and I've really seen the two working fine in parallel for over 3 months only then I will declare that version stable since it's more complicated. But I see your point, there is chicken and egg problem here, other than testing on clusters myself I can't get this out, and other than community trying it I have no way of knowing if there are any bugs. I've committed to slowly keep working on it until I reach the point. Again open to feedback.This sounds pretty disastrous from a data-consistency point-of-view.
Even if you accept that you need to do something in this scenario (which is why single-master avoids it entirely), if the operation to commit the data to the database has succeeded, the application has no way of knowing that the data has been silently changed underneath. The only safe thing to do is to reject the entire transaction, but of course with multiple-master, this becomes impossible too.
But consider one of the simplest examples you'll find in any OOP textbook. Bank accounts.
Transaction 1:
UPDATE ACCOUNT SET BALANCE=BALANCE+50 WHERE USER=1;
UPDATE ACCOUNT SET BALANCE=BALANCE-50 WHERE USER=2;
Transaction 2:
UPDATE ACCOUNT SET BALANCE=BALANCE+20 WHERE USER=2;
UPDATE ACCOUNT SET BALANCE=BALANCE-20 WHERE USER=3;
The whole reason sqlite has transactions is to ensure operations like this keep the database in a consistent state. If there's a conflict, the whole transaction has to be aborted, and the app knows it need to redo everything.
With the system you described, user 2's balance could end up -50, +20 or -30.
Now consider that you even find a solution for this in a single table, now consider that an transaction can affect multiple tables. For this example, you might have another table STOCK containing quantity of each item of stock. It's quite likely you'll have a conflict on the stock table but not the accounts table, for instance.
Yes this is eventually consistent just like Cassandra, the goal from get go was AP from CAP. It will not suite the transactional workload, but one important thing to notice here is that as long as the system makes progress the nodes will converge to same value. So if you are sensitive to order and consistency don't use Marmot. Having said that I can do 90% of the work with high availability and really really speed up reads due to amazing optimizations SQLite has, and local file means no SPOF. This will also enable you to have serverless DB with lambdas that can converge and really really scale out on the edge.
Hopefully this makes it clear, I've documented this on readme, but I will further improve documentation to make it clearer.
You might want to re-read my example again.
I'm certain all the nodes will converge to the same value. I'm also certain that there will be situations where all the nodes will converge to the same wrong value. Moreover, there will be no way to detect when your database becomes inconsistent.
To be clear, say you had a whole load of nodes, for example 10. They all ran this pseudo-code:
Assume that at the start ACCOUNT has a record for every user, all at balance 0.
If you ran this operation multiple times on a single node only, the following would always return 0:
SELECT SUM(BALANCE) FROM ACCOUNT
I guarantee that with the solution you have outlined, if you have multiple nodes executing this concurrently, the sum will not be 0 after a few hundred runs, even though it will be the same on all nodes. The problem will get worse as you have more latency between the nodes.
why am i getting tagged here all of a sudden
yall think i really can code this high level 🤣
it was a joke lmao
lmao ok
Nah. It’s the ‘need a server for free for a project’ part.
If I could really code I would have got a job by now.
Get a job first, then outsource.
Damn that was one heck of an article
@brzrkr - just wondering if you've made much more progress in the last month. I check the repository and it doesn't look like much has changed. How are the scaling experiments going?