Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Anyone here want to support OpenSource project?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Anyone here want to support OpenSource project?

Hey folks,

I have been working on an OSS project called Marmot. It's a masterless SQLite replication system built on top of NATS. It's still early days (2 months hardly) but we already have a lot of traction from community and people already starting to contribute. I am looking forward to build a site and demo for public to actually see it in action, for which I've started looking around for various host providers. I so far have basic sponsorship from Digital Ocean which I believe will help us do the staging testing, but it won't give me a big enough cluster of nodes (I am looking for at least 5 node cluster, and I can combine it with DO to show cross DC replication). I wanted to check with LET community to see if somebody will be interested to support an OSS project in exchange for putting them in my sponsors list?

P.S. I am already using it in production on a small cluster of 3 but that I can't put up for demo because that thing is not owned by me. So I know it works, it's matter of scaling it out and building a real serious demo out of it.

Thanked by 1ralf

Comments

  • Why not at least start with providers who easily dole out $100 credits per sign up for typically 30/60 days?

    Vultr, Kupilot as well as Linode supplies this (or close enough). You might need to shell out a bit to get the wheels going but I sure you can get this within your communities fairly easily.

    Throw in Karmatera, ServerMania and a few others into the mix and I sure you got a fair bit of instrafacture for insanity cheap if not free for at least 30 days.

  • Not_OlesNot_Oles Moderator, Patron Provider

    @yoursunny said:
    Use your real account @Otus9051.

    @yoursunny Please share how you know it's Otus. Thanks!

  • @yoursunny said:
    Use your real account @Otus9051.

    Beat me to it

    Thanked by 1yoursunny
  • yoursunnyyoursunny Member, IPv6 Advocate

    @Not_Oles said:

    @yoursunny said:
    Use your real account @Otus9051.

    @yoursunny Please share how you know it's Otus. Thanks!

    It's based on deep learning, with 26.9% certainty.

  • @Not_Oles said:

    @yoursunny said:
    Use your real account @Otus9051.

    @yoursunny Please share how you know it's Otus. Thanks!

    C’mon

  • Not_OlesNot_Oles Moderator, Patron Provider

    @yoursunny said:

    @Not_Oles said:

    @yoursunny said:
    Use your real account @Otus9051.

    @yoursunny Please share how you know it's Otus. Thanks!

    It's based on deep learning, with 26.9% certainty.

    Your wonderful sense of humor often makes my day! Thanks!

  • @yoursunny said:

    @Not_Oles said:

    @yoursunny said:
    Use your real account @Otus9051.

    @yoursunny Please share how you know it's Otus. Thanks!

    It's based on deep learning, with 26.9% certainty.

    2x+0.9 where x is Otus’ age and 0.9 is the cross-entropy error factor. Deep.

    Thanked by 1BlaZe
  • @brzrkr said:
    I have been working on an OSS project called [Marmot]

    It sounds like an interesting project, but it does worry me that you consider it "production ready", even though the point of this post is so you can test it on more than 3 machines.

    The link to the FAQ doesn't take you to FAQs at all, instead some kind of kanban board, and the link to "For more details and internal workings of marmot" doesn't point to a working page. Hunting around the docs I found a document called internals.md which just has a title and a single sentence.

    From this, it's impossible to tell if you've considered any edge cases at all, which would make me incredibly nervous about building anything on this other than insert-only applications. There's a reason most systems are single-master - it's because it's an incredibly hard problem to solve otherwise.

    Take the most basic problem - what happens if two different instances have the same row updated to different things "at the same time" (i.e. before they've managed to sync with each other)? You don't seem to even mention this as a possible issue, and yet it's absolutely essential to know the behaviour of the system at this point. Do the databases diverge? Does one update just get discarded? Do the updates get merged if they affect different columns? How would you know if there's a logical relationship between columns in that case? How does a system detect if it's out of sync with the others? How does a system know if data has been lost?

    I'd be very interested to see how you'd attempt to answer some of these questions. Because if you can produce a working solution, you might have invented the next paradigm shift. But if you can't, you've just created another way for people's data to silently get lost / corrupted.

    PS don't take this negatively - I'm actually very interested in a solution. I'm not particularly keen on litefs, and wanted to work on my own solution in this space too at some point, but for now I planned to evaluate lifefs, because I think that right now, it is actually the best solution out of litefs, rsqlite, dsqlite etc. I'm very interested to see if you can solve some of these problems.

  • @ralf said: Take the most basic problem - what happens if two different instances have the same row updated to different things "at the same time" (i.e. before they've managed to sync with each other)? You don't seem to even mention this as a possible issue, and yet it's absolutely essential to know the behaviour of the system at this point. Do the databases diverge? Does one update just get discarded? Do the updates get merged if they affect different columns? How would you know if there's a logical relationship between columns in that case? How does a system detect if it's out of sync with the others? How does a system know if data has been lost?

    Marmot works by using a pretty basic trick so that each process that's access database can capture changes,
    and then Marmot can publish them to rest of the nodes. This is how it works internally:

    • Each table gets a __marmot__<table_name>_change_log that will record a every change via triggers to
      change log.

    • Each table gets insert, update, delete triggers in that kicks in AFTER the changes have been
      committed to the table. These triggers record OLD or NEW values into the table.

    When you are running Marmot process, it's watching for changes on DB file and WAL file. Everytime there is a change
    Marmot scans these tables to publish them to other nodes in cluster by:

    • Gather all change records, and for each record calculate a consistent hash based on table name + primary keys.
    • Using the hash decide JetStream and subject the change belongs to. And publish the change into that specific JetStream.
    • Once JetStream has replicated the change log, mark the change published.
    • As soon as change is published to JetStream rest of the nodes replay that log, and row changes are applied via state machine
      to local tables of the node. This means every row in database due to RAFT consensus at stream level will have only one
      deterministic order of changes getting in cluster in case of race-conditions.

    • Once the order is determined for a change it's applied in an upsert or delete manner to the table. So it's quite
      possible that a row committed locally is overwritten or replaced later because it was not the last one
      in order of cluster wide commit order.

    @ralf said: From this, it's impossible to tell if you've considered any edge cases at all, which would make me incredibly nervous about building anything on this other than insert-only applications. There's a reason most systems are single-master - it's because it's an incredibly hard problem to solve otherwise.

    You are right about single-master is simpler to implement. And if you look closely with partitioning technique on top of NATS JetStream this is exactly what each partition is. A log of changes ordered via RAFT and applied by nodes in order to replicate changes.

    @ralf said: The link to the FAQ doesn't take you to FAQs at all, instead some kind of kanban board, and the link to "For more details and internal workings of marmot" doesn't point to a working page. Hunting around the docs I found a document called internals.md which just has a title and a single sentence.

    Yes, I am in middle of getting the site up and running and putting all the documents together. That board is my personal kanban board that I am using to track work. Thanks for pointing out I will get it fixed ASAP.

    @ralf said: It sounds like an interesting project, but it does worry me that you consider it "production ready", even though the point of this post is so you can test it on more than 3 machines.

    Thanks for interest, the only reason I marked it production ready is because that particular version is just bidirectional live-stream of data and has been in production with just 3 nodes handling a very read heavy traffic (and moderate mixed ~131 writes/second), and has been operating for well over a month now, multiple restarts and nodes catching up (there is manual snapshotting RN which I am converting to automated snapshotting in 0.6.x). Once I've done that and I've really seen the two working fine in parallel for over 3 months only then I will declare that version stable since it's more complicated. But I see your point, there is chicken and egg problem here, other than testing on clusters myself I can't get this out, and other than community trying it I have no way of knowing if there are any bugs. I've committed to slowly keep working on it until I reach the point. Again open to feedback.

    Thanked by 2yoursunny AndrewL64
  • @brzrkr said: Once the order is determined for a change it's applied in an upsert or delete manner to the table. So it's quite possible that a row committed locally is overwritten or replaced later because it was not the last one in order of cluster wide commit order.

    This sounds pretty disastrous from a data-consistency point-of-view.

    Even if you accept that you need to do something in this scenario (which is why single-master avoids it entirely), if the operation to commit the data to the database has succeeded, the application has no way of knowing that the data has been silently changed underneath. The only safe thing to do is to reject the entire transaction, but of course with multiple-master, this becomes impossible too.

    But consider one of the simplest examples you'll find in any OOP textbook. Bank accounts.

    Transaction 1:
    UPDATE ACCOUNT SET BALANCE=BALANCE+50 WHERE USER=1;
    UPDATE ACCOUNT SET BALANCE=BALANCE-50 WHERE USER=2;

    Transaction 2:
    UPDATE ACCOUNT SET BALANCE=BALANCE+20 WHERE USER=2;
    UPDATE ACCOUNT SET BALANCE=BALANCE-20 WHERE USER=3;

    The whole reason sqlite has transactions is to ensure operations like this keep the database in a consistent state. If there's a conflict, the whole transaction has to be aborted, and the app knows it need to redo everything.

    With the system you described, user 2's balance could end up -50, +20 or -30.

    Now consider that you even find a solution for this in a single table, now consider that an transaction can affect multiple tables. For this example, you might have another table STOCK containing quantity of each item of stock. It's quite likely you'll have a conflict on the stock table but not the accounts table, for instance.

  • brzrkrbrzrkr Member
    edited October 2022

    Yes this is eventually consistent just like Cassandra, the goal from get go was AP from CAP. It will not suite the transactional workload, but one important thing to notice here is that as long as the system makes progress the nodes will converge to same value. So if you are sensitive to order and consistency don't use Marmot. Having said that I can do 90% of the work with high availability and really really speed up reads due to amazing optimizations SQLite has, and local file means no SPOF. This will also enable you to have serverless DB with lambdas that can converge and really really scale out on the edge.

    Hopefully this makes it clear, I've documented this on readme, but I will further improve documentation to make it clearer.

  • ralfralf Member
    edited October 2022

    @brzrkr said:
    Yes this is eventually consistent just like Cassandra, the goal from get go was AP from CAP. It will not suite the transactional workload, but one important thing to notice here is that as long as the system makes progress the nodes will converge to same value.

    You might want to re-read my example again.

    I'm certain all the nodes will converge to the same value. I'm also certain that there will be situations where all the nodes will converge to the same wrong value. Moreover, there will be no way to detect when your database becomes inconsistent.

    To be clear, say you had a whole load of nodes, for example 10. They all ran this pseudo-code:

    userA = rand(20);
    userB = rand(20);
    amount = rand(100)+1;
    sql: UPDATE ACCOUNT SET BALANCE=BALANCE+amount WHERE USER=userA;
    sql: UPDATE ACCOUNT SET BALANCE=BALANCE-amount WHERE USER=userB;
    

    Assume that at the start ACCOUNT has a record for every user, all at balance 0.
    If you ran this operation multiple times on a single node only, the following would always return 0:

    SELECT SUM(BALANCE) FROM ACCOUNT

    I guarantee that with the solution you have outlined, if you have multiple nodes executing this concurrently, the sum will not be 0 after a few hundred runs, even though it will be the same on all nodes. The problem will get worse as you have more latency between the nodes.

    Thanked by 2yoursunny martheen
  • why am i getting tagged here all of a sudden

  • yall think i really can code this high level 🤣

  • @Otus9051 said:
    yall think i really can code this high level 🤣

    it was a joke lmao

  • @Otus9051 said:
    why am i getting tagged here all of a sudden

    lmao ok

  • @Otus9051 said:
    yall think i really can code this high level 🤣

    Nah. It’s the ‘need a server for free for a project’ part.

    Thanked by 1yoursunny
  • @jmaxwell said:

    @Otus9051 said:
    yall think i really can code this high level 🤣

    Nah. It’s the ‘need a server for free for a project’ part.

    If I could really code I would have got a job by now.

  • @Otus9051 said:

    If I could really code I would have got a job by now.

    Get a job first, then outsource.

  • @Dazzle said:

    @Otus9051 said:

    If I could really code I would have got a job by now.

    Get a job first, then outsource.

    Damn that was one heck of an article

  • @brzrkr - just wondering if you've made much more progress in the last month. I check the repository and it doesn't look like much has changed. How are the scaling experiments going?

Sign In or Register to comment.