New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
What's the most lightweight server side language for mysql server?
This discussion has been closed.
Comments
At 100k req/second, an access log is not practical. At about 150 bytes per connection, it would be 15MB per second of access.log. If it's just votes, then an
UPDATEinto a key value store makes more sense. Would suggest an in-memory DB with regular backups (Eg: https://sqlite.org/backup.html)Edit: If it was just votes, then you are just updating a counter in memory for each request. That would be a few hundred bytes in total.
Inserting will be a breeze, but parsing/etc.. not going to be great. Especially when someone realizes you are using JavaScript as a Language (JaaL), and beats you to death.
That's not a stupendous amount of logs compared to the web traffic (think of how many hits/sec google analytics gets, and they log a lot more than 150 bytes per hit) but yeah, it helps if it's spread across multiple servers.
Simply recording the number of votes takes almost no resources so that's easiest if it does what you need. OP is being very quiet about the actual requirements of this project.
Because the project is very simple.
It's just a vote. People go to the site. Input their User ID, and their vote, that's it. I don't need to verify whether the id is valid or not. I don't need to verify whether he already vote or not.
I can process the data later to remove the duplicate / invalid ID.
Imagine president election at its simplest form.
Flat file on tmpfs? Let's see, 10 billion votes max. 8 byte unsigned int64 for ID, 1 byte vote type, 1 byte line feed (optional).
10,000,000,000 x 10 bytes = 100,000,000,000 / 1024^3 = 93 GB total storage required. Just need a 128 GB server and hope it doesn't go down before tallying up the votes. nvme would work too if you wanted a durable format. Theoretically that's what 16 GB/second throughput?
Would advise locking the file to prevent race conditions. Once that's sorted start subtracting numbers from your performance figures to represent overhead from additional layers (SSL, frontend, etc).
Edit: I feel like this is a homework assignment.
Nothing with that much traffic is ever simple. It also can't be seen as a pure software problem: there is probably more network engineering than software involved. That's why there were questions about how you were hosting it, what your budget is, whether you have to record individual votes or just count them (you finally told us that just now), etc. How many total votes do you expect per election? How often will you run these elections? How reliable does the system have to be (affects whether you need realtime replication etc.)? Are the users all in the same geographic region? Yes this does begin to sound like a homework or interview question.
That's why it always helps to describe the actual application, e.g. "I'm working for the XYZ widget company that wants to poll users for widget preferences during livecasts" or "I'm working for the Swedish government and am organizing online parliamentary elections" or whatever.
What is the actual application? Who is it that wants this? If it's homework that's fine, just say so.
@yokowasis
Pardon me but, no, it's not a vote. It's willy-nilly collecting whatever happens to come along as "vote". But that's your problem; I don't care
Based on your "specification" I'd suggest to simply memory collect what comes in and to write it out from time to time, e.g. every second.
If you use multithreading (1 thread per avail. core) I'd use a very simple "scheduler", something like, e.g. 4 cores: current second mod 4 == 0 -> T1, == 1 -> T2 ... == 3 -> T4. Why? Because with a very high load microseconds count and you want an extremely simple and fast scheduler for the threads writing out their collected data.
Re. the format I would not play with json. Reason: Again, avoid any complexity and processing not needed. You have just 2 pieces of data, the vote (reducible to 0 or 1), and the user id, so you can write out the votes simply as 1 byte vote ('0' or '1' or 'Y' or 'N' plus x bytes user id plus newline, which as a nice side effect is also human readable and easy to post process. Hint: Do not use [f|s]printf or similar for writing '0','N','1','Y'.
No, not "might be" but definitely "is". And no, http/2 will not help a lot as the session with the back end consists only of session est. + 1 data packet (the GET or POST req), only 1 of which would need to handled by the back end app. As for TLS 1.3 most likely there isn't much choice anyway because the TLS version used will be what it happens to be (chosen by the client) unless one told the server to only use 1.3 which however is largely theory because that would exclude many clients/"voters".
If he wants https, no matter what version, and he doesn't have a big fat proxy,CDN, whatever de-TLSing and forwarding plain http he can forget about 100k+ req/s anyway.
_(Sorry Cloud%§$& forced me to split it)
_
No, no need, based on what @yokowasis said about his needs. Considering the high load (req/s) he should absolutely avoid any and all not vitally needed layers. Also, as a side note, keep in mind that fast hashing mechanisms often have non neglegible setup cost, i.e. they are fast only when hashing, say, > 128 or more byte strings.
"logging* - Unless he absolutely needs it I would in fact disable it. Reason: Logging can severely slow down a system when load is very high. I once had to design/build a system (on unix) doing 500k+ of connections/s and any kind of traditional logging eat up a very significant part of my cycles. In particular timestamp and IP printing had to be done by hand made specialized routines to bring it into an acceptable range.
"GET vs. POST" and "http server logging" - short version Do not do that! It will dramatically slow down your solution.
Instead just make your back end in nodejs or go back end look and behave like a (simple http/1) web server who knows only 1 single request (GET or POST, as you like) to 1 URI (that of your voting engine).
Reason: A normal http server has to do - and be ready to do - a lot of diverse things; being a jack of all trades it can't possibly be as fast as a specialized server. Plus, you pay for all those unneeded capabilities. All @yokowasis needs (according to his own spec) is basically a variation of a "hello world" http server, one that responds with "Your vote has been counted" or similar instead of "hello world" and that handles a particular fixed URL, collects the votes (and user ids) and writes them out every second or so, done.
Based on (just little) experience with internet voting systems I'd also consider an additional point not mentioned so far: diverse attempts to rig the votes, e.g. by scripts voting every second for hours (of which again there might be many). Being extremely simple and tailor fit to the task as well as being designed for speed is useful in that regard too.
This is a standard question, and TechEmpower runs regular benchmarks to see who is on top. You can even grab their test apps from github.
https://www.techempower.com/benchmarks/#section=data-r17&hw=ph&test=fortune
https://www.techempower.com/benchmarks/#section=data-r17&hw=ph&test=update
Note the database for both winners above is Postgres.
The wild thing is that #2 on the fortune benchmark is written in ... Java.
What do you want to vote? Why not for example build a simple Telegram bot api?
How is telegram relevant in my case?
Why are you crawling through + necroing > 1 month old threads?