All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
What's your forum software wish list?
Sometimes I mull over the idea of writing yet another forum program, since all the existing ones suck. What kind of wish list would you have for something like that? Some of my own ideas (not in any particular order):
Old-school presentation (like reddit, craigslist etc), site should work with no javascript, and not too much Discourse-like browser wizardry even if JS is enabled.
Fast fast fast, both in terms of performant server code and in serving minimal junk to your browser.
Thread ownership: depending on subforum settings by admin, anyone who starts a thread can moderate that thread (or that could be limited to established users, etc).
User-created subforums or isolated forums (like subreddits)
Good search functionality
Cross-server replication so you can have multiple endpoints.
Very private, encrypted PM system
Does anyone even care about this stuff any more? I've been a user of many forums over the years, but have never operated one from the admin side, so I'm sure there's a lot that I don't know.
Do you care much about RAM footprint? What's the biggest forum (# of users, # of posts per day) you could see yourself personally operating?
My basic target would be to run something LET sized on a typical midrange VPS, i.e. 0.5 or 1GB of ram and about as much SSD as the total amount of message data including archives. That's as opposed to targeting tiny VPS's though that's open for discussion. The fairly high specs are for performance reasons (ram for caching active threads, and SSD for the IOPS-heavy search indexes). The code would not be bloaty 
.
Comments
From a usability point of view I HATE Discourse. I want to be able to jump to any point on the page without having to wait for it to load, I also don't like all the flashy stuff like badges.
Vanilla (when it doesn't have quirks) is my ideal platform. Loads quick, everything loads at once and minimal ajax (only when it actually improves the user experience).
Some notes on your ideas that I didn't cover above:
Thread ownership: depending on subforum settings by admin, anyone who starts a thread can moderate that thread (or that could be limited to established users, etc).
As you mentioned with established users this would work. Having them meet minimum requirements like post count, visits & time registered.
User-created subforums or isolated forums (like subreddits)
This gets messy really quickly. Most communities have a primary objective, the rest of the threads go into a "everything else" category. This helps keep undesirable content out, if you allow users to create sub forums with their own permissions it will quickly go bad. It's like offering them free hosting.
Good search functionality
Very important, even just support for integration into current search stacks would be enough.
Cross-server replication so you can have multiple endpoints.
Making this easy is always a plus but people will figure out how to do it either way if they require it.
Very private, encrypted PM system
In for a world of hurt. Never deal with two way encrypted messaging between 3rd parties unless you are absolved from responsibility by law.
If you let google crawl your forum, this won't be too big a problem, right?
My wish list:
Trewq - your points are all good. The user-created subforum idea would be more for multi-topic forums like reddit. It could also be good for hosted installations that people pay to use, or (I hate this but it's how most of the web works) were supported by advertising.
Chihcherng, yes, good mobile usability is a must. I like to think that's most easily accomplished by using very simple formatting, but maybe I'm wrong about that.
3rd party auth (OpenID) is a reasonably easy feature if people want it, so ok (I don't use it myself).
I'd rather not rely on Google search and it also can't handle queries like "give me posts in a certain date range" which I think is useful on forums. "Give me the results most recent first" is also important, and Google doesn't do that. And lastly I'd want to be able to make subforums blocked from google search or only available to logged in users, while still being searchable with the forum search functions.
You really need to understand administration side of operating forums to properly develop a forum software. That's why Xenforo does it so well.
I have been working and optimising server end performance with some of the largest forums on the internet (mainly vbulletin since 2003 and xenforo since 2014) and it ultimately comes down to those 3 factors (Scalability, performance and security) from admin perspective. IIRC large forums I have worked on were around 60,000+ concurrent members online over 15 min session intervals and around 100-150 million posts and between 500,000 to 1.5 million unique ip visitors/day and 450+ GB of attachments.
Xenforo is what I use now and and definitely would be my go to
Some of my thoughts:
There are many mature forum software packages out there these days. If this endeavour will mostly be a one man job, it'll probably be a long time before you reach anywhere near feature parity, unless your aim is to be something very different, or you're some super-genius coder.
Very good.
To be honest, most forums out there don't have anywhere near enough traffic for (back-end) performance to really matter. Performance only really matters to the larger forums out there (and possibly LET users, but we're kinda a niche), but they'll usually be on beefier hardware and have people who know how to tweak things to make it handle the load. Also, large forums are less likely to go to the hassle of migrating to a different forum software, so it'll be some time before you get anyone with a sizable forum.
Most forum software runs on a PHP/MySQL stack, which isn't particularly known for performance, which probably gives you an indication of how much it really matters.
Front-end (loading) performance is always good, but that's also covered by your first point.
Rather niche. Reddit (and it's various clones) does this already, why would anyone try to compete with something way more established?
Most forums are fairly focused, so probably won't use this anyway. If this is going to be a special point of your software, then so be it, but just pointing out that this is likely not as useful as you may think it is.
You mean integration with ES/Sphinx? I'd honestly be suspicious if you tried to implement your own, assuming that you believe you can do better than existing popular solutions.
Something beyond typical database-level replication or clustering?
This seems like it'd be difficult to implement on a forum system. I'm not sure of your planned implementation details, but it could also be problematic, for example, when you need to moderate PMs (e.g. spam or abusive users).
For performance concerns, I'd say optimised DB queries (fast, non-problematic locking, and having few queries overall) would be the number 1 concern for high traffic forums, since it'll likely be your bottleneck. RAM would follow, then CPU usage.
Being able to integrate with reverse-proxy caches would be interesting (e.g. nginx caches pages, and the forum script purges the cache when updates happen).
A big one you've missed is customisability/extensibility. Every forum owner out there wants to make their forum unique, distinguished from the countless others out there. Particularly with social media these days, which covers a lot of generic stuff, forums do have to be focused and targeted, so a system which can be tailored to custom needs is quite important.
Eva2000 and xyz, thanks very much for the detailed thoughts. Again all the points are well taken. Regarding some specifics:
I would love to see a templating system that...
Do you have any existing favorites that I can look at?
Do you think anything like feature parity is important or worthwhile? One thing I hate about most forums is that they have too many features (Discourse is a big villain in that regard). The ones I use the most are LET, Reddit, and Hacker News, all of which have relatively few features, so I'm imagining something pretty light in terms of features.
You mean integration with ES/Sphinx?
I was thinking of maybe Xapian or CLucene but ES/Sphinx sounds like an option. I ran a Solr cluster for a while and Solr would work too, if you have the ram for it. I do sometimes think of implementing a search engine, come to think of it
. But that would be a big project in its own right, not part of a forum.
Most forum software runs on a PHP/MySQL stack, which isn't particularly known for performance, which probably gives you an indication of how much it really matters.
But even smaller forums are often pig slow, which is one of the persistent annoyances that makes me think of writing a fast one
. I'm partly inspired by Hacker News (written in Lisp and I think it serves active threads directly from ram with no database) and OKWS (okws.org, the C++ program behind OK Cupid). OKWS supposedly gave OK Cupid a significant financial advantage over competitors, since it let them run a huge site on modest hardware compared to what the typical PHP/MySQL (or worse, Rails) monstrosity would have needed. There have also been some big forums built around decades-old Usenet backends. Hmm. I also want to look into how Stack Exchange works.
ngx/mod_pagespeed looks interesting though I think it looks like those rewrites could be done statically? I hope that's how it works instead of rewriting whenever a page is served. Maybe basic page assembly can be done directly in nginx with the lua plugin (openresty?). More hmm.
Something beyond typical database-level replication or clustering?
Probably not, or anyway I don't have anything specific in mind. I wasn't thinking multi-master was necessary if that's what you meant.
Encrypted PM's is easy enough technically, but xyz and Trewq make good points about the potential for administrative hassle and abuse. So this would at most be a configurable option.
A robust scalable and distributed attachment/media storage layer would be useful.
Interesting, ok, will add it to list, including image proxying and caching, origin IP protection, etc. Thanks.
Wow, that's a lot
. How many new messages posted in a busy 15 minute period? How many active threads (most recent post less than 24 hours old? Less than 1 hour old?) How many posts and how many total bytes of text in the largest threads? What is the server setup? Do you think it's reasonable at all to expect forum admins (even for such large forums) to know how to tune SQL servers? I'd hope the software could work out of the box without needing tuning, beyond picking the server size based on the traffic level, so it could be put in a VPS template that you could spin up without much head scratching.
Well there was a forum I liked that moved to proboards.com (which sucks), so I liked the idea of giving an alternative to that. Reddit itself is pig slow--they deal with that by throwing more hardware at it, and it still gets wedged many times a day. I know there's a clone written in C# (voat.co uses it) but haven't checked into other ones. Stack Exchange (mentioned earlier) is also interesting, and blogs with comments (Wordpress, Slashdot) could also be seen as a type of forum. This is blue-sky of course.
I don't have as much feedback as some of the fantastic stuff you have already had however, the thing that I have ran into a number of times with a number of forums is the migration problem, it would be great to have a table mapping type migration system.
For example the LES forum is running on FluxBB which I chose because its so light weight but soon ran in to its limitations, without jumping through 100 hoops and being prepared to lose certain things I cant migrate it to something different.
probably not the best feedback but an import system that is both generic and flexible at that same time would be great.
Anthony, thanks, yes, migrating from other software is important and I had it in mind. What limitations did you hit with FluxBB and what are the migration obstacles? Is there much data in the LES forum that's not visible through the web interface or at worst MySQL? I'd like to think it wouldn't be too bad to read in a mysqldump and convert it. Starting with a web scraper would be messier and probably lose some things, but doesn't seem out of the question either.
This is from memory but it was things like avatars, signatures, passwords were blanked and forced resets, some formatting loss, some of the direct importers only supported a maximum number of posts (never figured out why).
I guess if you have experience with getting really hands on with this sort of thing its not to hard to do manually but its never been quite important enough to dedicate a few days of work to and certainly not important enough to consider writing an automation script for.
The funny thing is that the people behind FluxBB stopped work on fluxBB a while ago and maintain it only i.e. no improvements, they started developing Farum instead which looks great, but even they have not written an importer/migration tool about 2 years later.
@AnthonySmith do you mean flarum?
I personally feel the era for the old, traditional, forum is well over.
But a mobile-centric forum like Discourse is a let-down for me. It's over engineered. Vanillia is somewhere in between and I like it more than others.
Anthony I'm willing to take a look at migrating the LES forum. Let me know.
Yep.
yeah ability to migrate forums via importers is important but very hard to do as you'd have to be familiar with everyone elses forum software first ! So I'd focus on vbulletin, xenforo and invision board as majority of forum market are these 3 still. Probably look into those 3 software's current importers to see how things work.
none specially off the top of my head right now
Elasticsearch or Sphinx would be ideal for enhancing search as it's 2 most common with some forums like vBulletin, Invision Board and Xenforo so you have forum owners on most popular forum platforms already having the requirements.
Another suggestion is to have appropriate logging for failed log in attempts so folks can setup custom rules in modsecurity and fail2ban implementations to block bad actors/requests.
Also think about how cookies are setup so you have a clear distinction for logged in versus logged out cookie and not a cookie setup that persists after someone has logged in and logged out so makes caching via cookies i.e. Varnish Cache or Nginx proxy_cache or PHP-FPM fastcgi_cache easier to implement. Some forums required folks to add via addon/plugin additional cookies for members at login to use to determine that logged in vs logged out guest state.
This was many years ago (IIRC around 2011-2012) so specific details aren't fresh in my memory. Back then server cpu hardware and mysql were less performant that current generation of hardware and mysql/php versions heh. I think PHP 5.3 and MySQL 5.5 was the latest back then. Servers back then were the Intel Xeon X56xx or E56xx Westmere or Intel Xeon X34xx Lynnfield based processors mainly in web hosting world. For the 500,000 unique ip visitors/day setup it was basic 2x web server + 2 mysql (master/slave) running Litespeed web servers. For 60,000 concurrent users was multiple nginx based web servers with heavy use of PHP-FPM fastcgi_cache
And with 2 master/slaves IIRC.
Back then most forums I worked on were vBulletin so all would of had memcached server implemented. Some were pushing up to >10 billion memcached cache hits per month - so you can imagine how much savings there were to be had
So support for redis and memcached based caching in a forum is nice 
As to whether its reasonable to expect forum admins to know how to tune SQL servers, probably 95% of forum owners I have worked with in past 17+ yrs don't adequately know how to tune MySQL servers. But the larger forum owners usually have hired admins or teams to manage their servers so back then it was more about those hired admins trying to understand the forum software's ins/outs and how to make it work with what they do i.e. mysql replication, caching layers on server level etc. These days server hardware has gotten a lot more performant and MySQL/Percona/MariaDB 5.7/10 and PHP 7.x all perform much better. So a lot of times the less in the know forum admins are starting on a more scalable and better performing foundation.
But as other folks have touched on in this thread, big part of that is how optimised the mysql queries are in forum core software themselves. But as with wordpress and forums, alot of times the inefficiencies are introduced by 3rd party plugin/addons i.e. 3rd party addon/plugins that do a full scan of all users in mysql user table ! So you can't really expect a simple approach as pick server size and install forum software without potentially reaching the point of needing to tune the server.
It's on the fly optimisation but can be configured to be cached to file (default), memcached server and more recently to redis server caching. But yes if you can natively do alot of optimisations to css, js etc you can selectively disable pagespeed filters and only use some of them. You can see all the available pagespeed filters for mod_pagespeed and ngx_pagespeed under reference column at https://www.modpagespeed.com/doc/.
Some pagespeed filters are harder to implement manually so pagespeed auto optimisation on the fly makes it much easier i.e. optimising your css critical path rendering https://www.modpagespeed.com/doc/filter-prioritize-critical-css
I mainly use ngx_pagepseed for auto image conversion to webp format when pagespeed detects browsers that support webp. Though if your forum supported webp image format conversion for attachment uploads and embedded 3rd party image links, then you could natively do the same via web server i.e. on Nginx with webp images https://centminmod.com/webp/
Full demo of my optimise-images.sh script's webp conversion for nginx at https://github.com/centminmod/optimise-images/blob/master/examples/examples-optimise-webp-nginx-300417.md
So yes another forum feature to have would be support for webp images and attachments conditionally for web browsers that support webp
Oh speaking of performance related features, would be nice to have attachments that natively support X-Sendfile/X-Accel so PHP isn't left doing all the heavy lifting when dealing with large attachment processing/serving https://www.nginx.com/resources/wiki/start/topics/examples/x-accel/
check out https://stackoverflow.blog/ they post their configuration write ups over the years can't find specific ones but search should reveal some i.e.
from https://stackexchange.com/performance interesting statistics
Discourse is challenging your perspective here. Since when do you not have to wait for a page to load and, if you're looking for something specific, why are you scrolling and not searching? It's leaving old conversation behind (but making accessible), encouraging new threads, and challenging whether pagination really is as great as we think it is.
https://blog.codinghorror.com/the-end-of-pagination/
For example, a common problem here on vanilla is that we've moved on from a topic three pages later, and someone comes in and quotes a post from three pages ago and continues as though the conversation wasn't already had, often repeating something previously discussed. It becomes super annoying in a long thread, many of you saw it when people would ask me to take an action, followed by my response, followed by someone asking me to take action, repeat until I lose my mind and close thread. We have to challenge how we view conversation to get around these clear UX issues. And that's why discourse is so interesting to me. I feel your pains, then I challenge myself to answer why I'm doing it that way. It's like... For once someone decided not to make another version of the same software everyone else makes.
That's really what I want in forum software. I want it to toss out the ideas already implemented in existing software, not just iterate on what could be a plugin, but instead challenge how we think about using a forum. Give us a new way to interact that addresses the social issues, not just the technical ones.
tossing out might not be best idea, giving forum end user the choice between traditional and new method probably better
I still love phpBB. Managed many big forums based in vBulletin, phpBB and later Xenforo.
The thing is, for an admin, it's trivial to disable features you don't need, than implement ones you do.
I've honestly thought that current forums often have too much in their core though, instead of relying on extensions/plugins to implement the rest, so definitely don't think you need to have a lot of features. But if you're only targeting a low common demoninator feature set, which every other forum out there already does, you don't really have anything unique.
Maybe because they're running on oversold shared hosting? More seriously, you'd probably need to actually investigate the reason rather than assume what you write will be better - I'm not saying that you can't do better, rather, that there's many possible reasons for slowness.
Databases all cache stuff to memory, and if not, the OS does. You don't explicitly need to do it at the application layer - there are benefits, sure, but let's not overstate them.
And yet, they're still one of the most popular sites on the internet. Not trying to discourage you, but just reiterating the above theme that performance is probably not as important as you think it may be.
I typically estimate roughly 1KB per post. Thread sizes will heavily depend on the way the forum is run.
Migrations are a pain for a multitude of reasons - you'll see most forums don't bother because of that.
I wouldn't focus on an importer from the get go, to be honest. Many people don't migrate, and there's always the question of how to migrate away once you drop the project due to lost interest.
If your forum software gets popular, then people will think about migrating to it.
I personally couldn't recommend doing that. Most comparisons I've seen don't seem to show much benefit of WebP compared to well optimized JPEG implementations, like mozjpeg. WebP is also significantly slower, and doesn't seem to be getting that much adoption outside Chrome. Other formats exist on the horizon, e.g. HEIF and AVIF which also may make WebP obsolete.
That's an article advocating infinite scroll, which is something I absolutely hate because of the way it makes text squirm around on the screen, and you can't search for stuff further down the page because it hasn't been loaded yet, plus as they say you don't know how close to the end of the page you are. Google has that 10-item pagination so they can show you more ads on every page. They used to support 100 items per page but they got rid of it to show more ads.
One thing I miss about the old forums (anyone remember Webcrossing?) was being able to really get all the posts in one screen. If there were 500 posts in a thread at 1k each, that's maybe 150KB with mod_gzip, so you'd wait for it to load (that took a while on a dialup) and then you could scroll around at your leisure. These days you probably get more than 150K of JS bloat with every page. So I'd want to be able to serve very large pages if the user wanted them. I don't know why newer forums always paginate but I suspect it has to do with database-to-app traffic slowing things down.
Yes I've been considering Redis but also like the idea of caching messages directly in-process to avoid the (even local) IPC traffic to the Redis server. Another idea might be to let Redis serve post content directly, using lua scripting and some kind of hack to share the user socket between the forum software and redis (in geek talk, Linux's SCM_RIGHTS ancillary message lets you do that). There's also Pedis, a parallelized clone of Redis that's supposed to be faster (github.com/fastio/pedis) though I haven't tried it yet.
Supporting sendfile sounds worthwhile though I have no intention of using PHP
. Going beyond sendfile, a really fast setup might use DPDK which is a userspace network stack that's supposed to be faster than the Linux's stack.
Hmm I wonder what kinds of things those plugins did.
Good point, definitely would want to support this.
Hmm, I know in some cases the forums were running on dedicated servers and were still slow.
I don't know. Reddit is a privacy hazard (not as bad at Facebook yet but same idea) BECAUSE it has to sell user data to advertisers BECAUSE it has to bring in a lot of revenue BECAUSE it has to pay for so many expensive servers BECAUSE its code is so doggone slow. The admin staff is not that large. Again this is how OKCupid beat so many other dating sites back in the day. Because their server software was 10x faster than everyone else's, it used 1/10th as much hardware, needed far less administration (no big cluster just a few boxes), etc. I look at the stuff I've worked on in the past few years (midsized Rails sites), see six-figure monthly AWS bills, and think "jeez, if that code was written with any concept of speed, it could all run on a laptop". We're still in an era when fast enough software can change the economics of running a site. Maybe these ideas are overkill for something like a forum though.
Anyway, everything you folks are saying is thoughtful and well-taken and I appreciate it. Thanks as always.
not using PHP might limit your user base ?
DPDK ? You mean something similar to http://speedus.torusware.com/ ?
not using PHP might limit your user base ?
Well, Discourse is written in Ruby so non-PHP isn't an automatic stopper. Yeah it wouldn't run on PHP-only cheap shared hosting, but it should be ok as a VPS template.
That looks like a similar idea and maybe it's a wrapper around DPDK. See https://dpdk.org or https://en.wikipedia.org/wiki/DPDK .
http://www.seastar-project.org/ shows a benchmark graph of a DPDK-ified Seastar reimplementation of memcached beating the pants off the original memcached. Pedis (parallel Redis) also uses seastar.
Add-on question: are hosted forum services like Proboards still of much interest in the forum community? Would VPS hosts here have any interest in offering something like that to their customers, if there was a program that did it? That is, it could serve 1000s of moderate sized forums from a single app instance, something like serving 1000s of virtual hosts from a single httpd.
sweet more reading for me to do.. cheers
Something I might want to do with Centmin Mod's installed software i.e. memcached etc
ah there's more out there https://openfastpath.org/index.php/service/technicaloverview/
Have fun dealing with GDPR heh https://www.eugdpr.org/
There's that, and all the overhead involved, like parsing posts, additional work formatting templates, more data for gzip to compress etc. RAM usage per request also goes up, and can be particularly problematic on some PHP setups. Also forums these days aren't always just text - you've got avatars, signatures and images in posts, which will add up. This all consumes resources in the user's browser and you'll find large pages can be slow to load/render on the client.
I do somewhat like the idea of having all the text available for Ctrl+F'ing, but it's of limited use IMO. It also allows easier archiving of content.
(some forums have a 'Print thread' functionality which can be used to get the entire thread; it also conveniently removes ads)
Are you saying that Reddit wouldn't sell user data (or as much of it) if it paid less on servers? Don't kid yourself, they do it to profit, not because they have to cover a large bill. Faster software just means they profit more, and it just becomes a case of whether the costs of improving the software are worth the extra profit, which I bet, it isn't, hence why they don't bother.
* I don't actually specifically know anything about Reddit's case - maybe I'm wrong (I doubt it though), but for most websites, this is what I observe.
@Willie, it's often not the forum software but its content inside of it that attracts people.
I would say for performance, design and test it with Nginx, and make Nginx a first class citizen in your documentation. Then combine that with static caching of pages for non-authenticated users, and you'll have pretty much unbeatable performance without too much additional complexity from the likes of Varnish. For RAM caching, Redis is excellent.
As far as end-user features, I run a large forum, and probably the biggest feature that I would need is spam mitigation and moderation. I implemented a custom system to allow users to flag posts as spam, and depending on the number of different users that flag a post and the amount of time the poster has been registered, the post will get automatically unpublished and possibly ban the poster's account. The lack of this feature is what prevents me from switching to a different forum platform, although I would like to switch for other reasons.
I'd expect that stuff happens when the post is first made, and then the rendered form is cached. Is there some obstacle to that? I was thinking of even caching the gzipped rendered form, since gzip has the cute property that if you concatenate two gzipped strings and then ungzip the result, you get back the two original strings, i.e. ungzip(gzip(A)+gzip(B)) = A+B. So you can build up a page of posts by just concatenating the gzipped posts. You get slightly worse compression than if you gzipped everything in one block, but it uses no cpu.
(some forums have a 'Print thread' functionality which can be used to get the entire thread; it also conveniently removes ads)
I know VBB has this and it's nice, but the sites I know that enable it still limit the number of posts you can get in a single view. I don't know how much that is due to performance issues though. It comes down to what the forum admin wants, so maybe they'll prefer to set limits anyway.
True
. But it's in the nature of their business model that they have to pursue huge size in order to satisfy their investors. I don't know that Reddit is such a great example but that's a standard source of tension in the tech scene out here. There's tons of geeks who'd love to have a company where they get to do cool and fun things and make enough money to buy a nice house and a Tesla but they don't really care about becoming the next Jeff Bezos. But they instead have to take external financing which forces them to do stressful and unpleasant things all day, and take big risks where they end up being either billionaires or broke, when they'd rather just be millionaires with reasonable certainty.
At a smaller scale, there's another forum I used to frequent, I don't know the numbers but I'd consider it fairly large but not huge (a few thousand members and monthly traffic in the low TBs). Again it ran on a relatively expensive dedicated server cluster when with faster software it could have been on a cheap VPS (or back then, maybe a cheap single server). So it has to pay its hosting bills through advertising, fundraisers, etc. It's the usual setup with volunteer moderation so hosting is a big chunk of the site expense. That's one of the issues that keeps making me think we need better software.
In the case of VPSBoard, a change to the software (a [corrected] Invision "upgrade" that made it look more like facebook) drove people away. Vpsboard had modest but steady activity til that "upgrade" and then everyone left. I'm here on LET because I like the software here a lot better than the post-upgrade Vpsboard. They may have changed it again since then though.
Thanks, yes, I'll treat that as a high priority if I pursue this idea.
+1 that was main reason i stopped eventually the move to Invision Board. If they moved to Xenforo I would of stayed ! Not a fan of Vanilla board or Discourse but forums I frequent with them, I go for the content and topic discussed ^_^