Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Flat-file based blog performance for lot of posts - Page 2
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Flat-file based blog performance for lot of posts

2»

Comments

  • edanedan Member
    edited December 2015

    2bb3 said: If you can cache a bit that would work the same with a database, isn't it?

    Yes with much lower memory consumption.

    SilentFarter said: Flat-file sounds like a great idea, assuming you're nowhere near your inode limits.

    Yes. Hostgator shared hosting allow upto 250k inode limits. But moving to another host is very simple if its flat-file based.

    cgag said: sadly enough some file systems (most?) do linear scans of directories to get files, so opening a file in a heavily filled directory can be slow. I'm working on a flatfile comment system and was surprised to find how slow opening a file can be with 10s of thousands of files in a directory.

    Well interesting discussion here http://serverfault.com/questions/98235/how-many-files-in-a-directory-is-too-many-downloading-data-from-net

    I always know the filename and not doing glob every time since we use an index file.

  • Interested to know how you implement the search? Do you use a web service, or a web app?

  • @bookstack said:
    Interested to know how you implement the search? Do you use a web service, or a web app?

    This is the biggest weakness being flat-file based, currently it only can search the title and the tags for performance reason. Currently if we need content search as well, we need to use third-party services like Google CSE.

    But perhaps we can implemented content search as well in the future. How? example: when saving the content as .md it should also save the content to some kind of json encoded file, and use the filename as the key etc. later we just need to search those file. But it will take more server resources.

    Thanked by 1vRozenSch00n
  • Thanks for sharing that.

    Search is hard, not only the corpus needs to be tokenized, stemmed, you may want to search by synonyms, support exclusion, various strategy for the rank. That is why Lucene/Solr/ElasticSearch are invented to do that.

    My colleague uses SwiftType, and quite happy about its performance and accuracy.

    Thanked by 1edan
  • @bookstack said:
    Search is hard, not only the corpus needs to be tokenized, stemmed, you may want to search by synonyms, support exclusion, various strategy for the rank. That is why Lucene/Solr/ElasticSearch are invented to do that.

    My colleague uses SwiftType, and quite happy about its performance and accuracy.

    Yes but for flat-file based we just need simple searching. Swiftype? thanks, will look into it.

  • edan said: Yes. Hostgator shared hosting allow upto 250k inode limits. But moving to another host is very simple if its flat-file based.

    If they use CentOS 7 (and follow cPanel's configuration recommendations), inode limits aren't a concern.

    Otherwise, default limit is 100k inodes.

    Thanked by 1edan
  • @edan said:
    But perhaps we can implemented content search as well in the future.

    Take a look: https://www.petekeen.net/full-text-search-with-whistlepig

    Thanked by 1edan
  • netomxnetomx Moderator, Veteran

    Any flt file blog based on BASH? :P

  • tommy said: you kidding right

    you're*

  • @netomx said:
    Any flt file blog based on BASH? :P

    You could use socat to open a listener and process requests using bash, should be fast as hell. But you can write apps using bash as CGI too.

    Thanked by 1netomx
  • netomx said: flt file

    flat file*

  • PieHasBeenEatenPieHasBeenEaten Member, Host Rep

    @spellcheckguy Hey thanks dic!

    Thanked by 1netomx
  • netomxnetomx Moderator, Veteran

    @EkaatyLinux said:
    You could use socat to open a listener and process requests using bash, should be fast as hell. But you can write apps using bash as CGI too.

    You know, I like small things :P

    I may work on something as stupid as that on weekends :P

    Thanked by 1afterSt0rm
  • @netomx said:
    I may work on something as stupid as that on weekends :P

    I thought the same...

    Thanked by 1netomx
  • netomx said: Any flt file blog based on BASH? :P

    bashblog?

    Thanked by 2netomx edan
  • @souen said:

    Cool but it is a static site generator. I think that @netomx was talking about a dynamic blog with processing being made using bash and with flat files as a database. Something good to host on a small router without loosing dynamic capabilities and keeping the resource usage pretty low.

    Thanked by 2netomx souen
  • Thanks, will check it out.

Sign In or Register to comment.