New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Xeon E3 1230 and 10Gbit
bobsburgers
Member
in Help
will the cpu bottle neck ?
system will be in software raid 0 / 10gbit network port
4xHDDs in Raid0 can theoretically do 600MB/s read to which id be happy with but
system will be transferring 5-50GB Files to 5-20 different users at a time.
Comments
HDD will be a bottleneck. You can send files from RAM however, if most files that you transfer are the same then simply using ZFS ARC Cache can get you close to 10Gbit.
If you have lots of RAM (64GB+) then likely you can cache a lot and remember if its cached then IO is free, you may have just 70% hit ratio, but because you need disks for just 30% of traffic they'll be fine.
What HDDs you have in mind?
Realistically if all data is read from HDDs he wont get more than 300MB/s in RAID0/10, because he needs to read multiple files at once, so its not pure sequential. If these are lower denstity platters or some older HDDs (they likely are, given 13 year old CPU) then even 200MB/s would be amazing result if there is 5-20 reads at the same time.
It can be fixed only with RAM cache as HDDs are too slow, but for RAM cache to work he needs plenty of it, files need to be cacheable and cached. If its single download per file and delete (for example temporary file upload that expires after upload) then its not possible to cache anything. If its video platform where majority watch/download same video at the same time then its cacheable.
Upgrade that CPU
regards
If you take SSD or NVME your disks will not be the bottleneck anymore. It will probably be CPU.
@AXYZE exactly the info I was looking for. Appreciate it. With ZFS caching, are files loaded into ram as user downloads then deleted from cache or does cache delete as different files are loaded and no space is left in RAM?
With ZFS caching segments of files that you want to read are loaded into RAM. If there is this segment in RAM (cached) then its being sent from RAM.
If theres not enough memory for new segments then least used segments are removed from RAM.
ZFS does all add/remove from cache for you, it will do great job without tweaking.
On VoD site with 128GB ram (64GB ARC size) and 70TB data I have 85% cache hit-ratio because pretty much everyone watches newest/popular ones and most people do not watch until end.
@AXYZE say I have 4 different files being downloaded by 4 different users, and each file is 10-15GB and my ram is 64GB, would they ve loaded onto ram and sent from RAM ? What happens if for example I have larger multiple files being transferred and not enough RAM storage? Do all files start being sent at HDD limited speeds or is it partially still sent from ram and hdd at same time