Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Register

Quick Links


Shells Virtual Desktop
BMail.ag - Secure Email Service
Server.net
CPLicense.net
VPS Server
Buy VPN
Vultr
VMs for AI
HostDare
ReliableSite White-Label Dedicated Hosting for Resellers
InterServer VPS
BMail.ag - Secure Email Service
Best VPN
High-Performance Bare Metal Server Solutions
Karvl.com
Server Mania Cloud Hosting
DataWagon Hosting
AlphaVPS Hosting
Evoxt.com
Clouvider
VPS Hosting with NVMe
Residential IPs in the US & 4G Mobile Proxies in EU & US with Unlimited Bandwidth
ReliableSite White-Label Dedicated Hosting for Resellers
Rabisu - Hosting Solutions
Shells Virtual Desktop

Categories

In this Discussion

Home › Offers
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

SPANKING NEW DEALS! HAPPY NEW YEAR w/ PREMIUM GIVEAWAYS! FREEBIES GALORE by RackNerd.

1…478479481483484…1047

Comments

  • SaragoldfarbSaragoldfarb Member, Megathread Squad

    You guys do Yabs?

  • BeniskickbuttBeniskickbutt Member

    @Saragoldfarb said:
    The amount of shitposting in this thread is pretty impressive I must say.

    you inspire us <3

    Thanked by 1Saragoldfarb
  • cainyxuescainyxues Member

    @noob404 said:

    @DeusVult said:

    @noob404 said:

    @DeusVult said:
    But can be 25 dollars max too...

    One sure shot way to increase that budget is get RN credits

    They have storage VPS deals?

    Not exactly sure. Here's another idea. There are reseller hosting offers on this thread from time to time as well. Win one of these. Make some money selling hosting to friends and use it to maybe colocate with RackNerd.

    I also thought about selling hosting to friends even grabbed one offer but no one wanted to buy even though they all are cs students :smile:

  • BeniskickbuttBeniskickbutt Member

    @DeusVult said:
    I know this is possible because I've already saw deals (not flash sale ones!) with these specs and price, but I just can't find/contact these providers...

    something is bound to come up in the megathread i think. Get some people to bandwagon on your asks

  • noob404noob404 Member

    @Beniskickbutt said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Oh I see. The crawler coded in Node as well?

    He should try rust :)

    Heard good things of Rust as well.

    im only at the starting stages of learning but i like what i have seen so far

    Would you say it's a faster alternative than PHP for most purposes? Or, is Rust a language not meant for web development ?

    oh and yes much faster than php!

    Maybe I should try it out then. Is my knowledge of PHP be of any use when migrating to Rust?

  • BasToTheMaxBasToTheMax Member

    @TrK said:

    @BasToTheMax said:

    @TrK said:

    @BasToTheMax said:
    Also, i am not yet sure where to store the screenshots.

    s3? Preferably b2 with CF in front?

    Wouldn't cost that a lot when storing thousands of screenshots?

    thousands are easy, how many can you fit inside 1TB?

    If i would store screenshots of all pages that are currently on the internet (about 5 billion according to google), i would need about 250 TB of storage. This assumes the average image size is 50 KB. ChatGPT did the math for me.

  • BeniskickbuttBeniskickbutt Member

    @DeusVult said:
    Host-C has exactly what I want, but they are offline and haven't answered any of my contacts (email, DMs) for days. Apparently they are on the road and will get back tomorrow, so let's see...

    When you said offline, atfirst i thought you meant offline permantly and i was gonna say good thing you didnt get one there then :)

    The i realized my mistake in my interpretation

  • noob404noob404 Member

    @cainyxues said:

    @noob404 said:

    @DeusVult said:

    @noob404 said:

    @DeusVult said:
    But can be 25 dollars max too...

    One sure shot way to increase that budget is get RN credits

    They have storage VPS deals?

    Not exactly sure. Here's another idea. There are reseller hosting offers on this thread from time to time as well. Win one of these. Make some money selling hosting to friends and use it to maybe colocate with RackNerd.

    I also thought about selling hosting to friends even grabbed one offer but no one wanted to buy even though they all are cs students :smile:

    Oh yes. It might be a bit difficult to get clients in the beginning. Marketing is the tough part.

  • cainyxuescainyxues Member

    @noob404 said:

    @cainyxues said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Oh I see. The crawler coded in Node as well?

    Yes 😅

    Oh so you are a Node head. I love PHP. Want to learn something new. Confused between Python and Node.

    I would say python seems like a "better" thing for this but really all languages are capable of doing almots anything now with the amount of libraries they have

    Yah, also heard that with PHP8.3, PHP has gotten really fast and better. So, maybe I should stick with PHP and better my knowledge of it.

    i have nightmares from using php back in school. I havent touched it since then. Maybe a couple charity websites

    I kinda have gotten so used to it that I don't feel like migrating to anything else until everyone just decides to dump PHP.

    I have been recently loving the development in php as well as spa too

    What's spa?

    Single page application, there is this concept of designing whole application with only one single php file :smiley: [I haven't tried that out yet]

  • BeniskickbuttBeniskickbutt Member

    @TrK said:

    @Beniskickbutt said:

    @TrK said:

    @Beniskickbutt said:
    I wonder how ed gets his data.. he mentioned his mechanisms. Im trying to do the same and export more stats but it seems unfriendly to some ways to acquiring the data :) Maybe he'll let me poke on his stats page

    scraping html from residential connection, if you are into nodejs cheerio is a good start.

    I get 403'd i thought it was just wide open, i guess i may be wrong

    residential connection with authentication ofcourse :wink: here have a cookie :cookie:

    om nom nom nom thanks! Yeah i'll have to add that in. 99% of this code im running with is chatgpr generated right now. Will have to see if it can handle that

  • BeniskickbuttBeniskickbutt Member

    @DeusVult said:

    @noob404 said:

    @DeusVult said:
    But can be 25 dollars max too...

    One sure shot way to increase that budget is get RN credits

    They have storage VPS deals?

    I havent noticed any yet

  • TrKTrK Member

    @Saragoldfarb said:
    You guys do Yabs?

    naaah.... only SPAAAAAAAM a lot of it at that ...........

  • noob404noob404 Member

    @BasToTheMax said:

    @TrK said:

    @BasToTheMax said:

    @TrK said:

    @BasToTheMax said:
    Also, i am not yet sure where to store the screenshots.

    s3? Preferably b2 with CF in front?

    Wouldn't cost that a lot when storing thousands of screenshots?

    thousands are easy, how many can you fit inside 1TB?

    If i would store screenshots of all pages that are currently on the internet (about 5 billion according to google), i would need about 250 TB of storage. This assumes the average image size is 50 KB. ChatGPT did the math for me.

    Just a shot in the dark, would a bunch of coffee cloud storage options (free tiers) be any helpful?

  • noob404noob404 Member

    @cainyxues said:

    @noob404 said:

    @cainyxues said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Oh I see. The crawler coded in Node as well?

    Yes 😅

    Oh so you are a Node head. I love PHP. Want to learn something new. Confused between Python and Node.

    I would say python seems like a "better" thing for this but really all languages are capable of doing almots anything now with the amount of libraries they have

    Yah, also heard that with PHP8.3, PHP has gotten really fast and better. So, maybe I should stick with PHP and better my knowledge of it.

    i have nightmares from using php back in school. I havent touched it since then. Maybe a couple charity websites

    I kinda have gotten so used to it that I don't feel like migrating to anything else until everyone just decides to dump PHP.

    I have been recently loving the development in php as well as spa too

    What's spa?

    Single page application, there is this concept of designing whole application with only one single php file :smiley: [I haven't tried that out yet]

    Never heard of this as well. But doesn't this just widen the attack surface and add security issues?

  • SaragoldfarbSaragoldfarb Member, Megathread Squad

    @TrK said:

    @Saragoldfarb said:
    You guys do Yabs?

    naaah.... only SPAAAAAAAM a lot of it at that ...........

    Unfortunate... I need a RN yabs for a small box. Maybe like 1G, 1Core whatever else.

  • TrKTrK Member

    @BasToTheMax said:

    @TrK said:

    @BasToTheMax said:

    @TrK said:

    @BasToTheMax said:
    Also, i am not yet sure where to store the screenshots.

    s3? Preferably b2 with CF in front?

    Wouldn't cost that a lot when storing thousands of screenshots?

    thousands are easy, how many can you fit inside 1TB?

    If i would store screenshots of all pages that are currently on the internet (about 5 billion according to google), i would need about 250 TB of storage. This assumes the average image size is 50 KB. ChatGPT did the math for me.

    i mean will you need to screenshot each and every website from the get go? it's something to take care when you grow. as for 250TB it's gonna cost you dearly if you decide to go for it right now :joy:

  • SaragoldfarbSaragoldfarb Member, Megathread Squad

    Do any of you actually have servers with RN?

  • noob404noob404 Member

    @Saragoldfarb said:

    @TrK said:

    @Saragoldfarb said:
    You guys do Yabs?

    naaah.... only SPAAAAAAAM a lot of it at that ...........

    Unfortunate... I need a RN yabs for a small box. Maybe like 1G, 1Core whatever else.

    Can get you one tomorrow, cause I think I do have one of those, i.e., if youa re serious.

    Thanked by 1Saragoldfarb
  • TrKTrK Member

    @Saragoldfarb said:

    @TrK said:

    @Saragoldfarb said:
    You guys do Yabs?

    naaah.... only SPAAAAAAAM a lot of it at that ...........

    Unfortunate... I need a RN yabs for a small box. Maybe like 1G, 1Core whatever else.

    can do for you right now, but i don't have any 1gb 1vcore with me, only 2gb 2vcore or more, will it work? also it's compiling MYSQL right now so you will have to wait for an hour so :smile:

    Thanked by 1Saragoldfarb
  • cainyxuescainyxues Member

    @noob404 said:

    @cainyxues said:

    @noob404 said:

    @cainyxues said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Oh I see. The crawler coded in Node as well?

    Yes 😅

    Oh so you are a Node head. I love PHP. Want to learn something new. Confused between Python and Node.

    I would say python seems like a "better" thing for this but really all languages are capable of doing almots anything now with the amount of libraries they have

    Yah, also heard that with PHP8.3, PHP has gotten really fast and better. So, maybe I should stick with PHP and better my knowledge of it.

    i have nightmares from using php back in school. I havent touched it since then. Maybe a couple charity websites

    I kinda have gotten so used to it that I don't feel like migrating to anything else until everyone just decides to dump PHP.

    I have been recently loving the development in php as well as spa too

    What's spa?

    Single page application, there is this concept of designing whole application with only one single php file :smiley: [I haven't tried that out yet]

    Never heard of this as well. But doesn't this just widen the attack surface and add security issues?

    dunno about that :smile:

  • BeniskickbuttBeniskickbutt Member

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Oh I see. The crawler coded in Node as well?

    Yes 😅

    Oh so you are a Node head. I love PHP. Want to learn something new. Confused between Python and Node.

    I would say python seems like a "better" thing for this but really all languages are capable of doing almots anything now with the amount of libraries they have

    Yah, also heard that with PHP8.3, PHP has gotten really fast and better. So, maybe I should stick with PHP and better my knowledge of it.

    i have nightmares from using php back in school. I havent touched it since then. Maybe a couple charity websites

    I kinda have gotten so used to it that I don't feel like migrating to anything else until everyone just decides to dump PHP.

    its also easier to not have to learn a new language so makes sense

  • BeniskickbuttBeniskickbutt Member

    @Saragoldfarb said:

    @DeusVult said:

    @Saragoldfarb said:
    The amount of shitposting in this thread is pretty impressive I must say.

    Lol, how so

    It's a lot... With little content.

    DOES THIS NOT SATISFY YOU?

    <3

  • noob404noob404 Member

    @TrK said:

    @Saragoldfarb said:

    @TrK said:

    @Saragoldfarb said:
    You guys do Yabs?

    naaah.... only SPAAAAAAAM a lot of it at that ...........

    Unfortunate... I need a RN yabs for a small box. Maybe like 1G, 1Core whatever else.

    can do for you right now, but i don't have any 1gb 1vcore with me, only 2gb 2vcore or more, will it work? also it's compiling MYSQL right now so you will have to wait for an hour so :smile:

    BTW why compiling it when there are readymade packages for it? Compiling a custom one removing unnecessary features and options ?

    Thanked by 1Saragoldfarb
  • SaragoldfarbSaragoldfarb Member, Megathread Squad

    @noob404 said:

    @Saragoldfarb said:

    @TrK said:

    @Saragoldfarb said:
    You guys do Yabs?

    naaah.... only SPAAAAAAAM a lot of it at that ...........

    Unfortunate... I need a RN yabs for a small box. Maybe like 1G, 1Core whatever else.

    Can get you one tomorrow, cause I think I do have one of those, i.e., if youa re serious.

    Yes, cool.

    I am serious. I collect tiny VMS so I can flex it to the gang.

  • BeniskickbuttBeniskickbutt Member

    @Saragoldfarb said:
    You guys do Yabs?

    i read yabs. Dont do them myself

    Thanked by 1Saragoldfarb
  • SaragoldfarbSaragoldfarb Member, Megathread Squad

    @TrK said:

    @Saragoldfarb said:

    @TrK said:

    @Saragoldfarb said:
    You guys do Yabs?

    naaah.... only SPAAAAAAAM a lot of it at that ...........

    Unfortunate... I need a RN yabs for a small box. Maybe like 1G, 1Core whatever else.

    can do for you right now, but i don't have any 1gb 1vcore with me, only 2gb 2vcore or more, will it work? also it's compiling MYSQL right now so you will have to wait for an hour so :smile:

    Big=better.

  • BeniskickbuttBeniskickbutt Member

    @cainyxues said:

    @noob404 said:

    @DeusVult said:

    @noob404 said:

    @DeusVult said:
    But can be 25 dollars max too...

    One sure shot way to increase that budget is get RN credits

    They have storage VPS deals?

    Not exactly sure. Here's another idea. There are reseller hosting offers on this thread from time to time as well. Win one of these. Make some money selling hosting to friends and use it to maybe colocate with RackNerd.

    I also thought about selling hosting to friends even grabbed one offer but no one wanted to buy even though they all are cs students :smile:

    students have no mooney! :)

  • noob404noob404 Member

    @cainyxues said:

    @noob404 said:

    @cainyxues said:

    @noob404 said:

    @cainyxues said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Oh I see. The crawler coded in Node as well?

    Yes 😅

    Oh so you are a Node head. I love PHP. Want to learn something new. Confused between Python and Node.

    I would say python seems like a "better" thing for this but really all languages are capable of doing almots anything now with the amount of libraries they have

    Yah, also heard that with PHP8.3, PHP has gotten really fast and better. So, maybe I should stick with PHP and better my knowledge of it.

    i have nightmares from using php back in school. I havent touched it since then. Maybe a couple charity websites

    I kinda have gotten so used to it that I don't feel like migrating to anything else until everyone just decides to dump PHP.

    I have been recently loving the development in php as well as spa too

    What's spa?

    Single page application, there is this concept of designing whole application with only one single php file :smiley: [I haven't tried that out yet]

    Never heard of this as well. But doesn't this just widen the attack surface and add security issues?

    dunno about that :smile:

    Hmm. I am new to all of this as well. But, was just wondering. If everything is stored in a single file including MySQL connections, functions and everything, I "assume" it widens the attack surface for a hacker.

  • BeniskickbuttBeniskickbutt Member

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Oh I see. The crawler coded in Node as well?

    He should try rust :)

    Heard good things of Rust as well.

    im only at the starting stages of learning but i like what i have seen so far

    Would you say it's a faster alternative than PHP for most purposes? Or, is Rust a language not meant for web development ?

    oh and yes much faster than php!

    Maybe I should try it out then. Is my knowledge of PHP be of any use when migrating to Rust?

    i cant recall much about php but i would start out with something like this to get your feet wet and see how you like it

    https://tourofrust.com/

  • DeusVultDeusVult Member

    @Saragoldfarb said:
    Do any of you actually have servers with RN?

    i have two

Sign In or Register to comment.