Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Register

Quick Links


Shells Virtual Desktop
BMail.ag - Secure Email Service
Server.net
CPLicense.net
VPS Server
Buy VPN
Vultr
VMs for AI
HostDare
ReliableSite White-Label Dedicated Hosting for Resellers
InterServer VPS
BMail.ag - Secure Email Service
Best VPN
High-Performance Bare Metal Server Solutions
Karvl.com
Server Mania Cloud Hosting
DataWagon Hosting
AlphaVPS Hosting
Evoxt.com
Clouvider
VPS Hosting with NVMe
Residential IPs in the US & 4G Mobile Proxies in EU & US with Unlimited Bandwidth
ReliableSite White-Label Dedicated Hosting for Resellers
Rabisu - Hosting Solutions
Shells Virtual Desktop

Categories

In this Discussion

Home › Offers
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

SPANKING NEW DEALS! HAPPY NEW YEAR w/ PREMIUM GIVEAWAYS! FREEBIES GALORE by RackNerd.

1…477478480482483…1047

Comments

  • BeniskickbuttBeniskickbutt Member

    @Beniskickbutt said:

    @TrK said:

    @Beniskickbutt said:
    I wonder how ed gets his data.. he mentioned his mechanisms. Im trying to do the same and export more stats but it seems unfriendly to some ways to acquiring the data :) Maybe he'll let me poke on his stats page

    scraping html from residential connection, if you are into nodejs cheerio is a good start.

    I get 403'd i thought it was just wide open, i guess i may be wrong

    oops not using cherrio but just basic web requests i might add

  • DeusVultDeusVult Member

    @Beniskickbutt said:

    @DeusVult said:

    @Beniskickbutt said:

    @DeusVult said:
    I need 1TB HDD storage, 4-5TB BW and 1GBPS for 20 dollars yearly. Just that. But nothing...

    Aim your sights higher.. $10 yearly :)

    More 10 dollars 😢?

    oops my bad. I forgot where you were located! are you saying 20 in your currency?

    Nooooo. 20 dollars!

  • noob404noob404 Member

    @DeusVult said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @cainyxues said:

    @noob404 said:

    @cainyxues said:

    @noob404 said:

    @cainyxues said:
    I too want credits man like It would fund vps's if for instance even 100usd :lol: it would be good for 10 years :smile:

    You can easily convert your win's price value from trh main prize pool to RackNerd credits

    Yeah, I know that it's just that first of all I need to win :lol: lets just hope for the best for now................................................. :smile: :smiley:

    Sure, dude. Stay active and you won't regret it.

    Yeah, lets see btw why is dustinc not visible didn't see him yesterday and even today too

    Let's not forget the he's the CEO of RackNerd and that it's still the holiday season, being one of the busiest times for them. So, he might be extremely busy. But rest assured. He will be here with many GAs. Hope you can hang around till and each time it happens.

    Yeah I'm surprised he even has as much time as he does to drop in here

    Yah, he has some great time management skills

    I want that as well lol

    It takes a lot of efffort. I am trying to bring that into my life as well.

    Thanked by 1DeusVult
  • DeusVultDeusVult Member

    Which is pretty expensive for me

  • BeniskickbuttBeniskickbutt Member

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Oh I see. The crawler coded in Node as well?

    He should try rust :)

    Heard good things of Rust as well.

    im only at the starting stages of learning but i like what i have seen so far

  • noob404noob404 Member

    @DeusVult said:
    Which is pretty expensive for me

    What's your budget like?

  • DeusVultDeusVult Member

    But can be 25 dollars max too...

  • DeusVultDeusVult Member

    @Beniskickbutt said:

    @DeusVult said:
    Bro LET is one of the few places where ppl beg providers to sell them things... Like, I dmed so many providers and none of them answered...

    Yeah theres a lot of requests in the threads. I would be happy with many of htem

    True

  • BeniskickbuttBeniskickbutt Member

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Oh I see. The crawler coded in Node as well?

    Yes 😅

    Oh so you are a Node head. I love PHP. Want to learn something new. Confused between Python and Node.

    I would say python seems like a "better" thing for this but really all languages are capable of doing almots anything now with the amount of libraries they have

    Yah, also heard that with PHP8.3, PHP has gotten really fast and better. So, maybe I should stick with PHP and better my knowledge of it.

    i have nightmares from using php back in school. I havent touched it since then. Maybe a couple charity websites

  • SaragoldfarbSaragoldfarb Member, Megathread Squad

    The amount of shitposting in this thread is pretty impressive I must say.

  • DeusVultDeusVult Member

    @noob404 said:

    @DeusVult said:
    Which is pretty expensive for me

    What's your budget like?

    20-25 dollars yearly

  • BeniskickbuttBeniskickbutt Member

    @DeusVult said:

    @Beniskickbutt said:

    @DeusVult said:

    @Beniskickbutt said:

    @DeusVult said:
    I need 1TB HDD storage, 4-5TB BW and 1GBPS for 20 dollars yearly. Just that. But nothing...

    Aim your sights higher.. $10 yearly :)

    More 10 dollars 😢?

    oops my bad. I forgot where you were located! are you saying 20 in your currency?

    Nooooo. 20 dollars!

    oh ok! 10 would be even better than no? :) i was confused

  • noob404noob404 Member

    @Beniskickbutt said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Oh I see. The crawler coded in Node as well?

    He should try rust :)

    Heard good things of Rust as well.

    im only at the starting stages of learning but i like what i have seen so far

    Would you say it's a faster alternative than PHP for most purposes? Or, is Rust a language not meant for web development ?

  • BeniskickbuttBeniskickbutt Member

    @DeusVult said:
    Which is pretty expensive for me

    got it. I thought you said 20 was a good deal so i thought 10 even better :)

  • DeusVultDeusVult Member

    I know this is possible because I've already saw deals (not flash sale ones!) with these specs and price, but I just can't find/contact these providers...

  • noob404noob404 Member

    @DeusVult said:
    But can be 25 dollars max too...

    One sure shot way to increase that budget is get RN credits

  • DeusVultDeusVult Member

    Host-C has exactly what I want, but they are offline and haven't answered any of my contacts (email, DMs) for days. Apparently they are on the road and will get back tomorrow, so let's see...

  • BeniskickbuttBeniskickbutt Member

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Oh I see. The crawler coded in Node as well?

    He should try rust :)

    Heard good things of Rust as well.

    im only at the starting stages of learning but i like what i have seen so far

    Would you say it's a faster alternative than PHP for most purposes? Or, is Rust a language not meant for web development ?

    If i used rust on the web it would be for APIs but not for serving up the actual ui content. I did see there are some web frameworks for it thugh for static page generation

  • DeusVultDeusVult Member

    @Saragoldfarb said:
    The amount of shitposting in this thread is pretty impressive I must say.

    Lol, how so

    Thanked by 1Saragoldfarb
  • TrKTrK Member

    @Beniskickbutt said:

    @TrK said:

    @Beniskickbutt said:
    I wonder how ed gets his data.. he mentioned his mechanisms. Im trying to do the same and export more stats but it seems unfriendly to some ways to acquiring the data :) Maybe he'll let me poke on his stats page

    scraping html from residential connection, if you are into nodejs cheerio is a good start.

    I get 403'd i thought it was just wide open, i guess i may be wrong

    residential connection with authentication ofcourse :wink: here have a cookie :cookie:

  • DeusVultDeusVult Member

    @noob404 said:

    @DeusVult said:
    But can be 25 dollars max too...

    One sure shot way to increase that budget is get RN credits

    They have storage VPS deals?

  • DeusVultDeusVult Member

    @Beniskickbutt said:

    @DeusVult said:
    Which is pretty expensive for me

    got it. I thought you said 20 was a good deal so i thought 10 even better :)

    Ooh it definitely is but I guess they only do that on flash sales...

  • BasToTheMaxBasToTheMax Member

    @TrK said:

    @BasToTheMax said:
    Also, i am not yet sure where to store the screenshots.

    s3? Preferably b2 with CF in front?

    Wouldn't cost that a lot when storing thousands of screenshots?

  • noob404noob404 Member

    @Beniskickbutt said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Oh I see. The crawler coded in Node as well?

    Yes 😅

    Oh so you are a Node head. I love PHP. Want to learn something new. Confused between Python and Node.

    I would say python seems like a "better" thing for this but really all languages are capable of doing almots anything now with the amount of libraries they have

    Yah, also heard that with PHP8.3, PHP has gotten really fast and better. So, maybe I should stick with PHP and better my knowledge of it.

    i have nightmares from using php back in school. I havent touched it since then. Maybe a couple charity websites

    I kinda have gotten so used to it that I don't feel like migrating to anything else until everyone just decides to dump PHP.

  • TrKTrK Member

    @BasToTheMax said:

    @TrK said:

    @BasToTheMax said:
    Also, i am not yet sure where to store the screenshots.

    s3? Preferably b2 with CF in front?

    Wouldn't cost that a lot when storing thousands of screenshots?

    thousands are easy, how many can you fit inside 1TB?

  • noob404noob404 Member

    @DeusVult said:

    @noob404 said:

    @DeusVult said:
    But can be 25 dollars max too...

    One sure shot way to increase that budget is get RN credits

    They have storage VPS deals?

    Not exactly sure. Here's another idea. There are reseller hosting offers on this thread from time to time as well. Win one of these. Make some money selling hosting to friends and use it to maybe colocate with RackNerd.

  • cainyxuescainyxues Member

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Oh I see. The crawler coded in Node as well?

    Yes 😅

    Oh so you are a Node head. I love PHP. Want to learn something new. Confused between Python and Node.

    I would say python seems like a "better" thing for this but really all languages are capable of doing almots anything now with the amount of libraries they have

    Yah, also heard that with PHP8.3, PHP has gotten really fast and better. So, maybe I should stick with PHP and better my knowledge of it.

    i have nightmares from using php back in school. I havent touched it since then. Maybe a couple charity websites

    I kinda have gotten so used to it that I don't feel like migrating to anything else until everyone just decides to dump PHP.

    I have been recently loving the development in php as well as spa too

  • noob404noob404 Member

    @cainyxues said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Oh I see. The crawler coded in Node as well?

    Yes 😅

    Oh so you are a Node head. I love PHP. Want to learn something new. Confused between Python and Node.

    I would say python seems like a "better" thing for this but really all languages are capable of doing almots anything now with the amount of libraries they have

    Yah, also heard that with PHP8.3, PHP has gotten really fast and better. So, maybe I should stick with PHP and better my knowledge of it.

    i have nightmares from using php back in school. I havent touched it since then. Maybe a couple charity websites

    I kinda have gotten so used to it that I don't feel like migrating to anything else until everyone just decides to dump PHP.

    I have been recently loving the development in php as well as spa too

    What's spa?

  • BeniskickbuttBeniskickbutt Member

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @Beniskickbutt said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Oh I see. The crawler coded in Node as well?

    He should try rust :)

    Heard good things of Rust as well.

    im only at the starting stages of learning but i like what i have seen so far

    Would you say it's a faster alternative than PHP for most purposes? Or, is Rust a language not meant for web development ?

    oh and yes much faster than php!

  • SaragoldfarbSaragoldfarb Member, Megathread Squad

    @DeusVult said:

    @Saragoldfarb said:
    The amount of shitposting in this thread is pretty impressive I must say.

    Lol, how so

    It's a lot... With little content.

Sign In or Register to comment.