Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Register

Quick Links


Shells Virtual Desktop
BMail.ag - Secure Email Service
Server.net
CPLicense.net
VPS Server
Buy VPN
Vultr
VMs for AI
HostDare
ReliableSite White-Label Dedicated Hosting for Resellers
InterServer VPS
BMail.ag - Secure Email Service
Best VPN
High-Performance Bare Metal Server Solutions
Karvl.com
Server Mania Cloud Hosting
DataWagon Hosting
AlphaVPS Hosting
Evoxt.com
Clouvider
VPS Hosting with NVMe
Residential IPs in the US & 4G Mobile Proxies in EU & US with Unlimited Bandwidth
ReliableSite White-Label Dedicated Hosting for Resellers
Rabisu - Hosting Solutions
Shells Virtual Desktop

Categories

In this Discussion

Home › Offers
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

SPANKING NEW DEALS! HAPPY NEW YEAR w/ PREMIUM GIVEAWAYS! FREEBIES GALORE by RackNerd.

1…473474476478479…1047

Comments

  • BeniskickbuttBeniskickbutt Member

    @DeusVult said:

    @Beniskickbutt said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    wait a minute.. are you crwaling way back machine? Just looked back in the quote chain.. thats a bum load of data

    That's like zetas of data maybe?

    At a minium for sure

    Thanked by 1DeusVult
  • TrKTrK Member

    @xpress7 said:

    @TrK said:

    @xpress7 said:

    @noob404 said:

    @xpress7 said:

    @noob404 said:

    @xpress7 said:
    Lots of pages.. dont' want to go back now. Whats the topic currently? :D

    Img compression best methods. @BasToTheMax wants to know

    TrK knows better. He is building one optimization service.

    Yah pointed him to TrK, but he himself recommended imgproxy

    Thats also a good option. 👍

    Better than coding one and relying on something like CDN with custom endpoints and cache crawlers, just setup a docker host proxy with CF you get instant image optimization service up and running in minutes :smile:

    Differeance between a developer and an entreprenuer.. ;)

    and ofcourse between a free and busy guy :joy:

    Thanked by 1Beniskickbutt
  • BeniskickbuttBeniskickbutt Member

    @TrK said:

    @DeusVult said:

    @TrK said:

    @Beniskickbutt said: Whats just another idle vm 😀

    i mean everyone gets one or two idles in the portfolio :wink:

    I have a IPV6 VPS that I idle because I don't have IPV6 sadly...

    try tunnelbroker.net and enjoy :smile:

    wow, free??

  • BeniskickbuttBeniskickbutt Member

    @TrK said:

    @DeusVult said:

    @TrK said:

    @noob404 said:

    @TrK said:

    @cainyxues said:

    @TrK said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    or if you have an idle vm in your inventory you can always use imgproxy :wink:

    is this your commercial project ?

    nope it's an open source project, i used for quite some time :smile:

    Youa re gonna open source it?

    mine is entirely different, it's meant for shopify and likes a plugin :smile:

    Oooo you do sales?

    have a store with shopify, we recently started and i am constantly looking for ideas on tees and hoodies :smile:

    make some LET themed swag and throw it up there :)

  • TrKTrK Member

    @Beniskickbutt said: Something i can use perhaps if i even spin up my web app with shopify for my friends crafts

    sounds interesting what kind of crafts are we talking about?

  • TrKTrK Member

    @Beniskickbutt said: wow, free??

    always was :worried:

  • BeniskickbuttBeniskickbutt Member

    @cainyxues said:

    @TrK said:

    @DeusVult said:

    @TrK said:

    @noob404 said:

    @TrK said:

    @cainyxues said:

    @TrK said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    or if you have an idle vm in your inventory you can always use imgproxy :wink:

    is this your commercial project ?

    nope it's an open source project, i used for quite some time :smile:

    Youa re gonna open source it?

    mine is entirely different, it's meant for shopify and likes a plugin :smile:

    Oooo you do sales?

    have a store with shopify, we recently started and i am constantly looking for ideas on tees and hoodies :smile:

    Why didn't you code one ? shopify still charges a lot and you could get nice seo with next js fast rendering and ui too........................

    everything comes down to the time i think. TrK is a master of all but theres only 1 trk :(

  • DeusVultDeusVult Member

    @TrK said:

    @DeusVult said:

    @TrK said:

    @DeusVult said:

    @TrK said:

    @Beniskickbutt said: Whats just another idle vm 😀

    i mean everyone gets one or two idles in the portfolio :wink:

    I have a IPV6 VPS that I idle because I don't have IPV6 sadly...

    try tunnelbroker.net and enjoy :smile:

    Yoooo that's amazing. Thank u

    nothing better than some ipv6 connectivity at home, grateful to HE for the offerings. :smile:

    Hahahahaha yeah!!! That's amazing honestly

  • xpress7xpress7 Member

    @TrK said:

    @xpress7 said:

    @TrK said:

    @xpress7 said:

    @noob404 said:

    @xpress7 said:

    @noob404 said:

    @xpress7 said:
    Lots of pages.. dont' want to go back now. Whats the topic currently? :D

    Img compression best methods. @BasToTheMax wants to know

    TrK knows better. He is building one optimization service.

    Yah pointed him to TrK, but he himself recommended imgproxy

    Thats also a good option. 👍

    Better than coding one and relying on something like CDN with custom endpoints and cache crawlers, just setup a docker host proxy with CF you get instant image optimization service up and running in minutes :smile:

    Differeance between a developer and an entreprenuer.. ;)

    and ofcourse between a free and busy guy :joy:

    And personal project vs customer project. :D

  • SaragoldfarbSaragoldfarb Member, Megathread Squad

    Where the deals at?

  • DeusVultDeusVult Member

    @Beniskickbutt said:

    @DeusVult said:

    @noob404 said:

    @DeusVult said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Wow!!!that's a lot bro

    Wayback. That's expected.

    True

    Do they disclose how much storage space they actually use? They have to be one of the largest outside of search engines i would imagine

    Well over 100 petabytes (!!!)

  • BasToTheMaxBasToTheMax Member

    @Beniskickbutt said:

    @DeusVult said:

    @noob404 said:

    @DeusVult said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Wow!!!that's a lot bro

    Wayback. That's expected.

    True

    Do they disclose how much storage space they actually use? They have to be one of the largest outside of search engines i would imagine

    I once read an article about that, but i can't find it at the moment

  • TrKTrK Member

    @Beniskickbutt said: make some LET themed swag and throw it up there

    will be doing upcoming years, we need to figure out some short of featured collection by the end of Jan :neutral:

  • BeniskickbuttBeniskickbutt Member

    @TrK said:

    @Beniskickbutt said: Something i can use perhaps if i even spin up my web app with shopify for my friends crafts

    sounds interesting what kind of crafts are we talking about?

    All sorts of things. They hit flea markets and thrift stores all the time and just make all kinds of little doo dads and decor

    Thanked by 1TrK
  • TrKTrK Member

    @xpress7 said: And personal project vs customer project.

    atleast i finish customer's one.... :joy:

  • DeusVultDeusVult Member

    @TrK said:

    @Beniskickbutt said: wow, free??

    always was :worried:

    That's amazing honestly

  • DeusVultDeusVult Member

    @BasToTheMax said:

    @Beniskickbutt said:

    @DeusVult said:

    @noob404 said:

    @DeusVult said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Wow!!!that's a lot bro

    Wayback. That's expected.

    True

    Do they disclose how much storage space they actually use? They have to be one of the largest outside of search engines i would imagine

    I once read an article about that, but i can't find it at the moment

    I've searched and it's 100 petabytes

  • TrKTrK Member

    @Saragoldfarb said: Where the deals at?

    Sleeping......... got any in digirdpee?

  • BeniskickbuttBeniskickbutt Member

    @Saragoldfarb said:
    Where the deals at?

    we already got something better now.. you

  • BeniskickbuttBeniskickbutt Member

    @xpress7 said:

    @TrK said:

    @xpress7 said:

    @TrK said:

    @xpress7 said:

    @noob404 said:

    @xpress7 said:

    @noob404 said:

    @xpress7 said:
    Lots of pages.. dont' want to go back now. Whats the topic currently? :D

    Img compression best methods. @BasToTheMax wants to know

    TrK knows better. He is building one optimization service.

    Yah pointed him to TrK, but he himself recommended imgproxy

    Thats also a good option. 👍

    Better than coding one and relying on something like CDN with custom endpoints and cache crawlers, just setup a docker host proxy with CF you get instant image optimization service up and running in minutes :smile:

    Differeance between a developer and an entreprenuer.. ;)

    and ofcourse between a free and busy guy :joy:

    And personal project vs customer project. :D

    always good to pass the blame to another company if someone goes wrong for those professional projects ;)

  • BeniskickbuttBeniskickbutt Member

    @DeusVult said:

    @Beniskickbutt said:

    @DeusVult said:

    @noob404 said:

    @DeusVult said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Wow!!!that's a lot bro

    Wayback. That's expected.

    True

    Do they disclose how much storage space they actually use? They have to be one of the largest outside of search engines i would imagine

    Well over 100 petabytes (!!!)

    I'd like to have that kind of storage in my home :)

  • DeusVultDeusVult Member

    @Saragoldfarb said:
    Where the deals at?

    Hello Sara. How are you doing?

  • BeniskickbuttBeniskickbutt Member

    @BasToTheMax said:

    @Beniskickbutt said:

    @DeusVult said:

    @noob404 said:

    @DeusVult said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Wow!!!that's a lot bro

    Wayback. That's expected.

    True

    Do they disclose how much storage space they actually use? They have to be one of the largest outside of search engines i would imagine

    I once read an article about that, but i can't find it at the moment

    deus mentioned an amount but not sure if that was just another guess. It probably grows daily as well which is nuts

  • BeniskickbuttBeniskickbutt Member

    @TrK said:

    @Beniskickbutt said: make some LET themed swag and throw it up there

    will be doing upcoming years, we need to figure out some short of featured collection by the end of Jan :neutral:

    Can always go for new years items. Ironic things about having resolutions and breaking them?

    Thanked by 1TrK
  • DeusVultDeusVult Member

    Apparently DigiRDP will release new deals today... But honestly I don't think they will be storage ones

  • BeniskickbuttBeniskickbutt Member

    @TrK said:

    @xpress7 said: And personal project vs customer project.

    atleast i finish customer's one.... :joy:

    theres some more motivation there i think with the money involved :)

  • TrKTrK Member
    edited December 2024

    sara be ready.... them notifs coming hot :p

  • SaragoldfarbSaragoldfarb Member, Megathread Squad

    @DeusVult said:

    @Saragoldfarb said:
    Where the deals at?

    Hello Sara. How are you doing?

    Quite fine given circumstances. Caught a flu. Alcohol yesterday mostly took care of it.... Slow Sunday...

  • DeusVultDeusVult Member

    @Beniskickbutt said:

    @BasToTheMax said:

    @Beniskickbutt said:

    @DeusVult said:

    @noob404 said:

    @DeusVult said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Wow!!!that's a lot bro

    Wayback. That's expected.

    True

    Do they disclose how much storage space they actually use? They have to be one of the largest outside of search engines i would imagine

    I once read an article about that, but i can't find it at the moment

    deus mentioned an amount but not sure if that was just another guess. It probably grows daily as well which is nuts

    @Beniskickbutt said:

    @DeusVult said:

    @Beniskickbutt said:

    @DeusVult said:

    @noob404 said:

    @DeusVult said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Wow!!!that's a lot bro

    Wayback. That's expected.

    True

    Do they disclose how much storage space they actually use? They have to be one of the largest outside of search engines i would imagine

    Well over 100 petabytes (!!!)

    I'd like to have that kind of storage in my home :)

    Hahahahaha same

  • BasToTheMaxBasToTheMax Member

    @Beniskickbutt said:

    @BasToTheMax said:

    @Beniskickbutt said:

    @DeusVult said:

    @noob404 said:

    @DeusVult said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:

    @noob404 said:

    @BasToTheMax said:
    My vps is basically idle at the moment.

    The only VPS you have?

    Yes

    Waiting for a good project to make good use of the VPS?

    Yeah. I just don't know what to code.
    My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

    Oh that's cool. Yah, wayback machine would require a hell lot of storage.

    Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

    Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

    Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

    You can bring them down a bit more.

    How? I am currently using the avif image format.

    I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

    First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
    I am using the nodejs sharp library. I will send my code soon.

    That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

    Will try that. Don't have my laptop with me at the moment.

    Sure, let me know when you have. I am interested in this as well

    Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

    Wow!!!that's a lot bro

    Wayback. That's expected.

    True

    Do they disclose how much storage space they actually use? They have to be one of the largest outside of search engines i would imagine

    I once read an article about that, but i can't find it at the moment

    deus mentioned an amount but not sure if that was just another guess. It probably grows daily as well which is nuts

    Well, luckily, i will only store full-page screenshots, which should take way less storage!

Sign In or Register to comment.