SPANKING NEW DEALS! HAPPY NEW YEAR w/ PREMIUM GIVEAWAYS! FREEBIES GALORE by RackNerd.

DeusVult

@noob404 said:

@DeusVult said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:
My vps is basically idle at the moment.

The only VPS you have?

Yes

Waiting for a good project to make good use of the VPS?

Yeah. I just don't know what to code.
My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

Oh that's cool. Yah, wayback machine would require a hell lot of storage.

Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

You can bring them down a bit more.

How? I am currently using the avif image format.

I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
I am using the nodejs sharp library. I will send my code soon.

That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

Will try that. Don't have my laptop with me at the moment.

Sure, let me know when you have. I am interested in this as well

Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

Wow!!!that's a lot bro

Wayback. That's expected.

True

Beniskickbutt

@noob404 said:

@Beniskickbutt said:

@noob404 said:

@Beniskickbutt said:
Oh wow you guys been busy today 🙂

Covered some great topics. Since youa re here, let us choose a new topic, if you want

I noticed! Any interesting stand outs?

Img compression seems to be a topic under discussion. Got any good recommendation.

not something i have dealt with much. ARe we talking 100% lossless compression as well? :O

BasToTheMax

@Beniskickbutt said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:
My vps is basically idle at the moment.

The only VPS you have?

Yes

Waiting for a good project to make good use of the VPS?

Yeah. I just don't know what to code.
My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

Oh that's cool. Yah, wayback machine would require a hell lot of storage.

Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

You can bring them down a bit more.

How? I am currently using the avif image format.

I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
I am using the nodejs sharp library. I will send my code soon.

That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

Will try that. Don't have my laptop with me at the moment.

Sure, let me know when you have. I am interested in this as well

Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

Oh I see. The crawler coded in Node as well?

He should try rust

What about golang?

DeusVult

@noob404 said:
Okay will join back in a few hours maybe. You guys enjoy the party.

Thank u! Good rest

BasToTheMax

@Beniskickbutt said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:
My vps is basically idle at the moment.

The only VPS you have?

Yes

Waiting for a good project to make good use of the VPS?

Yeah. I just don't know what to code.
My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

Oh that's cool. Yah, wayback machine would require a hell lot of storage.

Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

You can bring them down a bit more.

How? I am currently using the avif image format.

I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
I am using the nodejs sharp library. I will send my code soon.

That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

Will try that. Don't have my laptop with me at the moment.

Sure, let me know when you have. I am interested in this as well

Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

wait a minute.. are you crwaling way back machine? Just looked back in the quote chain.. thats a bum load of data

No, i am currently just crawling the internet in general

TrK

@noob404 said:

@TrK said:

@cainyxues said:

@TrK said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:
My vps is basically idle at the moment.

The only VPS you have?

Yes

Waiting for a good project to make good use of the VPS?

Yeah. I just don't know what to code.
My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

Oh that's cool. Yah, wayback machine would require a hell lot of storage.

Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

or if you have an idle vm in your inventory you can always use imgproxy

is this your commercial project ?

nope it's an open source project, i used for quite some time

Youa re gonna open source it?

mine is entirely different, it's meant for shopify and likes a plugin

Beniskickbutt

@noob404 said:

@Beniskickbutt said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:
My vps is basically idle at the moment.

The only VPS you have?

Yes

Waiting for a good project to make good use of the VPS?

Yeah. I just don't know what to code.
My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

Oh that's cool. Yah, wayback machine would require a hell lot of storage.

Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

You can bring them down a bit more.

How? I am currently using the avif image format.

I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
I am using the nodejs sharp library. I will send my code soon.

That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

Will try that. Don't have my laptop with me at the moment.

Another site to add to the pool of bookmarks. Haven't heard of that one

Been using tinypng for a really long time. They good

is it pretty fast as well or do they have an api? I.e. can i use it to serve people compressed images of various compression levels at runtime?

DeusVult

@Beniskickbutt said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:
My vps is basically idle at the moment.

The only VPS you have?

Yes

Waiting for a good project to make good use of the VPS?

Yeah. I just don't know what to code.
My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

Oh that's cool. Yah, wayback machine would require a hell lot of storage.

Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

You can bring them down a bit more.

How? I am currently using the avif image format.

I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
I am using the nodejs sharp library. I will send my code soon.

That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

Will try that. Don't have my laptop with me at the moment.

Sure, let me know when you have. I am interested in this as well

Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

wait a minute.. are you crwaling way back machine? Just looked back in the quote chain.. thats a bum load of data

That's like zetas of data maybe?

TrK

@DeusVult said:

@TrK said:

@Beniskickbutt said: Whats just another idle vm 😀

i mean everyone gets one or two idles in the portfolio

I have a IPV6 VPS that I idle because I don't have IPV6 sadly...

try tunnelbroker.net and enjoy

Beniskickbutt

@noob404 said:

@Beniskickbutt said:

@noob404 said:

@Beniskickbutt said:

@noob404 said:

@TrK said:

@noob404 said:

@TrK said:
another random suggestion: Don't compile mariadb 11.4 on a slow VPS.... it will take hours to finish the task. or just compile it and go back to sleep... it will be done when you wake up... probably....

Remember trying to compile PHP. Afew years ago on a 2GB VPS. Made me wait for 2 hours and then said you need atleast 4GB RAM!

well different cases for different tasks you can bypass the ram requirements with swap but i don't usually wait more than an hour for my compilation tasks.

Oh yes. Infact there was a workaround (non-swap) for building it that I learned of back then. Don't remember now. I prefer my personal PC for compilation.

That's what I usually try to do as well. Local compile remote deploy

My preferred method as well. Somehow I feel I got more control when it's local.

My local box is also a little better than the remotes which is why i feel better doing it that way

I was offered a 128GB VPS for compiling Batocera. But being uncomfortable I decided to do it with my ancient i5-5250u

wow 128GB? OFfered to you just for the compile or to keep it for the compile and beyond? Thats huge

Beniskickbutt

@cainyxues said:

@Beniskickbutt said:

@noob404 said:

@Beniskickbutt said:

@noob404 said:

@cainyxues said:

@noob404 said:

@cainyxues said:

@noob404 said:

@cainyxues said:
I too want credits man like It would fund vps's if for instance even 100usd it would be good for 10 years

You can easily convert your win's price value from trh main prize pool to RackNerd credits

Yeah, I know that it's just that first of all I need to win lets just hope for the best for now.................................................

Sure, dude. Stay active and you won't regret it.

Yeah, lets see btw why is dustinc not visible didn't see him yesterday and even today too

Let's not forget the he's the CEO of RackNerd and that it's still the holiday season, being one of the busiest times for them. So, he might be extremely busy. But rest assured. He will be here with many GAs. Hope you can hang around till and each time it happens.

Yeah I'm surprised he even has as much time as he does to drop in here

Yah, he has some great time management skills

im still interested in knowing how he keeps up double bandwidth gang. Is it a cool script that helps him or does he really manually comb through it all??

I think there might be someone from his team too only viewing threads or maybe regex script

yeah thats what im wondering. Or Maybe mr ed helped him with a scrapper to find that stuff

DeusVult

@TrK said:

@DeusVult said:

@TrK said:

@Beniskickbutt said: Whats just another idle vm 😀

i mean everyone gets one or two idles in the portfolio

I have a IPV6 VPS that I idle because I don't have IPV6 sadly...

try tunnelbroker.net and enjoy

Yoooo that's amazing. Thank u

DeusVult

@TrK said:

@noob404 said:

@TrK said:

@cainyxues said:

@TrK said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:
My vps is basically idle at the moment.

The only VPS you have?

Yes

Waiting for a good project to make good use of the VPS?

Yeah. I just don't know what to code.
My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

Oh that's cool. Yah, wayback machine would require a hell lot of storage.

Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

or if you have an idle vm in your inventory you can always use imgproxy

is this your commercial project ?

nope it's an open source project, i used for quite some time

Youa re gonna open source it?

mine is entirely different, it's meant for shopify and likes a plugin

Oooo you do sales?

Beniskickbutt

@noob404 said:

@Beniskickbutt said:

@noob404 said:

@Beniskickbutt said:

@noob404 said:

@cainyxues said:

@noob404 said:

@cainyxues said:

@noob404 said:

@cainyxues said:
I too want credits man like It would fund vps's if for instance even 100usd it would be good for 10 years

You can easily convert your win's price value from trh main prize pool to RackNerd credits

Yeah, I know that it's just that first of all I need to win lets just hope for the best for now.................................................

Sure, dude. Stay active and you won't regret it.

Yeah, lets see btw why is dustinc not visible didn't see him yesterday and even today too

Let's not forget the he's the CEO of RackNerd and that it's still the holiday season, being one of the busiest times for them. So, he might be extremely busy. But rest assured. He will be here with many GAs. Hope you can hang around till and each time it happens.

Yeah I'm surprised he even has as much time as he does to drop in here

Yah, he has some great time management skills

im still interested in knowing how he keeps up double bandwidth gang. Is it a cool script that helps him or does he really manually comb through it all??

I'd like to think that he does it using a script or addon. But, I have a feeling he does it manually.

yeah it seems to be variable enough when it goes through it that theres some part of significant manual process to it. Hes also bumped up/responded to old comments of mine as well as others so it seems he is keeping an eye out there

Beniskickbutt

@noob404 said:

@Beniskickbutt said:

@noob404 said:
All the talk of local compilation just reminded me, I once compiled Batocera from source on my i5-5250u. Took 5 days!

thats obsurd! Any reason why you compiled rather than just got a built distribution of it? Are you hacking on linux kernels now too?

Was compiling it for the PS4, so, it did require many modifications.

ah got it. So you put linux onto a ps4? Very cool. Was it all from scratch or did you have some guides to do it? Did it give you access to more games?

TrK

@DeusVult said:

@TrK said:

@DeusVult said:

@TrK said:

@Beniskickbutt said: Whats just another idle vm 😀

i mean everyone gets one or two idles in the portfolio

I have a IPV6 VPS that I idle because I don't have IPV6 sadly...

try tunnelbroker.net and enjoy

Yoooo that's amazing. Thank u

nothing better than some ipv6 connectivity at home, grateful to HE for the offerings.

Beniskickbutt

@noob404 said:

@Beniskickbutt said:

@noob404 said:

@Beniskickbutt said:

@noob404 said:

@cainyxues said:

@noob404 said:

@cainyxues said:

@noob404 said:

@TrK said:
another random suggestion: Don't compile mariadb 11.4 on a slow VPS.... it will take hours to finish the task. or just compile it and go back to sleep... it will be done when you wake up... probably....

Remember trying to compile PHP. Afew years ago on a 2GB VPS. Made me wait for 2 hours and then said you need atleast 4GB RAM!

lol

Oh that was a disaster. To overcome this, I tried compiling on my PC and moving it to the VPS. only after I installed it did I know that the specific module for which I rebuilt PHP won't work without a few more dependencies. Learned akt that day

akt means?

Pardon I am on my phone. I meant "a lot"

Your English is pretty good. You mentioned wanting to get better. A lot of people use alot instead

We have been taught to use both depending on the sentence

interesting.. alot is actually not a word

Didn't know this. Got to learn a lot about grammar and vocabulary from you

i am by no means an expert so dont play too much attention Alot vs a lot is just one that I happen to know well : )

xpress7

@noob404 said:

@xpress7 said:

@noob404 said:

@xpress7 said:
Lots of pages.. dont' want to go back now. Whats the topic currently?

Img compression best methods. @BasToTheMax wants to know

TrK knows better. He is building one optimization service.

Yah pointed him to TrK, but he himself recommended imgproxy

Thats also a good option. 👍

TrK

@DeusVult said:

@TrK said:

@noob404 said:

@TrK said:

@cainyxues said:

@TrK said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:
My vps is basically idle at the moment.

The only VPS you have?

Yes

Waiting for a good project to make good use of the VPS?

Yeah. I just don't know what to code.
My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

Oh that's cool. Yah, wayback machine would require a hell lot of storage.

Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

or if you have an idle vm in your inventory you can always use imgproxy

is this your commercial project ?

nope it's an open source project, i used for quite some time

Youa re gonna open source it?

mine is entirely different, it's meant for shopify and likes a plugin

Oooo you do sales?

have a store with shopify, we recently started and i am constantly looking for ideas on tees and hoodies

Beniskickbutt

@cainyxues said:

@noob404 said:

@Beniskickbutt said:

@noob404 said:
All the talk of local compilation just reminded me, I once compiled Batocera from source on my i5-5250u. Took 5 days!

thats obsurd! Any reason why you compiled rather than just got a built distribution of it? Are you hacking on linux kernels now too?

Was compiling it for the PS4, so, it did require many modifications.

WTH what are you guys here I thought you were just talking about little bit of programming but you all are taking it to the next level

Theres just a few little chains going on! Noob just had a little tidbit of info to add on here with that regard

Beniskickbutt

@noob404 said:
Okay will join back in a few hours maybe. You guys enjoy the party.

sounds good! I will be in and out again today! Hope to catch you later. Party on my friend!

TrK

@xpress7 said:

@noob404 said:

@xpress7 said:

@noob404 said:

@xpress7 said:
Lots of pages.. dont' want to go back now. Whats the topic currently?

Img compression best methods. @BasToTheMax wants to know

TrK knows better. He is building one optimization service.

Yah pointed him to TrK, but he himself recommended imgproxy

Thats also a good option. 👍

Better than coding one and relying on something like CDN with custom endpoints and cache crawlers, just setup a docker host proxy with CF you get instant image optimization service up and running in minutes

Beniskickbutt

@DeusVult said:

@noob404 said:

@Beniskickbutt said:

@noob404 said:

@cainyxues said:

@noob404 said:

@cainyxues said:

@noob404 said:

@cainyxues said:
I too want credits man like It would fund vps's if for instance even 100usd it would be good for 10 years

You can easily convert your win's price value from trh main prize pool to RackNerd credits

Yeah, I know that it's just that first of all I need to win lets just hope for the best for now.................................................

Sure, dude. Stay active and you won't regret it.

Yeah, lets see btw why is dustinc not visible didn't see him yesterday and even today too

Let's not forget the he's the CEO of RackNerd and that it's still the holiday season, being one of the busiest times for them. So, he might be extremely busy. But rest assured. He will be here with many GAs. Hope you can hang around till and each time it happens.

Yeah I'm surprised he even has as much time as he does to drop in here

Yah, he has some great time management skills

I want that as well lol

I have been reading about different time management skills. Theres quite a lot of suggestions and guides out there but its hard to stick to some of them. One thing i found that works well is you really need to figure out how to focus on a single task at a time. Just moving around a lot can gt districating

cainyxues

@TrK said:

@DeusVult said:

@TrK said:

@noob404 said:

@TrK said:

@cainyxues said:

@TrK said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:
My vps is basically idle at the moment.

The only VPS you have?

Yes

Waiting for a good project to make good use of the VPS?

Yeah. I just don't know what to code.
My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

Oh that's cool. Yah, wayback machine would require a hell lot of storage.

Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

or if you have an idle vm in your inventory you can always use imgproxy

is this your commercial project ?

nope it's an open source project, i used for quite some time

Youa re gonna open source it?

mine is entirely different, it's meant for shopify and likes a plugin

Oooo you do sales?

have a store with shopify, we recently started and i am constantly looking for ideas on tees and hoodies

Why didn't you code one ? shopify still charges a lot and you could get nice seo with next js fast rendering and ui too........................

Beniskickbutt

@DeusVult said:

@noob404 said:

@DeusVult said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:
My vps is basically idle at the moment.

The only VPS you have?

Yes

Waiting for a good project to make good use of the VPS?

Yeah. I just don't know what to code.
My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

Oh that's cool. Yah, wayback machine would require a hell lot of storage.

Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

You can bring them down a bit more.

How? I am currently using the avif image format.

I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
I am using the nodejs sharp library. I will send my code soon.

That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

Will try that. Don't have my laptop with me at the moment.

Sure, let me know when you have. I am interested in this as well

Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

Wow!!!that's a lot bro

Wayback. That's expected.

True

Do they disclose how much storage space they actually use? They have to be one of the largest outside of search engines i would imagine

Beniskickbutt

@BasToTheMax said:

@Beniskickbutt said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:
My vps is basically idle at the moment.

The only VPS you have?

Yes

Waiting for a good project to make good use of the VPS?

Yeah. I just don't know what to code.
My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

Oh that's cool. Yah, wayback machine would require a hell lot of storage.

Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

You can bring them down a bit more.

How? I am currently using the avif image format.

I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
I am using the nodejs sharp library. I will send my code soon.

That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

Will try that. Don't have my laptop with me at the moment.

Sure, let me know when you have. I am interested in this as well

Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

Oh I see. The crawler coded in Node as well?

He should try rust

What about golang?

i support that decision as well

xpress7

@TrK said:

@xpress7 said:

@noob404 said:

@xpress7 said:

@noob404 said:

@xpress7 said:
Lots of pages.. dont' want to go back now. Whats the topic currently?

Img compression best methods. @BasToTheMax wants to know

TrK knows better. He is building one optimization service.

Yah pointed him to TrK, but he himself recommended imgproxy

Thats also a good option. 👍

Better than coding one and relying on something like CDN with custom endpoints and cache crawlers, just setup a docker host proxy with CF you get instant image optimization service up and running in minutes

Differeance between a developer and an entreprenuer..

Beniskickbutt

@BasToTheMax said:

@Beniskickbutt said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:
My vps is basically idle at the moment.

The only VPS you have?

Yes

Waiting for a good project to make good use of the VPS?

Yeah. I just don't know what to code.
My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

Oh that's cool. Yah, wayback machine would require a hell lot of storage.

Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

Yeah average image size is currently around 25-100 kb. Small pages are around 25-50 kb, while large pages can easily reach 300 kb.

You can bring them down a bit more.

How? I am currently using the avif image format.

I am not sure right now. But, I did work on this a few years ago. BTW, how are you compressing currently?

First, the page screenshot is using png, then i compress to webp and then i compress that webp image to avif, but this only saves like 5-30 kb at the moment. Also the avif compression takes like 30 seconds.
I am using the nodejs sharp library. I will send my code soon.

That's quite impressive. Just to test though, try running the original image through tinypng.com and see if you are already able to get it a lower size than your final compressed image, if you haven't already. Just to confirm.

Will try that. Don't have my laptop with me at the moment.

Sure, let me know when you have. I am interested in this as well

Oh by the way, the crawler already has added 800k urls in the db. About 5 of them are actually screenshotted (as that part is still work in progress).

wait a minute.. are you crwaling way back machine? Just looked back in the quote chain.. thats a bum load of data

No, i am currently just crawling the internet in general

im trying to whip up a fast crawler as well using chatgpt. Trying to get it to make one with python but its not spitting out something that works yet Dont have timeto put the work in it so wana see if it can do it for me and even dump things into pretty objects

TrK

@cainyxues said:

@TrK said:

@DeusVult said:

@TrK said:

@noob404 said:

@TrK said:

@cainyxues said:

@TrK said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:
My vps is basically idle at the moment.

The only VPS you have?

Yes

Waiting for a good project to make good use of the VPS?

Yeah. I just don't know what to code.
My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

Oh that's cool. Yah, wayback machine would require a hell lot of storage.

Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

or if you have an idle vm in your inventory you can always use imgproxy

is this your commercial project ?

nope it's an open source project, i used for quite some time

Youa re gonna open source it?

mine is entirely different, it's meant for shopify and likes a plugin

Oooo you do sales?

have a store with shopify, we recently started and i am constantly looking for ideas on tees and hoodies

Why didn't you code one ? shopify still charges a lot and you could get nice seo with next js fast rendering and ui too........................

i don't want to risk it, me being lazy and store being a passion for us two guys. It gets me free time to work on products and don't have to worry about patching things up. And honestly i might shift to headless shopify and do everything on a vue frontend later on. But it's not inthe todo list as of now and won't be in for atleast two to three years.

Beniskickbutt

@TrK said:

@noob404 said:

@TrK said:

@cainyxues said:

@TrK said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:

@noob404 said:

@BasToTheMax said:
My vps is basically idle at the moment.

The only VPS you have?

Yes

Waiting for a good project to make good use of the VPS?

Yeah. I just don't know what to code.
My wayback machine project is fun, but if i would actually deploy it, it would require a lot of storage. And i primarily make it to learn more about web crawling and puppeteer.

Oh that's cool. Yah, wayback machine would require a hell lot of storage.

Well one thing that is different than the WA, is that i only save full-page screenshots instead of the whole page with all the assets.

Oh so a waybackscreenshots. Sounds coolm.in that case you could compress the images using various tools. Infact, @TrK is working on a commercial image optimisation CDN that might be of help for your project

or if you have an idle vm in your inventory you can always use imgproxy

is this your commercial project ?

nope it's an open source project, i used for quite some time

Youa re gonna open source it?

mine is entirely different, it's meant for shopify and likes a plugin

Something i can use perhaps if i even spin up my web app with shopify for my friends crafts