New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Hmm, LET offers aren't really in any machine readable form so i guess the best that could be returned would be posts (maybe by user and/or date). Given Vanilla (the forum software used) is in kind of a weird state and persons (technically) running it also aren't exactly known for being uber-enthusiastic i don't think you will get anywhere with this.
If you manage to avoid cloudflare's challenges scraping would probably be your best option i fear.
I'm assuming that they wouldn't want an API since deal scalping is such a problem as it is. Something like Firecrawl might be able to scrape the site, but like @totally_not_banned said, captchas could be a problem.
Just vibecode some scraper.
While i pretty much hate all that code generation nonsense i would be kind of curious what some AI comes up with when told to "build a scraper that sorts LET offers into a mariadb table while evading cloudflare captchas". I figure it would either be a total clusterfluck or pure brilliance.
Lol, I feel ya. My bet is on a clusterfuck of pure brilliance
Open the network path in chrome dev tools and generate a .har, there are a lot of ways to continue and lots of APIs to find.
Welcome to the cockwomble forum, where everyone needs to ask AI and flood APIs with requests just to get a little data.
Before cockwombles, we had programmers. They were smart enough to code without overcomplicating everything. Let me introduce you to one of their wild inventions.
RSS feeds
https://lowendtalk.com/discussions/feed.rss
https://lowendtalk.com/categories/offers/feed.rss
WOW!
Sadly, nearly all providers on cockwomble forum are equally stupid, so you’ll have to monitor all categories to catch every offer, because placing an offer in the offers category is apparently a very hard and complicated task.
And/or lazy though. While i won't claim to be smart writing a scraper (especially for something as simple as LET) is often times just the path of least resistance. Before i figure out just about anything i have written 5 scrapers.
Well, until someone makes a post reminding everyone that there is actually RSS feeds. This is when i kick myself in the balls because i could have been even 3 times more lazy.
LowEndBox published a tutorial on this very thing
https://lowendbox.com/blog/how-to-monitor-new-lowendtalk-offers-with-python/
Well, i don't read LEB and like i've said: Path of least resistance. The time it takes to search around and (maybe - chances are i'd rather find out about Vanilla supporting RSS before anything else) stumble across that post is already half of a scraper (probably more)
Edit: Sorry for the "not uber-enthusiastic". It was meant mostly in relation to the idea of actually adding some kind of API. I guess that might not be instantly recognizable.
If you right-click and select “View Source” on the front and category pages, there’s a link tag in the head with the RSS URL.
I don’t get how anyone can miss it.
OP has a link in his signature to a personal page with his CV.
He claims to be a senior software engineer.
Can we please go back to when engineers actually researched and built things instead of asking “curling questions” on forums, so they can have their LLM to do their job.
We are one small step away from Ruflo taking over, with a swarm of AI agents handling the entire dev task while humans are reduced to dumb monkeys whose intellect is limited to pressing the “Start” button.
To make sure we take that small step, please use https://goal.ruv.io/ to research and plan your tasks and Claude and Ruflo to code it.
The end is nigh.
I built this for myself about 18 months ago, (without AI), and I've since expanded it to gather from LES and WHT, (and I'm currently working on Nodeseek).
LET has RSS feeds which give you the starting point, and then web scraping the content is easy because the format is simple and predictable. SeleniumBase has solutions for working around Cloudflare Captchas so it's not a very complicated project.
I use NTFY for notifications, but I've considered creating a Telegram channel to make it public. But the main reason I didn't bother so far is there's just so few good deals now.
Scrapping obviously, CF doesn't do jackshit.
I used to scrap LET to index it for a search engine.
Since the search here is dogshit slow.
It works fine.
Took a few weeks though.
Perfect for LLM shitposting.
I don't think it was an unreasonable question. Ideally I was hoping for an API where I could poll e.g. users to verify that they had the correct roles (and highlight where roles have changed), but clearly RSS is helpful and much better than screen scraping. I wasn't aware that the LET forums were powered by Vanilla, but happy to accept that's my bad.
Hard to argue against that.