Comments

• zGato Member

  October 9

  @AlbaHost has both Warsaw and Frankfurt. You'd need a custom VDS plan for the specs you need, but I'm sure they can accomodate with their VDS plans you don't share CPU cores with neighbors.

  With your budget you can get a quite beefy dedicated server from them too. But considering you'd rather not have to manage one, I'd stick with VDS. Not 100% sure if they got Ryzen 9 7950X/9950X VDS in Warsaw though, which is what you'd be aiming for to get 3k GB6 (or the EPYC/Intel alternative)

  I made a review about them here a while ago, my experience hasn't changed. If any, it has only improved they're great people

  Thanked by 3AlbaHost tzuli dimm
• oloke Member, Host Rep

  October 9 edited October 9

  With that budget and dedicated resource requirements, maybe a @Mevspace dedicated server would work as well?
  They have a DC in Wyszków, Poland.
  https://mevspace.com/dedicated-servers/amd

  edit:
  k.pl in Wrocław also seems suitable with their Ryzen 5 7600X dedi:
  https://k.pl/en/dedicated/?chassis=13&ram=32+GB+DDR5&disk=2x+500GB+NVMe&connection=1Gbps

  Thanked by 2tzuli dimm
• CraigRoeder Member

  October 10 edited October 10

  RISE-M
  AMD RYZEN 9 9900X
  12c / 24t - 4.4 GHz / 5.6 GHz
  64 GB 2 x 512 GB
  Public 1 Gbps
  Private 1 Gbps
  £85.99 Configure

  RISE-L
  AMD RYZEN 9 9950X
  16c / 32t - 4.3 GHz / 5.7 GHz
  128 GB 2 x 960 GB
  Public 1 Gbps
  Private 1 Gbps
  £128.99

  OVH right now, they also have Ryzen 9 5900X, Ryzen 7 9700X, Ryzen 7 5800X, ryzen 5 for less than £70.

  Go for this man. If you need help with OVH account, I can help. lol.

  For £120, you can have 4x of you asked for. I wish I can be in this business, I can be your ovh reseller

  Thanked by 1dimm
• CraigRoeder Member

  October 10

  BTW, over the years I optimized things, I dont think your stack need that much of power, I had to go up to 16 vcore, and now I am using only 1.5 dedi core, for 5 sites. 2 of them have 1GB db (400k rows, and 10M relation tbl. ).
  Calculate your db size, number requests per second and requests per day, p50, p90, p99 of your requests's time. You'll see.

  (Dedi resource is much better than you can expected, also php 8.5 is now 10x better than 7.4), also, with a dedicated one, you can setup some models with 0 down time roll back/backup/full vmd clones.... all yours!

  Thanked by 2LEBUserJoe linmobi
• dimm Member

  October 12

  @oloke said:
  With that budget and dedicated resource requirements, maybe a @Mevspace dedicated server would work as well?
  They have a DC in Wyszków, Poland.
  https://mevspace.com/dedicated-servers/amd

  edit:
  k.pl in Wrocław also seems suitable with their Ryzen 5 7600X dedi:
  https://k.pl/en/dedicated/?chassis=13&ram=32+GB+DDR5&disk=2x+500GB+NVMe&connection=1Gbps

  Thanks for the Mevspace suggestion! Somehow I missed them during my search. Their €111 dedicated server looks very interesting, so I'm keeping an eye on availability while discussing with my client whether moving to a dedicated server makes sense for us.

  As for k.pl, I actually bought a VPS from them. The performance is pretty good for €55/month (VPS Ryzen, 8vCPU, 16GB RAM, 250Gb Disk), so I can't complain about the value for money! 🙂 However, there are a few limitations for my use case: no custom ISO boot and no option for users to create their own VPS snapshots or backups. Being able to take a snapshot before a system upgrade or boot a Veeam recovery ISO to restore a full-machine backup would be really useful for me.

  Their dedicated server offers also don't seem quite as attractive as Mevspace's, and the rather slow support response times I've experienced with k.pl make me a bit hesitant. Mevspace also gives me the impression of having a more thoughtfully designed infrastructure, although that's just my impression — I haven't rented a dedicated server from either provider yet, so I can't really compare them firsthand.

  Thanks again for the recommendations!

  Thanked by 1oloke
• dimm Member

  October 12

  @CraigRoeder said:
  RISE-M
  AMD RYZEN 9 9900X
  12c / 24t - 4.4 GHz / 5.6 GHz
  64 GB 2 x 512 GB
  Public 1 Gbps
  Private 1 Gbps
  £85.99 Configure
  
  RISE-L
  AMD RYZEN 9 9950X
  16c / 32t - 4.3 GHz / 5.7 GHz
  128 GB 2 x 960 GB
  Public 1 Gbps
  Private 1 Gbps
  £128.99

  OVH right now, they also have Ryzen 9 5900X, Ryzen 7 9700X, Ryzen 7 5800X, ryzen 5 for less than £70.

  Go for this man. If you need help with OVH account, I can help. lol.

  For £120, you can have 4x of you asked for. I wish I can be in this business, I can be your ovh reseller

  Thanks for the suggestion! I somehow missed these configurations when browsing OVH's website. The RISE-M with the Ryzen 9 9900X looks particularly interesting, but unfortunately, it seems to be unavailable in Poland at the moment. The RISE-L is available, but only in France, which is a bit too far from my preferred location.

  I'll keep checking OVH's website from time to time in case something suitable becomes available. Thanks again for the tip! 🙂
• dimm Member

  October 12

  @CraigRoeder said:
  BTW, over the years I optimized things, I dont think your stack need that much of power, I had to go up to 16 vcore, and now I am using only 1.5 dedi core, for 5 sites. 2 of them have 1GB db (400k rows, and 10M relation tbl. ).
  Calculate your db size, number requests per second and requests per day, p50, p90, p99 of your requests's time. You'll see.

  (Dedi resource is much better than you can expected, also php 8.5 is now 10x better than 7.4), also, with a dedicated one, you can setup some models with 0 down time roll back/backup/full vmd clones.... all yours!

  Haha, to be fair, it's a fairly ordinary Laravel e-commerce store with around 60k products, but unfortunately, the code isn't exactly a masterpiece of optimization. 😂 There are quite a few unnecessary SQL queries, N+1 issues, and some PHP logic that could definitely be designed better.

  I'm more of a frontend developer, and I'm trying to improve my backend skills, but I don't have enough time to properly refactor everything at the moment. So for now, throwing more CPU power at the problem seems to be the easier workaround!

  I'm seriously considering going bare metal, partly because I'd also like more control over backups, snapshots and recovery. I've installed Proxmox VE on an old spare desktop at home and have been experimenting with it. So far, things are going reasonably well, but running it in production is a different story, and I'm a little nervous about getting something wrong.

  One thing I'm unsure about is hardware reliability. If a physical server fails, how quickly do providers typically repair or replace it? Are hardware failures something you encounter fairly often, or are they relatively rare with modern servers? I do have backups, but downtime while waiting for a replacement would still be a concern.

  Also, if you were setting up a production server with Proxmox for the first time, are there any particular pitfalls you'd watch out for, especially regarding networking, backups and recovery?

  I guess that's another reason to hang around here — lots of experienced people to learn from before I accidentally turn a production server into a very expensive learning experience. 😂

  By the way, apologies if my English sounds a bit awkward - I'm using a translator to write these posts))
• oloke Member, Host Rep

  October 12

  @dimm said: As for k.pl, I actually bought a VPS from them. The performance is pretty good for €55/month (VPS Ryzen, 8vCPU, 16GB RAM, 250Gb Disk), so I can't complain about the value for money! 🙂 However, there are a few limitations for my use case: no custom ISO boot and no option for users to create their own VPS snapshots or backups. Being able to take a snapshot before a system upgrade or boot a Veeam recovery ISO to restore a full-machine backup would be really useful for me.

  I also had a server with them, can't say for their support response time as I never had any issue, not a single downtime in a full year. Later cancelled as I didn't need that server anymore, but the experience was rather good.
  Right, I don't recall them allowing custom ISOs. There are workarounds (such as loading netboot.xyz from UEFI directly) but that's hacky on its own.

  @dimm said: Mevspace also gives me the impression of having a more thoughtfully designed infrastructure

  That's interesting. Korbank has been providing service for over 30 years now, I always assumed their infra is as thoughtful as it gets.

  @Mevspace on the other hand started operating in 2010 as Skynode, then transitioned to Mevspace in 2021. I know their new DC has their own "modular" architecture with containers and stuff. Not saying it's bad but I wouldn't consider korbank less experienced.

  Now I think you can also consider artnet in Gdańsk, they seem to also offer dedis with modern Ryzens:
  https://www.artnet.pl/en/dedicated-servers/configuration/

  Thanked by 1dimm
• dimm Member

  October 12

  @oloke said: That's interesting. Korbank has been providing service for over 30 years now, I always assumed their infra is as thoughtful as it gets.

  Oh, I see. To be fair, I was only referring to the management panel, not the underlying infrastructure. I've registered with both k.pl and Mevspace, and Mevspace's panel seems to offer more tabs, features and options to play around with.

  Of course, that's a very superficial impression, and I have no real experience with either provider's dedicated servers or infrastructure. I'm still a beginner in this area, so please don't take my impression as a serious comparison of the two providers))

  Thanked by 1oloke
• CraigRoeder Member

  October 12

  @dimm said:

  @CraigRoeder said:
  BTW, over the years I optimized things, I dont think your stack need that much of power, I had to go up to 16 vcore, and now I am using only 1.5 dedi core, for 5 sites. 2 of them have 1GB db (400k rows, and 10M relation tbl. ).
  Calculate your db size, number requests per second and requests per day, p50, p90, p99 of your requests's time. You'll see.

  (Dedi resource is much better than you can expected, also php 8.5 is now 10x better than 7.4), also, with a dedicated one, you can setup some models with 0 down time roll back/backup/full vmd clones.... all yours!

  Haha, to be fair, it's a fairly ordinary Laravel e-commerce store with around 60k products, but unfortunately, the code isn't exactly a masterpiece of optimization. 😂 There are quite a few unnecessary SQL queries, N+1 issues, and some PHP logic that could definitely be designed better.

  I'm more of a frontend developer, and I'm trying to improve my backend skills, but I don't have enough time to properly refactor everything at the moment. So for now, throwing more CPU power at the problem seems to be the easier workaround!

  I'm seriously considering going bare metal, partly because I'd also like more control over backups, snapshots and recovery. I've installed Proxmox VE on an old spare desktop at home and have been experimenting with it. So far, things are going reasonably well, but running it in production is a different story, and I'm a little nervous about getting something wrong.

  One thing I'm unsure about is hardware reliability. If a physical server fails, how quickly do providers typically repair or replace it? Are hardware failures something you encounter fairly often, or are they relatively rare with modern servers? I do have backups, but downtime while waiting for a replacement would still be a concern.

  Also, if you were setting up a production server with Proxmox for the first time, are there any particular pitfalls you'd watch out for, especially regarding networking, backups and recovery?

  I guess that's another reason to hang around here — lots of experienced people to learn from before I accidentally turn a production server into a very expensive learning experience. 😂

  By the way, apologies if my English sounds a bit awkward - I'm using a translator to write these posts))

  English is not my language, i dont use AI/translator, as long as it is understandable, no need to sorry. Just focus on the points.

  For optimizing, it's fairly easy. Just mark down some slow pages, (DO: ask AI how to identify slow pages on my localhost), and then timing/profiling on that. 60k is a small number, I guess your db is 100mb or less, innodb cache to RAM is definitely help, with proper indexes. On your localhost, if it is 40ms or less for any page, that'd be perfect, 1 core can handle 25 CCU or 2M requests/day. Mostly, you just need to attack a few slow pages only. 1 day work man. No need to refactor thing, sometime, rewriting an sql query really changes the game. I bet your stack need to rewrite some SQL queries. And I also bet that you need 2 cores or less, 4 cores for peaks.... and even the vcore, of 2015 cpu models, php/mariadb shouldn't require game cpu (like ryzen)...

  For hardware reliability, My 2017 OVH bare metal is perfectly reliable (9 year old). People complained mostly about disk and its changing time, my disks are barely used. I have heard bad reviews here and there, but it'd be the same for VPS any where. You do soft raid, and that'd be fine. Over the last year, my OVH vps and KS game 2 are 100% up time. But you could expect 99.xx only. If you want 100% for real, that'd require more work.

  Proxmox is good, I tried that. one of my game ((KS game 2)) is old, disk is on vmware centos 32 bit, I had to install windows and vmware, and copy and done. So, any way, it's up to you. just block all the ports, do fail2ban, perm/key only.... you can forward the ports all you want, clone vm any time...

  OVH Poland: I bet this one is still 4 times more than you ever needed, since you'll have raid already, no need to worry about disk failure. But for 100% uptime, order one in PL, one in DE if you want to donate to OVH.

  SYS-GAME-1
  CPU AMD Ryzen 5 3600X
  6 c / 12 t
  3.8 GHz / 4.4 GHz
  Storage
  2 x 512 GB
  Bandwidth
  Public 500 Mbps
  Price
  $55.30

  This RISE-3 is available in Poland, too. And I bet that it is 10x than you ever needed.

  RISE-3
  CPU AMD Ryzen 9 5900X
  12 c / 24 t
  3.7 GHz / 4.8 GHz
  32 GB to 128 GB
  Storage
  2 x 512 GB -4 x 3.84 TB
  Bandwidth
  Public 1 Gbps - 3 Gbps
  Private 1 Gbps - 2 Gbps
  Price
  $102

  wanna bet?

  Thanked by 1dimm
• CraigRoeder Member

  October 12

  This is my python script, vibe coded, it focused on SEO, but you can use it for profiling, too. Just enable localhost pointing domain, the running time on localhost is closest to CPU-time. Also enable error log, error showing... and then attack the slowest pages. MAX_PAGES can be 1k, or depth=10, depending on your structures... The average loading time and slow threshold will help you on optimizing things...

  import requests
  from urllib.parse import urljoin, urlparse
  from collections import defaultdict
  import time
  from bs4 import BeautifulSoup
  import csv
  from datetime import datetime
  import urllib3
  import gc
  import json
  from collections import deque

  Disable SSL warnings for localhost without cert

  urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

  class SEOAuditor:
  def init(self, base_url, max_pages=1000, max_depth=3):
  self.base_url = base_url
  self.max_pages = max_pages
  self.max_depth = max_depth
  self.visited = set()
  self.to_visit = deque([(base_url, 0)]) # O(1) queue
  self.enqueued = {base_url} # URLs already queued

      
      # SEO issue tracking
      self.duplicate_titles = defaultdict(list)
      self.duplicate_descriptions = defaultdict(list)
      self.page_depths = {}
      self.missing_titles = []
      self.missing_descriptions = []
      self.broken_links = []
      self.thin_content = []
      self.missing_h1 = []
      self.duplicate_h1 = defaultdict(list)
      
      # Redirect tracking
      self.redirects_301 = []
      self.redirects_302 = []
      self.redirect_chains = []
      
      # Performance tracking
      self.load_times = []
      
      # Video tracking
      self.video_pages = []
      self.videos_by_depth = defaultdict(int)
      
      # Progress tracking
      self.start_time = time.time()
      
      # Session with SSL verification disabled
      self.session = requests.Session()
      self.session.verify = False
      self.session.max_redirects = 10  # Track redirect chains
      
  def get_page_depth(self, url):
      """Calculate how many clicks from homepage"""
      path = urlparse(url).path.strip('/')
      if not path:
          return 0
      return path.count('/') + 1
  
  def is_video_page(self, soup, url):
      """Detect if this is a video page"""
      # Check for common video indicators
      indicators = [
          soup.find('video'),  # HTML5 video tag
          soup.find('meta', property='og:type', content='video'),
          soup.find('script', type='application/ld+json'),  # Could contain VideoObject
          'video' in url.lower(),
          '/watch' in url.lower(),
          '/v/' in url.lower(),
      ]
      return any(indicators)
  
  def fetch_page(self, url):
      """Fetch and parse a page, track redirects and load time"""
      try:
          start_time = time.time()
          
          # Allow redirects but track them
          response = self.session.get(url, timeout=10, allow_redirects=True)
          
          load_time = time.time() - start_time
          self.load_times.append({'url': url, 'load_time': load_time, 'status': response.status_code})
          
          # Track redirect chain
          if response.history:
              chain = [r.url for r in response.history] + [response.url]
              
              for i, hist_response in enumerate(response.history):
                  if hist_response.status_code == 301:
                      self.redirects_301.append({
                          'from': hist_response.url,
                          'to': response.history[i+1].url if i+1 < len(response.history) else response.url,
                          'status': 301
                      })
                  elif hist_response.status_code == 302:
                      self.redirects_302.append({
                          'from': hist_response.url,
                          'to': response.history[i+1].url if i+1 < len(response.history) else response.url,
                          'status': 302
                      })
              
              if len(chain) > 2:  # More than one redirect = chain
                  self.redirect_chains.append({
                      'original': url,
                      'chain': chain,
                      'chain_length': len(chain) - 1
                  })
          
          if response.status_code == 200:
              return BeautifulSoup(response.content, 'html.parser'), response.status_code
          else:
              self.broken_links.append({'url': url, 'status': response.status_code})
              
      except Exception as e:
          self.broken_links.append({'url': url, 'error': str(e)})
          
      return None, None
  def extract_links(self, soup, current_url):
      """Extract all internal links"""
      text = soup.get_text()
  
      if "Warning:" in text or "Fatal error:" in text or "Undefined variable" in text or "unexpected token" in text:
          print("\n⚠️ PHP ERROR FOUND")
          print("PAGE:", current_url)
          exit(1);
  
  
      links = []
      for link in soup.find_all('a', href=True):
          href = link['href']
          full_url = urljoin(current_url, href)
          
          # Only internal links
          if urlparse(full_url).netloc == urlparse(self.base_url).netloc:
              # Remove fragment for deduplication but keep query
              parsed = urlparse(full_url)
              clean_url = parsed._replace(fragment='', query='').geturl()
  
  
  
  
  
              if current_url!=clean_url and (clean_url == "https://myvideo.net/" or "https://myvideo.net/tags/shemale" in clean_url):
                  print("\nFOUND LINK:")
                  print("FROM PAGE:", current_url)
                  print("RAW HTML:", str(link))
                  exit();
  
              # Normalize trailing slash (remove it unless it's the root)
              #if clean_url.endswith('/') and parsed.path != '/':
              #    clean_url = clean_url.rstrip('/')
              links.append(clean_url)
      return links
  
  def analyze_page(self, url, soup, depth):
      """Analyze SEO elements of a page"""
      self.page_depths[url] = depth
      
      # Check if it's a video page
      if self.is_video_page(soup, url):
          self.video_pages.append({'url': url, 'depth': depth})
          self.videos_by_depth[depth] += 1
      
      # Title analysis
      title_tag = soup.find('title')
      title = title_tag.text.strip() if title_tag else None
      if not title:
          self.missing_titles.append(url)
      else:
          # Only store duplicates if we haven't already stored too many
          if len(self.duplicate_titles[title]) < 100:
              self.duplicate_titles[title].append(url)
      
      # Meta description
      meta_desc = soup.find('meta', attrs={'name': 'description'})
      description = meta_desc.get('content', '').strip() if meta_desc else None
      if not description:
          self.missing_descriptions.append(url)
      else:
          self.duplicate_descriptions[description].append(url)
      
      # H1 analysis
      h1_tags = soup.find_all('h1')
      if not h1_tags:
          self.missing_h1.append(url)
      else:
          h1_text = ' '.join([h1.text.strip() for h1 in h1_tags])
          self.duplicate_h1[h1_text].append(url)
      
      # Content length
      body = soup.find('body')
      if body:
          word_count = sum(1 for _ in body.stripped_strings)
  
          if word_count < 300:
              self.thin_content.append({'url': url, 'word_count': word_count})
  
  def print_progress(self, current, total):
      """Print crawl progress with estimation"""
      elapsed = time.time() - self.start_time
      progress = (current / total) * 100 if total > 0 else 0
      
      # Estimate time remaining
      if current > 0:
          avg_time_per_page = elapsed / current
          remaining_pages = total - current
          estimated_remaining = avg_time_per_page * remaining_pages
          
          hours = int(estimated_remaining // 3600)
          minutes = int((estimated_remaining % 3600) // 60)
          seconds = int(estimated_remaining % 60)
          
          eta = f"{hours}h {minutes}m {seconds}s" if hours > 0 else f"{minutes}m {seconds}s"
      else:
          eta = "calculating..."
      
      print(f"\rProgress: {current}/{total} ({progress:.1f}%) | "
            f"Videos found: {len(self.video_pages)} | "
            f"Queue: {len(self.to_visit)} | "
            f"ETA: {eta}", end='', flush=True)
  
  def save_checkpoint(self, pages_crawled):
      """Save progress checkpoint in case of crash"""
      checkpoint = {
          'pages_crawled': pages_crawled,
          'videos_found': len(self.video_pages),
          'timestamp': datetime.now().isoformat()
      }
      try:
          with open('seo_audit_checkpoint.json', 'w') as f:
              json.dump(checkpoint, f)
      except:
          pass  # Don't crash if checkpoint fails
  
  def crawl(self):
      """Main crawling function with depth control"""
      print(f"Starting SEO Audit")
      print(f"{'='*80}")
      print(f"Site: {self.base_url}")
      print(f"Max pages: {self.max_pages}")
      print(f"Max depth: {self.max_depth}")
      print(f"{'='*80}\n")
      
      pages_crawled = 0
      
      while self.to_visit and pages_crawled < self.max_pages:
          url, depth = self.to_visit.popleft()
  
          
          # Skip if already visited or too deep
          if url in self.visited or depth > self.max_depth:
              continue
          
          self.print_progress(pages_crawled, self.max_pages)
          self.visited.add(url)
          pages_crawled += 1
          
          soup, status = self.fetch_page(url)
          if not soup:
              continue
          
          # Analyze the page
          self.analyze_page(url, soup, depth)
          
          # Find new links (at next depth level) BEFORE cleaning up soup
          if depth < self.max_depth:  # Only extract links if we haven't hit max depth
              links = self.extract_links(soup, url)
              MAX_QUEUE = 100000
              for link in links:
                  if link not in self.visited and link not in self.enqueued:
                      if len(self.to_visit) < MAX_QUEUE:
                          self.to_visit.append((link, depth + 1))
                          self.enqueued.add(link)
          
          # Clean up soup object to free memory
          soup.decompose()
          del soup
          
          # Periodic garbage collection every 100 pages
          if pages_crawled % 1000 == 0:
              gc.collect()
              # Save checkpoint
              self.save_checkpoint(pages_crawled)
          
          # No rate limiting needed for localhost
          # time.sleep(0.05)
      
      print("\n")  # New line after progress bar
  
  def generate_report(self):
      """Generate comprehensive SEO audit report"""
      timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
      report_file = f'seo_audit_{timestamp}.txt'
      csv_file = f'seo_issues_{timestamp}.csv'
      
      total_time = time.time() - self.start_time
      avg_load_time = sum(item['load_time'] for item in self.load_times) / len(self.load_times) if self.load_times else 0
      
      with open(report_file, 'w', encoding='utf-8') as f:
          f.write("=" * 80 + "\n")
          f.write("SEO AUDIT REPORT\n")
          f.write("=" * 80 + "\n\n")
          f.write(f"Site: {self.base_url}\n")
          f.write(f"Date: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n")
          f.write(f"Pages Crawled: {len(self.visited)}\n")
          f.write(f"Max Depth: {self.max_depth}\n")
          f.write(f"Total Time: {total_time:.2f} seconds\n")
          f.write(f"Average Load Time: {avg_load_time:.3f} seconds\n\n")
          
          # Video Statistics
          f.write("\n" + "=" * 80 + "\n")
          f.write("VIDEO STATISTICS\n")
          f.write("=" * 80 + "\n")
          f.write(f"Total video pages found: {len(self.video_pages)}\n\n")
          f.write("Videos by depth:\n")
          for depth in sorted(self.videos_by_depth.keys()):
              count = self.videos_by_depth[depth]
              percentage = (count / len(self.video_pages) * 100) if self.video_pages else 0
              f.write(f"  Depth {depth}: {count} videos ({percentage:.1f}%)\n")
          f.write("\n")
          
          # Redirect Analysis
          f.write("\n" + "=" * 80 + "\n")
          f.write("REDIRECT ANALYSIS\n")
          f.write("=" * 80 + "\n")
          f.write(f"301 Permanent Redirects: {len(self.redirects_301)}\n")
          f.write(f"302 Temporary Redirects: {len(self.redirects_302)}\n")
          f.write(f"Redirect Chains (2+ hops): {len(self.redirect_chains)}\n\n")
          
          if self.redirect_chains:
              f.write("Redirect Chains (fix these!):\n")
              for chain in self.redirect_chains[:10]:
                  f.write(f"\nChain length: {chain['chain_length']}\n")
                  for i, url in enumerate(chain['chain']):
                      f.write(f"  {i+1}. {url}\n")
          
          if self.redirects_302:
              f.write("\n302 Temporary Redirects (consider changing to 301):\n")
              for redir in self.redirects_302[:20]:
                  f.write(f"  {redir['from']} → {redir['to']}\n")
          
          # Performance Statistics
          f.write("\n" + "=" * 80 + "\n")
          f.write("PERFORMANCE STATISTICS\n")
          f.write("=" * 80 + "\n")
          f.write(f"Average load time: {avg_load_time:.3f}s\n")
          slow_threshold = avg_load_time * 4  # 4x average is considered slow
          slow_pages = [item for item in self.load_times if item['load_time'] > slow_threshold]
          f.write(f"Slow pages (>{slow_threshold:.3f}s): {len(slow_pages)}\n\n")
          
          # Top 20 slowest pages
          f.write("Top 20 slowest pages:\n")
          slowest_sorted = sorted(self.load_times, key=lambda x: x['load_time'], reverse=True)[:20]
          for i, item in enumerate(slowest_sorted, 1):
              f.write(f"  {i}. {item['load_time']:.3f}s - {item['url']}\n")
          
          # Duplicate Titles
          f.write("\n" + "=" * 80 + "\n")
          f.write("DUPLICATE TITLES\n")
          f.write("=" * 80 + "\n")
          duplicate_title_count = sum(1 for urls in self.duplicate_titles.values() if len(urls) > 1)
          f.write(f"Duplicate title groups: {duplicate_title_count}\n\n")
          for title, urls in list(self.duplicate_titles.items())[:10]:
              if len(urls) > 1:
                  f.write(f"Title: {title}\n")
                  f.write(f"Pages ({len(urls)}):\n")
                  for url in urls[:5]:
                      f.write(f"  - {url}\n")
                  if len(urls) > 5:
                      f.write(f"  ... and {len(urls) - 5} more\n")
                  f.write("\n")
          
          # Page Depth Analysis
          f.write("\n" + "=" * 80 + "\n")
          f.write("PAGE DEPTH ANALYSIS\n")
          f.write("=" * 80 + "\n")
          depth_distribution = defaultdict(int)
          for depth in self.page_depths.values():
              depth_distribution[depth] += 1
          
          f.write("Pages by depth:\n")
          for depth in sorted(depth_distribution.keys()):
              count = depth_distribution[depth]
              percentage = (count / len(self.page_depths) * 100) if self.page_depths else 0
              f.write(f"  Depth {depth}: {count} pages ({percentage:.1f}%)\n")
          
          # Missing Elements
          f.write("\n" + "=" * 80 + "\n")
          f.write("MISSING ELEMENTS\n")
          f.write("=" * 80 + "\n")
          f.write(f"Missing titles: {len(self.missing_titles)}\n")
          if self.missing_titles:
              for url in self.missing_titles[:20]:
                  f.write(f"  - {url}\n")
              if len(self.missing_titles) > 20:
                  f.write(f"  ... and {len(self.missing_titles) - 20} more\n")
          f.write(f"\nMissing descriptions: {len(self.missing_descriptions)}\n")
          if self.missing_descriptions:
              for url in self.missing_descriptions[:20]:
                  f.write(f"  - {url}\n")
              if len(self.missing_descriptions) > 20:
                  f.write(f"  ... and {len(self.missing_descriptions) - 20} more\n")
          f.write(f"\nMissing H1: {len(self.missing_h1)}\n")
          if self.missing_h1:
              for url in self.missing_h1[:20]:
                  f.write(f"  - {url}\n")
              if len(self.missing_h1) > 20:
                  f.write(f"  ... and {len(self.missing_h1) - 20} more\n")
          f.write("\n")
          
          # Thin Content
          f.write("\n" + "=" * 80 + "\n")
          f.write("THIN CONTENT (< 300 words)\n")
          f.write("=" * 80 + "\n")
          f.write(f"Total pages: {len(self.thin_content)}\n\n")
          for item in self.thin_content[:20]:
              f.write(f"{item['word_count']} words: {item['url']}\n")
          
          # Broken Links
          f.write("\n" + "=" * 80 + "\n")
          f.write("BROKEN LINKS & ERRORS\n")
          f.write("=" * 80 + "\n")
          f.write(f"Total: {len(self.broken_links)}\n\n")
          for item in self.broken_links[:20]:
              f.write(f"{item}\n")
      
      # CSV export
      with open(csv_file, 'w', newline='', encoding='utf-8') as csvf:
          writer = csv.writer(csvf)
          writer.writerow(['URL', 'Issue Type', 'Details', 'Depth', 'Load Time'])
          
          for title, urls in self.duplicate_titles.items():
              if len(urls) > 1:
                  for url in urls:
                      load_time = next((item['load_time'] for item in self.load_times if item['url'] == url), 0)
                      writer.writerow([url, 'Duplicate Title', title[:100], self.page_depths.get(url, 0), f"{load_time:.3f}s"])
          
          for url in self.missing_titles:
              load_time = next((item['load_time'] for item in self.load_times if item['url'] == url), 0)
              writer.writerow([url, 'Missing Title', '', self.page_depths.get(url, 0), f"{load_time:.3f}s"])
          
          for redir in self.redirects_301:
              writer.writerow([redir['from'], '301 Redirect', redir['to'], '', ''])
          
          for redir in self.redirects_302:
              writer.writerow([redir['from'], '302 Redirect', redir['to'], '', ''])
          
          for chain in self.redirect_chains:
              writer.writerow([chain['original'], 'Redirect Chain', f"{chain['chain_length']} hops", '', ''])
      
      print(f"\n{'=' * 80}")
      print(f"AUDIT COMPLETE")
      print(f"{'=' * 80}")
      print(f"Report saved to: {report_file}")
      print(f"CSV saved to: {csv_file}")
      print(f"{'=' * 80}\n")
      
      # Console Summary
      print("SUMMARY:")
      print(f"  Total pages crawled: {len(self.visited)}")
      print(f"  Total videos found: {len(self.video_pages)}")
      print(f"  Videos within depth {self.max_depth}: {len(self.video_pages)}")
      print(f"  Duplicate titles: {sum(1 for urls in self.duplicate_titles.values() if len(urls) > 1)}")
      print(f"  301 redirects: {len(self.redirects_301)}")
      print(f"  302 redirects: {len(self.redirects_302)}")
      print(f"  Redirect chains: {len(self.redirect_chains)}")
      print(f"  Missing titles: {len(self.missing_titles)}")
      print(f"  Missing descriptions: {len(self.missing_descriptions)}")
      print(f"  Thin content: {len(self.thin_content)}")
      print(f"  Broken links: {len(self.broken_links)}")
      print(f"  Average load time: {avg_load_time:.3f}s")
      slow_threshold = avg_load_time * 4
      print(f"  Slow pages (>{slow_threshold:.3f}s): {len([item for item in self.load_times if item['load_time'] > slow_threshold])}")
      
      print(f"\n  Videos by depth:")
      for depth in sorted(self.videos_by_depth.keys()):
          count = self.videos_by_depth[depth]
          percentage = (count / len(self.video_pages) * 100) if self.video_pages else 0
          print(f"    Depth {depth}: {count} videos ({percentage:.1f}%)")
  

  if name == "main":
  # Configuration
  BASE_URL = "https://myvideo.net/"
  MAX_PAGES = 10000 # Number of pages to crawl
  MAX_DEPTH = 5 # Maximum depth to crawl (0=homepage, 1=one click away, etc.)
  
  print(f"\nSEO Auditor Configuration:")
  print(f" Base URL: {BASE_URL}")
  print(f" Max Pages: {MAX_PAGES}")
  print(f" Max Depth: {MAX_DEPTH}")
  print(f" SSL Verification: DISABLED (for localhost)\n")
  
  auditor = SEOAuditor(BASE_URL, max_pages=MAX_PAGES, max_depth=MAX_DEPTH)
  auditor.crawl()
  auditor.generate_report()

  Thanked by 1dimm
• dimm Member

  October 14

  @CraigRoeder said: For optimizing, it's fairly easy. Just mark down some slow pages, (DO: ask AI how to identify slow pages on my localhost), and then timing/profiling on that. 60k is a small number, I guess your db is 100mb or less, innodb cache to RAM is definitely help, with proper indexes. On your localhost, if it is 40ms or less for any page, that'd be perfect, 1 core can handle 25 CCU or 2M requests/day. Mostly, you just need to attack a few slow pages only. 1 day work man. No need to refactor thing, sometime, rewriting an sql query really changes the game. I bet your stack need to rewrite some SQL queries. And I also bet that you need 2 cores or less, 4 cores for peaks.... and even the vcore, of 2015 cpu models, php/mariadb shouldn't require game cpu (like ryzen)...

  Thanks a lot for the detailed advice and for sharing your script! I really appreciate it. I'll definitely take a closer look at profiling the application and identifying the slowest pages.

  My store has around 60k clothing products, many of which have 5–10 sizes each, so the total number of product variants is considerably higher. And yes, I'm well aware that parts of my code aren't particularly well optimized)) I do plan to refactor things, but probably not immediately.

  My more urgent priority right now is moving the website out of Kyiv. It's currently hosted in a data center there, and given the risk of infrastructure damage during the coming winter, I'd rather move everything to a safer location in Europe, probably Poland, within the next month. Once the migration is done, I plan to take some time off work and focus on optimizing the Laravel application properly.

  For reference, my current VPS scores around 1,148 single-core and 5,945 multi-core in Geekbench 7, with 8 vCPUs. So you're probably right that a better VPS could be sufficient.

  However, I've somehow become fascinated with the idea of running my own dedicated server with Proxmox)) I like the extra control over virtualization, snapshots, backups and recovery, and I'd also like to learn how to manage this kind of setup properly. So part of this is a practical decision, and part of it is simply wanting to learn something new.

  Thanks again for taking the time to share your experience. It's really helpful!
• dimm Member

  October 14

  I honestly don't understand how dedicated server pricing works. Prices can differ by more than 2x between providers for seemingly identical hardware specifications.

  My guess is that the differences might come down to the actual hardware used — for example, RAM speed and configuration, NVMe drive performance, or even network quality and stability.

  I find it hard to believe that one provider can offer significantly lower prices than its competitors with truly identical hardware and service quality. After all, they're all buying their hardware in the same market, more or less at the same prices.

  I'm planning to contact the providers and ask them what specific RAM modules and NVMe drives they use in these servers. Hopefully, that will help me understand where the price difference comes from.
• CraigRoeder Member

  October 14

  @dimm said: My store has around 60k clothing products, many of which have 5–10 sizes each

  I found out that around 100k+ rows, mysql became a big bottleneck. and I have a strong feeling that this is the case.

  @dimm said: I do plan to refactor things

  You really dont need to, just exam the slow pages, log the sql queries and their times, ask AI to optimize the queries, and you see the big difference right away.

  @dimm said: out of Kyiv

  unless your clients are in ucr, move to the places that you have the most clients, or a big hub that is nearest to the clients

  @dimm said: a better VPS could be sufficient

  actually this is not true, disk speed plays big role when it comes to sql without optimizations. Let's assume a query of 200ms, calculating in php is actually 20ms, the other is reading from the disk (not the RAM - 180ms), then 2x of CPU power can lower 20ms to 10ms. but dedicated disk can change 180ms->80ms. same old CPU will make the total request time to 100ms (2x faster). So, in your case, I'll pick dedicated disk over gaming CPU. (a large RAM and RAM cached optimizing sql will make this 180ms into 30ms easily. That will give you 50ms request time, which is 4x faster without changing CPU).

  @dimm said: 1,148 single-core and 5,945 multi-core in Geekbench 7, with 8 vCPUs

  Metric 8-vCPU VPS Dedicated E5-4660 v4
  Single-core 1,148 Roughly 1,050–1,150 as a directional estimate
  Multi-core 5,945 Likely higher with all 16 cores available
  CPU allocation 8 vCPUs 16 physical cores / 32 threads
  Performance consistency Depends on host contention Depends on sustained clocks and cooling

  yes, your vcpu is not as good as 2016 Broadwell-EP processor. So, SYS-1 $33.20 of OVH is a big improvement already, larger RAM to cache, full disk power to yourself.

  @dimm said: own dedicated server with Proxmox

  Yes, you should do this, it's good to know, also very easy with AI copy-paste CLI.

  @dimm said: dedicated server pricing works

  Unless your business is shady, OVH is one of the best our there. Some others give you crypto payment, which you dont need, some give you extra privacy you dont want, some support you quicker (this you might need, but when it comes to hardwares, it'd be the same).

  In the same OVH, they moved servers around from advance (newest), to the SYS/KS, if the hardwares are identical, lower prices mean that chance of failure is higher (RAM/CPU are almost never an issues, just disk TBW for nvme). but for OVH you can trust. For smaller providers, they just have a few ready for you, when it comes to hardware replacement, it'd be trouble. Anyway, cheaper means used. and you'll never know the level of being used.

  (OVH dont have to buy those old hardwares, they recycled from previous renters, that's why it's cheaper)