Migrating Hosting. Fediverse / Mastodon Instance - Issues with gem/nginx/default-sites

bamboo4409 · March 2023

Greetings:

I am trying to migrate hosting from one provider to another. Not sure if anyone can help, but happy to send cash or beers your way if we can figure this out.

I already backed up my files and backed up my database.

Here is the instructions for migration:

https://docs.joinmastodon.org/admin/migrating/

The part that I am stuck on, is the "bundle install" section. No matter what it gives the following error:

mastodon@GreenCloud:~/live$ bundle config deployment 'true'
mastodon@GreenCloud:~/live$ bundle config without 'development test'
Your application has set without to "development:test". This will override the global value you are currently setting
mastodon@GreenCloud:~/live$ bundle install -j$(getconf _NPROCESSORS_ONLN)
Fetching source index from https://rubygems.org/
Network error while fetching https://rubygems.org/quick/Marshal.4.8/rake-13.0.6.gemspec.rz (execution expired)

I even tried with --retry flag but still has same error. Anyone think of a workaround?

JabJab · March 2023

Have you tried easiest test - aka download that file via wget/curl to see if it's even working?
Maybe it's IPv6 issue, maybe it's route between, maybe something totally else - but I would assume execution expired and Network error while fetching means can't download the file

Yeah, I would assume this this famous IPv6 network production ready.

jab@green:~$ wget -4 https://rubygems.org/quick/Marshal.4.8/rake-13.0.6.gemspec.rz
--2023-03-29 23:21:05--  https://rubygems.org/quick/Marshal.4.8/rake-13.0.6.gemspec.rz
Resolving rubygems.org (rubygems.org)... 151.101.1.227, 151.101.129.227, 151.101.65.227, ...
Connecting to rubygems.org (rubygems.org)|151.101.1.227|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 716 [application/octet-stream]
Saving to: ‘rake-13.0.6.gemspec.rz’

rake-13.0.6.gemspec.rz                                      100%[========================================================================================================================================>]     716  --.-KB/s    in 0s

2023-03-29 23:21:05 (110 MB/s) - ‘rake-13.0.6.gemspec.rz’ saved [716/716]

jab@green:~$ wget -6 https://rubygems.org/quick/Marshal.4.8/rake-13.0.6.gemspec.rz
--2023-03-29 23:21:09--  https://rubygems.org/quick/Marshal.4.8/rake-13.0.6.gemspec.rz
Resolving rubygems.org (rubygems.org)... 2a04:4e42:200::483, 2a04:4e42:600::483, 2a04:4e42::483, ...
Connecting to rubygems.org (rubygems.org)|2a04:4e42:200::483|:443... failed: Network is unreachable.
Connecting to rubygems.org (rubygems.org)|2a04:4e42:600::483|:443... failed: Network is unreachable.
Connecting to rubygems.org (rubygems.org)|2a04:4e42::483|:443... failed: Network is unreachable.
Connecting to rubygems.org (rubygems.org)|2a04:4e42:400::483|:443... failed: Network is unreachable.

Set your system to prefer IPv4 over IPv6 and I guess it will work - first google result for "Prefer IPv4 over IPv6 [insert_distribution_name]" but it will most likely be edit /etc/gai.conf and uncommenting

#
#    For sites which prefer IPv4 connections change the last line to
#
precedence ::ffff:0:0/96  100

Not_Oles · March 2023

Hi @bamboo4409!

Have you seen https://gist.github.com/zulhfreelancer/e8eb723e97a818b6727fb53bd94a0a96 which suggests disabling IPv6?

If you get something like this when running bundle install...

Network error while fetching https://rubygems.org/quick/Marshal.4.8/xxx-x.x.x.gemspec.rz (execution expired)

...this solution might work for you.

Best wishes!

Tom

bamboo4409 · March 2023

OMG that worked. Thank you! Never even thought about testing the file using wget =X

Going to continue the process and try to get NGINX up and running ... wish me luck!

Not_Oles · March 2023

Haha, I get lucky sometimes. Congrats on your Mastodon instance!

bamboo4409 · May 2023

Hey, @Not_Oles or @JabJab or anyone who can help me.

Still running issues as I finish the process. I know your time is important, so more than happy to send cash or beers for any help getting this running.

I went through the guides:

https://docs.joinmastodon.org/admin/migrating/
https://docs.joinmastodon.org/admin/install/

And I completed everything with the exception of requesting a new cert & starting the mastodon services.

I migrated the database, copied env production, and also restored the /etc/nginx/sites-available files. Yet despite this, can't seem to get the service running:

https://pastebin.com/fRDWbJAA

But when I try to request a certificate, I also get another error:

https://pastebin.com/gQ4BPMws

is there anything I can to do help or troubleshoot? Any help or guidance would be really appreciated, thank you in advance!

Don_Keedic · May 2023

@bamboo4409 said:

Can you post up your nginx virthualhost config? And can you run "nginx -t" and post the output?

I have a feeling it's a straightforward fix but just want to be certain.

bamboo4409 · May 2023

Hey, thanks for offering to help - really appreciate it. Sorry for the pastebin links, this forum was blocking me for posting sql snippets.

root@GreenCloud:/etc/nginx/sites-available# vi default
https://pastebin.com/Vaw6LTYK

root@GreenCloud:/etc/nginx/sites-available# vi mastodon
https://pastebin.com/DejbU86q

Hopefully this is what you mean by the configuration? sorry, I'm new to this; tinkering with linux was one of my covid19 projects.

root@GreenCloud:/etc/nginx/sites-available# nginx -t
https://pastebin.com/HcjXhv7z

Hope this helps..

Don_Keedic · May 2023

Comment all this out:

} <- (End bracket for your port 80 server section)

server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name my0.net;

ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;

# Uncomment these lines once you acquire a certificate:
ssl_certificate /etc/letsencrypt/live/my0.net/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/my0.net/privkey.pem; # managed by Certbot

So it looks like this (remove the ' ) (sorry..markdown isn't cooperating with octothropes)

'#}'

'# server {'
'# listen 443 ssl http2;'
'# listen [::]:443 ssl http2;'
'# server_name my0.net;'

'# ssl_protocols TLSv1.2 TLSv1.3;'
'# ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;'
'# ssl_prefer_server_ciphers on;'
'# ssl_session_cache shared:SSL:10m;'
'# ssl_session_tickets off;'

'# Uncomment these lines once you acquire a certificate:'
'# ssl_certificate /etc/letsencrypt/live/my0.net/fullchain.pem; # managed by Certbot'
'# ssl_certificate_key /etc/letsencrypt/live/my0.net/privkey.pem; # managed by Certbot'

Then restart nginx. It should start successfully.

Run certbot again. It should run successfully.

Uncomment out the lines you commented out, restart nginx and you should be good to go.

The problem is you're transferring an instance that already has an SSL cert. Your current configuration is trying to utilize a cert you don't have yet. Once you comment out the SSL stuff and restart Nginx, it'll be running on port 80. Certbot will reach out, see your server is good and issue you your certificates. Then you uncomment those lines and restart nginx, it'll load up your new certificate and you'll be back up and running!

bamboo4409 · May 2023

So, still getting an error. But I might not be commending out the right lines. Here is a screenshot of my terminal:

https://snipboard.io/MjOa9c.jpg

But when restarting nginx says there is a duplicate root?

https://pastebin.com/Zrf44Nj4

My apologies, for some reason struggling with basic instructions. Let me know if I should comment out more. Or what else I can do to try...

Don_Keedic · May 2023

No worries! Nope, you commented out the lines perfectly.

We fixed two issues there and found another. Let me look back through your mastodon config.

Don_Keedic · May 2023

Line 25 + Line 51 are duplicates in your mastodon configuration (we temporarily extended the port 80 config, so that's why it became a duplicate)

The one other thing you'll need to comment out (in your mastodon config) that I totally missed was your redirects

' if ($host = my0.net) {'
' return 301 https://$host$request_uri;'
' } # managed by Certbot''

and on line 27

'location / { return 301 https://$host$request_uri; }'

The other thing that needs to be commented out was line 51 (the duplicate root) and once that's done, restart nginx and That should get your nginx running.

Give certbot a run again, that should work.

Once you get your cert, you just need to uncomment everything you commented out, restart nginx and it should be back to normal.

bamboo4409 · May 2023

Looks like another duplicate error:

https://pastebin.com/MMH7PX3S

Do you think I should comment that line 62? or what whole section/block?

Don_Keedic · May 2023

Instead of butchering your configuration with comments... keep a copy of it around and just try using a bare bones nginx config just to get your cert.

https://pastebin.com/raw/jSaygCYR - use this as your mastodon config

Restart nginx and then run this

mkdir /var/www/
mkdir /var/www/html
touch /var/www/html/index.html

chown -R www-data:www-data /var/www/html

That should be enough to get a basic server up and running on port 80 and that should be enough to have certbot work for you. Get your certs and then you can replace that bare bones config with the configuration you were using pre-migration, restart nginx and you should be set.

(all of this is assuming you've already opened up port 80/443 on your firewall.) I've migrated and forgot about the firewall a few times. Anyways, hope this gets you up and running man.

bamboo4409 · May 2023

Ok. so I was able to use that barebones config file and start nginx. then I requested a certificate without an issue, then replaced the config with my mastodon config from my backup.

Now

here is a copy of my nginx error log:
https://dpaste.com//36ALHVUVE

Does this mean anything to you?

I am so lost. The contents in this log look greek to me.

bamboo4409 · May 2023

The 3x services were able to start just fine. And it looks like everything is running. but the browser just says 500 error

Don_Keedic · May 2023

@bamboo4409 said:
The 3x services were able to start just fine. And it looks like everything is running. but the browser just says 500 error

Hey sorry I'm back!

This is looking better (less of an NGINX problem anyways..)

Can you verify your mastodon instance is running?

Run

netstat -peanut

And look for a binding on port 3000

If you see a port 3000 and mastodon is binded to the port - temporarily allow port 3000 through your firewall and run

curl -I yourmastodonipaddresshere:3000 and see if you receive a 200 response.

I have a feeling NGINX is good and something's up with Mastodon.

bamboo4409 · May 2023

@Don_Keedic no apology needed, I really appreciate your time and help!

This is what netstat -peanut is giving me:
https://pastebin.com/mnT8dinS

Looks like its 80 & 443. Would you recommend to still run that curl command though regardless?

Don_Keedic · May 2023

tcp 0 0 127.0.0.1:3000 0.0.0.0:* LISTEN 1000 14262871 911049/puma 5.6.5 (

Looks good!

Is there any way you might be able to try a direct connection from another device? Try and narrow down if it's mastodon or Nginx giving us issues.

If you could,

Allow 3000/TCP (temporarily)

Then use your mastodon client to login via IP instead of domain - so

mymastodonip:3000

instead of

mydomain.com

And see if you're able to log in successfully. If you can, it's an Nginx problem and we can get that figured out.

bamboo4409 · May 2023

So what I tried to do, is to use a different browser in a Private Window to connect via the IP and port.

Here are a couple screenshots that the connection failed.

Pina Client:
https://pasteboard.co/h4RTk7Aqti5g.png

Elk Client:
https://pasteboard.co/mFWfH0omyuNb.png

I can keep trying, and share logs with mast or nginx if you think that would be helpful? Or I can just give you the pw to re-do nginx and troubleshoot.

I am okay with either option, just feel awful taking up so much of your time.

Don_Keedic · May 2023

Did you happen allow that port through your firewall? I'm getting a "connection refused" error when a request was made to that ip:port. I should be getting a 20x,30x, 40x or 50x error

I'll hold off on logging into your server directly... too many liabilities, but I do appreciate you offering! I will keep trying to help you get this figured out!

bamboo4409 · May 2023

I am so sorry, but please remind me what to do - to allow the port through firewall?

Don_Keedic · May 2023

Hopefully you have UFW installed to make it easy!

on your command line:

ufw allow 3000/tcp

That will open the port up.

Then when we're done diagnosing (just in case I have to jet off somewhere) to re-enable the rule do

ufw status numbered

note the number next to the 3000/tcp rule

ufw delete rulenumberhere

and that will remove access to 3000/tcp from the outside (it will still be accessible to nginx on your server via localhost but not from the outside)

Just trying to knock out any variables, making sure the base program is running properly and then we can get it tweaked to work from there.

bamboo4409 · May 2023

So I did the following:

root@GreenCloud:~# ufw allow 3000/tcp
Rules updated
Rules updated (v6)
root@GreenCloud:~# systemctl restart nginx
root@GreenCloud:~# systemctl restart mastodon-web
root@GreenCloud:~# systemctl restart mastodon-sidekiq
root@GreenCloud:~# systemctl restart mastodon-streaming

I tried restarting the services to be safe. But when visiting with my browser, still says:

This page isn’t working
my0.net is currently unable to handle this request.
HTTP ERROR 500

Howdy, Stranger!

Categories

In this Discussion

Migrating Hosting. Fediverse / Mastodon Instance - Issues with gem/nginx/default-sites

Comments

Howdy, Stranger!

Quick Links

Categories

In this Discussion

Migrating Hosting. Fediverse / Mastodon Instance - Issues with gem/nginx/default-sites

Comments