Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Register

Quick Links


Shells Virtual Desktop
BMail.ag - Secure Email Service
Server.net
CPLicense.net
VPS Server
Buy VPN
Vultr
VMs for AI
HostDare
ReliableSite White-Label Dedicated Hosting for Resellers
InterServer VPS
BMail.ag - Secure Email Service
Best VPN
High-Performance Bare Metal Server Solutions
Karvl.com
Server Mania Cloud Hosting
DataWagon Hosting
AlphaVPS Hosting
Evoxt.com
Clouvider
VPS Hosting with NVMe
Residential IPs in the US & 4G Mobile Proxies in EU & US with Unlimited Bandwidth
ReliableSite White-Label Dedicated Hosting for Resellers
Rabisu - Hosting Solutions
Shells Virtual Desktop

Categories

In this Discussion

Home › Providers › Outages
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Crunchbits outage 2024/11/27

ckyteckyte Member
in Outages

Crunchbits Spokan, WA DC is experiencing another outage due to a fiber cut, services are reachable but seeing speeds <1 Mbps up on their failover circuit (I presume)

https://uptime.crunchbits.net/status/public

Bummer

Thanked by 1JasonM

Comments

  • itsTomHarperitsTomHarper Member, Megathread Squad

  • yoursunnyyoursunny Member, IPv6 Advocate

    There's also lots of flooding inside the network:

    tcpdump shows this, 2000 packets per second:

    23:56:57.359057 IP 38.175.202.122.44565 > 255.255.255.255.138: UDP, length 197
23:56:57.359563 IP 38.175.202.122.44565 > 255.255.255.255.138: UDP, length 197
23:56:57.360072 IP 38.175.202.122.44565 > 255.255.255.255.138: UDP, length 197
23:56:57.360578 IP 38.175.202.122.44565 > 255.255.255.255.138: UDP, length 197

    If the owner of 38.175.202.122 is here, please stop sending this traffic.

  • jarjar Patron Provider, Top Host, Veteran
    edited November 2024

    @yoursunny said: If the owner of 38.175.202.122 is here, please stop sending this traffic.

    root@accounts:~# ping 38.175.202.122
PING 38.175.202.122 (38.175.202.122) 56(84) bytes of data.
64 bytes from 38.175.202.122: icmp_seq=1 ttl=63 time=0.426 ms
64 bytes from 38.175.202.122: icmp_seq=1 ttl=127 time=36.7 ms (DUP!)
64 bytes from 38.175.202.122: icmp_seq=2 ttl=63 time=0.546 ms
64 bytes from 38.175.202.122: icmp_seq=2 ttl=127 time=46.0 ms (DUP!)
64 bytes from 38.175.202.122: icmp_seq=3 ttl=63 time=0.661 ms
64 bytes from 38.175.202.122: icmp_seq=3 ttl=127 time=25.4 ms (DUP!)

    I assume this isn't just anyone's server. If it is I'm calling them weird.

  • yoursunnyyoursunny Member, IPv6 Advocate

    @jar said:

    @yoursunny said: If the owner of 38.175.202.122 is here, please stop sending this traffic.

    root@accounts:~# ping 38.175.202.122
> PING 38.175.202.122 (38.175.202.122) 56(84) bytes of data.
> 64 bytes from 38.175.202.122: icmp_seq=1 ttl=63 time=0.426 ms
> 64 bytes from 38.175.202.122: icmp_seq=1 ttl=127 time=36.7 ms (DUP!)
> 64 bytes from 38.175.202.122: icmp_seq=2 ttl=63 time=0.546 ms
> 64 bytes from 38.175.202.122: icmp_seq=2 ttl=127 time=46.0 ms (DUP!)
> 64 bytes from 38.175.202.122: icmp_seq=3 ttl=63 time=0.661 ms
> 64 bytes from 38.175.202.122: icmp_seq=3 ttl=127 time=25.4 ms (DUP!)

    I assume this isn't just anyone's server. If it is I'm calling them weird.

    I guess they are running net-speeder, which transmits every packet twice, or some other similar weirdos.
    https://github.com/snooda/net-speeder

    When a broadcast packet is generated, it gets duplicated to infinity, so that we see 2000 packets per second.

    We call for @crunchbits to suspend this server as it violates Internet protocol.

    Thanked by 6jar Peppery9 bdl JerryHou nick_ fluffernutter
  • ckyteckyte Member

    Looks like they are in the affected subnet Cognet is cracking down on

    IPv4 Renumbering: Separately, and unrelated to just the Yearly plans, we will be renumbering all VPSs in the 38.0.0.0/8 IPv4 space. This is due to a recent AUP change by Cogent, which has created operational risks by threatening to revoke BGP sessions/announcements for unrelated and lawful content hosted in our separate completely non-Cogent facilities. To safeguard your rights and maintain stability, we have already secured new IP space and affected customers will be notified of migration details soon.

    https://discord.com/channels/1115069545896489053/1173725088881840138/1308850061387632681

  • titustitus Member

    Spokane, WA - Intermittent Network Connectivity Issues
    We are currently experiencing intermittent network connectivity issues in the Spokane, WA area. Our team is actively working to stabilize the network as quickly as possible.
    At this time, we do not have an estimated time for full resolution (ETR). We will keep you updated as we make progress.
    Thank you for your patience and understanding as we work to resolve this issue.

    Update as of 16:18:00 PST:
    An upstream provider has identified a fiber cut as the cause of the intermittent network issues in the Spokane, WA area.
    The upstream provider has dispatched a team to assess and address the damage. At this time, there is still no estimated time for resolution (ETR).
    We will continue to provide updates as more information becomes available. Thank you for your patience and understanding.

    -

    @ckyte said: Looks like they are in the affected subnet Cognet is cracking down on

    104.36.87.0/24 affected also :(

  • titustitus Member

    Update as of 18:04:00 PST:
    The upstream provider's team is now on-site and actively working to repair the fiber cut causing the intermittent network issues in the Spokane, WA area.
    There is still no estimated time for resolution (ETR), but we will continue to share updates as the situation progresses.
    Thank you for your continued patience and understanding.

    https://i.postimg.cc/SxzKwPFj/2024-11-28-031404.png

  • newlannewlan Member

    ye Waiting for fix

  • Protocol903Protocol903 Member

    @yoursunny said: We call for @crunchbits to suspend this server as it violates Internet protocol.

    When you say "we", I always wonder, who else do you refer to? Your borg syndicate? Or you work for some org?

    Thanked by 1yoursunny
  • cybertechcybertech Member
    edited November 2024

    @shaikhmanal said:

    @yoursunny said: We call for @crunchbits to suspend this server as it violates Internet protocol.

    When you say "we", I always wonder, who else do you refer to? Your borg syndicate? Or you work for some org?

    he identifies as "we" in public forum.

    Thanked by 2kheng86 yoursunny
  • zedzed Member

    @cybertech said:

    @shaikhmanal said:

    @yoursunny said: We call for @crunchbits to suspend this server as it violates Internet protocol.

    When you say "we", I always wonder, who else do you refer to? Your borg syndicate? Or you work for some org?

    he identifies as "we" in public forum.

    He's obviously speaking for all ipv6 addresses.

    Thanked by 3yoursunny lothos Protocol903
  • MannDudeMannDude Patron Provider, Veteran

    @zed said:

    @cybertech said:

    @shaikhmanal said:

    @yoursunny said: We call for @crunchbits to suspend this server as it violates Internet protocol.

    When you say "we", I always wonder, who else do you refer to? Your borg syndicate? Or you work for some org?

    he identifies as "we" in public forum.

    He's obviously speaking for all ipv6 addresses.

    After all them pushups he's given names to each arm.

    Thanked by 2yoursunny maverick
  • yoursunnyyoursunny Member, IPv6 Advocate

    It seems that 38.175.202.122 has stopped the UDP flooding.
    Latency returns to normal:

    debian@bbb:~$ mtr -bwzc20 104.36.86.147
Start: 2024-11-28T03:59:44+0000
HOST: bbb                                                                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    AR750.lan (192.168.5.1)                                           0.0%    20    1.3   1.0   0.8   1.3   0.2
  2. AS701    lo0-100.WASHDC-VFTTP-325.verizon-gni.net (108.31.110.1)           0.0%    20    9.3   9.3   3.5  37.3   6.9
  3. AS701    ae1325-20.WASHDCDN-MSE01-AA-IE1.verizon-gni.net (100.41.22.168)   0.0%    20    8.2  24.6   3.8  71.6  17.3
  4. AS???    ???                                                              100.0    20    0.0   0.0   0.0   0.0   0.0
  5. AS1299   ash-b2-link.ip.twelve99.net (80.239.135.178)                     20.0%    20   13.9  14.7   4.4  90.6  20.5
  6. AS1299   ash-bb2-link.ip.twelve99.net (62.115.123.124)                     0.0%    20   11.0   9.4   4.9  13.4   2.4
  7. AS1299   nyk-bb2-link.ip.twelve99.net (62.115.136.200)                     0.0%    20   12.0  12.5   8.7  18.3   2.6
  8. AS1299   chi-bb2-link.ip.twelve99.net (62.115.132.135)                     0.0%    20   22.9  27.2  22.5  32.2   3.6
  9. AS1299   sea-b1-link.ip.twelve99.net (62.115.132.155)                     55.0%    20   68.0  69.1  64.2  73.0   2.7
 10. AS1299   tierpoint-ic-357229.ip.twelve99-cust.net (62.115.188.229)         0.0%    20   69.3  79.7  65.7 117.7  16.9
 11. AS17378  te0021.corertr-02.sea.tierpoint.net (173.240.48.213)              0.0%    20   68.7  71.2  67.4  78.0   2.8
 12. AS17378  be1000.corertr-02.spo.tierpoint.net (65.61.125.35)                0.0%    20   81.9  78.7  74.6  83.6   3.2
 13. AS17378  65.61.105-86.ptr.tierpoint.com (65.61.105.86)                    40.0%    20   80.1  80.7  74.5  92.8   4.7
 14. AS400304 147.86.36.104.crunchbits.us (104.36.86.147)                      30.0%    20   79.3  79.7  74.8  93.0   4.6
debian@bbb:~$ mtr -bwzc20 2606:a8c0:3:1fd::a
Start: 2024-11-28T04:00:16+0000
HOST: bbb                                                                     Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS701    2600:4040:2c98:e700::1                                           0.0%    20    1.4   1.2   0.9   1.4   0.2
  2. AS701    2600:4040:2c90::1                                                0.0%    20   10.3  12.3   3.2  65.6  12.9
  3. AS701    2600:4000:1:236::c4                                              0.0%    20    8.0  14.1   3.8  59.9  11.8
  4. AS701    loopback1.ASBNVAEG-PPR01-CC.ALTER.NET (2600:803::5b)             0.0%    20    6.8   7.2   4.4  12.5   1.8
  5. AS1299   ash-b2-link.ip.twelve99.net (2001:2035:0:bb3::1)                 0.0%    20    4.2   8.0   4.1  38.6   7.7
  6. AS1299   ash-bb2-v6.ip.twelve99.net (2001:2034:1:74::1)                   0.0%    20   10.9  11.2   4.6  14.7   2.0
  7. AS1299   nyk-bb2-v6.ip.twelve99.net (2001:2034:1:b8::1)                   0.0%    20   12.6  12.9  11.0  16.0   1.4
  8. AS1299   chi-bb2-v6.ip.twelve99.net (2001:2034:1:c8::1)                  95.0%    20   22.7  22.7  22.7  22.7   0.0
  9. AS???    ???                                                             100.0    20    0.0   0.0   0.0   0.0   0.0
 10. AS1299   tierpoint-ic-357229.ip.twelve99-cust.net (2001:2035:0:205e::2)   0.0%    20   70.1  70.1  66.7  72.2   1.5
 11. AS2119   2001:4670:ffff:2::1                                              0.0%    20   68.2  68.8  67.1  72.0   1.0
 12. AS17378  be1000.corertr-02.spo.tierpoint.net (2001:19b8::1:b)             0.0%    20   73.7  75.6  73.0  83.5   2.9
 13. AS17378  2001:19b8:252::10b                                              20.0%    20   83.3  81.9  73.2  93.2   5.7
 14. AS400304 2606:a8c0:3:1fd::a                                              25.0%    20   89.7  80.3  76.1  89.7   4.1

    Fiber cut is either NTT or Crunchbits' own dark fiber.
    MTR from 10 hours ago shows NTT routes with no loss except the last hop that is caused by the UDP floods, now it's "tierpoint" routes with packet loss in last two hops probably due to congestion.

  • DataRecoveryDataRecovery Member

    @MannDude said:

    @zed said:

    @cybertech said:

    @shaikhmanal said:

    @yoursunny said: We call for @crunchbits to suspend this server as it violates Internet protocol.

    When you say "we", I always wonder, who else do you refer to? Your borg syndicate? Or you work for some org?

    he identifies as "we" in public forum.

    He's obviously speaking for all ipv6 addresses.

    After all them pushups he's given names to each arm.

    IIRC, the left arm is Lucy.

    What's the name of the right one? Rebecca?

  • NeoonNeoon Community Contributor, Veteran
    edited November 2024

    @yoursunny said:
    It seems that 38.175.202.122 has stopped the UDP flooding.
    Latency returns to normal:

    debian@bbb:~$ mtr -bwzc20 104.36.86.147
Start: 2024-11-28T03:59:44+0000
HOST: bbb                                                                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    AR750.lan (192.168.5.1)                                           0.0%    20    1.3   1.0   0.8   1.3   0.2
  2. AS701    lo0-100.WASHDC-VFTTP-325.verizon-gni.net (108.31.110.1)           0.0%    20    9.3   9.3   3.5  37.3   6.9
  3. AS701    ae1325-20.WASHDCDN-MSE01-AA-IE1.verizon-gni.net (100.41.22.168)   0.0%    20    8.2  24.6   3.8  71.6  17.3
  4. AS???    ???                                                              100.0    20    0.0   0.0   0.0   0.0   0.0
  5. AS1299   ash-b2-link.ip.twelve99.net (80.239.135.178)                     20.0%    20   13.9  14.7   4.4  90.6  20.5
  6. AS1299   ash-bb2-link.ip.twelve99.net (62.115.123.124)                     0.0%    20   11.0   9.4   4.9  13.4   2.4
  7. AS1299   nyk-bb2-link.ip.twelve99.net (62.115.136.200)                     0.0%    20   12.0  12.5   8.7  18.3   2.6
  8. AS1299   chi-bb2-link.ip.twelve99.net (62.115.132.135)                     0.0%    20   22.9  27.2  22.5  32.2   3.6
  9. AS1299   sea-b1-link.ip.twelve99.net (62.115.132.155)                     55.0%    20   68.0  69.1  64.2  73.0   2.7
 10. AS1299   tierpoint-ic-357229.ip.twelve99-cust.net (62.115.188.229)         0.0%    20   69.3  79.7  65.7 117.7  16.9
 11. AS17378  te0021.corertr-02.sea.tierpoint.net (173.240.48.213)              0.0%    20   68.7  71.2  67.4  78.0   2.8
 12. AS17378  be1000.corertr-02.spo.tierpoint.net (65.61.125.35)                0.0%    20   81.9  78.7  74.6  83.6   3.2
 13. AS17378  65.61.105-86.ptr.tierpoint.com (65.61.105.86)                    40.0%    20   80.1  80.7  74.5  92.8   4.7
 14. AS400304 147.86.36.104.crunchbits.us (104.36.86.147)                      30.0%    20   79.3  79.7  74.8  93.0   4.6
debian@bbb:~$ mtr -bwzc20 2606:a8c0:3:1fd::a
Start: 2024-11-28T04:00:16+0000
HOST: bbb                                                                     Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS701    2600:4040:2c98:e700::1                                           0.0%    20    1.4   1.2   0.9   1.4   0.2
  2. AS701    2600:4040:2c90::1                                                0.0%    20   10.3  12.3   3.2  65.6  12.9
  3. AS701    2600:4000:1:236::c4                                              0.0%    20    8.0  14.1   3.8  59.9  11.8
  4. AS701    loopback1.ASBNVAEG-PPR01-CC.ALTER.NET (2600:803::5b)             0.0%    20    6.8   7.2   4.4  12.5   1.8
  5. AS1299   ash-b2-link.ip.twelve99.net (2001:2035:0:bb3::1)                 0.0%    20    4.2   8.0   4.1  38.6   7.7
  6. AS1299   ash-bb2-v6.ip.twelve99.net (2001:2034:1:74::1)                   0.0%    20   10.9  11.2   4.6  14.7   2.0
  7. AS1299   nyk-bb2-v6.ip.twelve99.net (2001:2034:1:b8::1)                   0.0%    20   12.6  12.9  11.0  16.0   1.4
  8. AS1299   chi-bb2-v6.ip.twelve99.net (2001:2034:1:c8::1)                  95.0%    20   22.7  22.7  22.7  22.7   0.0
  9. AS???    ???                                                             100.0    20    0.0   0.0   0.0   0.0   0.0
 10. AS1299   tierpoint-ic-357229.ip.twelve99-cust.net (2001:2035:0:205e::2)   0.0%    20   70.1  70.1  66.7  72.2   1.5
 11. AS2119   2001:4670:ffff:2::1                                              0.0%    20   68.2  68.8  67.1  72.0   1.0
 12. AS17378  be1000.corertr-02.spo.tierpoint.net (2001:19b8::1:b)             0.0%    20   73.7  75.6  73.0  83.5   2.9
 13. AS17378  2001:19b8:252::10b                                              20.0%    20   83.3  81.9  73.2  93.2   5.7
 14. AS400304 2606:a8c0:3:1fd::a                                              25.0%    20   89.7  80.3  76.1  89.7   4.1

    Fiber cut is either NTT or Crunchbits' own dark fiber.
    MTR from 10 hours ago shows NTT routes with no loss except the last hop that is caused by the UDP floods, now it's "tierpoint" routes with packet loss in last two hops probably due to congestion.

    Did you report that to the abuse department?

  • HostSlickHostSlick 🚩 Host Rep Tag Suspended
    edited November 2024

    @ckyte said:
    Looks like they are in the affected subnet Cognet is cracking down on

    IPv4 Renumbering: Separately, and unrelated to just the Yearly plans, we will be renumbering all VPSs in the 38.0.0.0/8 IPv4 space. This is due to a recent AUP change by Cogent, which has created operational risks by threatening to revoke BGP sessions/announcements for unrelated and lawful content hosted in our separate completely non-Cogent facilities. To safeguard your rights and maintain stability, we have already secured new IP space and affected customers will be notified of migration details soon.

    https://discord.com/channels/1115069545896489053/1173725088881840138/1308850061387632681

    What kind of Aids Cogent are with this.
    I usually avoid IP Leasing companies that are crying about abuse fast.

    Thanked by 2Andreix sillycat
  • AndreixAndreix Member, Host Rep

    @yoursunny said:

    23:56:57.359057 IP 38.175.202.122.44565 > 255.255.255.255.138: UDP, length 197
23:56:57.359563 IP 38.175.202.122.44565 > 255.255.255.255.138: UDP, length 197
23:56:57.360072 IP 38.175.202.122.44565 > 255.255.255.255.138: UDP, length 197
23:56:57.360578 IP 38.175.202.122.44565 > 255.255.255.255.138: UDP, length 197

    If the owner of 38.175.202.122 is here, please stop sending this traffic.

    At least is not 4letters_c that's flooding you...

    Thanked by 1yoursunny
  • fxdjclubfxdjclub Member

    Unfortunately I have 2 machines in Spokane and now my speed is just terrible.

  • ckyteckyte Member

    Seems to be returning to normal!

  • netguynetguy Member

    Is Crunchbits alive still?

  • Petey_LongPetey_Long Barred
    edited November 2024

    @netguy said:
    Is Crunchbits alive still?

    It's not like Eric is out there with a ditch witch trenching their own fiber lines and neglected to call 811 before he dug but it's likely that some other stupid contractor did. Nothing to do with Crunchbits.

    It's also Thanksgiving and if he's like the rest of us, we're spending time with our families between binge eating trips.

Sign In or Register to comment.