Hey Folks,

Hub 3 might be a bit iffy today. I'm trying to track down 2 issues that I cant seem to create in my test envrionment.

I've put it in debug mode, and may need to keep increasing debugging in particular areas until I find it.

Hopefully regular programming will return soon.


...ëîåï
--- SBBSecho 3.20-Linux
* Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)

Re: Hub 3
By: deon to All on Thu Jun 27 2024 08:11 am

Howdy,

Hub 3 might be a bit iffy today. I'm trying to track down 2 issues that I cant seem to create in my test envrionment.

So the first issue I tracked down and fixed (clrghouz had mail for an upstream node, but for some reason wasnt sending it).

The 2nd issue has me stumped. I've been doing some updates after the last couple of days, and I started seeing session failures.

I've spent the last 24-36 hrs debugging and tracing and cant figure it out, so I'm now wondering if it is the upstream system (or my testing hasnt triggered the sequence for the problem).

I can see mail flowing between *most* nodes connected to Hub 3, with the exception of 3:
* 21:1/100 - although a test message just got a response from 6/100 which is via 1/100.
* 21:3/110
* 21:3/119
* 21:3/170

The only commonn thing I can see is that those nodes are using binkd (might be a coincidence?)

The issue I'm seeing is when those poll in, I'm sending a mail packet, and while waiting for the M_GOT, I'm getting random 3 chars (and those characters make me wait for more data which never arrives. After 5 mins, the binkd session times out.

I think if I poll outbound, I can successful send mail (not 100% confident that this is right).

Anybody else having issues?

Jay/Stephan/Drew if you get this I'd be keen to see your logs and maybe turn it up (especially if you are successful getting mail from other systems).


...ëîåï
--- SBBSecho 3.20-Linux
* Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)

On 27 Jun 2024 at 05:33p, deon pondered and said...

The 2nd issue has me stumped. I've been doing some updates after the last couple of days, and I started seeing session failures.
I've spent the last 24-36 hrs debugging and tracing and cant figure it out, so I'm now wondering if it is the upstream system (or my testing hasnt triggered the sequence for the problem).

Hi there

I've sent you some logging from my end at 1/100 which I hope helps :)

Kerr Avon [Blake's 7] 'I'm not expendable, I'm not stupid and I'm not going' avon[at]bbs.nz | bbs.nz | fsxnet.nz

--- Mystic BBS v1.12 A48 (Linux/64)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)

On 28 Jun 2024, deon said the following...

Jay/Stephan/Drew if you get this I'd be keen to see your logs and maybe turn it up (especially if you are successful getting mail from other systems).

Sure, I can send some logs your way.

I'm currently set to loglevel 4 but can ramp that up for a bit as well.


Jay

... A baker's calendar has 13 months

--- Mystic BBS v1.12 A49 2024/05/29 (Linux/64)
* Origin: Northern Realms (21:3/110)

Re: Re: Hub 3
By: Warpslide to deon on Thu Jun 27 2024 08:12 am

Howdy,

Jay/Stephan/Drew if you get this I'd be keen to see your logs and maybe turn it up (especially if you are successful getting mail from other systems).

Sure, I can send some logs your way.

The verbose logging would be helpful, thanks.

In the mean time, I've spun up an old instance of clrghouz and are redirecting you and the others affected to it - so mail appears to be flowing for you now. Those not affected are on the latest clrghouz.

Just need to figure how I broke it. So I'll be progressing each commit until I find the commit that broke it - please bear with me if you see mail errors while I'm chasing this. (Least I know which commit works.)

(The wonders of containers and git that make this kinda easy to debug.)


...ëîåï
--- SBBSecho 3.20-Linux
* Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)

On 28 Jun 2024, deon said the following...

I can see mail flowing between *most* nodes connected to Hub 3, with the exception of 3:
* 21:1/100 - although a test message just got a response from 6/100
which is via 1/100.
* 21:3/110

I don't know if it's related/relevant but when I poll you I'm using IPv6 & SSL in binkd:

node 21:3/100@fsxnet -md -6 -pipe "openssl s_client -6 -quiet -alpn binkp -connect *H:*I" n3.z21.bbs.dege.au:24553 <password>


Jay

... Some things have got to be believed to be seen

--- Mystic BBS v1.12 A49 2024/05/29 (Linux/64)
* Origin: Northern Realms (21:3/110)

Re: Re: Hub 3
By: Warpslide to deon on Thu Jun 27 2024 08:47 am

Howdy,

I don't know if it's related/relevant but when I poll you I'm using IPv6 & SSL in binkd:

Not related unfortunately. I have IPv6 and IPv4 nodes failing, as well as TLS and non-TLS.

The only common thing is binkd strangely.


...ëîåï
--- SBBSecho 3.20-Linux
* Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)

Re: Re: Hub 3
By: deon to Warpslide on Thu Jun 27 2024 10:40 pm

Howdy,

Just need to figure how I broke it. So I'll be progressing each commit until I find the commit that broke it - please bear with me if you see mail errors while I'm chasing this. (Least I know which commit works.)

OK, not sure how I broke it, reverted the patch that did it and we are back in action. Strange it was only affecting inbound connections (when that patch was addressing outbound ones), and only binkd systems.

Anyway, back to regular programming - let me know if you notice anything odd.


...ëîåï
--- SBBSecho 3.20-Linux
* Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)

Howdy,

For hub 3 nodes, sorry, I've been down the last few days.

A neighbours tree (across the road), which is about 25m high snapped in half due to high winds on Wednesday and landed on the power lines that supplies my house and neighbours house. It also included the HFC internet line.

It all happened at a time, when my kids were walking home from school, and my son got hit by the powerlines and the branches of the tree. Sparks flew everywhere, freaked him out, but thankfully the power line was insulated and he is OK.

(Also, fortunately, his sister was annoyed with him, so she was 5m or more behind him.)

Power was restored yesterday and I have the internet company coming out on Wednesday morning to restore the HFC capable. SO hopefully hub 3 will be back in business then.


...ëîåï
--- SBBSecho 3.20-Linux
* Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)

Re: Hub 3
By: deon to All on Mon Sep 02 2024 03:00 pm

Howdy,

For hub 3 nodes, sorry, I've been down the last few days.

Hub 3 is back online and processing mail.

Sorry for the outage folks :)


...ëîåï
--- SBBSecho 3.20-Linux
* Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)

On 04 Sep 2024, deon said the following...

Hub 3 is back online and processing mail.

Heeeeeey! We're back! :)


Jay

... What do you call a cow on a trampoline? A milk shake

--- Mystic BBS v1.12 A49 2024/05/29 (Linux/64)
* Origin: Northern Realms (21:3/110)

On 04 Sep 2024 at 11:41a, deon pondered and said...

Hub 3 is back online and processing mail.

Sorry for the outage folks :)

welcome back :)

Sorry to hear of the weather issues etc... I hope things are on the mend for you now.

Kerr Avon [Blake's 7] 'I'm not expendable, I'm not stupid and I'm not going' avon[at]bbs.nz | bbs.nz | fsxnet.nz

--- Mystic BBS v1.12 A48 (Linux/64)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)

On 04 Sep 2024 at 05:24a, Warpslide pondered and said...

On 04 Sep 2024, deon said the following...

Hub 3 is back online and processing mail.

Heeeeeey! We're back! :)

heeey :)

Kerr Avon [Blake's 7] 'I'm not expendable, I'm not stupid and I'm not going' avon[at]bbs.nz | bbs.nz | fsxnet.nz

--- Mystic BBS v1.12 A48 (Linux/64)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)

Hub 3 is back online and processing mail.

Sorry for the outage folks :)

Thanks for bringing things back online once you were able to. Hope everything else is good since the storm. No need to apologize :)

---
Ganiman
bbs.madetoraid.com:[2323/2222]

--- Mystic BBS v1.12 A47 2021/12/24 (Linux/64)
* Origin: Made to Raid BBS (21:3/174)