chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.278

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1449 lines read.
1430 lines found with ASCII only.
18 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.285

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1449 lines read.
1430 lines found with ASCII only.
18 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

---
* Origin: http://www.vlist.eu (2:280/5555)

Hello Michiel!

12 Oct 24 00:00, you wrote to all:

---
* Origin: http://www.vlist.eu (2:280/5555)

nice ..


Dennis


--- GoldED+/LNX 1.1.5-b20240309
* Origin: ---- BOFH: Problem solved, user deleted. (2:280/2060)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.292

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1457 lines read.
1438 lines found with ASCII only.
18 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.299

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1450 lines read.
1431 lines found with ASCII only.
18 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.306

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1452 lines read.
1433 lines found with ASCII only.
18 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.313

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1446 lines read.
1427 lines found with ASCII only.
18 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.320

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1448 lines read.
1429 lines found with ASCII only.
18 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.327

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1443 lines read.
1424 lines found with ASCII only.
18 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.334

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1429 lines read.
1410 lines found with ASCII only.
18 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.096

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1485 lines read.
1467 lines found with ASCII only.
17 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.348

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1437 lines read.
1418 lines found with ASCII only.
18 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.355

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1491 lines read.
1447 lines found with ASCII only.
43 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.362

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1488 lines read.
1444 lines found with ASCII only.
43 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.003

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1494 lines read.
1450 lines found with ASCII only.
43 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.010

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1482 lines read.
1438 lines found with ASCII only.
43 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.017

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1482 lines read.
1438 lines found with ASCII only.
43 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.024

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1445 lines read.
1426 lines found with ASCII only.
18 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.031

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1436 lines read.
1417 lines found with ASCII only.
18 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.045

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1436 lines read.
1417 lines found with ASCII only.
18 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.038

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1435 lines read.
1416 lines found with ASCII only.
18 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.052

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1430 lines read.
1411 lines found with ASCII only.
18 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

Hello Michiel,

can you please, if you mark your message as "UTF-8", post erroneus string in safe way, e.g. as hex dump, or replacing invalid codepoints with some character like question mark?
If you copy from nodelist INVALID string with BINARY AS IS, it obviously makes your message invalid too.
I was to forcibly mark your message as CP437 to read it, becouse trying to decode it as UTF-8 ends with error.

orig.message to echo UTF-8 on 21 Feb 25 00:15:50

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.052

,208,-=P=I=X=-,S?ve,Bj÷rn_Felten,46-31-960447,9600,CM,XA,V34
^
1430 lines read.
1411 lines found with ASCII only.
18 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

Sergey

... vim
--- PyFTN
* Origin: fluid.fidoman.ru (2:5020/12000)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.053

,208,-=P=I=X=-,S?ve,Bj�rn_Felten,46-31-960447,9600,CM,XA,V34
^
1430 lines read.
1411 lines found with ASCII only.
18 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.053

,208,-=P=I=X=-,S?ve,Bj?rn_Felten,46-31-960447,9600,CM,XA,V34
^
1430 lines read.
1411 lines found with ASCII only.
18 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

Hello Sergey,

On Friday February 21 2025 03:30, you wrote to me:

can you please, if you mark your message as "UTF-8", post erroneus
string in safe way, e.g. as hex dump, or replacing invalid codepoints
with some character like question mark? If you copy from nodelist
INVALID string with BINARY AS IS, it obviously makes your message
invalid too. I was to forcibly mark your message as CP437 to read it, becouse trying to decode it as UTF-8 ends with error.

Hmm.. I see your point.

The program I wrote many years ago only detects the first appearance of a non valid utf8 sequence in a line. It ignores any other invalid sequences in the same line. So only the first occurence is replaced by a question mark, the rest is copied as is.

I think I have found a work around.


Cheers, Michiel

--- GoldED+/W32-MSVC 1.1.5-b20170303
* Origin: Nieuw Schnøørd (2:280/5555)

chkwf8 2.0.1 W32 UTF-8 well formed sequence test.
© 2015-2017, Michiel van der Vlist, PA0MMV (2:280/5555)

File: dailyutf.066

,208,-=P=I=X=-,S?ve,Bj?rn_Felten,46-31-960447,9600,CM,XA,V34
^
1434 lines read.
1415 lines found with ASCII only.
18 lines found with well formed UTF-8 sequences.
1 line found with an ill formed UTF-8 sequence.
0 lines found with a BOM.

... Αναπληρωτής Επίτροπος Αστυνομίας
--- Deputy Nodelist Police Commissioner
* Origin: Главный заместитель комиссара полиции нодлиста (2:280/5555)

Hello Michiel,

orig.message to echo UTF-8 on 07 Mar 25 00:15:58

File: dailyutf.066

,208,-=P=I=X=-,S?ve,Bj?rn_Felten,46-31-960447,9600,CM,XA,V34
^

Got it without issues.
Next, are Björn going to do anything with his nodelist record?

Sergey

... vim
--- PyFTN
* Origin: fluid.fidoman.ru (2:5020/12000)

Hello Sergey,

On Friday March 07 2025 03:22, you wrote to me:

File: dailyutf.066

,208,-=P=I=X=-,S?ve,Bj?rn_Felten,46-31-960447,9600,CM,XA,V34
^

Got it without issues.

Good.

Next, are Björn going to do anything with his nodelist record?

It doesn't look like it. Björn does not participate in de UTF nodelist project. He has his own ideas.. :(

He insists on entering the 'a' and 'o' with umlaut in Säve and Björn in 202/208 in Latin-1 in the normal ASCII nodelist. So in the ASCII list they are replaced by question marks by MakeNl. In the UTF list which in his case is just a copy of the ASCII segment submitted, they appear "as submitted" and the line is flagged as in error by my program.

To get it right Björn would have to be convinced to let go of the idea that entering LATIN-1 in the regular nodelist is not a good idea. Plus that he would have to start participating in the UTF nodelist project.

I have tried and sort of given up... :-(


Cheers, Michiel

--- GoldED+/W32-MSVC 1.1.5-b20170303
* Origin: Nieuw Schnøørd (2:280/5555)

Hello Michiel,

orig.message to echo UTF-8 on 07 Mar 25 09:36:19

He insists on entering the 'a' and 'o' with umlaut in Säve and Björn in 202/208 in Latin-1 in the normal ASCII nodelist. So in the ASCII list they are replaced by question marks by MakeNl. In the UTF list which in his case is just a copy of the ASCII segment submitted, they appear "as submitted" and the line is flagged as in error by my program.

I think it is not very contradictory. I he will success in entering non-ASCII chars in nodelist (making it full 8-bit), encoding must be defined. Ok, if it will be latin-1, but let it be only for European segments.
That is, lets define encoding on per-region or even per-network basis. So when importing nodelist, it must be split back on segments and correctly transcoded.
E.g. default encoding if ASCII, so Zone records must be ASCII. But zone may specify own encoding, so regions in it may use it in own record, and define encoding for underlying regions. Further, region record use zone encoding, and may define encoding for networks. Network record use region encoding and may define encoding for node records.

To get it right Björn would have to be convinced to let go of the idea that entering LATIN-1 in the regular nodelist is not a good idea. Plus that he would have to start participating in the UTF nodelist project.

Usage of latin-1 has the issue, as it is windows' encoding, but in FIDO traditionally used DOS charset such as CP437 (IBMPC), CP850, CP866.

I have tried and sort of given up... :-(

If encoding for defaut nodelist will be settled, there will be no issue to transcode it to UTF-8.

Sergey

... vim
--- PyFTN
* Origin: fluid.fidoman.ru (2:5020/12000)

Hello Sergey,

On Friday March 07 2025 15:01, you wrote to me:

He insists on entering the 'a' and 'o' with umlaut in Säve and
Björn in 202/208 in Latin-1 in the normal ASCII nodelist. So in
the ASCII list they are replaced by question marks by MakeNl. In
the UTF list which in his case is just a copy of the ASCII
segment submitted, they appear "as submitted" and the line is
flagged as in error by my program.

I think it is not very contradictory. I he will success in entering non-ASCII chars in nodelist (making it full 8-bit), encoding must be defined.

The encoding for the regular nodelist IS defined: ASCII and ASCII only. For backward compatibility it must stay that way. There still may be nodelist processing software around that breaks when he highest bit is not zero. That is why MakeNl (without the ALLOW8BIT setting) substitutes a question mark for characters with the highest bit set.

The encoding for the UTF nodelist is also defined: UTF-8.

Ok, if it will be latin-1, but let it be only for European
segments. That is, lets define encoding on per-region or even
per-network basis.

Very bad idea. Having more than one encoding within the same file is a bad idea anyway, not just for the nodelist but for ANY text file.

So when importing nodelist, it must be split back on segments and correctly transcoded. E.g. default encoding if ASCII, so Zone records
must be ASCII. But zone may specify own encoding, so regions in it may
use it in own record, and define encoding for underlying regions.
Further, region record use zone encoding, and may define encoding for networks. Network record use region encoding and may define encoding
for node records.

Are you serious? You really still want every back alley in Fidonet to have its own 8 bit encoding? With all the forward and backward re-encoding and other limitations? C'mon.. That's chaos! Unicode was invented for the very purpose of getting rid of all this codepage shit.

Why do you think Microsoft went full Unicode internally? Three decades ago. Why do you think 99% of what is on the web is UTF-8? To get rid of the mess of all the hundreds of 8 bit encodings that floated around!

Nah, as far as the nodelist goes, it is either just ASCII or UTF-8. No more codepage shit.


Cheers, Michiel

--- GoldED+/W32-MSVC 1.1.5-b20170303
* Origin: Nieuw Schnøørd (2:280/5555)

Hey Michiel!

On Sun, 09 Mar 2025 11:42:16 +0100, you wrote:

Nah, as far as the nodelist goes, it is either just ASCII or UTF-8. No
more codepage shit.

Agreed.

As an aside, one could simply run the UTF-8 generated nodelist through a conversion utility (like iconv or similar) to ensure that it is proper UTF-8, before being distributed to the rest of the world, rather than having a weekly report that shows an error that you say will most likely never be fixed (due to someone being stubborn and/or proud that they are the one creating the error on purpose?).

Regards,
Nick

... Sarcasm: because beating people up is illegal.
--- GoldED+/LNX 1.1.5-b20240309
* Origin: _thePharcyde telnet://bbs.pharcyde.org (Wisconsin) (1:154/10)

Hello Nicholas,

On Sunday March 09 2025 08:13, you wrote to me:

Nah, as far as the nodelist goes, it is either just ASCII or UTF-8.
No more codepage shit.

Agreed.

Thank you.

My notoriously unreliable memory tells me that you name should actually be spelled "Boël" with two dots on the 'e'. Is that correct?

As an aside, one could simply run the UTF-8 generated nodelist through
a conversion utility (like iconv or similar) to ensure that it is
proper UTF-8, before being distributed to the rest of the world,
rather than having a weekly report that shows an error that you say
will most likely never be fixed (due to someone being stubborn and/or proud that they are the one creating the error on purpose?).

This particular problem could indeed be easely fixed at the ZC's side. But it should not be needed. Plus that it is hard to fight against someone knwowingly and willingly looking for loopholes to derail the system. If this particular loophole were blocked it could easely turn into an arms race that benefits no one. The weekly error may be the lesser evil...


Cheers, Michiel

--- GoldED+/W32-MSVC 1.1.5-b20170303
* Origin: Nieuw Schnøørd (2:280/5555)

Hey Michiel!

On Sun, 09 Mar 2025 18:04:36 +0100, you wrote:

My notoriously unreliable memory tells me that you name should actually
be spelled "Boël" with two dots on the 'e'. Is that correct?

That is correct. However:

1) The US keyboard doesn't easily allow for it

2) I don't care to type some kind of alt-key combination to produce diaereses every time I want to type my name.

3) Whether I write it that way or not, most Americans don't realize what they are actually there for (or even the definition of a diaereses and what it actually means). All through life it has been pronounced "bowl" or something random like that, unless they have actually asked myself, a family member, or friend in the past.

This particular problem could indeed be easely fixed at the ZC's side.
But it should not be needed. Plus that it is hard to fight against
someone knwowingly and willingly looking for loopholes to derail the
system. If this particular loophole were blocked it could easely turn
into an arms race that benefits no one. The weekly error may be the
lesser evil...

This 'loophole' was originally done for the ASCII nodelist, though, correct? I seem to recall it being in the original nodelist long before it was an issue in the UTF-8 version.

Regards,
Nick

... Sarcasm: because beating people up is illegal.
--- GoldED+/LNX 1.1.5-b20240309
* Origin: _thePharcyde telnet://bbs.pharcyde.org (Wisconsin) (1:154/10)

Hello Nicholas,

On Sunday March 09 2025 12:47, you wrote to me:

My notoriously unreliable memory tells me that you name should
actually be spelled "Boël" with two dots on the 'e'. Is that
correct?

That is correct. However:

OK.

1) The US keyboard doesn't easily allow for it

It is not the keyboard. Contrary to what they do in Belgium, in The Netherlands we do not use a special Dutch keyboard with dedicated keys fo the special characters. We use the same physical US keyboard that you have. It is the keyboard driver that takes care of the special characters. Some keys have been made into "dead keys" by the driver. E,g, to get an 'e' with diaresis I first press the key for the double quote, followed by the 'e'. To type the double quote itself, i have to press that key twice. It is easy once you are used to it.

2) I don't care to type some kind of alt-key combination to produce diaereses every time I want to type my name.

3) Whether I write it that way or not, most Americans don't realize
what they are actually there for (or even the definition of a
diaereses and what it actually means). All through life it has been pronounced "bowl" or something random like that, unless they have
actually asked myself, a family member, or friend in the past.

Understood. So I take it you are not really interested in the UTF nodelist project to get your name with an 'ë' in the nodelist?

This particular problem could indeed be easely fixed at the ZC's
side. But it should not be needed. Plus that it is hard to fight
against someone knwowingly and willingly looking for loopholes to
derail the system. If this particular loophole were blocked it
could easely turn into an arms race that benefits no one. The
weekly error may be the lesser evil...

This 'loophole' was originally done for the ASCII nodelist, though, correct? I seem to recall it being in the original nodelist long
before it was an issue in the UTF-8 version.

I do not recall the exact time frame, but it is correct that the main issue is with the ASCII nodelist. That it also appears as an error in the UTF-8 nodelist is "colleteral damage".

The ZC does double processing. One for the ASCII list and one for the UTF list. For the UTF list MakeNl is run with ALLOW8BIT set. For regions that dot not participate in the UTF list, the ZC uses the ASCII segment for both list. For regions that do participate, their seperate UTF region segment is used for the UTF list.

RC20 does not participate in the UTF nodelist project. He does not send two different segments. But his segment is not pure ASCII, it contain some characters in Latin-1. So for the ASCII list, the ZC's MakNl substitutes question marks for the non ASCII characters. For the UTF list, they are passed "as is". But that was only was noticed after I started my weekly error report.


Cheers, Michiel

--- GoldED+/W32-MSVC 1.1.5-b20170303
* Origin: Nieuw Schnøørd (2:280/5555)

Re: UTF-8 nodelist report
By: Nicholas Boel to Michiel van der Vlist on Sun Mar 09 2025 12:47 pm

3) Whether I write it that way or not, most Americans don't realize what they are actually there for (or even the definition of a diaereses and what it actually means). All through life it has been pronounced "bowl" or something random like that, unless they have actually asked myself, a family member, or friend in the past.

So what is the proper pronunciation? :-)
--
digital man (rob)

Rush quote #82:
The clouds prepare for battle in the dark and brooding silence
Norco, CA WX: 70.6�F, 28.0% humidity, 11 mph W wind, 0.00 inches rain/24hrs
--- SBBSecho 3.23-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Hello Michiel,

orig.message to echo UTF-8 on 09 Mar 25 11:42:16

The encoding for the regular nodelist IS defined: ASCII and ASCII only. For backward compatibility it must stay that way. There still may be nodelist processing software around that breaks when he highest bit is not zero. That is why MakeNl (without the ALLOW8BIT setting) substitutes a question mark for characters with the highest bit set.

About question mark, is it valid symbol for sysop name? Can somebody name himself as "???? ???? ????" for example?

Very bad idea. Having more than one encoding within the same file is a bad idea anyway, not just for the nodelist but for ANY text file.

Agree, thought it is technically possible, but it will kill 99% programmers.

Are you serious? You really still want every back alley in Fidonet to have its own 8 bit encoding? With all the forward and backward re-encoding and other limitations? C'mon.. That's chaos! Unicode was invented for the very purpose of getting rid of all this codepage shit.

That's the thing that I do to keep all messages on my node in UTF-8... without stable success though.

Why do you think Microsoft went full Unicode internally? Three decades ago. Why do you think 99% of what is on the web is UTF-8? To get rid of the mess of all the hundreds of 8 bit encodings that floated around!

There was two ways... make encoding a part of type, as if you use int for range, how to know, is it in miles or kilometers... or set the one and only encoding.
Second way was chosen.

Nah, as far as the nodelist goes, it is either just ASCII or UTF-8. No more codepage shit.

May be it is duty of ZC, to drop invalid segment, and not "help" sysop to publish errors, replacing them with any mark.

Sergey

... vim
--- PyFTN
* Origin: fluid.fidoman.ru (2:5020/12000)

Hey Michiel!

On Sun, 09 Mar 2025 22:23:24 +0100, you wrote:

It is not the keyboard. Contrary to what they do in Belgium, in The Netherlands we do not use a special Dutch keyboard with dedicated keys
fo the special characters. We use the same physical US keyboard that you have. It is the keyboard driver that takes care of the special
characters. Some keys have been made into "dead keys" by the driver.
E,g, to get an 'e' with diaresis I first press the key for the double
quote, followed by the 'e'. To type the double quote itself, i have to
press that key twice. It is easy once you are used to it.

After 30+ years of not doing it, it may be difficult to get used to it. I'm not saying it's impossible, as I can do ALT-0235 to get the desired "ë", but it's a bit of a challenge to try to remember to do every time I type my name. :)

Understood. So I take it you are not really interested in the UTF
nodelist project to get your name with an 'ë' in the nodelist?

I actually thought we discussed this some time ago. I also believe I was getting the DAILYUTF from you at some point, but possibly when your system went down, and stayed down for some time, it was stopped for some reason (I don't think I ever turned it off).

Is anyone in zone 1 involved (I'm unsure if ZC1 is able to forward it on for me)? Or would the submissions need to go to Ward, since I'm an RC?

The ZC does double processing. One for the ASCII list and one for the
UTF list. For the UTF list MakeNl is run with ALLOW8BIT set. For regions that dot not participate in the UTF list, the ZC uses the ASCII segment
for both list. For regions that do participate, their seperate UTF
region segment is used for the UTF list.

I suppose anyone involved would have to double process with and without the ALLOW8BIT option, correct?

RC20 does not participate in the UTF nodelist project. He does not send
two different segments. But his segment is not pure ASCII, it contain
some characters in Latin-1. So for the ASCII list, the ZC's MakNl substitutes question marks for the non ASCII characters. For the UTF
list, they are passed "as is". But that was only was noticed after I
started my weekly error report.

Is it even LATIN-1? Or does it just end up that way because of CP437 probably turns into ISO-8859-1 on Windows? Isn't CP437 -> ISO-8859-1 basically the same as ASCII -> UTF-8 where everything in the former is also represented in the latter (then again, ASCII would be part of many, if not every charset)?

Regards,
Nick

... Sarcasm: because beating people up is illegal.
--- GoldED+/LNX 1.1.5-b20240309
* Origin: _thePharcyde telnet://bbs.pharcyde.org (Wisconsin) (1:154/10)

Hey Sergey!

On Mon, 10 Mar 2025 03:07:16 +0300, you wrote:

That's the thing that I do to keep all messages on my node in UTF-8... without stable success though.

What is unstable about it, if you don't mind me asking?

or set the one and only encoding. Second way was chosen.

Do you still translate all incoming mails to UTF-8 using iconv so they're readable?

May be it is duty of ZC, to drop invalid segment, and not "help" sysop
to publish errors, replacing them with any mark.

I would love to see those "?" entries in DAILYUTF as "🖕" instead. At least then it wouldn't cause errors. :)

Regards,
Nick

... Sarcasm: because beating people up is illegal.
--- GoldED+/LNX 1.1.5-b20240309
* Origin: _thePharcyde telnet://bbs.pharcyde.org (Wisconsin) (1:154/10)

Hey Rob!

On Sun, 09 Mar 2025 17:12:52 -0700, you wrote:

3) Whether I write it that way or not, most Americans don't realize
what they are actually there for (or even the definition of a
diaereses and what it actually means). All through life it has been
pronounced "bowl" or something random like that, unless they have
actually asked myself, a family member, or friend in the past.

So what is the proper pronunciation? :-)

The same as Noël. Not sure I've ever heard anyone pronounce that wrong. :)

Regards,
Nick

... Sarcasm: because beating people up is illegal.
--- GoldED+/LNX 1.1.5-b20240309
* Origin: _thePharcyde telnet://bbs.pharcyde.org (Wisconsin) (1:154/10)

Hello Sergey,

On Monday March 10 2025 03:07, you wrote to me:

About question mark, is it valid symbol for sysop name? Can somebody
name himself as "???? ???? ????" for example?

Onde Fidonet sysop can ask more question than 10 FTSC members can answer. So I pass.

Very bad idea. Having more than one encoding within the same file
is a bad idea anyway, not just for the nodelist but for ANY text
file.

Agree, thought it is technically possible, but it will kill 99% programmers.

There is no generally agreed mechanism to change the encoding in the middle of a text file. Is say: don't even try it.

Are you serious? You really still want every back alley in
Fidonet to have its own 8 bit encoding? With all the forward and
backward re-encoding and other limitations? C'mon.. That's chaos!
Unicode was invented for the very purpose of getting rid of all
this codepage shit.

That's the thing that I do to keep all messages on my node in UTF-8... without stable success though.

OK. he message base however is something different. The best may be to archive them "as is". Or not, I don't know. But now we are no discussing the nodelist.

Why do you think Microsoft went full Unicode internally? Three
decades ago. Why do you think 99% of what is on the web is UTF-8?
To get rid of the mess of all the hundreds of 8 bit encodings
that floated around!

There was two ways... make encoding a part of type, as if you use int
for range, how to know, is it in miles or kilometers... or set the one
and only encoding. Second way was chosen.

So let us stick to that choice.

Nah, as far as the nodelist goes, it is either just ASCII or
UTF-8. No more codepage shit.

May be it is duty of ZC, to drop invalid segment, and not "help" sysop
to publish errors, replacing them with any mark.

I say it is the resposibiliy of the RCs to submit an error free segment to the ZC. But let's not make a mauntain out of a mole hil. We have ONE RC submitting a segment that causes ONE line in the nodlist to be in error. An error that is largely cosmetic.


Cheers, Michiel

--- GoldED+/W32-MSVC 1.1.5-b20170303
* Origin: Nieuw Schnøørd (2:280/5555)

Hello Nicholas,

On Sunday March 09 2025 21:03, you wrote to me:

Some keys have been made into "dead keys" by the driver. E,g, to get an
'e' with diaresis I first press the key for the double quote, followed
by the 'e'. To type the double quote itself, i have to press that key
twice. It is easy once you are used to it.

After 30+ years of not doing it, it may be difficult to get used to
it.

After the dead key method was introduced I got accustomed to it fairly quickly. And that was also after several decades of not doing it...

Then again here in The Neterlands we are used to dealing with accents and diaresis. They are paty of out language.

I'm not saying it's impossible, as I can do ALT-0235 to get the
desired "ë",

ALT 137 in DOS.
There is the alternative of "AltGr". The right Alt key acts as a second "shift" key. E.g. "AltGr 5" is the Euro sign.

but it's a bit of a challenge to try to remember to do
every time I type my name. :)

Yeah, that's awfull.

In DOS there was the option of defining a keyboard macro for things like that. I don't know about Windows or Linus. Did not need it any mor4e.

Understood. So I take it you are not really interested in the UTF
nodelist project to get your name with an 'ë' in the nodelist?

I actually thought we discussed this some time ago.

Yes we did, but I do not recall the details.

I also believe I was getting the DAILYUTF from you at some point, but possibly when your system went down, and stayed down for some time, it
was stopped for some reason (I don't think I ever turned it off).

You want me to turn it on again?

Is anyone in zone 1 involved (I'm unsure if ZC1 is able to forward it
on for me)? Or would the submissions need to go to Ward, since I'm an
RC?

At the moment no one in Z1 participates in de dailyutf,

The ZC does double processing. One for the ASCII list and one for
the UTF list. For the UTF list MakeNl is run with ALLOW8BIT set.
For regions that dot not participate in the UTF list, the ZC uses
the ASCII segment for both list. For regions that do participate,
their seperate UTF region segment is used for the UTF list.

I suppose anyone involved would have to double process with and
without the ALLOW8BIT option, correct?

Yes. When I was RC I also did it. It is not hard, the UTF part is mostly a copy of the ASCII setup. Except for the file names and the ALLOW8BIT setting.

RC20 does not participate in the UTF nodelist project. He does not
send two different segments. But his segment is not pure ASCII, it
contain some characters in Latin-1. So for the ASCII list, the ZC's
MakNl substitutes question marks for the non ASCII characters. For
the UTF list, they are passed "as is". But that was only was
noticed after I started my weekly error report.

Is it even LATIN-1? Or does it just end up that way because of CP437 probably turns into ISO-8859-1 on Windows?

No. Here in Western Europe the default code page for DOS was 850. That is still the case in the Windows CLI. In the Linux community Latin-1 is popular.

Isn't CP437 -> ISO-8859-1 basically the same as ASCII -> UTF-8 where everything in the former is also represented in the latter (then
again, ASCII would be part of many, if not every charset)?

ASCII is a subset of nearly all charactersets in use, with the exception of the obsolete national 7 bit character sets.

I recall Björn mentioning that is is Latin-1. But does it mnatter? The point is that it is NOT UTF-8 and therefore an error in he UTF nodelist.


Cheers, Michiel

--- GoldED+/W32-MSVC 1.1.5-b20170303
* Origin: Nieuw Schnøørd (2:280/5555)

Hey Michiel!

On Mon, 10 Mar 2025 15:26:06 +0100, you wrote:

Then again here in The Neterlands we are used to dealing with accents
and diaresis. They are paty of out language.

Very true, and it's the complete opposite here. Not only my name, but I've seen plenty of words that should have an accent or diaeresis and is quite often typed without.

I'm not saying it's impossible, as I can do ALT-0235 to get the
desired "ë",

ALT 137 in DOS.
There is the alternative of "AltGr". The right Alt key acts as a second "shift" key. E.g. "AltGr 5" is the Euro sign.

I would have to do some reconfiguration to get that. Both of my alt keys do the same thing, as we don't normally have an AltGr key here, but I do know about it by word of mouth.

I also believe I was getting the DAILYUTF from you at some point,
but possibly when your system went down, and stayed down for some
time, it was stopped for some reason (I don't think I ever turned
it off).

You want me to turn it on again?

Sure, please. I still have all the same config in place here to receive it.

I believe I was also receiving this echo from you before you went down as well. It doesn't look like I paused or disabled anything with our link (I just linked with someone else at the time), so it must have stopped at your end when you reconfigured.

At the moment no one in Z1 participates in de dailyutf,

That's a bummer. I figured Andrew, at the very least, would have been interested.. as he was the only one from Z1 that jumped on the IPV6 bandwagon before I did.

Yes. When I was RC I also did it. It is not hard, the UTF part is mostly
a copy of the ASCII setup. Except for the file names and the ALLOW8BIT setting.

Got it. I'll take a look at it when I have some free time this weekend.

No. Here in Western Europe the default code page for DOS was 850. That
is still the case in the Windows CLI. In the Linux community Latin-1 is popular.

I think most of my Windows applications are now either ISO8859-1 or UTF-8. But I'm on Windows 11, so it probably started to change some time ago. As for Linux, ISO-8859-1 may have been popular at some point, but these days most distributions are installed with UTF-8 as the default (which is a good thing, IMO).

ASCII is a subset of nearly all charactersets in use, with the exception
of the obsolete national 7 bit character sets.

I imagine we probably won't see much of those national 7 bit sets any more, unless someone comes online with some very vintage hardware. ;)

I recall Björn mentioning that is is Latin-1. But does it mnatter? The point is that it is NOT UTF-8 and therefore an error in he UTF nodelist.

True, and let's not forget that it's an error in the original nodelist, as well.

Regards,
Nick

... Sarcasm: because beating people up is illegal.
--- GoldED+/LNX 1.1.5-b20240309
* Origin: _thePharcyde telnet://bbs.pharcyde.org (Wisconsin) (1:154/10)

Nicholas Boel wrote to Michiel van der Vlist <=-

Is it even LATIN-1? Or does it just end up that way because of CP437 probably turns into ISO-8859-1 on Windows? Isn't CP437 -> ISO-8859-1 basically the same as ASCII -> UTF-8 where everything in the former is also represented in the latter (then again, ASCII would be part of
many, if not every charset)?

,208,-=P=I=X=-,Säve,Björn_Felten,46-31-960447,9600,CM,XA,V34

The ä is 0xe4
The ö is 0xf6

That is indeed ISO-8859-1
... MultiMail, the new multi-platform, multi-format offline reader!
--- MultiMail/Win v0.52
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Hello Nicholas,

On Monday March 10 2025 18:28, you wrote to me:

Then again here in The Netherlands we are used to dealing with
accents and diaresis. They are part of our language.

Very true, and it's the complete opposite here. Not only my name, but
I've seen plenty of words that should have an accent or diaeresis and
is quite often typed without.

How was that before the computer age? How is it written on your birth certificate? Your father's?

[..]

I also believe I was getting the DAILYUTF from you at some
point, but possibly when your system went down, and stayed down
for some time, it was stopped for some reason (I don't think I
ever turned it off).

You want me to turn it on again?

Sure, please. I still have all the same config in place here to
receive it.

Done. You should receive the next dailyutf tomorrow. (Later today for you)

I believe I was also receiving this echo from you before you went down
as well. It doesn't look like I paused or disabled anything with our
link (I just linked with someone else at the time), so it must have stopped at your end when you reconfigured.

@PATH: 154/10 280/5555

Hmmm....

At the moment no one in Z1 participates in de dailyutf,

That's a bummer. I figured Andrew, at the very least, would have been interested.. as he was the only one from Z1 that jumped on the IPV6 bandwagon before I did.

So you could be the first...

Yes. When I was RC I also did it. It is not hard, the UTF part is
mostly a copy of the ASCII setup. Except for the file names and the
ALLOW8BIT setting.

Got it. I'll take a look at it when I have some free time this
weekend.

Naming conventions here in Z2 for Region and net segments are UTFRxx and UTFNyyy. You can follow that or - as you would be the first in Z1 - create you own naming scheme. ;-)

After you have created UTFR11.xyz the next logical step would be to persuade your ZC to also set it up for Z1 to create an UTFZ1 and send that to Ward to incorpoate it into the DAILYUTF. A little bird tells me that it will be better it I stay out of that part of the process.

If Z1C does not want to cooperate we have to look for other ways to get your UTF region segment into the DAILYUTF.

No. Here in Western Europe the default code page for DOS was 850.
That is still the case in the Windows CLI. In the Linux community
Latin-1 is popular.

I think most of my Windows applications are now either ISO8859-1 or
UTF-8. But I'm on Windows 11, so it probably started to change some
time ago.

There is a difference between Windows in general and the Windows Command Line Inerface. In the CLI of Windows 10, the default is still CP850.

As for Linux, ISO-8859-1 may have been popular at some
point, but these days most distributions are installed with UTF-8 as
the default (which is a good thing, IMO).

Also for he CLI? That is relevant since most Fidonet sowwtare is still CLI based.

ASCII is a subset of nearly all charactersets in use, with the
exception of the obsolete national 7 bit character sets.

I imagine we probably won't see much of those national 7 bit sets any more, unless someone comes online with some very vintage hardware. ;)

I for one have not seen any of it for over a quarter of a century. That's why I labelled them as "obsolete".

I recall Björn mentioning that is is Latin-1. But does it mnatter?
The point is that it is NOT UTF-8 and therefore an error in he UTF
nodelist.

True, and let's not forget that it's an error in the original
nodelist, as well.

Indeed.


Cheers, Michiel

--- GoldED+/W32-MSVC 1.1.5-b20170303
* Origin: Nieuw Schnøørd (2:280/5555)

Hey Chris!

On Mon, 10 Mar 2025 17:22:26 -0700, you wrote:

The ä is 0xe4
The ö is 0xf6

That is indeed ISO-8859-1

Thanks for testing, and letting us know. I sure as heck didn't care enough to check for myself. :)

Regards,
Nick

... Sarcasm: because beating people up is illegal.
--- GoldED+/LNX 1.1.5-b20240309
* Origin: _thePharcyde telnet://bbs.pharcyde.org (Wisconsin) (1:154/10)

Hey Michiel!

On Tue, 11 Mar 2025 12:10:08 +0100, you wrote:

How was that before the computer age? How is it written on your birth certificate? Your father's?

You sparked my interest, so I dug it up from the bottom of a drawer in my bedroom, and no. None of the names on my birth certificate have it (they all look to be typed with a typewriter), except for my mother's signature. :)

Done. You should receive the next dailyutf tomorrow. (Later today for
you)

Thanks!

@PATH: 154/10 280/5555

Perfect. I guess I never turned that off, either. I just set up another link to it at some point when your system went down.

So you could be the first...

Maybe. If I can figure out a good way to send it up the chain of clerks. :)

Naming conventions here in Z2 for Region and net segments are UTFRxx and UTFNyyy. You can follow that or - as you would be the first in Z1 -
create you own naming scheme. ;-)

Noted.

If Z1C does not want to cooperate we have to look for other ways to get
your UTF region segment into the DAILYUTF.

I'll see what happens, I suppose.

There is a difference between Windows in general and the Windows Command Line Inerface. In the CLI of Windows 10, the default is still CP850.

Understood. Here it still looks like it is CP437.

As for Linux, ISO-8859-1 may have been popular at some
point, but these days most distributions are installed with UTF-8
as the default (which is a good thing, IMO).

Also for he CLI? That is relevant since most Fidonet sowwtare is still
CLI based.

Mainly for the CLI, yes. And that is why I chose what I did so that most, if not all of my CLI based Fidonet software works with UTF-8 and IPv6. If it didn't at first, it definitely does now! ;)

Regards,
Nick

... Sarcasm: because beating people up is illegal.
--- GoldED+/LNX 1.1.5-b20240309
* Origin: _thePharcyde telnet://bbs.pharcyde.org (Wisconsin) (1:154/10)