Context (somewhat irrelevant to the bug report): I have an InSim application written in C#, using InSimDotNet library. InSimDotNet offers classes that help convert LFS-encoded bytes into Unicode strings and back. I wanted to have a way of confirming that decoding and encoding operations are lossless, thus wrote a class called 'LfsEncodingChecker': a class that does a round-trip, and checks if input matches round-tripped data. For example: GetString() takes in bytes and gives a unicode string, then I use the string and call GetBytes() on it, and check if the original input bytes match the bytes generated from GetBytes(). Same check applies for GetBytes() function. In case of a data loss, it logs a warning message. I saw this in my logs: (explanation below)
Explanation: My checker worked as intended but the server sent me incorrect LFS-encoded bytes, which subsequently made the checker warn for a different reason. A player named Epicproes had a nickname ("｢OK｣ Epic (G29)") that contained Japanese (CP932) characters '｢' and '｣' which should have been sent as '\xA2' and '\xA3' respectively but they arrived as "\xEF\xBF\xBD".
Here are different representations of what I got and what the server should have sent:
Note: In all cases, nickname was truncated by the server to fit in 24 bytes with a null terminator. This is why the nickname ends with "(G2" instead of "(G29)".
I asked ChatGPT about this mysterious EF BF BD sequence, and it seems to be UTF-8 encoded "replacement character" represented as '�'.
I used Epicproes'es nickname and tried reproducing the same issue but InSim was working as intended. I tried locally (client), on a temporary server, and on the same couple (insim server & lfs server, though I had restarted InSim earlier for maintainance, which might have reset things on server side, I don't know) but couldn't reproduce it.
[2025-10-18 02:06:51 INF] [ModularInSim.Program] --- Started ---
[2025-10-18 15:30:26 WRN] [ModularInSim.PortableEncoding.LfsEncodingChecker] GetString(): Round-trip data integrity check failed.
bytes1 '5E4AEFBFBD4F4BEFBFBD2045706963202847325E4C20636F6E6E656374656420284570696370726F657329'
bytes2 '5E4A3FBD4F4B3FBD2045706963202847325E4C20636F6E6E656374656420284570696370726F657329'
result '^J?ｽOK?ｽ Epic (G2^L connected (Epicproes)'
at System.Environment.get_StackTrace()
at ModularInSim.PortableEncoding.LfsEncodingChecker.GetString(Byte[], Int32, Int32)
at InSimDotNet.Packets.IS_MSO..ctor(Byte[])
[TRUNCATED]
[2025-10-18 15:30:26 WRN] [ModularInSim.PortableEncoding.LfsEncodingChecker] GetString(): Round-trip data integrity check failed.
bytes1 '5E4A5E33EFBFBD4F5E374BEFBFBD204570696320284732'
bytes2 '5E4A5E333FBD4F5E374B3FBD204570696320284732'
result '^J^3?ｽO^7K?ｽ Epic (G2'
at System.Environment.get_StackTrace()
at ModularInSim.PortableEncoding.LfsEncodingChecker.GetString(Byte[], Int32, Int32)
at InSimDotNet.Packets.IS_NCN..ctor(Byte[])
[TRUNCATED]
[2025-10-18 15:30:35 WRN] [ModularInSim.PortableEncoding.LfsEncodingChecker] GetString(): Round-trip data integrity check failed.
bytes1 '5E4AEFBFBD4F4BEFBFBD2045706963202847325E4C20646973636F6E6E656374656420284570696370726F657329'
bytes2 '5E4A3FBD4F4B3FBD2045706963202847325E4C20646973636F6E6E656374656420284570696370726F657329'
result '^J?ｽOK?ｽ Epic (G2^L disconnected (Epicproes)'
at System.Environment.get_StackTrace()
at ModularInSim.PortableEncoding.LfsEncodingChecker.GetString(Byte[], Int32, Int32)
at InSimDotNet.Packets.IS_MSO..ctor(Byte[])
[TRUNCATED]
Explanation: My checker worked as intended but the server sent me incorrect LFS-encoded bytes, which subsequently made the checker warn for a different reason. A player named Epicproes had a nickname ("｢OK｣ Epic (G29)") that contained Japanese (CP932) characters '｢' and '｣' which should have been sent as '\xA2' and '\xA3' respectively but they arrived as "\xEF\xBF\xBD".
Here are different representations of what I got and what the server should have sent:
Hex:
"5E4A5E33 EFBFBD 4F5E374B EFBFBD 20457069632028473200" (received) (24 bytes)
"5E4A5E33 A2 4F5E374B A3 204570696320284732392900" (expected) (24 bytes)
String:
"^J^3\xEF\xBF\xBDO^7K\xEF\xBF\xBD Epic (G2" (received)
"^J^3\xA2O^7K\xA3 Epic (G29)" (expected)
Unicode:
"^J^3�O^7K� Epic (G2" (received)
"^J^3｢O^7K｣ Epic (G29)" (expected)
Note: In all cases, nickname was truncated by the server to fit in 24 bytes with a null terminator. This is why the nickname ends with "(G2" instead of "(G29)".
I asked ChatGPT about this mysterious EF BF BD sequence, and it seems to be UTF-8 encoded "replacement character" represented as '�'.
I used Epicproes'es nickname and tried reproducing the same issue but InSim was working as intended. I tried locally (client), on a temporary server, and on the same couple (insim server & lfs server, though I had restarted InSim earlier for maintainance, which might have reset things on server side, I don't know) but couldn't reproduce it.