State of the new server

[zeus]

I want to complain that ingame it feels laggy. Very is an understatement of the actual game feel

Before someone comes and says it is my internet connection then I ran a standard ping at the same time when playing. I think it establishes quite nicely what the state of my internet usage was at the time

Code: [Select]

Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=55ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=54ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116
Reply from 62.254.88.27: bytes=32 time=53ms TTL=116

Ping statistics for 62.254.88.27:
    Packets: Sent = 1423, Received = 1423, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 52ms, Maximum = 114ms, Average = 53ms

How it feels ingame - incoming packets drop when a) someone joins (especially when there is over 8 players in the server), significant drop b) when the bot joins, significant drop c) when there is overtime combined with how many people are still alive - just the overtime part alone quite often creates a constant negative incoming packet count

When the packets drop - there is not enough information to run the game, game feels jumpy and hence "laggy". I do not have the same effects when playing in some other servers, neither did I have these effects 3 months ago when I first started to play in this server.

So something has changed, whether its the mods running on the server or the settings being changed / played around with on the server. Either case it unplayable

[zeus]

As I posted before reading the other posts it could be easily connected to the discoveries under https://miasma.rocks/index.php?topic=1499.msg17944;topicseen#new

[Piglet]

I want to complain that ingame it feels laggy. Very is an understatement of the actual game feel

What are the symptoms you are seeing? Up to about two months ago the server was being run on a VPS with a CPU not powerful enough to run the game with 20+ players. The CPU usage used to reach 100%.

We now have a brand new physical machine with a CPU that doesn't reach much over 50% on a single core when heading toward 30 players.

Previously the server had 2GB RAM shared across the different servers. The new one has 16.

Previously the bandwidth available was shared. For the last month we've had an uncontended 200Mbps up an down speed connection.

Previously the server was in Germany. After a test in the Nederlands on a less powerful machine than we have now, we're now running from a UK datacentre.

How it feels ingame - incoming packets drop when a) someone joins (especially when there is over 8 players in the server), significant drop b) when the bot joins, significant drop c) when there is overtime combined with how many people are still alive - just the overtime part alone quite often creates a constant negative incoming packet count

A "negative incoming packet count" is impossible. If you mean a reduced packet count, that's unlikely. It may be that with everyone in proximity the engine's unable to give everyone the full information in the bandwidth the game engine has chosen to send your client (as per the thread you link to). Previously that would have been worse, not better, as the server itself wasn't able to do the calculations in the time available.

Please can you do a "Stat Net | stat fps" at each of these points to show what you're sending/getting through the network, and your framerate your client is able to do.

When the packets drop - there is not enough information to run the game, game feels jumpy and hence "laggy". I do not have the same effects when playing in some other servers, neither did I have these effects 3 months ago when I first started to play in this server.

So something has changed, whether its the mods running on the server or the settings being changed / played around with on the server. Either case it unplayable

The mods running have been different over Christmas - Santa Hats, Christmas Decoration. snowy ONS scoreboard, and Capture the tree. They're now not running.

Other differences are:

1. The player highlighting/bright-skins. That's client side only.
2. Ping compensation. That is running client side again, but shouldn't have any impact on what you're talking about.
3. Anti-cheat. We have a few more checks being run. Unfortunately we had some cheaters and needed to check who was who. Three regular players have been banned for cheating. These things will have an impact on replication and client load. I'll turn them on and off as needed.
4. Server version. We have been on an unreleased version of the engine that I was helping Epic test. For compatibility reasons with the anti-cheat we've had to revert to 3369. Again, that can be flipped as needed.
5. Hitsounds. We've been using one that modulates pitch based on damage amount rather than serverside code spotting damage and replicating the sound event to the client. Unlikely to be a problem as written by Wormbo, but worth looking at again.

[zeus]

What are the symptoms you are seeing?

Usual in-game incoming packet count (from stat net) is around 57-58 packets / sec. For the moments when the lag is experienced, the incoming packet count drops to 35-40 range, sometimes also accompanied with incoming PL. With this drop also the FPS (stat fps) drops into less than the HZ on monitor -> non-smooth gaming experience where one feels lost for third a second and even sliding on the map...
My FPS has been capped in the ini to ~125 FPS range (netspeed 10001), offline FPS against 7 bots is ~225 range (not much, but should be more than enough)

3. Anti-cheat. We have a few more checks being run. Unfortunately we had some cheaters and needed to check who was who. Three regular players have been banned for cheating. These things will have an impact on replication and client load. I'll turn them on and off as needed.

Does this create disk read / write events with some interval / under certain conditions? In TR old servers a somewhat similar incoming packet drop could be experienced when they were running hundreds of zounds and someone typed something in the chat - the game made a HDD read as the chat could have contained any "triggerword", but apparently for some players it was not able to read fast enough for the game to be smooth -> HDD IO was blocking. I assume this problem could be solved by upgrading to SSD, but this seems idiotic for a game that is 15 years old :/

[Piglet]

If you're seeing packet loss and reduced FPS that's an indication to me that it's your PC that is struggling. As detailed above, the physical machine running the game server has massive headroom on CPU, memory, and bandwidth.

The anticheat is probably doing disc reads, but not all through the game. I have a pretty old machine which is far less powerful than the new server. It uses "state of the ark" spinny rust drives, and I don't see the issues you describe.

[Piglet]

Another thought is that the maps we're using aren't optimised. Many of them are very open, which makes replication limitations of the engine very apparent.

[holyspam]

I get the exact same problems only when i run the 64bit version of the game, there's packetloss IN and reduced packet count in random places, i dont get this with the normal version.

The only consistent low packet count event which is "reproducible" across maps/ut versions, only happens after player death or between rounds which is intentional, it's how 3spn is coded.

[hagis]

disk access can cause weird things,. I don't know that it helps but I have the game files loaded on a USB connected disk - which is fine most of the time but if the disk goes to sleep during the game I see a 1/2 second freeze when the disk spins back up

note one fix is to make sure the disk doesn't power off (which is how I have it set generally)

[zeus]

Running the FOOFOO magic midgame is idiotic - I experienced a massive FPS drop, the screen froze for like 20 seconds with additional constant 60FPS for the next moments. For a whole minute I was literally unable to do anything :@

[Piglet]

When was this? I'll check the log

[zeus]

17:25:47 UTC

[Piglet]

Yeah - I see at about then three things were manually kicked off when a maximum of one should ever be running. I'm not surprised you saw problems.

[tirofijo!!]

i cant change my skin, i was playing with the option in the f5 menu, but now im always othello

[Piglet]

Not one I've heard of before. I have only ever changed skin by a editing the ini file.

As far as I know there's nothing on the server preventing you change

[tirofijo!!]

      :cryface: :cryface: :cryface: