If anyone's noticed any lag the past few days

[luisc99 (OP)]

Hey,

I'm not sure who's been on the past few days with it being Christmas and all, but I noticed today the server is seemingly really laggy. I'm still trying to work out what's causing it, and how to fix it, but if anyone's played over the past few days, I'd like to hear from you. When did you first notice this lag? What are you doing when it happens? Is there anything major you know of that's changed recently? Anything else that could be useful? Thanks

Sorry about the lag. I'm trying to see what I can do to fix it.
~ Luis

[OctoGamer]

Hey,

I'm not sure who's been on the past few days with it being Christmas and all, but I noticed today the server is seemingly really laggy. I'm still trying to work out what's causing it, and how to fix it, but if anyone's played over the past few days, I'd like to hear from you. When did you first notice this lag? What are you doing when it happens? Is there anything major you know of that's changed recently? Anything else that could be useful? Thanks

Sorry about the lag. I'm trying to see what I can do to fix it.
~ Luis

map i think.

[CyphurTheFox]

Hey Luis, Couple of things that I think may be useful.

First, a note, I've been measuring the server's tps by leveraging the fact that /time also provides the time in ticks, and I have a custom script on my computer that runs the command twice with a delay of one second.

1: the lag is definitely server side. The tps measurements I've been taking reliably report back 2-3 tps, 4 if lucky.

2: the lag is not related to player structures outside qualia, or anything loaded by them. this was tested by first measuring the tps inside my village, then warping to a home I have set way out ~500k blocks from qualia, then repeating the measurement method. the tps does not change. I was the only person on the server at the time. Thus, the lag is either related to an internal server function or something occurring in the spawn chunks, as the lag is consistent regardless of location. (#2 was performed when I was the only person on the server)

3: the lag is not cleared by a server restart. test #2 was performed immediately after a server restart.

Hey,

I'm not sure who's been on the past few days with it being Christmas and all, but I noticed today the server is seemingly really laggy. I'm still trying to work out what's causing it, and how to fix it, but if anyone's played over the past few days, I'd like to hear from you. When did you first notice this lag? What are you doing when it happens? Is there anything major you know of that's changed recently? Anything else that could be useful? Thanks

Sorry about the lag. I'm trying to see what I can do to fix it.
~ Luis

map i think.

A quick theory:
If the lag is related to the dynmap trying to reconstruct the map, then the server restarts may be
interrupting that. Not sure, just a theory, so I threw out a post on the support forums for dynmap asking about the effects of a server restart on the dynmap render.

Also if it is a issue with the dynmap render,maybe it would be a good idea to see if its possible to render it on a different computer with a bit more graphics punch, then transfer the rendered files to the server?

Update: So I got an answer, apparently there's a dynmap config option that needs to be set so dynmap can save its work across a server restart.

If the following Option is set to true, Dynmap will save pending tile renders on shutdown and start over where it was when the server has started again.

Code: [Select]

# Save and restore pending tile renders - prevents their loss on server shutdown or /reload
saverestorepending: true

[hobbes13]

Yea I have been experiencing it too. On my end I have been getting 60 fps and no chunk updates but still lags.

[luisc99 (OP)]

-- Snip --

@CyphurTheFox Interesting results, thanks for taking the time do look into that Have a llama in thanks

1. Your estimation of TPS is correct. I did a timing breakdown over the past hour, and the average TPS was 2.2. Yesterday when I checked it was around the same, but it's good to know that you've confirmed that it's like that all the time, and not just in spikes.

2. Lag being present regardless of location is interesting, as (for the most part), it eliminates structures or entities as the cause, unless they're in the spawn chunks. Based on the timings log I collected, it seems sounds and pistons are taking longer than they should, but I'd be willing to bet that's just a side effect of everything else lagging

3. I never manage to catch the restart, so I was unable to test this myself, so that's good to know. I wasn't sure if the restart fixed it and it gradually got worse over the news few hours, or if it was an instant effect

About the map:
- I've got the plugin set to only render the map when the TPS is above 15. If it ever drops below that, it'll pause the map until it stabilises again. Similarly, it should pause the map rendering if there's more than a few people online, to avoid lag then too
- From the timings report I got, dynmap is taking up 1.51% of the tick, which while that's high enough to give me a yellow warning, it's nowhere near high enough to be the source of the lag
- The saverestorepending flag is set in our config, so it should be keeping its progress over restarts
- From what I can tell though, the queue is empty, so even if the map were a problem, it shouldn't be rendering anything right now

Based on the timings I took, the things that are taking up the most time are doTick and doTickPending, which are taking up 825% and 768% of the tick time each, respectively. The doTickPending I'm pretty sure is only taking up so much time because it's being postponed by doTick taking longer than it should. I'm trying to see what I can do about this, but there's not too much info out there that I can find. We had this issue about 2 years ago, which our host tried to resolve by moving some stuff around and seeing what could be done, but from what I remember it just randomly fixed itself. I'll try and keep you updated with any progress I make on fixing it.

If anyone notices anything else that could be useful, please let me know
Thanks, and sorry again for the lag

[Akomine]

Does anyone have a base/spawner/anything an incredible number of millions of blocks away from Qualia?

[CyphurTheFox]

Does anyone have a base/spawner/anything an incredible number of millions of blocks away from Qualia?

I have a home set 500k blocks out (/home CyphurTheFox:valley), which should unload anything but the spawn chunks and the immediate area around you when you’re way out there, given the fact that the server render dist is 9 chunks.

Speaking of the spawn chunks, Luis, I loaded up the WDL mod just now and grabbed the spawn chunks. I’m going to load it into a locally hosted session of gnebmons carpet mod and run some tests to see how the game handles just the spawn chunks. Will update later.

 On a side note, walking around quaila because elytra aren’t working with the lag  really puts how beautiful some of the back areas are into perspective. Kudos to whomever built it.

[OctoGamer]

On a side note, walking around quaila because elytra aren’t working with the lag  really puts how beautiful some of the back areas are into perspective. Kudos to whomever built it.

yes it is and tenretni built most of it I think 

[CyphurTheFox]

Speaking of the spawn chunks, Luis, I loaded up the WDL mod just now and grabbed the spawn chunks. I’m going to load it into a locally hosted session of gnebmons carpet mod and run some tests to see how the game handles just the spawn chunks. Will update later.

Ok, so the results of my testing suggest that the lag is unrelated to the overworld spawn chunks. When I loaded up carpet mod with qualia, I stood in the world spawn and used the tick warp function to jump ahead about 20 mins worth of ticks. I then waited the test out. The spawn chunks when loaded in mostly vanilla minecraft are not a source of lag, as during the tickwarp I was pulling 75ish tps. Then to simulate the spawn chunks when a player is not in the immediate, I started the same test again, but I was in spectator mode. More or less the same result. I think this effectively rules out spawn chunk based lag as well, unless its related to one of the plugins.

I did find your old reddit post about this happening a year ago and there you mentioned that 1600 chunks loaded with 3 players was normal for VC? Is that still the case? because if so, the lag is prob there, if its not a plugin issue.

There was also an even older post on the spigot forums mentioning doTick lag, you probably found it already, but it does suggest either a world guard issue and/or a problem with the number of worlds the server has (1-2 reccomended). Maybe thats the issue, given that VC has 6?

Something I failed to mention earlier that is likely just coincidence is something that happened a few days prior to the whole lag issue starting. I was on the server with frog, and we were messing around with red stone. At some point we ended up on lag machines, how I'm not sure. Two pretty intricate devices with pistons and clocks and observers ensued. However, I do also recall that the machines were 1: built outside spawn chunks and/or in the superflat world and 2:turned off/disabled/destroyed. Nonetheless I did double check that the one that I did know the location of is completely destroyed. 

On an unrelated note, because I was in spectator mode, and I had qualia in front of me, I couldn't help but poke around under qualia and check it out. Underground areas can be just as interesting as the places on top. This absolutely holds true for qualia. Even including the new metro system:
Nice Easter Egg, Luis <<Link

[luisc99 (OP)]

I did find your old reddit post about this happening a year ago and there you mentioned that 1600 chunks loaded with 3 players was normal for VC? Is that still the case? because if so, the lag is prob there, if its not a plugin issue.

1600 was normal for a while, but it fluctuates quite a bit based on where the players are, and what people are doing. Right now, there's 273 chunks loaded in the overworld, and either 0 or 256 loaded in other worlds, which are spawn chunks (16*16)

There was also an even older post on the spigot forums mentioning doTick lag, you probably found it already, but it does suggest either a world guard issue and/or a problem with the number of worlds the server has (1-2 reccomended). Maybe thats the issue, given that VC has 6?

VC Actually has 10 worlds loaded now, but many of them are empty, and we've had them all for years. While worlds could be an issue, the timings seem to indicate only the overworld is having trouble ticking, which leads me to believe it's something else. I could well be wrong though

Something I failed to mention earlier that is likely just coincidence is something that happened a few days prior to the whole lag issue starting. I was on the server with frog, and we were messing around with red stone. At some point we ended up on lag machines, how I'm not sure. Two pretty intricate devices with pistons and clocks and observers ensued. However, I do also recall that the machines were 1: built outside spawn chunks and/or in the superflat world and 2:turned off/disabled/destroyed. Nonetheless I did double check that the one that I did know the location of is completely destroyed. 

It's likely a coincidence, although maybe not. The report does indicate sounds and tile entity ticks are taking longer than usual, including block physics checks, which comes from block updates. TileEntityPiston is listed as one of the tile entities, so maybe there is some residual side effect of that. If you happen to know the coords of either, I can try and see if that has an effect. MC has this weird quirk with chunks where you can sometimes keep them loaded with pistons and/or hoppers, which may be where the 19 extra chunks come from as mentioned above.

On an unrelated note, because I was in spectator mode, and I had qualia in front of me, I couldn't help but poke around under qualia and check it out. Underground areas can be just as interesting as the places on top. This absolutely holds true for qualia. Even including the new metro system:
Nice Easter Egg, Luis <<Link

Qualia has loads of stuff under it, and lots of things have been burred over the years. That specific easter egg was when I decided to re-direct the metro to somewhere else, and got lazy filling it all in. There's plenty more easter eggs around though, both in Quaila and other builds we've done. I'll give you your reward next time I log in, which will likely be in a few days

Does anyone have a base/spawner/anything an incredible number of millions of blocks away from Qualia?

CRH does I'm pretty sure, as I had to change some config settings to keep everything working with him being out so far

[CRH244]

Does anyone have a base/spawner/anything an incredible number of millions of blocks away from Qualia?

I have a base 1mil blocks away but it isnt large to any degree and its been there for well over a year i think

[luisc99 (OP)]

By changing literally nothing with the server, the lag has gone away, which is good I guess? Like last time this happened though, I still don't know what caused it to start, or what caused it to stop...

I've added a TPS logger to VcStatus, so a graph of the server's TPS is available on the website. I'll get an alert on my phone if it ever drops below 15. It's been stable at just under 20 for the past day though, so it should be good.

So yeah, let me know if it actually is okay, or if the stats are lying.