Farms Causing Lag?

[Naomi (OP)]

Hello everyone! Disclaimer I don't want to cause any issues, and this isn't at anyone in particular I just want a happy solution for everyone.

I know the past few days have had debates of whether or not farms are causing lags and I have seen both perspectives. I know that whenever I am on the server building or mining, and someone gets on and starts using a large automatic farm I have issues with lag. I know a few days ago when this happened I was on with at least 10 people on. Now the debate is that farms don't cause the lag and Luis said he is looking into that. . As a player who enjoys playing the game I just want to be able to do the things I enjoy as I'm sure many of us players do. I completely understand wanting to make money to do a build project, but making money for the sake of being rich and lagging players in the process is disheartening to me who loves this server and wants everyone to have a good time. I know some people are in favor for waiting until 1.16, but as someone who loves playing, why make them lag while we wait for that update. Here are some solutions I feel could work to benefit the server.
1. Disable all large automatic farms, make people work for their money in a way that doesn't lag the server (This is a harsh solution)
2. Make the items from large automatic farms economically invaluable so that large automatic farms are unappealing
3. Ban the use of large automatic farms when there are more then 5 players on (This is so that you can still use your farms, but it won't lag as much when there are large groups trying to get things done) (This is my personal favorite)

Thank you all for your time and I hope we can reach a solution together. I look forward to everyone's suggestions! 

[Brutalfive]

Or, as everyone is in agreement that 1.14 seems to be a big factor to lag, why don't we update to 1.15. I know many people, including myself, voted to wait and skip 1.15, but firstly we don't know when 1.16 is coming out in 2020, so we could have a longgggg time still to deal with this lag, but by updating we may be able to enjoy a lag free server once more. Secondly it will become possible for us to get the answer we are all in search for, what is causing all this lag, and is 1.14 the main culprit?!

This will help sort the farm causing lag disagreement amongst the community

[Shensley]

3. Ban the use of large automatic farms when there are more then 5 players on (This is so that you can still use your farms, but it won't lag as much when there are large groups trying to get things done) (This is my personal favorite)

I do like this idea, or what brutal said about just updating to 1.15.

[Akomine]

Option 2 is my favourite solution, but the implementation requires quite a good dynamic market plugin, and I can't seem to find one that uses physical shops.

IF YOU CAN FIND ONE, TELL ME ASAP.


Otherwise, a discussion about regulating farms can happen. As for upgrading to 1.15, does Luis think it's less laggy?

[luisc99]

I'm not going to comment too much on this yet, I've still got tests going on before I say things without evidence.

As for upgrading to 1.15, does Luis think it's less laggy?

I've not done loads of testing on it beyond plugin tests, but I've seen some very mixed opinions from server owners who are usually quite reliable. I know 1.15 implemented some (but not much) multithreading, which does improve performance slightly in some areas, but also introduces other bugs. Some people have described it as essentially 1.14.5, in regards to the fact it didn't really add much, and mainly served to patch some of the bugs that came around in 1.14. I've also seen lots of comments that 1.15 is basically no different in terms of performance, but then again others say it's slightly better.

One thing to note in 1.15 is that in almost every case I've read, client-side FPS will drop, due to some poor optimisation in some rendering changes they've made.

When it comes to 1.16, it's anyone's guess as to when it'll be released. I made this quick chart showing all the major updates since 1.1, and the time between the first snapshot and the first "major" version (some versions had a .1 or .2 patch a couple of days later). The ones I've coloured in orange are the updates which I consider to be major changes to the game in terms of the codebase (1.3 unified MP and SP, 1.6 completely changed resource management, 1.8 recoded commands, 1.13 was "The Flattening", etc). Considering 1.16 seems to be relatively minor in terms of things changed, if I had to guess, I'd say early-to-mid May for a release, but I refuse to be held to that estimate. Mojang are a weird company, nobody can predict how they'll work.

[TheLegend12369]

Personally like the 3rd option, and it's something that a group has already been discussing.

[Naomi (OP)]

Thank you all for the feedback! I prefer option 3 because everyone wins and it doesn't take as much regulation such as changing the shops and completely removing farms. I think it gives everyone a chance to be happy.

[luisc99]

Right, I've still got a lot more data to go through, but instead of enjoying the sunny weather or doing uni work, I decided to share what I've spent all day doing. I'm doing more than just this, but at least this is complete enough I can waffle about it for a bit...

Some experiments
I've done some tests in 3 locations on the server, in 3 different versions. Namely, I've tested the area immediately surrounding /warp Qualia, as well as both Octo and Cyphur's farms (as Octo seems to be the one everyone's complaining about, and Cyohur admitted to having a large farm). I tested each of these for 30 minutes, on a clone of VC running Paper 1.14.4, Paper 1.15.2, and Vanilla 1.16 (20w15a). I'd take the 1.16 test with a very large pinch of salt, considering it's not running any of the plugins we use, and so the results aren't directly transferrable. I just included it for a comparison.

What is lag?
Before I start, I'd just like to describe what lag actually is. Minecraft has 3 main types of lag: server lag, client lag, and connection lag.
- The easiest one to understand is client lag, which is simply your computer's ability to process and render everything the server tells it. If it can't do that at the frame rate you want, you'll get a lower FPS.
- Connection lag is what it sounds like, it's lag caused by your connection to the server. If you have slow internet, or like me you're located on the other side of the world to the server and have a low ping, then you'll have some connection lag. This is mostly noticeable by things having a slight delay in them
- Server lag is probably the most difficult to understand, as it can look a lot like connection lag from a player's point of view. Basically, the server has a metric called "Ticks per Second" (TPS), which is a measure of how the game is running. Minecraft is designed to run with 20 ticks in every second, meaning it's got to try and process every single change between one tick and the next, every 50 ms (0.05 s). That means every entity, every block, every chunk, every player, every item, every inventory, everything that has changed since the server last looked at it, it's got to see what's changed, check with each plugin if they want to do something about it, and then move on to the next item. If each tick's checks takes longer than 50 ms, it starts to eat into the next tick's time, which means the server runs slightly slower. If each tick took 100 ms, the server would be running at 10 TPS, meaning the game would be running at half the usually speed. This doesn't mean you're lagging, your client could still be getting 500 FPS and run as smooth as silk, but the server itself will be struggling to keep up with what's going on. Now, every so often, your client will ping the server, to make sure it's still alive, and if the server doesn't respond in time, you'll get what's called a "timeout", which will disconnect you. This isn't actually the server's fault, it's just that it didn't manage to get around to replying to your ping in time, because it was too busy doing everything else in its queue.

What affects TPS?
The TPS can be affected by a lot of things, as every single line of code in the server takes a little bit of time to run, so simply banning or disabling things isn't a good solution. I'm not saying they won't work, but it's a horribly inefficient way of dealing with problems. For example, say hoppers are a problem. Sure, if you ban farms then the lag might go down, but that's only because less hoppers are being processed. However, even though it looks like farms were the problem, you would never know because too many things were banned at once. On a semi-related note, I made some changes to hoppers to significantly reduce their processing time, which looking at my status logger does seem to have helped a little bit with the lag. There's still some major changes I could make, but then it would start to have a potentially noticeable effect on gameplay, which I try to avoid at all costs with my changes. MY approach to fixing things like this is if you notice something's changed, I've done it wrong. Anyway, as I was saying, TPS is affected by millions of tiny things, and it's a long process to try and solve, so arguing over whether major changes should be made will cause us to miss the actual underlying problems.


That being said, I've noticed a lot of childish arguments discussion about if farms should be banned, which brings me back to the point I'm trying to make in this post. I did some simple tests, and here are the results.

1. Qualia
As a baseline, I performed each of the tests at the Qualia warp point, to get a picture of how a typical area should perform. Some processing involved here will be the items in all of the showcases, and those cool little floating text things we've got dotted around Qualia. This is the graph of the TPS and average time between ticks in all three versions (red is 1.14, green is 1.15, blue is 1.16), but this is the general summary:

Version	Mean TPS	Median TPS	Standard Deviation
1.14	19.80	20.30	2.27
1.15	19.92	19.51	2.01
1.16	19.98	19.89	2.77

As you can see, there is very little difference between the versions, and the TPS is pretty much what it should be. Please ignore the trail off on the 1.16 chart, I messed up the test towards the end, it's not representative of the actual performance, and is simply a statistical error.

2. Octo's Farm
Here, the major things for the server to process is the redstone and pistons involved in the farms, every entity that is in the farm, each of the item drops it creates, and all the hoppers and chests that are required to store all the drops. Again, this is the graph of how it performed in different versions, and this is how it breaks down:

Version	Mean TPS	Median TPS	Standard Deviation
1.14	19.52	19.97	4.29
1.15	20.03	20.59	2.97
1.16	20.06	20.24	3.44

There's a few things to take away from this specific experiment.
- Firstly, it appears that the farm itself does not lower the TPS of the server, but it does make it fluctuate more. This is to be expected. As much as I complain about Minecraft's coding, I will admit that it does try and recover from TPS drops where it can, and if possible, it'll try and run the server slightly faster some times, in order to ensure the overall average TPS is as close to 20 as possible. This is why the TPS is sometimes above 20, and why it averages out to almost 20 despite the fluctuations.
- Secondly, it appears that there is little difference between the three versions, aside from a slightly lower amplitude of oscillations in the TPS in later versions. Considering I only did these tests once, and only for 30 minutes (which already took around 6 hours), I'd take these with a pinch of salt, as you should with any experiment with no repeats. For example, if I were to only sample between 600 and 800 seconds, it would appear that 1.14 is significantly worse than the other versions, whereas if I were to sample between 0 and 200 seconds, 1.14 looks the best.
- Thirdly, it appears that Octo's farm isn't causing nearly as much lag as you all claim it to. I will admit, these tests aren't perfect, and it doesn't 100% directly correlate to what's going on on the server, but it looks like there isn't too much impact.

3. Cyphur's Farm
I picked this as a third test to ensure I've got multiple data points for farms, rather than just the single one. As Cyphur admitted in a different thread that they have a large farm, I took that as an invitation to test it. Much like Octo's farm, there are a lot of things to process here, mainly entities and water flows, but this farm also includes nether portals, which will involve moving entities between worlds, which may be a source of strain on the server by requiring more chunks to be loaded and unloaded. As with before, here is the chart, and here are the results:

Version	Mean TPS	Median TPS	Standard Deviation
1.14	19.48	20.06	6.35
1.15	20.03	20.00	2.37
1.16	20.16	20.06	2.29

Again, there's some key points to take away from this:
- The TPS drop is relatively low, but it does fluctuate. In fact, it fluctuates the most out of all three tests on 1.14.
- This experiment is a very good example of how the average TPS can be 20, but the instantaneous TPS can vary wildly. You can see it drops down to 12 at some points, and rises to 24 at others, all of which will have a noticeable effect in gameplay in the form of lag spikes.
- In this specific instance, it appears that the fluctuations are less noticeable in newer versions of the game. This could be due to a wide variety of reasons, such as certain bugs being fixed, or optimisations in areas of the code improving performance in certain areas.
- Based on these results, it looks like chunk loading/unloading contributes to TPS fluctuations, which would correlate with the poor performance of chunks in 1.14.

Conclusions
- Farms don't appear to cause a prolonged TPS drop from my tests on a clone of the server, but they do cause the TPS to fluctuate more than normal.
- TPS is affected by a variety of things, not just farms in general (eg: hoppers vs water vs chunk loading/unloading vs redstone)
- 1.15 does appear to have a marginally better performance, but not drastically. While in test 3 the performance does seem to be improved quite a bit, this same order of magnitude of improvement isn't really seen in the other two tests. Don't get me wrong, there is an improvement in test 2, but I'd expect there to be a similar trend among all three tests if performance were improved across the board.
- I personally do not see any clear evidence in here that flat-out banning farms or restricting their use to low player numbers should be enforced server-wide. Yes, it causes performance to fluctuate, but it doesn't appear to cause the significant prolonged drops we're seeing on VillageCraft. If individual farm owners want to optionally do that, then it's up to them, but I don't agree with enforcing a ban based on this evidence.
- While performance may be slightly improved in 1.15, that's not the only thing we should consider when deciding to update. We also need to consider the effort that would be needed to actually upgrade, in making sure all the plugins are ready, as well as the requirement for everyone to update to a new version of Minecraft, possibly only for a few weeks. I'll continue to think about this in the upcoming days/weeks

A bit on the tests
I performed these tests on a simple clone of VillageCraft. What do I mean by that? I mean I'm running every single plugin with exactly the same configuration and options as we are on VillageCraft, as well as running the same software and server-wide options. However, I am not running the full world. Instead, I am loading a 3-region square centred around the relevant points in the relevant worlds (1536x1536 blocks). This was done for two reasons, firstly, most of the world would be unloaded anyway, so there wasn't too much benefit in including it; and secondly, our worlds are probably way over 200 GB right now, and while I do enjoy data hoarding, and as much as it would please me to have that full clone, I do not have anywhere near enough patience to download it all over my slow internet connection. All the tests were performed with the same Java parameters as we use on VillageCraft, and the 1.14 and 1.15 tests were performed with the same amount of physical RAM as we have on VillageCraft. Due to an oddity with how Vanilla allocates RAM compared to Paper, combined with the fact I only have 8 GB of RAM, I had to run the 1.16 test on a lower memory. That being said, it did still perform well, so if run on exactly the same configuration I imagine the performance would be even better. This lower RAM is probably the reason for the occasional lag spikes, where I suspect some GC was being performed. While I did try and match all those settings, there are some other things I didn't match, partly because I couldn't be bothered, but mainly because I actually couldn't match them. I can't match the server's hardware (CPU, etc), which will likely have quite a large impact on the results, but also VillageCraft runs on Linux, on server-grade equipment, whereas I'm running this on macOS on a 7-year-old desktop computer. That too could have a potential effect. Finally, due to the way I'm collecting this data, there are some occasional spikes in TPS which claim to reach in the order of 10⁴. These are clearly outliers in the data, created by the fact I'm using a very simplistic method to calculate TPS at every tick. To overcome this, I've filtered the TPS (but not time since last tick), to remove any obviously wrong values. If I were doing this properly, I'd use a low-pass filter with a very low cut-off frequency, but I didn't really feel like coding that, and this data is too different to the low-pass filters I've used before to be able to copy-and-paste that. So, I used the cheap method of (if the TPS is over a certain limit, just set it to be the same as the previous value), which seems to work nicely. It does annoy me though, but I've already spent long enough on this.

TL;DR: Farms don't seem to cause prolonged TPS lag, but they do cause fluctuations, which may be slightly visible in-game depending on what else is going on. TPS lag comes from millions of different sources, and finding out what's actually causing the lag and solving that problem is much harder than you think, and banning major things like farms won't actually solve the problem, but rather hide it until it shows itself some other way. Finally, on a personal note, I'd appreciate it if you all stopped arguing over this and speculating as to what you think is causing the lag, and just let me actually try and solve it using the data I have. If you've got any comments or things you've noticed about the lag, I'd love to hear from you about it. PM me on here, or on Discord (@Luis). I'm all ears.

As always, I'm continually working on the back-end of the server, making changes every few days based on the data I have available, to try and solve the lag. There's a good deal of things that I'm doing to fix it that won't really show up in-game, but trust me, I'm doing stuff about it. I've still got a bunch more things I'm investigating, and will continue to tweak things as I go

[Yvette]

I read through that whole thing, and while I appreciate the effort being put into everything & the minor changes being made to alleviate problems, it's generally known Farms don't cause many issues when there are low player counts; hence why the urgency is to use farms when there are not many players on, and though you didnt specify, I can assume you were the sole player online for your tests.

The most impactful times of Farm Usage is during peak hours, where we get between 10-15 people online at once, and the behavior pattern over the last few weeks has shown that during those times, the impact of large Pumpkin farms running majorly impacts the experience of the large amount of players online because it's combined with the strain of those people online. It seems as though combining both of those separate strains on the server at the same time is what causes the detrimental lag spikes - and in my opinion, people come over farms, hence the discussion of banning farms during peak hours - not altogether.

A 4-6 TPS fluctuation, when the TPS is sitting at 20, clearly doesn't cause too noticeable of a difference in gameplay, but when TPS is sitting between 12-15, and then you add a 4-6 fluctuation on top of that, it makes the server unstable and difficult to play on.

I may not have expressed this well enough in the past but again, the issues is not solely farms, and the blame lies primarily on poor optimisation in 1.14. However, when it comes to solving the issue, I think it's a no brainer to not allow the farms during peak hours, until a more optimised update comes out.

Once again, thanks for the tests and experiments; knowing the amount of fluctuation caused by these sorts of farms is helpful in figuring out solutions. Luis

[luisc99]

A 4-6 TPS fluctuation, when the TPS is sitting at 20, clearly doesn't cause too noticeable of a difference in gameplay, but when TPS is sitting between 12-15, and then you add a 4-6 fluctuation on top of that, it makes the server unstable and difficult to play on.

The question shouldn't be about what effect farms have on a laggy server, it should be why is the server lagging in the first place. I'm not too worried about those fluctuations, my goal here is to try and figure out why the TPS is 12-15 during peak times. If I can solve that, then it will have a knock-on benefit which will be very noticeable. If we ban farms during peak hours, the TPS will still be lower than it should, and that's the problem we should be focusing on.

[Naomi (OP)]

Thank you Luis for looking into this!

[Yvette]

A 4-6 TPS fluctuation, when the TPS is sitting at 20, clearly doesn't cause too noticeable of a difference in gameplay, but when TPS is sitting between 12-15, and then you add a 4-6 fluctuation on top of that, it makes the server unstable and difficult to play on.

The question shouldn't be about what effect farms have on a laggy server, it should be why is the server lagging in the first place. I'm not too worried about those fluctuations, my goal here is to try and figure out why the TPS is 12-15 during peak times. If I can solve that, then it will have a knock-on benefit which will be very noticeable. If we ban farms during peak hours, the TPS will still be lower than it should, and that's the problem we should be focusing on.

For sure. But in the meantime, I don't think we should leave our players behind to suffer in peak hours due to these fluctuations. When a solution is out, I'm all for having no restrictions, but if there's no solution atm, a temporary solution suffices until something more permanent is figured out. To clarify, my perspective isn't that we just ban farms and say "Fuck it" to trying to improve the baseline situation.

[Akomine]

A 4-6 TPS fluctuation, when the TPS is sitting at 20, clearly doesn't cause too noticeable of a difference in gameplay, but when TPS is sitting between 12-15, and then you add a 4-6 fluctuation on top of that, it makes the server unstable and difficult to play on.

The question shouldn't be about what effect farms have on a laggy server, it should be why is the server lagging in the first place. I'm not too worried about those fluctuations, my goal here is to try and figure out why the TPS is 12-15 during peak times. If I can solve that, then it will have a knock-on benefit which will be very noticeable. If we ban farms during peak hours, the TPS will still be lower than it should, and that's the problem we should be focusing on.

For sure. But in the meantime, I don't think we should leave our players behind to suffer in peak hours due to these fluctuations. When a solution is out, I'm all for having no restrictions, but if there's no solution atm, a temporary solution suffices until something more permanent is figured out. To clarify, my perspective isn't that we just ban farms and say "Fuck it" to trying to improve the baseline situation.

Limiting farms to off-peak hours or when the player count is low enough might help alleviate TPS problems eh? Might not be the worst idea

[TheLegend12369]

A 4-6 TPS fluctuation, when the TPS is sitting at 20, clearly doesn't cause too noticeable of a difference in gameplay, but when TPS is sitting between 12-15, and then you add a 4-6 fluctuation on top of that, it makes the server unstable and difficult to play on.

The question shouldn't be about what effect farms have on a laggy server, it should be why is the server lagging in the first place. I'm not too worried about those fluctuations, my goal here is to try and figure out why the TPS is 12-15 during peak times. If I can solve that, then it will have a knock-on benefit which will be very noticeable. If we ban farms during peak hours, the TPS will still be lower than it should, and that's the problem we should be focusing on.

For sure. But in the meantime, I don't think we should leave our players behind to suffer in peak hours due to these fluctuations. When a solution is out, I'm all for having no restrictions, but if there's no solution atm, a temporary solution suffices until something more permanent is figured out. To clarify, my perspective isn't that we just ban farms and say "Fuck it" to trying to improve the baseline situation.

Limiting farms to off-peak hours or when the player count is low enough might help alleviate TPS problems eh? Might not be the worst idea

I appreciate all that Luis is doing, but he even said it does have an effect, and I think that helping the already struggling server during peak times, at least as a temporary solution, is better than just waiting until you can find the true cause, because who knows how long that will be. For all we know, the main cause is probably just Mojang's shitty coding, and no matter how good Luis may be, which is very, I don't think he's willing to recode a whole game to solve the problem. I think a temporary fix to help during peak hours is better than nothing.

[Akomine]

A 4-6 TPS fluctuation, when the TPS is sitting at 20, clearly doesn't cause too noticeable of a difference in gameplay, but when TPS is sitting between 12-15, and then you add a 4-6 fluctuation on top of that, it makes the server unstable and difficult to play on.

The question shouldn't be about what effect farms have on a laggy server, it should be why is the server lagging in the first place. I'm not too worried about those fluctuations, my goal here is to try and figure out why the TPS is 12-15 during peak times. If I can solve that, then it will have a knock-on benefit which will be very noticeable. If we ban farms during peak hours, the TPS will still be lower than it should, and that's the problem we should be focusing on.

For sure. But in the meantime, I don't think we should leave our players behind to suffer in peak hours due to these fluctuations. When a solution is out, I'm all for having no restrictions, but if there's no solution atm, a temporary solution suffices until something more permanent is figured out. To clarify, my perspective isn't that we just ban farms and say "Fuck it" to trying to improve the baseline situation.

Limiting farms to off-peak hours or when the player count is low enough might help alleviate TPS problems eh? Might not be the worst idea

I appreciate all that Luis is doing, but he even said it does have an effect, and I think that helping the already struggling server during peak times, at least as a temporary solution, is better than just waiting until you can find the true cause, because who knows how long that will be. For all we know, the main cause is probably just Mojang's shitty coding, and no matter how good Luis may be, which is very, I don't think he's willing to recode a whole game to solve the problem. I think a temporary fix to help during peak hours is better than nothing.

This should be voted on by the Parliament, and if that's going to happen, someone needs to draft wording to vote on before we actually put it to a vote. If people actually do want to do this, then someone should start.

[TheLegend12369]

A 4-6 TPS fluctuation, when the TPS is sitting at 20, clearly doesn't cause too noticeable of a difference in gameplay, but when TPS is sitting between 12-15, and then you add a 4-6 fluctuation on top of that, it makes the server unstable and difficult to play on.

The question shouldn't be about what effect farms have on a laggy server, it should be why is the server lagging in the first place. I'm not too worried about those fluctuations, my goal here is to try and figure out why the TPS is 12-15 during peak times. If I can solve that, then it will have a knock-on benefit which will be very noticeable. If we ban farms during peak hours, the TPS will still be lower than it should, and that's the problem we should be focusing on.

For sure. But in the meantime, I don't think we should leave our players behind to suffer in peak hours due to these fluctuations. When a solution is out, I'm all for having no restrictions, but if there's no solution atm, a temporary solution suffices until something more permanent is figured out. To clarify, my perspective isn't that we just ban farms and say "Fuck it" to trying to improve the baseline situation.

Limiting farms to off-peak hours or when the player count is low enough might help alleviate TPS problems eh? Might not be the worst idea

I appreciate all that Luis is doing, but he even said it does have an effect, and I think that helping the already struggling server during peak times, at least as a temporary solution, is better than just waiting until you can find the true cause, because who knows how long that will be. For all we know, the main cause is probably just Mojang's shitty coding, and no matter how good Luis may be, which is very, I don't think he's willing to recode a whole game to solve the problem. I think a temporary fix to help during peak hours is better than nothing.

This should be voted on by the Parliament, and if that's going to happen, someone needs to draft wording to vote on before we actually put it to a vote. If people actually do want to do this, then someone should start.

There was a movement that was brought to my attention to make a CM-initiated vote, I'll get on that. Just wanted to see where discussion was going.

[PengBunny]

To begin I absolutely love your mature response to players concerns:

That being said, I've noticed a lot of childish arguments discussion about if farms should be banned, which brings me back to the point I'm trying to make in this post. I did some simple tests, and here are the results.

This comes off of as condescending. I understand you do a lot for the server in the background, working with all of its frustrating code and plugins and everything and sure you may understand a lot of what is going on more than the average VC player but you are human and you do miss things. Just because you know how to code does not mean you are always totally right on anything server related.

Luis, while you did run these tests on a VC clone to "prove" that farms don't cause a consistent TPS dip, that doesn't completely let farms off the hook. For one, a massive problem I see with the tests conducted is that a consistently low TPS or high fluctuations in TPS are most observable at peak hours -- when we have many players on the server. In your tests you don't mention taking into account multiple players being on the server so I am assuming your tests are with a sole player (you) on the test server. Obviously, this isn't a very good way to measure whether or not we should implement a rule to limit farm use when a certain amount of players are on -- in fact it doesn't do anything to prove that it wouldn't help with lag. A proper test of this clone server would have around the same number of players that are on during peak hours when we have these issues. While these results can help you narrow down what the lag may be it does not mean that other forms of data are to be ignored.

The anecdotal evidence from a large majority of players that claim to experience lag when OctoGamer gets on (presumably spawning at his farm) has been dismissed routinely when referring to this issue. While it isn't numerical data that you can graph and find the mean or deviation, IT IS data. Player feedback should not be ignored. It means there is something that your tests obviously aren't picking up on that deals with Octo's farm that OTHER players experience constantly and that should be looked into. A lot of the frustration that caused these "childish arguments" is that in general the ones protecting farms are both staff and owners of large farms and always claim that it isn't the issue when everyone else is experiencing the effects. As the lead programmer/administrator of the server it is your job to listen to people's frustrations and not immediately dismiss them, as this can guide you to a better understanding of the problem at hand. It's likely that these complaints from players are valid and way you are handling data recording prevents you from seeing these issues.

The question shouldn't be about what effect farms have on a laggy server, it should be why is the server lagging in the first place. I'm not too worried about those fluctuations, my goal here is to try and figure out why the TPS is 12-15 during peak times. If I can solve that, then it will have a knock-on benefit which will be very noticeable. If we ban farms during peak hours, the TPS will still be lower than it should, and that's the problem we should be focusing on.

The community's argument isn't that farms are severely slowing down the server consistently but rather the extreme fluctuations that cause rubber banding and make VillageCraft unplayable. Why are fluctuations not an issue? Yes, the big picture is to solve the overall lag problem that the server has, but that doesn't mean we should ignore solutions to make the server a little more playable. Waiting until you figure out the big mystery to the lag isn't doing us much of anything when players constantly feel the effects of large farms on the already laggy server that SHOULD be a priority to lessen. Your data doesn't include any of the many large pumpkin farms on VC which is interesting as this is the most prevalent large automated farm on VillageCraft with many players owning them. It would prove to be a good idea to do some tests on these massive pumpkin farms that are much bigger and have way more redstone than these other large farms though I understand these are not the farms in question that have been constantly complained about by the community.

I'd appreciate it if you all stopped arguing over this and speculating as to what you think is causing the lag, and just let me actually try and solve it using the data I have.

You are speculating just as much as the players are. You aren't getting anywhere, and there hasn't been results for the time that this issue has been around. I don't see why there is any issue with doing a small change to limit large farms until we move toward a permanent solution when you DO happen to solve the underlying problems. It is obviously causing noticeable lag for many players including myself and I find it ridiculous that you so vehemently deny that it could be causing the issue that players are experiencing. I say we vote on this somewhat quickly so it doesn't get swept to the side like it has been for awhile now. That way if the community is wrong, the change can be reverted and crossed off the list as being an issue causing the lag -- and if the community is right -- it will heighten the current experience on VillageCraft by limiting the amount of rubber-banding and skipping when large farms are functioning in high player counts.




btw Octo I'm not targeting you, you are just the prime example everyone refers to when arguing against large farm usage

[OctoGamer]

btw Octo I'm not targeting you, you are just the prime example everyone refers to when arguing against large farm usage

yea but my name always seems to be the one dropped by people. even when not at it I still get accused. even when its turned off. sure is starting to feel like i'm being targeted.

[luisc99]

First off, I'd like to apologise if it sounded condescending. I did not intend for it to come across this way.

Just because you know how to code does not mean you are always totally right on anything server related.

I am fully aware of this, and at no point in the past 7 years of me doing this have I ever claimed I am totally right. I have been proved wrong before, and I fully expect I will be proven wrong in the future. That's the point. Every time I'm wrong, it teaches me a lot more about how the game works and how to solve things in the future. If I were always right, I would never learn anything.

Luis, while you did run these tests on a VC clone to "prove" that farms don't cause a consistent TPS dip, that doesn't completely let farms off the hook.

I did not run these tests to prove that farms aren't the problem. I did these tests to see what effect farms have on an otherwise ordinary server. Going into a test with a motive to prove will only result in you doing the test until it agrees with what you're trying to show. I didn't say that farms don't cause lag, and I didn't say that they aren't off the hook. I was very careful not to say this. What I said is they do not appear to cause prolonged lag, but they do cause (in some cases quite strong) fluctuations. If people look at my tests and decide to interpret them as farms causing problems, then that's perfectly fine by me. That's exactly why I did them, so people can draw conclusions based on it.

For one, a massive problem I see with the tests conducted is that a consistently low TPS or high fluctuations in TPS are most observable at peak hours -- when we have many players on the server. In your tests you don't mention taking into account multiple players being on the server so I am assuming your tests are with a sole player (you) on the test server. Obviously, this isn't a very good way to measure whether or not we should implement a rule to limit farm use when a certain amount of players are on -- in fact it doesn't do anything to prove that it wouldn't help with lag. A proper test of this clone server would have around the same number of players that are on during peak hours when we have these issues.

You are correct, I was the only person on the test server, however this was only because I didn't have the computing power nor enough accounts to actually simulate 10+ players online, in a way which would be consistent in every test. That lack of consistency is why I didn't do these tests on the server itself, as if there's one less player or people are doing different things on the server, it creates a new variable in which the results could be affected. I agree, a proper test would've had a number of people doing typical activities, but this is hard to achieve in a test condition. Again though, the point of these tests wasn't to prove banning it wouldn't help with lag, but it was to see what effect a farm would have on an otherwise smooth server. The way the server deals with multiple players (if they're not near each other) is it pretty much treats them as individual mini-servers, processing each player and their surroundings separately. If one player causes the server to take 20ms to process everything, and 8 others require 5ms of processing each, then the overall time per tick will be 60ms, so the TPS will be 16.7. Yes, it would be great to do a test with everything running at once, but the point of this was to see the effect farms have. This just gets added to the rest of the processing the server has to do. Obviously there's other factors, but in the very simplest assumptions, if you remove the farms, you'd have the effect of removing that processing, and the time required to do so.

While these results can help you narrow down what the lag may be it does not mean that other forms of data are to be ignored.

I'm not ignoring other data. In fact, my very first sentence in that post was "Right, I've still got a lot more data to go through". I'm looking at everything I have, but as you said, I'm only human, and there's only so many hours in the day for me to spend looking at it. This was just the thing I did that could be presented most clearly.

The anecdotal evidence from a large majority of players that claim to experience lag when OctoGamer gets on (presumably spawning at his farm) has been dismissed routinely when referring to this issue. While it isn't numerical data that you can graph and find the mean or deviation, IT IS data. Player feedback should not be ignored. It means there is something that your tests obviously aren't picking up on that deals with Octo's farm that OTHER players experience constantly and that should be looked into.

I'm not dismissing any evidence, and I never have. Not all my data is numerical data that I can make graphs out of and find averages from, I only did that so it's easier to talk each test as a whole and compare them here. I've been very open throughout my time on VC that I welcome player feedback, and frequently say to get in touch with me about anything you like. I like to think I'm easy to get in touch with and approachable, but if this is not the case, I do apologise, and would appreciate it if you told me how I could improve. I acknowledge there's plenty of things that test didn't pick up on, which is why I'm collecting lots of other data to try and cover as many possibilities as I can, one of those is a correlation between TPS and the list of players logged in a any given time, to try and see if there is any correlation between when people log in, and where they are on the server. I also idle on the Discord about 16 hours a day and am taking notes of when I notice people saying they're feeling lag, and what condition the server is in at that time. I'm also working on getting a copy of the full server process tree at every instant instead of just the ones I ask for, to get more detail on those moments when people notice the lag.

A lot of the frustration that caused these "childish arguments" is that in general the ones protecting farms are both staff and owners of large farms and always claim that it isn't the issue when everyone else is experiencing the effects. As the lead programmer/administrator of the server it is your job to listen to people's frustrations and not immediately dismiss them, as this can guide you to a better understanding of the problem at hand. It's likely that these complaints from players are valid and way you are handling data recording prevents you from seeing these issues.

As with the point above, I try not to dismiss anything unless I can categorically prove it us not the case, which I have not yet done with anything so far dealing with this lag. I'm always listening to everyone's comments about anything server-related, and trying to do what I can to address them. AS you say, they're very important in figuring out what's wrong, as sometimes people feel lag when the numbers suggest there isn't much, and sometimes the numbers imply there's lots of lag but people don't really notice it. This is all important to know, as it helps me narrow down what is causing the lag people experience, and what could be done to fix it. I don't doubt that complaints from players are valid, I know for a fact they're some of the most valuable pieces of data around, and as always I welcome any and all comments about anything, regardless of what they are. The way I did this specific test may hide some of those issues, but there's plenty of other things I'm looking at which are centred around these issues.

As the lead programmer/administrator of the server

I am not the lead admin the server, and don't really want to be seen as like I am. The staff team as a whole works together, as do the admins. There's no real "lead" in the team, and I don't think there should be. I frequently discuss non-trivial backend changes with other admins, rather than just doing things on my own.

The community's argument isn't that farms are severely slowing down the server consistently but rather the extreme fluctuations that cause rubber banding and make VillageCraft unplayable. Why are fluctuations not an issue? Yes, the big picture is to solve the overall lag problem that the server has, but that doesn't mean we should ignore solutions to make the server a little more playable.

I'm not saying the fluctuations aren't an issue, in fact I consider fluctuations to be as bad as a consistent lag. Fluctuations themselves sometimes aren't as noticeable as they sound though, on average the server will fluctuate quite a bit in regular use. Every time the server saves the world, or every time someone logs in and out, the TPS fluctuates, but because it's so brief it's almost never noticeable. These very short period fluctuations aren't a major problem, as they're often over before anyone notices, unless you catch it on the tick(s) it's fluctuating. However, larger fluctuations are just as much of a concern as persistent lag. Once again, I was very careful to avoid saying that fluctuations aren't an issue. As you say, the overarching goal of this is to solve the overall lag problem, but getting rid of noticeable fluctuations is a big part of that. Yes, changes should be made to make the server more playable, but major changes such as farms aren't ones I'm going to do myself, or get behind without reasonable evidence. I've made plenty of other solutions which have improved things, which I'll talk about below.

Waiting until you figure out the big mystery to the lag isn't doing us much of anything when players constantly feel the effects of large farms on the already laggy server that SHOULD be a priority to lessen. Your data doesn't include any of the many large pumpkin farms on VC which is interesting as this is the most prevalent large automated farm on VillageCraft with many players owning them. It would prove to be a good idea to do some tests on these massive pumpkin farms that are much bigger and have way more redstone than these other large farms though I understand these are not the farms in question that have been constantly complained about by the community.

I'm more than happy to investigate large pumpkin farms too. I only picked those two farms in my test because they were the two I noticed most in discussions (Octo's was the most complained about, and Cyphur's was mentioned in the thread). If you've got a good example of a large pumpkin farm, let me know where it is and I'll do some testing on that. For what it's worth, it did look like Octo's farm included an automated pumpkin farm, but I didn't check the size of it, so there may be better examples as you say.

You are speculating just as much as the players are. You aren't getting anywhere, and there hasn't been results for the time that this issue has been around. I don't see why there is any issue with doing a small change to limit large farms until we move toward a permanent solution when you DO happen to solve the underlying problems.

You say I'm not getting anywhere and there's been no results, but I would politely disagree. Over the course of this, I've made many changes, each of which has had a notable effect on average TPS, without any changes in gameplay. That has been my goal up to now, I've been trying to change things which wouldn't affect gameplay to allow everyone to continue as normal. There are other changes I have in mind which I could do but which would start to be noticeable in-game, and I've been careful to avoid these until I need to. To give some examples, I've recently changed quite a bit with how hoppers process things, which had a significant impact on timings in regions with lots of hoppers. There's a lot more I can do with hoppers, but many now would start to break some edge-case situations in game. I've also changed quite a bit with how entities work, which too has had an effect on the average TPS in normal gameplay. Similarly I've changed things with regards to chunk loading, which also appears to have helped slightly. I'm not afraid to admit these changes haven't solved the problem, but I am definitely changing things, and these changes are definitely making improvements (albeit some of them small) to the lag. You say "there hasn't been results for the time that this issue has been around", but I'm interested to know what you consider as the "start" of this issue. From my point of view, there's always been lag on VC, for as long as I can remember. Yes, it did get worse in August when we switched to 1.14, but it's always been around. I remember multiple times over the years trying to debug things that cause lag, many of which we ended up managing to fix. I find it hard to believe that this lag is new though.

It is obviously causing noticeable lag for many players including myself and I find it ridiculous that you so vehemently deny that it could be causing the issue that players are experiencing. I say we vote on this somewhat quickly so it doesn't get swept to the side like it has been for awhile now. That way if the community is wrong, the change can be reverted and crossed off the list as being an issue causing the lag -- and if the community is right -- it will heighten the current experience on VillageCraft by limiting the amount of rubber-banding and skipping when large farms are functioning in high player counts.

I'm not denying it could be causing problems, I'm just saying that based on this test alone, it does not appear to be the primary cause of the major lag we're seeing. Those are two very different statements. Other people on that thread read what I said and interpreted it as me saying that they are causing lag, so to say I'm denying it is confusing. In the conclusions part of the post, I gave some factual statements about that specific data, and then gave a personal opinion on how I would interpret those. And that's why I've done these tests: to allow people to draw their own conclusions from data, rather than listening to what everyone else is saying. I've never stopped anyone voting on it, Legend has been CM since November, and the rules are clear that if 5 players want a Parliament vote on something then they can have a vote. If there's a community backing for limiting farms, there should be no difficulty finding the names required to trigger a vote. However, this vote should not be rushed. There should be a reasonable discussion beforehand where people can share opinions, and ideally there should be suggestions on how the community wants this to be enforced. If we get a vote that just says "people can't use large farms when there is a high player count" then how are we supposed to interpret that? What defines "large"? How many players are required for a "high player count"? Should people be punished for breaking the rules? If so, what is an acceptable punishment? If people log in at a farm after the rules come into force, are they immediately breaking it? Can people be in the vicinity of farms if they're disabled? Can people quickly go to a farm to collect supplies they've left there? What would happen if there's 10 people on the server but every one of them wants to use farms? Is that allowed, or should they be prevented from doing so? As far as I can tell, there's never been anyone stopping a vote on this, so if there's the numbers then a vote should go ahead. That's the point of Parliament.

At the end of my previous post I said "If you've got any comments or things you've noticed about the lag, I'd love to hear from you about it. PM me on here, or on Discord (@Luis). I'm all ears.", and that stands. I do what I do to keep the server going for everyone else to play on, if there's something you want to say about it, talk to me. I won't share things you say without permission, so don't worry about people finding out about stuff. If you've got any comments/suggestions/ideas, please tell me them.

[luisc99]

For all we know, the main cause is probably just Mojang's shitty coding, and no matter how good Luis may be, which is very, I don't think he's willing to recode a whole game to solve the problem.

I've done some patches in the past. Mostly back when I was messing around with Forge I'd re-code some stuff to get things working the way I wanted, but it was always messy. More recently though I've occasionally done some hacky re-coding in Bukkit to get some awkward plugins working, and I also did that thing with 1.13 when I added some form of dynamic block/item/biome registry, which if coded in a better way would've made quite a good modding API. They've all been private patches through. There was that one time I found a bug and told Mojang how I patched it in my tests, but that was over 3 years ago and I've not heard from them about it. The bug still exists.

Some of the other ideas I have on my list would involve recoding bits of the game to account for edge cases, but it's not super easy to do in Bukkit/Spigot/Paper without being very careful not to break stuff.

That being said, Paper is open source, and Mojang have obf mappings now. It would be a pain, but not *too* hard, to patch stuff that's seriously broken. However, Mojang pays people do do that, so it's not my job to do it too

[Naomi (OP)]

I know Shen's farm lags me a lot, it could be a different type then Octo's if you need others to look at Luis

[Shensley]

I've used my farm 3 times since this whole thing started. I understand how the first time caused serious lag because I hadn't used it in many month so all the pumpkins had grown but i purposely avoid using it now so please dont bring me into this.

[CyphurTheFox]

I must say, I quite liked the data you provided luis, as it matches up very cleanly with data I already have about these stuff. Namely the lag situation on my farm becoming much much better on 1.16, likely due to backend nether portal updates. Something interesting to note is that once that farm actually gets spun up it stops loading new chunks. the flow of guardians to the nether is enough to prevent unloading. The thing that causes the most lag on that farm is when a portal doesn't get a guardian through it in at least 15 seconds, which allows its link cache to expire. The process of reestablishing that cache is one of the more annoyingly laggy parts of portals.

I'm more than happy to investigate large pumpkin farms too. I only picked those two farms in my test because they were the two I noticed most in discussions (Octo's was the most complained about, and Cyphur's was mentioned in the thread). If you've got a good example of a large pumpkin farm, let me know where it is and I'll do some testing on that. For what it's worth, it did look like Octo's farm included an automated pumpkin farm, but I didn't check the size of it, so there may be better examples as you say.

If you're interested in checking out a pumpkin farm to investigate, which I personally would find very interesting, I would reccomend one of these:
my /home CyphurTheFox:farm (Mountain Dwelling) (27329 Pumpkin Seeds) (A slightly more reasonable size)
the one at /warp warp (85653 Pumpkin Seeds) (90% Sure biggest on server)

Both are the largest pumpkin farms on server that I know of. If you need me to I could pop back and reinstall the clock and on-off lever for my /home farm if you'd like to test it, as I removed the control redstone a while ago as at the time it was undeniably causing people to complain about server lag, and I didn't use it anyway. (That was when server was not on paper. Eigencraft redstone has brought a lot of optimizations there, so take that observation with a large grain of salt.)

IIRC the farm at /warp warp has also had its redstone mostly removed, so a reinstall there might be necessary as well.

EDIT:

I know Shen's farm lags me a lot, it could be a different type then Octo's if you need others to look at Luis

Just did a simlar analysis for shen's farm, for any curious, and it comes in around 13550 pumpkin seeds

The Pumpkin farm at uhhhhh comes in at 5840 pumpkin seeds