TQ health status
Just thought I would go through what's happening regarding the current TQ problems and share the information that I have.
We are losing a lot of nodes over a 23 hour period. Ideally we do not want to lose any, perhaps 1-2 at most. For the last few weeks, we've lost up to as many as 20 nodes. Once we lose so many, we have no choice but to reboot the whole cluster and why we have had so many unscheduled reboots lately. Valar and Papasmurf are monitoring TQ very closely and are working on fixing this problem asap.
In regards to starbases, when a node dies there are rare occasions when the star systems are remapped. All starbases in those systems or constellations break resulting in starbases starting up in the wrong location, without a forcefield and/or are unusable. The cause for this is not yet known, but we do have a possible fix in testing.
The server is using more RAM than it should, thus resulting in a memory leak. We deployed a hotfix today with server debugging code written by Porkbelly. Over the next few days he and Mike will be monitoring the nodes checking on what's using the memory and why it’s filling up. Mike is also looking at market data taken from TQ yesterday to see if this is having an impact on memory usage.
So, there you go. TQ's sick, but we have a good team working on restoring her health.