Bot lags every 15-60 secs

[BadassOverlord]

Hi guys,

I looked around to see if anyone else is having the same problem as me, but I couldn't find anything in English.

Around 1 month ago, but my Sinusbot has been getting a lag spike every 15-60 seconds (it's completely random). It makes listening to the bots completely unbearable.

This never happened before. I remember the bot running smoothly with no problems before, on the same host and same configuration. My teamspeak ping fluctuation is around 3 ping, and we all have very low ping (5-15 ping), and there is 0.00% packet loss on both the bot and the listener, even when the lag happens. Furthermore, my cpu usage is always around 30%, and my RAM is only 50% used. When the hiccup happens, nothing unusual occurs except for the bot's ping spiking up from 0 to 1-2. No CPU spike, no ping spike for the listener, no packet loss on both bot and listener, but the lag is there. In fact, I've even noticed the bot's mic icon turn off (like when someone talks, the mic light turns on in the channel tree. I've seen that briefly turning off during moments of intense lag).

Also, when the bot lags, it really is just the bot. I tried an experiment where I had two teamspeak tabs open on my server, and both were in the same channel. I used a soundboard to play some music through one of my teamspeak clients, and it played perfectly. There were no noticeable hiccups, and both me and some other people confirmed it.

I'm at a complete loss on what to do. I've already asked my VPS provider (Ramnode) and they said nothing appears unusual on their end. My network is 100% stable and I have pretty good internet.

I am using version 0.13.37-f7e9ece.

 

[Kir-à]

You have this problem since the last version or even before?

 

[BadassOverlord]

No I've had this problem for like 2 versions now.

 

[flyth]

Scripts maybe? VPS could be overbooked?

 

[BadassOverlord]

Scripts maybe? VPS could be overbooked?

How do you figure out if your VPS is overbooked?

As for scripts, I'm only running stream2icecast, digital clock, and remember last channel.

Once I get home I'll try running without scripts, but these were the same scripts I was running a month ago when it worked fine.

I ran cat /proc/cpuinfo and I didn't find anything interesting. Pasted full output below.

 

[BadassOverlord]

Here's the full output of cat /proc/cpuinfo:

processor    : 0
vendor_id    : GenuineIntel
cpu family    : 6
model        : 13
model name    : QEMU Virtual CPU version (cpu64-rhel6)
stepping    : 3
microcode    : 0x1
cpu MHz        : 2399.998
cache size    : 4096 KB
physical id    : 0
siblings    : 1
core id        : 0
cpu cores    : 1
apicid        : 0
initial apicid    : 0
fpu        : yes
fpu_exception    : yes
cpuid level    : 4
wp        : yes
flags        : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm nopl pni cx16 hypervisor lahf_lm abm
bugs        :
bogomips    : 4799.99
clflush size    : 64
cache_alignment    : 64
address sizes    : 46 bits physical, 48 bits virtual
power management:

processor    : 1
vendor_id    : GenuineIntel
cpu family    : 6
model        : 13
model name    : QEMU Virtual CPU version (cpu64-rhel6)
stepping    : 3
microcode    : 0x1
cpu MHz        : 2399.998
cache size    : 4096 KB
physical id    : 1
siblings    : 1
core id        : 0
cpu cores    : 1
apicid        : 1
initial apicid    : 1
fpu        : yes
fpu_exception    : yes
cpuid level    : 4
wp        : yes
flags        : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm nopl pni cx16 hypervisor lahf_lm abm
bugs        :
bogomips    : 4799.99
clflush size    : 64
cache_alignment    : 64
address sizes    : 46 bits physical, 48 bits virtual
power management:

 

[BadassOverlord]

It's been getting worse and worse now. Already tried with all scripts turned off, no difference. I tried again with a soundboard and it sounded perfectly fine. No lag spikes or anything, but the moment I turn on the bot, it lags really really bad. Every 2-5 seconds there will be a hiccup, and every 10-15 there will be major lag where it will become extremely robotic and sometimes audio stops playing and all that can be heard is static. I've also noticed the bot's ping going from 0ms (it's hosted on the same server as ts) all the way to 6-7ms. That's even higher than my own ping on the teamspeak server, and the server is hosted on a VPS.

If there are any logs or anything that shed some more info, please let me know.

 

[siysco]

Q: How do you figure out if your VPS is overbooked?
You might misunderstand. When you purchase a VPS your VM is running on shared hardware. Lets say your VM is running on a xen cluster of 20 servers - but there are also 10,000 other customers also running their virtual machines on the same hardware/cluster. If everyone maxes their vCPU's you may have to wait until your instructions are executed, since the real CPU's are at 0% idle. It's like overselling parking spaces. They can oversell because they know everyone won't be using the parking garage at the same time or day - at least they hope so . So, you MIGHT be "waiting in line" to use your virtual processor - at least that is what has been suggested in this thread.

You can test this really easily [on Linux]!
the top command on linux has the st statistic that monitors just what we have been discussing.

From the man page:
st : time stolen from this vm by the hypervisor

You could also create an AWS EC2 instance (.micro [running linux] is FREE for a year) and migrate there. You can test there and see if you experience the same issues.

I hope this helps!

 

[BadassOverlord]

Q: How do you figure out if your VPS is overbooked?
You might misunderstand. When you purchase a VPS your VM is running on shared hardware. Lets say your VM is running on a xen cluster of 20 servers - but there are also 10,000 other customers also running their virtual machines on the same hardware/cluster. If everyone maxes their vCPU's you may have to wait until your instructions are executed, since the real CPU's are at 0% idle. It's like overselling parking spaces. They can oversell because they know everyone won't be using the parking garage at the same time or day - at least they hope so . So, you MIGHT be "waiting in line" to use your virtual processor - at least that is what has been suggested in this thread.

You can test this really easily!
the top command on linux has the st statistic that monitors just what we have been discussing.

From the man page:
st : time stolen from this vm by the hypervisor

You could also create an AWS EC2 instance (.micro [running linux] is FREE for a year) and migrate there. You can test there and see if you experience the same issues.

I hope this helps!

Ah shoot it looks like you're correct. My st statistic is showing upwards of 60%. The only thing is, is this out of the actual CPU chip itself, or is it out of my allocated resources?

EDIT: Just filed a support ticket on my VPS provider calling them out on this BS, let's see what happens :/

 

[siysco]

If I understand your question correctly; it could be either. Check other top statistics, like id, and see how your CPU is behaving. You will be able to come up with your own conclusion, once you understand what the values mean.

 

[BadassOverlord]

If I understand your question correctly; it could be either. Check other top statistics, like id, and see how your CPU is behaving. You will be able to come up with your own conclusion, once you understand what the values mean.

It's quite clear that my CPU resources are being stolen from me. The ping spikes and hard lag on the bot correlate perfectly with spikes on the "ST" counter.

Thanks for your help though, I finally have some proof to show my VPS host that no, my VPS is not running normal, like they always claim it does.

 

[siysco]

Awesome! As a final suggestion, you may want to look into installing a SNMP server so you can track these events (with pretty graphs). Having this historical data will help your case.

You can do this all internally so there are no vulnerabilities. Probably not appropriate for this forum though.

 

[siysco]

Oh! I forgot. You also could have just upgraded your kernel to patch Meltdown.
The patch can potentially cause up to a 20% CPU performance drop - depending on the instructions, but more commonly 2-5%.

If you were already close to maxing your CPU, this could be the culprit! It would be fun to know. Maybe check your yum transaction log.

 

[veaex]

Scripts maybe? VPS could be overbooked?

i hvae 48gb of ram ddr4 | 6 vcores | 512 gb ssd and my ram is to 0,5 % loaded :-/ My bot is lagging too. so i dont think the problem is in the hardware.

EDIT// and a 10gbits internet connection | centralized in munich/germany

 

[irgendwr]

i hvae 48gb of ram ddr4 | 6 vcores | 512 gb ssd and my ram is to 0,5 % loaded :-/ My bot is lagging too. so i dont think the problem is in the hardware.

EDIT// and a 10gbits internet connection | centralized in munich/germany

Virtual hardware does not guarantee any performance. The usage you see does not reflect performance or usage of the actual hardware.

Read the posts from @siysco, he/she explained VPS overbooking pretty well:

Q: How do you figure out if your VPS is overbooked?
You might misunderstand. When you purchase a VPS your VM is running on shared hardware. Lets say your VM is running on a xen cluster of 20 servers - but there are also 10,000 other customers also running their virtual machines on the same hardware/cluster. If everyone maxes their vCPU's you may have to wait until your instructions are executed, since the real CPU's are at 0% idle. It's like overselling parking spaces. They can oversell because they know everyone won't be using the parking garage at the same time or day - at least they hope so . So, you MIGHT be "waiting in line" to use your virtual processor - at least that is what has been suggested in this thread.

You can test this really easily [on Linux]!
the top command on linux has the st statistic that monitors just what we have been discussing.

From the man page:
st : time stolen from this vm by the hypervisor

 

[PaInKiLLeR]

You need a KVM Root Server not a V-Server for a better Performance

 

[Lolipop]

I'm having a very similar experience.. I reinstalled my whole dedicated server and put the backup of my ts3 on the new version of ubuntu and installed sinusbot again to fix youtube-dl bcuz it was downloading very slow..

Anyway youtube-dl fixed.. the songs download literally almost instantly.. around 1 sec max.. (2gbit down / up connection ) but now there is a prob with music stutter that didn't happen b4.

 

[CUBE]

just the past few days im getting up to a 2% Packet loss in Teamspeak, on only the bot, but couldn't find any updated info in english regarding why this would happen..... I just increased my sampleinterval to 100 and it seems to be a little more stable although still fluctuating... I will keep an eye on it but so far its sounding a lot better.. fingers crossed... love the program.. my friends and I are having a lot of fun with it ...

 

[Alfredo Lima Verde]

You need a KVM Root Server not a V-Server for a better Performance

I use 8 bots total in my panel (with the license extended) and when putting 6 bots playing together when using the extras (for the staff) when trying to play using the webinteface they get very lagged being that I use a KVM and the CPU stays in 20 % and Ram by 20%

 

[Lolipop]

I'm sorry.. 8 bots (6 bots constantly working) + extra addons shit and using only 20% (CPU and RAM).. Wtf are the specs ?

I have multiple servers with bots rn.. Basically 1 cpu / 1gb ram devoted to 2 bots.