[Docker] SinusBot is taking all the ram after 2 days

[lennard0711]

Hello,

I encountered a problem with the official Docker Version of the Bot. If you have the container running for about 48h it will start to play chrome and tries to get all the RAM possible.
I don't know if it's because I have 2 bots running at the same time (but i don't really think so) or if it's a general Bug.

My current solution is a Systemd Timer that restarts the Bots every 24h but it's a pretty bad solution. I would try to find it out by myself what the exact cause is but I don't really have the time for that right now.

But here is some (probably) usless info about the system:

My 'docker-compose.yml':

version: '3.1'

volumes:
  teamspeak:

services:
  app:
    image: teamspeak
    restart: always
    ports:
      - 9987:9987/udp
      - 10011:10011
      - 30033:30033
    volumes:
      - teamspeak:/var/ts3server/
    environment:
      TS3SERVER_LICENSE: accept
  bot1:
    image: sinusbot/docker
    restart: always
    ports:
      - 8020:8087
    volumes:
      - ./bot1/scripts:/opt/sinusbot/scripts
      - ./bot1/data:/opt/sinusbot/data
  bot2:
    image: sinusbot/docker
    restart: always
    ports:
      - 8025:8087
    volumes:
      - ./bot2/scripts:/opt/sinusbot/scripts
      - ./bot2/data:/opt/sinusbot/data

diagSinusbot.sh output for Bot1:

==========================================================
SINUSBOT RELATED
SYSTEM INFORMATION
- Operating System: Debian GNU/Linux 9.5 (stretch) (Docker)
- Kernel: Linux 4.9.0-8-amd64 x86_64
- Load Average: 2.41 1.48 1.27
- Uptime: 14 days, 16 hours, 32 minutes, 59 seconds
- OS x64 check: OK
- OS Updates: 0 (well done!)
- OS Missing Packages: None
- OS APT Last Update: 27.08.2018 11:54:34 UTC +00:00:00
- SHELL LOCALE LANG: (not set)
- Bot Start Script: not found
- DNS resolution check: www.sinusbot.com resolved to 104.28.15.74 -> OK
- HTTPS check with IPv4 mode: SUCCESS [Connection was established to www.sinusbot.com, CODE #200]
- HTTPS check with IPv6 mode: IGNORE
- CPU:
    Architecture:          x86_64
    CPU(s):                4
    Thread(s) per core:    1
    Core(s) per socket:    4
    Socket(s):             1
    Model name:            Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
    CPU MHz:               2199.996
    Hypervisor vendor:     KVM
    Virtualization type:   full
- RAM: 4.75 GB/11.73 GB in use (40%)
- SWAP: 0 B/0 B in use (0%) (SWAP disabled)
- DISK: 53.41 GB/293.37 GB in use (18%)
- Package versions:
   + libglib: 2.50.3-2

BOT INFORMATION
- Status: running (PIDs: 11, User: root)
- Webinterface: port locally reachable (Port: 8087)
- Binary: /opt/sinusbot/sinusbot
- Binary Info: MD5 Hash: 457224ee75d8a25d64eab6fc7465cd48, Perms: 755, User: root
- Version: 0.14.3-0e747fd
- TS3 Plugin: installed (md5 hash match)
   - Bot Plugin: 9463787dd59286dea7a4a284409993c3
   - TS3 Client: 9463787dd59286dea7a4a284409993c3
- Config:
   - LogLevel =
   - TS3Path = /opt/sinusbot/TeamSpeak3-Client-linux_amd64/ts3client_linux_amd64 (Version 3.2.1)
   - YoutubeDLPath = /usr/local/bin/youtube-dl (does exist, version: 2018.10.05)
- Installed scripts: Animated_Nickname.js; CountryGroup.js; CoverLoader_[Radio_Station_Support].js; Next_Track_Please.js; Reconnect.js; Steam_Server_Information.js; Youtube_Search.js; advertising.js; alonemode.js; bookmark.js; come.js; defaultChannel+.js; followme.js; groupProtection.js; norecording.js; osuStats.js; pizzaTimer.js; registerNotificator.js; rememberChannel.js; songBan.js; volumeKeeper.js; welcome.js

TIME INFORMATION
- Time (local): 21.10.2018 16:34:32 UTC +00:00:00
- Time (remote): <Failed retrieving remote time!>
- Time (difference): n/a secs
- Timezone: Etc/UTC

OTHER INFORMATION
- TeamSpeak 3 Version: 3.2.1
- youtube-dl Version: 2018.10.05
- DiagScript Version: 0.7.1
==========================================================

diagSinusbot.sh output for Bot2:

==========================================================
SINUSBOT RELATED
SYSTEM INFORMATION
- Operating System: Debian GNU/Linux 9.5 (stretch) (Docker)
- Kernel: Linux 4.9.0-8-amd64 x86_64
- Load Average: 1.86 1.43 1.28
- Uptime: 14 days, 16 hours, 36 minutes, 14 seconds
- OS x64 check: OK
- OS Updates: 0 (well done!)
- OS Missing Packages: None
- OS APT Last Update: 27.08.2018 11:54:34 UTC +00:00:00
- SHELL LOCALE LANG: (not set)
- Bot Start Script: not found
- DNS resolution check: www.sinusbot.com resolved to 104.28.15.74 -> OK
- HTTPS check with IPv4 mode: SUCCESS [Connection was established to www.sinusbot.com, CODE #200]
- HTTPS check with IPv6 mode: IGNORE
- CPU:
    Architecture:          x86_64
    CPU(s):                4
    Thread(s) per core:    1
    Core(s) per socket:    4
    Socket(s):             1
    Model name:            Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
    CPU MHz:               2199.996
    Hypervisor vendor:     KVM
    Virtualization type:   full
- RAM: 4.76 GB/11.73 GB in use (40%)
- SWAP: 0 B/0 B in use (0%) (SWAP disabled)
- DISK: 53.46 GB/293.37 GB in use (18%)
- Package versions:
   + libglib: 2.50.3-2

BOT INFORMATION
- Status: running (PIDs: 12, User: root)
- Webinterface: port locally reachable (Port: 8087)
- Binary: /opt/sinusbot/sinusbot
- Binary Info: MD5 Hash: 457224ee75d8a25d64eab6fc7465cd48, Perms: 755, User: root
- Version: 0.14.3-0e747fd
- TS3 Plugin: installed (md5 hash match)
   - Bot Plugin: 9463787dd59286dea7a4a284409993c3
   - TS3 Client: 9463787dd59286dea7a4a284409993c3
- Config:
   - LogLevel = 3
   - TS3Path = /opt/sinusbot/TeamSpeak3-Client-linux_amd64/ts3client_linux_amd64 (Version 3.2.1)
   - YoutubeDLPath = /usr/local/bin/youtube-dl (does exist, version: 2018.10.05)
- Installed scripts: Animated_Nickname.js; CountryGroup.js; CoverLoader_[Radio_Station_Support].js; Next_Track_Please.js; Reconnect.js; Steam_Server_Information.js; Youtube_Search.js; advertising.js; alonemode.js; bookmark.js; come.js; defaultChannel+.js; followme.js; groupProtection.js; norecording.js; osuStats.js; pizzaTimer.js; registerNotificator.js; rememberChannel.js; songBan.js; volumeKeeper.js; welcome.js

TIME INFORMATION
- Time (local): 21.10.2018 16:37:47 UTC +00:00:00
- Time (remote): <Failed retrieving remote time!>
- Time (difference): n/a secs
- Timezone: Etc/UTC

OTHER INFORMATION
- TeamSpeak 3 Version: 3.2.1
- youtube-dl Version: 2018.10.05
- DiagScript Version: 0.7.1
==========================================================

 

[Lala Sabathil]

I've often seen, that docker itself takes all ram.
Check your docker config.

 

[Xuxe]

I've often seen, that docker itself takes all ram.
Check your docker config.

Never seen in 3 years with Docker and Kubernetes.
Post the output of

docker stats

and

free -m

please.

A possible easy solution (should be there anyways) would be a memory limitation and health check with docker. A OOM (Out of Memory) condition should crash the bot and the health check should restart the container automatically.

@mxschmitt

 

[lennard0711]

Never seen in 3 years with Docker and Kubernetes.
Post the output of

docker stats

and

free -m

please.

A possible easy solution (should be there anyways) would be a memory limitation and health check with docker. A OOM (Out of Memory) condition should crash the bot and the health check should restart the container automatically.

@mxschmitt

I've often seen, that docker itself takes all ram.
Check your docker config.

I'll do it as soon as possible but I don't know if I have enough time today or even the next two weeks.

 

[Lala Sabathil]

Never seen in 3 years with Docker and Kubernetes.
Post the output of

Seen it many times myself and at other suers system

 

[Multivitamin]

Probably just a script which has a memory leak, what scripts are enabled?

 

[lennard0711]

Here is the wished output of $ free -m:

              total        used        free      shared  buff/cache   available
Mem:          12019        7090        4240         161         688        4513
Swap:             0           0           0

and of $ docker stats:

CONTAINER ID        NAME                          CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
923138f09866        nextcloud_db_1                0.02%               102.4MiB / 11.74GiB   0.85%               49.8MB / 204MB      37.6MB / 1.29GB     31
9217cac88654        wordpress_db_1                0.02%               94.64MiB / 11.74GiB   0.79%               6.99MB / 40MB       47.2MB / 147MB      30
0d248265ec0a        gitlab_app_1                  2.38%               3.015GiB / 11.74GiB   25.69%              164MB / 120MB       8.91GB / 39.8GB     312
0377b15cfbff        wordpress_app_1               0.03%               105.8MiB / 11.74GiB   0.88%               89.6MB / 95.4MB     650MB / 65.5kB      11
cb2233dd5016        nextcloud_app_1               0.07%               104MiB / 11.74GiB     0.87%               415MB / 190MB       3.64GB / 0B         11
7a728dd635c9        openvpn_openvpn_1             0.01%               1.289MiB / 11.74GiB   0.01%               1.69GB / 1.75GB     79MB / 0B           1
49a3a44f5615        teamspeak_bot1_1              2.22%               1.87GiB / 11.74GiB    15.93%              1.34GB / 909MB      122MB / 152MB       86
db91b20d4614        teamspeak_bot2_1              1.63%               1.075GiB / 11.74GiB   9.16%               1.75GB / 530MB      262MB / 163MB       91
91311e01ec33        teamspeak_app_1               0.54%               12.05MiB / 11.74GiB   0.10%               1.84GB / 3.08GB     90.7MB / 16.4kB     27
bc663ef52bc4        mumbleserver_app_1            0.00%               4.648MiB / 11.74GiB   0.04%               85.9kB / 23.3kB     33.3MB / 328kB      2
6c99d2e1fc61        gitlabrunner_gitlabRunner_1   0.22%               7.793MiB / 11.74GiB   0.06%               128MB / 366MB       155MB / 0B          12
51d712d085fc        haproxy_haproxy_1             0.07%               12.49MiB / 11.74GiB   0.10%               0B / 0B             3.48MB / 0B         2
63b01b60d126        mail_mail_1                   0.01%               6.445MiB / 11.74GiB   0.05%               225kB / 0B          1.69GB / 32.8kB     7
1c81da2f6cde        watchtower_watchtower_1       0.01%               3.426MiB / 11.74GiB   0.03%               54.3kB / 0B         16.1MB / 0B         11

A possible easy solution (should be there anyways) would be a memory limitation and health check with docker. A OOM (Out of Memory) condition should crash the bot and the health check should restart the container automatically.

Well my Users say the Bot starts lagging when the Ram fills up, so it's not the best solution. Right now my Systemd timer solution is working fine but the real question is: Why does it want wall the RAM?

 

[lennard0711]

Probably just a script which has a memory leak, what scripts are enabled?

On Bot1: Animated Nickname, Next track please!, Alone Mode, !come, DefaultChannel+, Osu!Stats, PizzaTimer, Volume keeper, Register Notificator, Song blacklist/ban Group Protection, Reconnect

On Bot2: Animated Nickname, !come, DefaultChannel+, Osu!Stats, PizzaTimer, Volume keeper, Alone Mode, Reconnect

 

[lennard0711]

Someting new is happening now:

Since I use watchtower, my containers update themself when there's a new version. So since this update my CPU idle went from max 10% to min 18% and when in use up to 35% instead of max 15-20%.

I noticed this behaviour on September 25th around 0 o'clock. This is the last commit/update that's on the server right now that probably raised the CPU load: GitHub

Edit: The commit I posted is just the latest but probably not the one that actually increased the CPU usage that much.

 

[irgendwr]

September

I assume you mean Oktober?

So since this update my CPU idle went from max 10% to min 18% and when in use up to 35% instead of max 15-20%.

The only thing that changed is that we added a health check which allows docker to restart the container if the bot is not responding. This doesn't noticeably increase the CPU usage. The rest of the commits just restructured things which doesn't affect you.

On Bot1: Animated Nickname, Next track please!, Alone Mode, !come, DefaultChannel+, Osu!Stats, PizzaTimer, Volume keeper, Register Notificator, Song blacklist/ban Group Protection, Reconnect

On Bot2: Animated Nickname, !come, DefaultChannel+, Osu!Stats, PizzaTimer, Volume keeper, Alone Mode, Reconnect

Use a license to get more instances, this isn't allowed.

Please delete all of the scripts, restart and see if you still have a high CPU/RAM usage.

 

[lennard0711]

I assume you mean Oktober?

Yes.... I'm going to edit it...

The only thing that changed is that we added a health check which allows docker to restart the container if the bot is not responding. This doesn't noticeably increase the CPU usage. The rest of the commits just restructured things which doesn't affect you.

Yeah I've seen that. Now it makes even less sense.

Use a license to get more instances, this isn't allowed.

Well I'm owning 50% of the VPS while a friend holds the other 50% and each one of us has their own BOT. Is this still not allowed?

Please delete all of the scripts, restart and see if you still have a high CPU/RAM usage.

I was about to just reinstall the Bots but I can test that first.

 

[lennard0711]

Ok now the Bot idles around 10% CPU usage which is still higher than before but its back to the normal 15% CPU usage while playing music.
So I will be probably searching for the evil script sometime this week and will watch the RAM usage to see if it still wants to play chrome.

Use a license to get more instances, this isn't allowed.

And coming back to this: I'm probably going to "buy a license"/donate soon. But for testing purposes it would be pretty convenient to have up to 4 Instances for about 2 weeks and then have the option to either downgrade to the free version with 2 instances or to buy a license. This is maybe an idea for the Bot as soon it's out of the Beta

 

[irgendwr]

Ok now the Bot idles around 10% CPU usage

Can you be a bit more specific or show a screenshot what CPU usage you are looking at? There are multiple usage values: The CPU Usage of the server, the CPU usage of the docker container, the CPU usage of the sinusbot combined with all of the TS Clients and yt-dl sessions, and the CPU usage of only the sinusbot.
If you are looking at the highest value where it says "sinusbot" in htop in your container you are looking at the CPU and MEM usage of the Sinusbot + the usage of all of the TS Clients combined.

Btw, I think I know why the RAM (and maybe also CPU) usage have increased: The only thing that changed in the docker image is that the TS Client was automatically updated.
The TS Client is probably to blame for most of the RAM usage (the sinusbot often only uses 100-200MB depending on what is running).
Do you have a serverbanner on the TS Server that the bot connects to? If your serverbanner is large or a GIF the TS Client will download it and might fill up the RAM. (other people had this issue already)

 

[lennard0711]

Can you be a bit more specific or show a screenshot what CPU usage you are looking at? There are multiple usage values: The CPU Usage of the server, the CPU usage of the docker container, the CPU usage of the sinusbot combined with all of the TS Clients and yt-dl sessions, and the CPU usage of only the sinusbot.
If you are looking at the highest value where it says "sinusbot" in htop in your container you are looking at the CPU and MEM usage of the Sinusbot + the usage of all of the TS Clients combined.

Sorry, since I don't have that much time I forgot to define it. Most of the time I'm refering to the output of $ docker stats if not otherwise stated.

Btw, I think I know why the RAM (and maybe also CPU) usage have increased: The only thing that changed in the docker image is that the TS Client was automatically updated.
The TS Client is probably to blame for most of the RAM usage (the sinusbot often only uses 100-200MB depending on what is running).
Do you have a serverbanner on the TS Server that the bot connects to? If your serverbanner is large or a GIF the TS Client will download it and might fill up the RAM. (other people had this issue already)

Well actually the CPU usage increased and the RAM usage decreased now. And my Server is pretty vanilla, so no server banner, not many groups (just admins, normals and guests) and mostly no one is using a ts avatar and if they use one it's just a jepg.

 

[lennard0711]

So this is how it is currently if I look at $ docker stats

CONTAINER ID        NAME                          CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
441fb416e704        teamspeak_bot2_1              17.45%              168.4MiB / 11.74GiB   1.40%               707kB / 1.85MB      60.5MB / 1.94MB     88
8488947d1a70        teamspeak_bot1_1              7.80%               145.2MiB / 11.74GiB   1.21%               551kB / 353kB       243MB / 2.02MB      86

So Bot1 is just in idle mode and doing nothing while Bot2 is getting used by 4 people right now. So if it's the new Teamspeak Client that uses "all" that CPU, well that's just how it is (Wap Bap....). I will turn off my Systemd timer for now to test if the RAM "Bug" is also gone, maybe it's fixed now.