[00:01:22] Why it's PetScan down so often? [00:13:10] @kimkelting: because it is a solo volunteer created and managed service that has a number of scaling and stability challenges? [00:18:00] !log petscan Soft reboot petscan5.petscan.eqiad1.wikimedia.cloud via horizon. Instance not responsive to ssh and console logs shows OOMKiller reaping a number of processes. [00:18:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Petscan/SAL [00:22:45] !log petscan Started webservice following instance reboot using instructions from https://wikitech.wikimedia.org/wiki/Nova_Resource:Petscan#Starting_the_service [00:22:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Petscan/SAL [00:27:04] !log petscan Reported actions upstream at https://github.com/magnusmanske/petscan_rs/issues/184 [00:27:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Petscan/SAL [00:27:27] * bd808 is done playing rescue petscan for the day [03:15:55] What is the main scale bottleneck? (re @wmtelegram_bot: @kimkelting: because it is a solo volunteer created and managed service that has a number of scaling and stability chall...) [04:04:49] I’m really not certain @harej. I haven’t actually ever tried to monitor or profile it. I was probably reacting to the question more than providing facts. [18:23:40] :) [19:30:13] !log lucaswerkmeister@tools-bastion-13 tools.lexeme-forms deployed 8cdbda6ce3 (upgrade dependencies, including Werkzeug 3.0.6) [19:30:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [19:38:07] !log lucaswerkmeister@tools-bastion-13 tools.quickcategories deployed 13fc15a961 (upgrade dependencies, including Werkzeug 3.0.6) [19:38:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [19:42:37] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed 4bf78a4c42 (upgrade dependencies, including Werkzeug 3.0.6) [19:42:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [19:53:00] !log lucaswerkmeister@tools-bastion-13 tools.speedpatrolling deployed a56dc3d07c (upgrade dependencies, including Werkzeug 3.0.6) [19:53:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.speedpatrolling/SAL [19:57:22] !log lucaswerkmeister@tools-bastion-13 tools.pagepile-visual-filter deployed a2df6bd3de (upgrade dependencies, including Werkzeug 3.0.6) [19:57:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.pagepile-visual-filter/SAL [20:04:18] !log lucaswerkmeister@tools-bastion-13 tools.translate-link deployed 02b7a10a31 (upgrade dependencies, including Werkzeug 3.0.6) [20:04:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL [20:08:34] !log lucaswerkmeister@tools-bastion-13 tools.ranker deployed c034355650 (upgrade dependencies, including Werkzeug 3.0.6) [20:08:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ranker/SAL [20:08:48] ^ I will be *very happy* once we have push-to-deploy :D [20:11:15] such lazy developers [20:26:16] ikr. too lazy to even set up a hacky version of it today [20:27:45] (I assume it would be doable if I was comfortable storing an SSH key in GitLab CI variables)