[00:40:18] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Scap, 10Infrastructure-Foundations, 10Puppet: Fatal error: Uncaught ConfigException: Failed to load configuration from etcd - https://phabricator.wikimedia.org/T296125 (10AlexisJazz) >>! In T296125#7518351, @LucasWerkmeister wrote: > Web reques... [00:47:38] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [09:38:05] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10Release, 10Train Deployments: 1.38.0-wmf.9 seems to have introduced a memory leak - https://phabricator.wikimedia.org/T296098 (10akosiaris) p:05Unbreak!→03High Both appserver cluster and parsoid cluster have been back to normal memory patterns since th... [12:43:31] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10Release, 10Train Deployments: 1.38.0-wmf.9 seems to have introduced a memory leak - https://phabricator.wikimedia.org/T296098 (10Jdforrester-WMF) p:05High→03Unbreak! Still UBN as it's a train blocker. [13:16:18] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T293950 (10Urbanecm) [16:52:17] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10Release, 10Train Deployments: 1.38.0-wmf.9 seems to have introduced a memory leak - https://phabricator.wikimedia.org/T296098 (10daniel) >>! In T296098#7518762, @Jdforrester-WMF wrote: > Still UBN as it's a train blocker. But it's a blocker for wmf.11, no... [16:53:53] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10Release, 10Train Deployments: 1.38.0-wmf.9 seems to have introduced a memory leak - https://phabricator.wikimedia.org/T296098 (10Majavah) >>! In T296098#7518928, @daniel wrote: >>>! In T296098#7518762, @Jdforrester-WMF wrote: >> Still UBN as it's a train b... [16:55:31] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10Release, 10Train Deployments: 1.38.0-wmf.9 seems to have introduced a memory leak - https://phabricator.wikimedia.org/T296098 (10daniel) >>! In T296098#7517073, @thcipriani wrote: >> Edit: wrong ticket I wanted to mention. > > ^ @daniel does this seem pos... [17:01:34] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10Release, 10Train Deployments: 1.38.0-wmf.9 seems to have introduced a memory leak - https://phabricator.wikimedia.org/T296098 (10daniel) >>! In T296098#7518932, @Majavah wrote: > No, wmf.9 was rolled back due to it. Hold on, now I'm confused. My take it t... [18:11:41] 10Phabricator, 10Project-Admins: Rename - https://phabricator.wikimedia.org/T296156 (10MediaJS) [18:12:31] 10Phabricator, 10Project-Admins: Rename - https://phabricator.wikimedia.org/T296156 (10MediaJS) p:05Triage→03Lowest [18:13:39] 10Phabricator, 10Project-Admins: Rename MediaJS to Bongo-Cat - https://phabricator.wikimedia.org/T296156 (10AntiCompositeNumber) [19:39:20] 10Phabricator, 10Project-Admins: Rename MediaJS to Bongo-Cat - https://phabricator.wikimedia.org/T296156 (10Aklapper) 05Open→03Resolved a:03Aklapper Done. [19:48:43] from 110 to 52 to 110 users in the channel in few hours :) [21:10:12] PROBLEM - SSH on contint1001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [22:11:16] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook