[00:56:22] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [00:59:45] !log unstick beta ci [00:59:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:15:22] 10Phabricator, 10Release-Engineering-Team (Done by Thu 04 Nov🔥): give visibility for "in progress" tasks on a work board - https://phabricator.wikimedia.org/T291593 (10AntiCompositeNumber) Does the indicator have to be shown for open tasks? As open is the default state, it seems a bit redundant and takes up ex... [01:17:03] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Release-Engineering-Team: Make icinga-wm complain if beta CI jobs haven't run in some period of time - https://phabricator.wikimedia.org/T295241 (10Reedy) [09:04:37] 10Continuous-Integration-Config, 10Release-Engineering-Team (Seen), 10Scap, 10MW-on-K8s, and 6 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10awight) ^ I've added a patch for discussion, once these files are read during startup we... [14:42:47] 10Beta-Cluster-Infrastructure, 10Infrastructure-Foundations, 10Puppet: Add memcached to mwmaint01 using puppet - https://phabricator.wikimedia.org/T240263 (10Majavah) 05Open→03Resolved mwmaint02 was created recently and didn't need this procedure. [15:15:04] 10Continuous-Integration-Config, 10phan-taint-check-plugin: Taint-check demo broken with: munmap() failed: [28] Invalid argument - https://phabricator.wikimedia.org/T295228 (10Daimona) I could reproduce this issue on the toolforge instance at https://taintcheck.toolforge.org/. At this point, I think it's cause... [16:47:24] 10Beta-Cluster-Infrastructure: Migrate beta cluster to ELK7 - https://phabricator.wikimedia.org/T283013 (10Majavah) 05Open→03Resolved a:03Majavah [16:47:26] 10Beta-Cluster-Infrastructure, 10SRE Observability, 10Wikimedia-Logstash, 10observability: logstash-beta.wmflabs.org does not receive any mediawiki events - https://phabricator.wikimedia.org/T233134 (10Majavah) [17:09:02] 10Beta-Cluster-Infrastructure, 10Chinese-Sites: Request temporary admin and interface admin right on zhwiki beta cluster - https://phabricator.wikimedia.org/T249819 (10Cosine02) [17:26:27] 10Continuous-Integration-Config, 10phan-taint-check-plugin: Taint-check demo broken with: munmap() failed: [28] Invalid argument - https://phabricator.wikimedia.org/T295228 (10Daimona) a:03Daimona This is indeed caused by some change in emscripten, since it works with 2.0.21 but not 2.0.34 (latest version).... [17:30:32] (03PS1) 10Daimona Eaytoy: dockerfiles: [mediawiki-phan-taint-check-demo] Lock emscripten to 2.0.21 [integration/config] - 10https://gerrit.wikimedia.org/r/737205 (https://phabricator.wikimedia.org/T295228) [20:17:00] PROBLEM - SSH on contint1001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [21:31:55] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [mediawiki-phan-taint-check-demo] Lock emscripten to 2.0.21 [integration/config] - 10https://gerrit.wikimedia.org/r/737205 (https://phabricator.wikimedia.org/T295228) (owner: 10Daimona Eaytoy) [21:34:02] (03Merged) 10jenkins-bot: dockerfiles: [mediawiki-phan-taint-check-demo] Lock emscripten to 2.0.21 [integration/config] - 10https://gerrit.wikimedia.org/r/737205 (https://phabricator.wikimedia.org/T295228) (owner: 10Daimona Eaytoy) [21:35:46] !log Docker: Publishing mediawiki-phan-taint-check-demo 0.1.5 for T295228 [21:35:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:35:49] T295228: Taint-check demo broken with: munmap() failed: [28] Invalid argument - https://phabricator.wikimedia.org/T295228 [21:41:34] 10Continuous-Integration-Config, 10Codex, 10Design-Systems-team (Design Systems Team FY2021-22 Kanban Board), 10Patch-For-Review: Deploy current main branch's docs site to doc.wikimedia.org - https://phabricator.wikimedia.org/T293704 (10Jdforrester-WMF) Any way to make the GitHub link instead link to our G... [21:42:46] (03PS1) 10Jforrester: jjb: [mw-tools-phan-demos-publish] Bump to pinned emscripten image [integration/config] - 10https://gerrit.wikimedia.org/r/737211 (https://phabricator.wikimedia.org/T295228) [21:47:14] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/737211 (https://phabricator.wikimedia.org/T295228) (owner: 10Jforrester) [21:49:00] (03Merged) 10jenkins-bot: jjb: [mw-tools-phan-demos-publish] Bump to pinned emscripten image [integration/config] - 10https://gerrit.wikimedia.org/r/737211 (https://phabricator.wikimedia.org/T295228) (owner: 10Jforrester)