[00:14:06] (03CR) 10Jeena Huneidi: "recheck" [blubber] - 10https://gerrit.wikimedia.org/r/650266 (owner: 10Dduvall) [00:49:10] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10MW-on-K8s, 10Release Pipeline, 10serviceops: Request volume for Docker images and container filesystems on releases machines - https://phabricator.wikimedia.org/T272092 (10Dzahn) a:05dduvall→03Dza... [05:00:56] 10Release-Engineering-Team (Pipeline), 10Add-Link, 10Growth-Structured-Tasks, 10Growth-Team (Current Sprint), 10Patch-For-Review: Add Link engineering: Deployment Pipeline setup - https://phabricator.wikimedia.org/T265893 (10Tgr) Talked this over with @kostajh and the options we saw for the batch job wer... [05:06:25] 10Release-Engineering-Team (Pipeline), 10Add-Link, 10Growth-Structured-Tasks, 10Growth-Team (Current Sprint), 10Patch-For-Review: Add Link engineering: Deployment Pipeline setup - https://phabricator.wikimedia.org/T265893 (10Tgr) (The task for this is {T272419} btw. Currently it is written based on the f... [08:00:03] Project mwcore-phpunit-coverage-master build #1165: 04STILL FAILING in 5 hr 0 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1165/ [08:07:38] how to feel like a newbie again, compiler yields gibberish message: "Future is a raw type. References to generic type Future should be parameterized" [08:07:41] * hashar sigh [08:44:06] 10:26:26 [Composer\Downloader\TransportException] [08:44:09] 10:26:26 Content-Length mismatch, received 169191 bytes out of the expected 2379602 [08:44:12] what's this? [09:28:29] 10Phabricator-Bot-Requests, 10observability: Create Phabricator bot for Alertmanager - https://phabricator.wikimedia.org/T272391 (10fgiunchedi) Amazing, thank you very much @Aklapper ! We'd like to change the bot's avatar (although we don't have one yet), is that something we can do autonomously ? [09:34:47] 10Release-Engineering-Team (Pipeline), 10Add-Link, 10Growth-Structured-Tasks, 10Growth-Team (Current Sprint), 10Patch-For-Review: Add Link engineering: Deployment Pipeline setup - https://phabricator.wikimedia.org/T265893 (10kostajh) 05Resolved→03Open Re-opening while we discuss T265893#6760579 [09:37:11] (03Abandoned) 10Hashar: Raise client timeout in simpleAttachedJobBulk test [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/657187 (owner: 10Hashar) [09:37:46] (03CR) 10Hashar: "recheck" [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/655663 (https://phabricator.wikimedia.org/T271683) (owner: 10Hashar) [10:07:50] (03CR) 10Hashar: [C: 03+2] "thx" [integration/config] - 10https://gerrit.wikimedia.org/r/657110 (owner: 10Yaron Koren) [10:09:03] (03Merged) 10jenkins-bot: layout: Add CI for mediawiki/extensions/Acrolinx [integration/config] - 10https://gerrit.wikimedia.org/r/657110 (owner: 10Yaron Koren) [10:10:53] 10Phabricator-Bot-Requests, 10observability: Create Phabricator bot for Alertmanager - https://phabricator.wikimedia.org/T272391 (10Aklapper) If you're an admin you should be able via https://phabricator.wikimedia.org/people/picture/27816/ . If not then please throw some image at me. :) [10:26:20] Nikerabbit: just a network error? [11:08:13] 10Phabricator-Bot-Requests, 10observability: Create Phabricator bot for Alertmanager - https://phabricator.wikimedia.org/T272391 (10Aklapper) [12:46:55] 10Phabricator: "New WMF-NDA Task" form does not allow subscribers to access the task - https://phabricator.wikimedia.org/T272476 (10Urbanecm) [13:43:19] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T271341 (10Nikerabbit) [14:02:46] 10Release-Engineering-Team (Pipeline), 10Add-Link, 10Growth-Structured-Tasks, 10Growth-Team (Current Sprint), 10Patch-For-Review: Add Link engineering: Deployment Pipeline setup - https://phabricator.wikimedia.org/T265893 (10akosiaris) >>! In T265893#6760579, @Tgr wrote: > Talked this over with @kostajh... [14:24:51] (03CR) 10Thcipriani: Refactor other builder types to use RequirementsConfig [blubber] - 10https://gerrit.wikimedia.org/r/650266 (owner: 10Dduvall) [14:24:54] (03CR) 10Thcipriani: [C: 03+2] Refactor other builder types to use RequirementsConfig [blubber] - 10https://gerrit.wikimedia.org/r/650266 (owner: 10Dduvall) [14:27:23] (03CR) 10jerkins-bot: [V: 04-1] Refactor other builder types to use RequirementsConfig [blubber] - 10https://gerrit.wikimedia.org/r/650266 (owner: 10Dduvall) [15:00:43] 10phan-taint-check-plugin: Hardcode hashing functions as removing taintedness? - https://phabricator.wikimedia.org/T272492 (10Daimona) [15:01:22] 10Release-Engineering-Team (Pipeline), 10Add-Link, 10Growth-Structured-Tasks, 10Growth-Team (Current Sprint), 10Patch-For-Review: Add Link engineering: Deployment Pipeline setup - https://phabricator.wikimedia.org/T265893 (10kostajh) >>! In T265893#6761792, @akosiaris wrote: >>>! In T265893#6760579, @Tgr... [15:12:58] (03PS4) 10Thiemo Kreuz (WMDE): Add test for index error in AssignmentInReturnSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/606157 [15:17:12] (03CR) 10jerkins-bot: [V: 04-1] Add test for index error in AssignmentInReturnSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/606157 (owner: 10Thiemo Kreuz (WMDE)) [15:19:17] 10Phabricator-Bot-Requests, 10observability: Create Phabricator bot for Alertmanager - https://phabricator.wikimedia.org/T272391 (10fgiunchedi) Sounds great, thank you @Aklapper ! I'll update the task when we have the bot deployed and working [16:17:57] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T271341 (10brennen) [16:23:54] 10phan-taint-check-plugin: Hardcode hashing functions as removing taintedness? - https://phabricator.wikimedia.org/T272492 (10sbassett) Should be very low risk, IMO, given the extremely low likelihood of malicious data being created by the output of a standard hash function. [16:28:33] hey all, a week or two ago someone told me how to get a PHP mw shell on deploy1001 (or maybe is was mwmain1002) [16:28:42] i did not take a note and it is no longer in my shell history [16:31:05] OH WAIT GOT IT [16:31:09] wikitech search success [16:31:10] https://wikitech.wikimedia.org/wiki/Wikimedia_binaries [16:31:15] mwscript shell.php --wiki testwiki [16:31:16] thank you. [16:53:34] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T271341 (10brennen) [17:00:37] ottomata: here, have a rubber ducky ;) [17:01:50] :) [17:49:39] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [17:50:23] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [17:51:15] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [17:51:51] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [18:14:13] (03CR) 10Jforrester: [C: 03+2] Rewrite ParenthesesAroundKeywordSniff to fix whitespace handling [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/650595 (https://phabricator.wikimedia.org/T265741) (owner: 10Umherirrender) [18:15:58] (03Merged) 10jenkins-bot: Rewrite ParenthesesAroundKeywordSniff to fix whitespace handling [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/650595 (https://phabricator.wikimedia.org/T265741) (owner: 10Umherirrender) [18:21:45] Project mediawiki-core-doxygen-docker build #22030: 04FAILURE in 17 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/22030/ [18:22:55] oh [18:25:23] https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/22030/artifact/errors.txt [18:25:28] not sure why that is an error [18:25:56] cause a success build is working ( https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/22029/artifact/errors.txt/*view*/ ) [18:33:14] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2325.codfw.wmnet'] ` an... [18:33:47] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2316.codfw.wmnet'] ` an... [18:35:31] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2327.codfw.wmnet'] ` an... [18:36:07] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2329.codfw.wmnet'] ` an... [18:41:59] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T271341 (10brennen) [18:57:09] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [18:58:09] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [18:59:14] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:00:00] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:04:17] brennen: your welcome [19:05:59] * James_F hands RhinosF1 an 'e' and an apostrophe. :-) [19:06:18] James_F: it's been a long day [19:06:34] Sorry. :-) [19:08:46] James_F: you're not to know but yeah today if can't go wrong, it's blown up spectacularly. If it could wrong, it definately has. [19:08:49] 10Phabricator (Upstream), 10Developer-Wishlist (2017), 10Upstream: Phabricator should suggest possible duplicates when creating a new task - https://phabricator.wikimedia.org/T45 (10Elitre) Any changes that would make this possible? ^_^ [19:08:57] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T271342 (10Ladsgroup) [19:09:59] You're free to yaw on your tales of yore. [19:11:50] James_F: someone caught covid for the second time and they not doing great [19:12:19] Sorry to hear that. It's a frightening time. [19:12:50] Yeah [19:12:57] They already had asthma [19:13:15] * RhinosF1 doesn't enjoy waking up and the first thing he sees being a sweary text message [19:17:14] :( [19:19:40] Yippee, build fixed! [19:19:41] Project mediawiki-core-doxygen-docker build #22031: 09FIXED in 15 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/22031/ [19:39:01] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2331.codfw.wmnet'] ` an... [19:40:12] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2333.codfw.wmnet'] ` an... [19:40:44] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2335.codfw.wmnet'] ` an... [19:41:35] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2337.codfw.wmnet'] ` an... [19:59:28] 10Phabricator (Upstream), 10Developer-Wishlist (2017), 10Upstream: Phabricator should suggest possible duplicates when creating a new task - https://phabricator.wikimedia.org/T45 (10Aklapper) See my previous comments here; has some situation changed, or have new arguments arisen? [20:00:05] Project mwcore-phpunit-coverage-master build #1166: 04STILL FAILING in 5 hr 0 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1166/ [20:23:54] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T271341 (10brennen) [20:41:43] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10MW-on-K8s, 10Release Pipeline, 10serviceops: Request volume for Docker images and container filesystems on releases machines - https://phabricator.wikimedia.org/T272092 (10Dzahn) New disks have been... [20:55:53] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:56:15] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:57:08] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:57:26] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:37:22] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2338.codfw.wmnet'] ` an... [21:37:57] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2339.codfw.wmnet'] ` an... [21:38:29] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2351.codfw.wmnet'] ` an... [21:39:02] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2353.codfw.wmnet'] ` an... [22:22:13] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:23:55] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:24:11] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2355.codfw.wmnet'] ` Of... [22:24:23] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:24:48] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:25:04] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:41:28] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/WikiToLDAP] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/657423 [22:41:30] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [extensions/WikiToLDAP] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/657423 (owner: 10QChris) [22:41:36] (03PS1) 10QChris: Import done. Revoke import grants [extensions/WikiToLDAP] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/657424 [22:41:38] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [extensions/WikiToLDAP] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/657424 (owner: 10QChris) [22:51:28] yo, so to add the new virtual disks that were requested on releases* servers.. we need to reboot the VMs [22:52:00] it will be really quick but they are the backends for releases.wikimedia.org of course [22:52:17] but i think we can JFDI it [22:52:28] j #wikimedia-operations [22:52:31] oops :) [23:03:27] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2355.codfw.wmnet'] ` an... [23:06:20] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2361.codfw.wmnet'] ` an... [23:06:59] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2359.codfw.wmnet'] ` an... [23:07:38] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2357.codfw.wmnet'] ` an... [23:14:24] mutante: yeah, I don't think that's a big deal, that machine cuts branches on Tuesdays, but as you mentioned it's mostly used for downloading releases [23:15:19] thcipriani: cool, thanks. I will just do that and start with codfw [23:15:43] checking icinga to see if we should downtime something [23:26:52] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [23:27:28] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [23:28:06] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [23:28:36] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [23:36:25] thcipriani: this is just making me laugh now.. releases2002 did not come back from reboot.. ferm/iptables failed to start so no SSH access and now debugging .. and all because I claimed it's trivial.. this guarantees it [23:36:42] good that it's just codfw:) [23:37:05] haha the "t"-word will curse all your endevours [23:37:23] yes, whenever it's used it raises the probablity of an issue, for sure:) [23:38:20] Jan 20 23:35:10 releases2002 ferm[997]: DNS query for 'backup1001.eqiad.wmnet' f [23:38:30] that's bad [23:38:44] when the failed DNS query makes the firewall service fail [23:39:02] it's kind of what Antoine always talked about why he wanted IPs [23:39:13] but first.. why is it failing [23:40:02] that's not a decom'ed host ... grr [23:42:03] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T271341 (10brennen) End-of-day status: Quiet on group1 for the last several hours. [23:46:59] hrm, weirdly I can't get to releases1001.wikimedia.org via ssh [23:47:13] thcipriani: eqiad.wmnet [23:47:58] releases.wikimedia.org is still up but let's definitely not reboot releases1001 [23:48:22] went with wikimedia.org since that was in my history, but I can't get to releases1001.eqiad.wmnet either...strange [23:48:23] thcipriani: ah, also it's 1002 [23:48:28] ah ha [23:48:49] 2002 .. does not even have an IP address right now [23:48:56] so it cant talk to a DNS server [23:49:14] yep 1001 works, sorry for the noise. thought it was odd. [23:49:16] something is bad with ganeti networking maybe [23:49:24] er 1002 [23:49:31] ACK:) [23:50:29] ah, this would explain my issues with mwmaint1002 [23:50:36] (I also cannot ssh to that) [23:51:28] foks: uhm. no, that should not explain it.. wah [23:51:42] huh [23:51:54] (it might be me? I don't know where to report this. :) ) [23:51:54] [mwmaint1002:~] $ uptime 23:51:47 up 139 days, [23:51:57] pheew [23:52:17] lol, sorry for the heart attack [23:52:20] foks: report it to me.. what is your full SSH command [23:52:32] and try it with -vvv [23:52:52] foks: oh, I think i know! [23:52:54] it's `ssh mwmaint1002.eqiad.wmnet` [23:52:56] there are new bastion hosts [23:52:58] ohhh [23:53:00] you have to edit your ssh config [23:53:05] I had to do that as well today [23:53:07] that would explain `ssh: connect to host bast4002.wikimedia.org port 22: Operation timed out` [23:53:13] foks: are you on the ops mailing list? [23:53:15] try just counting up [23:53:18] I am not, legoktm [23:53:19] to 4003 [23:53:21] ok [23:53:23] thank you! [23:53:32] you should be on ops if you have shell, tbh [23:53:36] for this kind of thing [23:53:44] foks: https://lists.wikimedia.org/mailman/listinfo/ops [23:53:50] danke [23:54:41] just forwarded the announcement to you [23:54:56] thanks legoktm! [23:57:51] TIL: https://config-master.wikimedia.org/ \o/ [23:58:01] yup, seems good now. Thanks mutante :) [23:58:07] foks: great:) [23:59:08] thcipriani: it used to be on noc.wikimedia.org but got separated [23:59:24] well the pybal part [23:59:29] I had no idea there was a list of key fingerprints anywhere [23:59:44] aaah.. that :)