[01:12:06] (03CR) 10Jeena Huneidi: apt.go: Add ability to target releases (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/658432 (https://phabricator.wikimedia.org/T272759) (owner: 10Jeena Huneidi) [01:22:44] (03PS1) 10Legoktm: rust: Install default-libmysqlclient-dev [integration/config] - 10https://gerrit.wikimedia.org/r/658471 [01:50:42] 10VPS-project-Codesearch: Create just puppet search profile for codesearch - https://phabricator.wikimedia.org/T272947 (10Legoktm) [02:12:28] 10VPS-project-Codesearch: Create just puppet search profile for codesearch - https://phabricator.wikimedia.org/T272947 (10Legoktm) 05Open→03Resolved [02:35:35] James_F: if you had let LibUp make the patches on the canaries it would've also upgraded minus-x :| [02:36:03] legoktm: Ah well. Next time. [02:36:17] I wanted to be sure the upgrades would work sanely. [02:38:10] I suppose I could have not pushed them to gerrit for the ones which were no-ops. [02:39:04] James_F: I -1'd https://gerrit.wikimedia.org/r/c/mediawiki/tools/api-testing/+/658405 over the indentation change [02:39:14] others all looked good [02:39:33] Hmm. That was just an auto fix by eslint. [02:39:37] Did it break? [02:40:48] Odd. [02:40:51] https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/api-testing/+/refs/heads/master/.eslintrc.json [02:40:56] is the "indent" rule there throwing it off? [03:12:56] Maybe. [03:13:01] Will prod. [03:39:24] legoktm: Yes. See the next patch fixing things. I could re-stack them if you want? [04:04:57] 10phan, 10CheckUser: CheckUser Phan Failures - https://phabricator.wikimedia.org/T272950 (10Reedy) [04:18:34] 10Phabricator: Change form 90 (create vm-request) not to be editable by subscribers by default - https://phabricator.wikimedia.org/T272951 (10Peachey88) [04:20:32] 10phan, 10CheckUser: CheckUser Phan Failures - https://phabricator.wikimedia.org/T272950 (10Reedy) I'm confused. Fine on a no-op commit https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CheckUser/+/658373 / https://integration.wikimedia.org/ci/job/mwext-php72-phan-docker/92956/console Fails on chained ht... [04:23:50] 10phan, 10CheckUser: CheckUser Phan Failures - https://phabricator.wikimedia.org/T272950 (10Reedy) {b4a377d714c716809fc6106c6a0b3c91907ef46d} added the |null... So I guess, the CU code maybe needs to be more defensive.. Or just the nullable type hints added (have just done in a newer PS) [04:30:46] 10phan, 10CheckUser: CheckUser Phan Failures - https://phabricator.wikimedia.org/T272950 (10Reedy) 05Open→03Invalid Marking the string paramters in the docs as `?string` seems enough Unexpected side effects! [04:36:28] James_F: thanks and +2'd [04:45:44] legoktm: Thanks. [05:36:07] 10Release-Engineering-Team (Logspam), 10StructuredDiscussions, 10Growth-Team (Current Sprint), 10User-brennen, 10Wikimedia-production-error: Flow: could not locate workflow for revision {rev} - https://phabricator.wikimedia.org/T270334 (10Tgr) There's also {T154796} which looks fairly similar. [05:40:18] (03CR) 10Legoktm: [C: 03+2] [labs/tools/logo-test] Add Rust jobs [integration/config] - 10https://gerrit.wikimedia.org/r/658355 (owner: 10Legoktm) [05:42:18] (03Merged) 10jenkins-bot: [labs/tools/logo-test] Add Rust jobs [integration/config] - 10https://gerrit.wikimedia.org/r/658355 (owner: 10Legoktm) [05:43:16] !log reloaded zuul for https://gerrit.wikimedia.org/r/658355 [05:43:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:09:46] 10Release-Engineering-Team (Logspam), 10StructuredDiscussions, 10Growth-Team (Current Sprint), 10User-brennen, 10Wikimedia-production-error: Flow: could not locate workflow for revision {rev} - https://phabricator.wikimedia.org/T270334 (10Tgr) This is triggered by a CheckUser hook that generates related... [06:25:59] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Wikimedia-General-or-Unknown, 10Performance-Team (Radar): Document and automate updating of static/project-logos in mediawiki-config - https://phabricator.wikimedia.org/T98640 (10Legoktm) * https://meta.wi... [06:32:25] (03PS1) 10Legoktm: Review access change [extensions/UploadWizard] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/658375 [06:32:44] (03PS2) 10Legoktm: Allow all label bots to vote Verified, not just jenkins-bot [extensions/UploadWizard] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/658375 [06:32:57] (03CR) 10Legoktm: [V: 03+2 C: 03+2] Allow all label bots to vote Verified, not just jenkins-bot [extensions/UploadWizard] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/658375 (owner: 10Legoktm) [09:00:03] Project mwcore-phpunit-coverage-master build #1178: 04STILL FAILING in 6 hr 0 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1178/ [09:23:54] 10Phabricator, 10DBA: Restart m3 (phabricator) database master db1132 - https://phabricator.wikimedia.org/T272596 (10Marostegui) Procedure: Pre restart [] Silence m3 hosts [] buffer pool dump + disablement in advance to make the restart faster Restart [] `!log m3 master restart - T272596` [] set phabricato... [09:29:05] 10Phabricator, 10DBA: Restart m3 (phabricator) database master db1132 - https://phabricator.wikimedia.org/T272596 (10Marostegui) [09:30:21] 10MediaWiki-Releasing, 10Upstream: Mediawiki 1.34.2 tarball incompatible with 7zip on windows due to Pax format - https://phabricator.wikimedia.org/T257102 (10Nasirkhan) 7zip shows warnings that it is going to replace files. If it extracts without any warning, then I assume 7zip fixed the issue. I use 19.00 a... [11:02:28] 10Phabricator: Change form 90 (create vm-request) not to be editable by subscribers by default - https://phabricator.wikimedia.org/T272951 (10akosiaris) 05Open→03Resolved a:03akosiaris Done, it's `All users` now. Thanks! [11:28:44] 10Phabricator: Add a Phabricator policy capability "Can create Countdowns" - https://phabricator.wikimedia.org/T258599 (10Aklapper) Given that many things take me years I think 6mo are quite alright :D (Thanks!) [12:38:11] 10Project-Admins: Request for creation of Wikimedia Affiliates Data Portal work board - https://phabricator.wikimedia.org/T272974 (10DAlangi_WMF) [13:16:30] 10Phabricator: Unhandled Exception ("Exception") - https://phabricator.wikimedia.org/T272976 (10IN) [13:18:19] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Wikidata-Campsite, 10Patch-For-Review, and 2 others: Run browser tests as part of "npm test" of wikidata/query/gui in CI - https://phabricator.wikimedia.org/T222200 (10Addshore) [13:18:51] 10Phabricator: Unhandled Exception ("Exception") - https://phabricator.wikimedia.org/T272976 (10IN) [13:30:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Wikidata, and 3 others: Create docker based CI job to build the wikidata-query-gui - https://phabricator.wikimedia.org/T210286 (10Addshore) [13:45:48] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Wikidata-Campsite, 10Patch-For-Review, and 2 others: Run browser tests as part of "npm test" for wikidata/query/gui in CI - https://phabricator.wikimedia.org/T222200 (10Addshore) [13:45:51] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Wikidata-Campsite, 10Patch-For-Review, and 2 others: Run browser tests as part of "npm test" for wikidata/query/gui in CI - https://phabricator.wikimedia.org/T222200 (10Addshore) [13:51:06] 10Phabricator-Bot-Requests, 10observability: Create Phabricator bot for Alertmanager - https://phabricator.wikimedia.org/T272391 (10fgiunchedi) @Aklapper could you set https://commons.wikimedia.org/wiki/File:Emoji_u1f526.svg as the bot's icon? I'm assuming Phabricator wants a rasterized version, so perhaps htt... [13:53:37] 10Phabricator: Unhandled Exception ("Exception") - https://phabricator.wikimedia.org/T272976 (10Aklapper) [13:55:08] 10Phabricator: Unhandled Exception ("Exception") - https://phabricator.wikimedia.org/T272976 (10Aklapper) 05Open→03Stalled Hi,how to "Change your browser UI on this page"? Please include full and complete steps to reproduce, which don't leave room for interpretation - thanks! :) [13:58:49] 10Phabricator: Unhandled Exception ("Exception") - https://phabricator.wikimedia.org/T272976 (10IN) >>! In T272976#6777187, @Aklapper wrote: > Hi,how to "Change your browser UI on this page"? Please include full and complete steps to reproduce, which don't leave room for interpretation - thanks! :) Sorry, I typ... [14:14:24] hello, looking for a merger for https://gerrit.wikimedia.org/r/c/integration/config/+/658449 [14:14:35] adding a dep in EventLogging ext on EventBus ext [14:23:53] 10Phabricator (Upstream): Unhandled Exception ("Exception") - https://phabricator.wikimedia.org/T272976 (10Aklapper) 05Stalled→03Invalid You manipulated the user agent string, so the anti-CSRF token doesn't work validate. There is no bug here; this is expected. [14:38:50] 10Project-Admins: Request for creation of Wikimedia Affiliates Data Portal work board - https://phabricator.wikimedia.org/T272974 (10Aklapper) Hi, thanks for requesting this! Just one question to potentially clarify for the description: When would I be supposed to create a ticket in Phabricator about the Wikimed... [15:02:35] 10Project-Admins: Request for creation of Wikimedia Affiliates Data Portal work board - https://phabricator.wikimedia.org/T272974 (10DAlangi_WMF) Thanks for your question @Aklapper, that's a great question. The board will be about to create technical tasks related to the gadgets powering the portal. Meanwhile,... [15:04:08] 10Gerrit: Evaluate Gerrit's zuul-results-summary plugin - https://phabricator.wikimedia.org/T269803 (10hashar) [15:21:39] Project mediawiki-core-doxygen-docker build #22171: 04FAILURE in 17 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/22171/ [15:32:19] (03PS3) 10Lars Wirzenius: test: make test failures under Python3 fail tox test run, and fix [tools/scap] - 10https://gerrit.wikimedia.org/r/656951 [15:33:01] (03CR) 10Lars Wirzenius: "I pushed a new patch set, and would appreciate a re-review. Thanks!" (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/656951 (owner: 10Lars Wirzenius) [15:43:57] paladox: yt? does https://gerrit.wikimedia.org/r/c/integration/config/+/658449 look ok to you? [15:44:15] hi [15:45:13] that seems fine ottomata [15:46:16] ok if I merge then? [15:52:02] (03CR) 10Ottomata: [C: 03+2] Load EventBus for EventLogging tests [integration/config] - 10https://gerrit.wikimedia.org/r/658449 (https://phabricator.wikimedia.org/T272863) (owner: 10Mholloway) [15:52:05] (03PS2) 10Ottomata: Load EventBus for EventLogging tests [integration/config] - 10https://gerrit.wikimedia.org/r/658449 (https://phabricator.wikimedia.org/T272863) (owner: 10Mholloway) [15:52:09] (mergingg) [15:58:06] oh paladox i don't have perms to merge [15:58:35] thcipriani ^ [15:59:43] sbassett: ping, pm? [16:00:13] RhinosF1: I'm around. Need something? [16:00:49] Hello. I'm around too. [16:01:15] sbassett: see pm [16:02:52] paladox: ottomata that lgtm, but I'm headed into a meeting and don't have a time/focus to deploy, dancy might be able to help [16:03:20] i'm here as well if needed (cc: dancy). [16:04:17] (03CR) 10Ahmon Dancy: [C: 03+2] Load EventBus for EventLogging tests [integration/config] - 10https://gerrit.wikimedia.org/r/658449 (https://phabricator.wikimedia.org/T272863) (owner: 10Mholloway) [16:05:23] 10Project-Admins, 10Security, 10user-sbassett: Create security-related project tags secteam-reviewed and wikimedia-project-event - https://phabricator.wikimedia.org/T271729 (10sbassett) >>! In T271729#6775487, @Aklapper wrote: > I created #secteam-wikimedia-project-event and #secteam-reviewed; please review.... [16:05:37] (03Merged) 10jenkins-bot: Load EventBus for EventLogging tests [integration/config] - 10https://gerrit.wikimedia.org/r/658449 (https://phabricator.wikimedia.org/T272863) (owner: 10Mholloway) [16:05:48] 10Project-Admins, 10Security, 10user-sbassett: Create security-related project tags secteam-reviewed and wikimedia-project-event - https://phabricator.wikimedia.org/T271729 (10sbassett) 05Open→03Resolved p:05Triage→03Low a:03Aklapper [16:06:17] thank you! [16:06:50] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/658449 [16:06:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:07:32] ottomata: Should be in place now. [16:07:40] Can you verify? [16:08:44] looks like mholloway is running a rrecheck here: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventLogging/+/658441 [16:08:45] :) [16:10:38] 👍🏾 [16:13:59] Looks like it didn't do the trick. [16:17:55] Yippee, build fixed! [16:17:56] Project mediawiki-core-doxygen-docker build #22172: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/22172/ [16:21:10] ya hm 11:08:15 A dependency error was encountered while installing the extension "EventLogging": Could not find the registration file for the extension "EventBus" [16:34:11] 10Phabricator: Protect Phabricator/Help subpages in the same way as Help page itself - https://phabricator.wikimedia.org/T272989 (10danshick-wmde) [16:39:12] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [16:40:43] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [16:41:16] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10MW-on-K8s, 10Release Pipeline: Move private settings to a k8s compatible location - https://phabricator.wikimedia.org/T271475 (10akosiaris) >>! In T271475#6769543, @dduvall wrote: >>>! In T271475#67690... [16:43:23] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [16:45:20] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [16:45:25] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2321.codfw.wmnet'] ` Of... [16:45:54] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [17:01:55] (03CR) 10Ahmon Dancy: test: make test failures under Python3 fail tox test run, and fix (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/656951 (owner: 10Lars Wirzenius) [17:05:18] dancy: it looks like wmf-quibble needs something else too? [17:05:21] getting A dependency error was encountered while installing the extension "EventLogging": Could not find the registration file for the extension "EventBus" [17:05:26] ^ mholloway [17:06:13] maybe brennen knows more ^? [17:07:04] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Scap, 10MediaWiki-Release-Tools, 10Patch-For-Review: Output of scap lint.py isn't helpful if it fails the php -l check - https://phabricator.wikimedia.org/T272760 (10thcipriani) [17:07:23] brennen and I are in a meeting at the moment. Will respond in about an hour [17:09:34] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Discovery-Search (Current work), 10Patch-For-Review: Remove dependency on Maven for CI for Java projects and align Jenkins job configuration. - https://phabricator.wikimedia.org/T271541 (... [17:10:21] ok thank you! [17:13:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Release Pipeline: Have PipelineBot link directly to build console output on test - https://phabricator.wikimedia.org/T271107 (10thcipriani) [17:15:19] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review: Upgrade Jenkins Gearman plugin from a forked repo - https://phabricator.wikimedia.org/T271683 (10thcipriani) [17:17:30] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10WVUI, 10Release Pipeline (Blubber): Use npm version 6.x on blubber-generated docker images - https://phabricator.wikimedia.org/T269957 (10thcipriani) [17:18:14] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1409.eqiad.wmnet'] ` an... [17:19:42] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1408.eqiad.wmnet'] ` an... [17:20:29] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10serviceops, 10Release Pipeline (Blubber): Blubber needs to check if a user is present before creating it as part of its runs stanza - https://phabricator.wikimedia.org/T268819 (10thcipriani) [17:21:11] 10Beta-Cluster-Infrastructure, 10Quality-and-Test-Engineering-Team (QTE): Upgrade deployment-prep-db hosts to buster/MariaDB 10.4 - https://phabricator.wikimedia.org/T268628 (10thcipriani) [17:21:42] 10Release-Engineering-Team (Deployment services), 10Phatality, 10Developer Productivity: Phatality: Move Request ID back to task description - https://phabricator.wikimedia.org/T237683 (10mmodell) [17:22:05] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Phatality, 10Developer Productivity: Phatality: Move Request ID back to task description - https://phabricator.wikimedia.org/T237683 (10mmodell) [17:22:14] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Phatality, 10Developer Productivity: Phatality: Move Request ID back to task description - https://phabricator.wikimedia.org/T237683 (10mmodell) p:05Triage→03High [17:24:39] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: Create WMF CI image for Node.js 14 - https://phabricator.wikimedia.org/T267888 (10thcipriani) [17:24:53] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services): Upgrade CI jobs for WMF-deployed projects from Node 10 to Node 14 LTS (tracking) - https://phabricator.wikimedia.org/T267890 (10thcipriani) [17:25:03] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: Create WMF CI image for Node.js 14 - https://phabricator.wikimedia.org/T267888 (10thcipriani) 05Open→03Stalled [17:26:29] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T271342 (10brennen) [17:27:17] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [17:27:30] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [17:30:15] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2321.codfw.wmnet'] ` an... [17:30:32] 10Release-Engineering-Team, 10Wikimedia-Site-requests, 10User-Nikerabbit: Need an easy way to get list of wikis using Translate - https://phabricator.wikimedia.org/T266097 (10thcipriani) > I don't know where the project is at, I'm afraid; I think that's now something @jeena is working on. I think this refer... [17:30:56] 10Release-Engineering-Team-TODO, 10Wikimedia-Site-requests, 10User-Nikerabbit: Need an easy way to get list of wikis using Translate - https://phabricator.wikimedia.org/T266097 (10thcipriani) [17:31:21] 10Phabricator, 10Release-Engineering-Team, 10User-Addshore: "Production Error" tasks do not have a Story Points field - https://phabricator.wikimedia.org/T240343 (10mmodell) 05Open→03Resolved a:03mmodell I've disabled form 68 as I think it is a useless duplicate. [17:32:06] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10serviceops, 10Patch-For-Review: Replace production deployment servers and update them to Buster - https://phabricator.wikimedia.org/T265963 (10thcipriani) [17:32:35] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10serviceops, 10Patch-For-Review: Replace production deployment servers and update them to Buster - https://phabricator.wikimedia.org/T265963 (10thcipriani) [17:33:08] 10Beta-Cluster-Infrastructure, 10Quality-and-Test-Engineering-Team (QTE), 10UploadWizard: 404 error for SVG image thumbnail in UploadWizard on Beta Cluster - https://phabricator.wikimedia.org/T265580 (10thcipriani) [17:37:09] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10DBA, and 2 others: Create integration test env for wmfmariadbpy - https://phabricator.wikimedia.org/T265266 (10thcipriani) [17:39:05] 10Release-Engineering-Team-TODO, 10MediaWiki-Release-Tools, 10MW-1.35-release: mediawiki/extensions and mediawiki/skins missing a REL1_35 branch - https://phabricator.wikimedia.org/T264365 (10thcipriani) [17:39:24] 10Release-Engineering-Team (Deployment services), 10Scap, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Localisation cache must be purged after or during train deploy, not (just) before - https://phabricator.wikimedia.org/T263872 (10mmodell) [17:40:24] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10User-brennen: Explore tools for remote pairing - https://phabricator.wikimedia.org/T240484 (10brennen) [17:41:19] 10Release-Engineering-Team (Deployment services), 10Scap, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Localisation cache must be purged after or during train deploy, not (just) before - https://phabricator.wikimedia.org/T263872 (10mmodell) a:03mmodell [17:41:31] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10User-brennen: Explore tools for remote pairing - https://phabricator.wikimedia.org/T240484 (10brennen) [17:43:25] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Scap, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Localisation cache must be purged after or during train deploy, not (just)... - https://phabricator.wikimedia.org/T263872 [17:49:51] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1337.eqiad.wmnet'] ` an... [17:54:41] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO: Remove "php-" from wiki version numbers - https://phabricator.wikimedia.org/T63733 (10hashar) 05Open→03Stalled [18:00:08] (03CR) 10C. Scott Ananian: [C: 04-2] "This is waiting on the next quibble release, which will include the dependent patch." [integration/config] - 10https://gerrit.wikimedia.org/r/655742 (https://phabricator.wikimedia.org/T271863) (owner: 10C. Scott Ananian) [18:01:08] Hey all. I have a few questions relating to logstash / client side errors / deploys. [18:01:12] 1) How does one set up alerts on logstash? we've had a few issues where editors have introduced large error spikes https://phabricator.wikimedia.org/T264665 - when there is a sudden change in frequency i'd like alerts sent on email / irc / potentially phab. Are we able to do this? [18:01:13] 2) From now on, any newly introduced bug that throws over 1000 errors in 12 hr period on https://logstash.wikimedia.org/app/dashboards#/view/AXDBY8Qhh3Uj6x1zCF56 should be UBN and block the train - what would be the best way to fit this into our deploy process? [18:01:16] 3) During the backport window, we should be checking https://logstash.wikimedia.org/app/dashboards#/view/AXDBY8Qhh3Uj6x1zCF56 as part of our backport process and rolling back if a new error is introduced. How best to do that? [18:01:18] 4) How should client error dashboard sit with the mediawiki-new-errors one for PHP errors ? Should they stay separate or be merged? [18:01:23] Can also open a phab ticket for this if you let me know what tags would be most appropriate :) [18:01:24] thanks in advance! [18:02:15] Jdlrobson: reading qs, will respond in order [18:03:07] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Scap: Applying security patches should be robust and also give some useful output - https://phabricator.wikimedia.org/T269153 (10LarsWirzenius) "apply-patches" should be atomic: either all pa... [18:04:38] Jdlrobson: #1 is probably a question for... #wikimedia-observability and/or sre, i think. it seems like if you want that kind of alerting, maybe it should follow the same pattern used for icinga alerts on server-side mw errors, but i'm not best qualified to answer that one. [18:06:10] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1409.eqiad.wmnet'] ` an... [18:06:22] 👍 [18:06:32] Jdlrobson: #2 is a substantial process change to the train, i'd like to hear what thcipriani thinks about that, but i think some combination of a) working it into our usual log triage process and b) releng could use some help on that one. i'd suggest filing a task for capturing discussion. [18:07:14] probably #2 and #3 roll up into the same discussion; deployer docs for backports will need updated to reflect. [18:07:17] I've had similar conversations with thcipriani and jr on this one. What tags should I file the ticket under? [18:07:43] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1408.eqiad.wmnet'] ` an... [18:08:47] Jdlrobson: Release-Engineering-Team for a start, we can triage from there i think. [18:10:10] Jdlrobson: re: #4, my immediate gut feeling is that client errors and mw-new-errors should remain separate buckets for purposes of diagnosing things, but maybe we can/should get a view that combines them. [18:11:20] i don't think i have a good feel for how that would play out in practice - seems like a view on both would be useful for getting a feel about what's going on, but errors of either class might get lost in the noise. [18:12:09] (i.e., some client error blows up and a deployer misses a php exception that should be a blocker but is happening at a much lower volume.) [18:13:49] ottomata: Regarding `A dependency error was encountered while installing the extension "EventLogging": Could not find the registration file for the extension "EventBus" 9:05 AM`: Unfortunately I don't know what is required to get past this. I bet hashar knows. I think he's offline right now. [18:20:29] i can dig a bit. [18:20:30] ok thanks [18:26:34] !log updating git plugins on releases-jenkins.wikimedia.org [18:26:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:26:56] !log updating pipeline plugins on releases-jenkins.wikimedia.org [18:26:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:27:47] !log restarting jenkins on releases-jenkins.wikimedia.org following plugin updates [18:27:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:35:51] Jdlrobson: re: #2, that's a good heuristic and I think that's helpful since it doesn't require knowing too much about the error, we'd need to update documentation in a few places but that should be doable. Please file a ticket for Release-Engineering-Team. I also think #3 is effectively a documentation update. Your last question: I don't think I have a fully formed opinion about. We will [18:35:53] need it seperate for the 1000 errors in 12 hour period thing. Together feels like a more holistic view of production health. [18:42:47] Jdlrobson: RE: 1) We recommend setting up a query to export metrics about what you want to alert on in prometheus-es-exporter and set up an alert with monitoring::check_prometheus configured to alert how you like it. This will give you IRC alerts at least and open the door for other notification options. [18:43:04] (03CR) 10Ahmon Dancy: [C: 04-1] apt.go: Add ability to target releases (035 comments) [blubber] - 10https://gerrit.wikimedia.org/r/658432 (https://phabricator.wikimedia.org/T272759) (owner: 10Jeena Huneidi) [18:57:17] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by legoktm on cumin1001.eq... [19:18:01] 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10MediaWiki-Internationalization, 10MW-1.36-notes (1.36.0-wmf.20; 2020-12-01): Add flag to rebuildLocalisationCache.php to skip MessageBlobStore::clearGlobalCacheEntry - https://phabricator.wikimedia.org/T268698 (10dancy) 05Open→03Resolved [19:18:03] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)): l10n process for MW containerization - https://phabricator.wikimedia.org/T260827 (10dancy) [19:38:25] brennen: any luck? [19:39:28] ottomata: not a ton - was just reading T232413, which seems kinda relevant [19:39:28] T232413: mwext-php72-phan-docker job fails on WikimediaMessages: A dependency error was encountered while installing the extension "MinervaNeue": Could not find the registration file for the extension "MobileFrontend" - https://phabricator.wikimedia.org/T232413 [19:41:03] or maybe T263101 [19:41:03] T263101: Make mediawiki core installer stop trying to find extensions early and check if they are loaded or not - https://phabricator.wikimedia.org/T263101 [19:41:35] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2317.codfw.wmnet'] ` an... [19:42:29] huh [19:42:42] i thought maybe we had just not confgured something properly [19:43:27] that's entirely plausible. :) [19:43:39] this is not an area i know very much about. [19:45:25] ok, added a comment to https://phabricator.wikimedia.org/T272863 and pinged has har [19:45:48] ack. sorry i'm not more use. [20:05:49] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:09:17] thanks for looking! [20:09:27] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:15:58] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:16:12] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:22:30] (03CR) 10Jeena Huneidi: apt.go: Add ability to target releases (034 comments) [blubber] - 10https://gerrit.wikimedia.org/r/658432 (https://phabricator.wikimedia.org/T272759) (owner: 10Jeena Huneidi) [20:31:55] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:45:44] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1411.eqiad.wmnet'] ` an... [20:49:20] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1413.eqiad.wmnet'] ` an... [20:52:18] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:56:20] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:58:06] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10Patch-For-Review, 10Sustainability (Incident Followup): Add option in Scap to restart php-fpm for emergency deployments, and skip depoolin... - https://phabricator.wikimedia.org/T243009 [20:58:08] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10Patch-For-Review: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10Krinkle) [20:58:10] 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10Sustainability (Incident Followup), 10User-brennen: Add a quick rollback command to scap that skips all unnecessary steps - https://phabricator.wikimedia.org/T244544 (10Krinkle) [21:00:04] Project mwcore-phpunit-coverage-master build #1179: 04STILL FAILING in 6 hr 0 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1179/ [21:02:15] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10Patch-For-Review: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10Krinkle) [21:02:17] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Patch-For-Review, 10Sustainability (Incident Followup), 10User-brennen: Scap feature: restart php-fpm on deployment - https://phabricator.wikimedia.org/T264362 (10Krinkle) [21:03:27] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T271342 (10dancy) group0 rolled back due to T273006 [21:03:56] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Patch-For-Review, 10Sustainability (Incident Followup), 10User-brennen: Scap feature: restart php-fpm on deployment - https://phabricator.wikimedia.org/T264362 (10Krinkle) 05Open→03Reso... [21:03:58] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10Patch-For-Review: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10Krinkle) [21:06:13] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10Patch-For-Review: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10Krinkle) [21:06:24] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10Patch-For-Review: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10Krinkle) [21:08:18] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10Patch-For-Review: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10Krinkle) The feature flag has landed meanwhile. Highlights f... [21:14:30] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2308.codfw.wmnet'] ` an... [21:20:53] 10Phabricator, 10Release-Engineering-Team, 10User-Addshore: "Production Error" tasks do not have a Story Points field - https://phabricator.wikimedia.org/T240343 (10Dzahn) Now there is no more way to set the priority of a ticket with the Production Error tag. [21:23:00] 10Phabricator, 10Release-Engineering-Team, 10User-Addshore: "Production Error" tasks do not have a Story Points field - https://phabricator.wikimedia.org/T240343 (10RhinosF1) @Legoktm had to make https://phabricator.wikimedia.org/transactions/editengine/maniphest.task/view/46/#2010 as it broke priority on al... [21:28:46] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1338.eqiad.wmnet'] ` an... [21:29:50] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1337.eqiad.wmnet'] ` an... [21:29:56] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:31:53] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T271342 (10dancy) Seeing several hits for T265720 error messages too. [21:34:33] dancy: raised blocker to UBN, I know the form was broke for a bit but priority change is working again now [21:34:59] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2306.codfw.wmnet'] ` an... [21:36:38] Thanks RhinosF1|NotHere [21:37:25] (03CR) 10Jeena Huneidi: [C: 03+2] Refactor other builder types to use RequirementsConfig [blubber] - 10https://gerrit.wikimedia.org/r/650266 (owner: 10Dduvall) [21:37:36] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by legoktm on cumin1001.eq... [21:38:59] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2304.codfw.wmnet'] ` an... [21:40:14] (03Merged) 10jenkins-bot: Refactor other builder types to use RequirementsConfig [blubber] - 10https://gerrit.wikimedia.org/r/650266 (owner: 10Dduvall) [21:44:26] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T271342 (10dancy) [21:51:36] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10Patch-For-Review: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10thcipriani) >>! In T266055#6775216, @Krinkle wrote: >>>! In... [21:56:44] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:57:24] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:01:17] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:02:48] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:07:33] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T271342 (10dancy) [22:12:29] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2302.codfw.wmnet'] ` an... [22:20:19] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2300.codfw.wmnet'] ` an... [22:40:20] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2298.codfw.wmnet'] ` an... [22:40:57] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2299.codfw.wmnet'] ` an... [22:45:52] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2297.codfw.wmnet'] ` an... [22:55:17] (03PS1) 10QChris: Add .gitreview [blubber-doc/example/calculator-service] - 10https://gerrit.wikimedia.org/r/658689 [22:55:19] (03CR) 10QChris: [V: 03+2 C: 03+2] Add .gitreview [blubber-doc/example/calculator-service] - 10https://gerrit.wikimedia.org/r/658689 (owner: 10QChris) [23:00:42] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10MediaWiki-Docker, 10dev-images, and 2 others: Create PHP 7.3 and PHP 7.4 variants of docker-registry.wikimedia.org/dev/stretch-php72-fpm-apache2-xdebug - https://phabricator.wikimedia.org/T245444 (10br... [23:17:20] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1264.eqiad.wmnet'] ` an... [23:22:13] Project mediawiki-core-doxygen-docker build #22179: 04FAILURE in 17 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/22179/ [23:48:07] (03PS1) 10Ahmon Dancy: Refactor _call_rebuildLocalisationCache [tools/scap] - 10https://gerrit.wikimedia.org/r/658700 [23:52:36] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T271342 (10dancy)