[00:05:45] 10phan-taint-check-plugin: Taint-check doesn't retrieve taintedness from alternate method implementations - https://phabricator.wikimedia.org/T273562 (10Daimona) [00:12:28] 10phan-taint-check-plugin: taint-check does not checks undeclared class properties - https://phabricator.wikimedia.org/T216254 (10Daimona) FTR, I think this should be resolved upstream, see [[https://phan.github.io/demo/?code=%3C%3Fphp%0A%0Aclass+Foo+%7B%0A%7D%0A%0Afunction+doStuff%28%29+%7B%0A++++%24a+%3D+new+F... [00:16:08] 10Release-Engineering-Team (Pipeline), 10SRE, 10Kubernetes: Helm install fails in CI namespace - https://phabricator.wikimedia.org/T273563 (10jeena) [00:27:35] 10Release-Engineering-Team (Pipeline), 10SRE, 10Kubernetes: Helm install fails in CI namespace: apparmor failed to apply profile - https://phabricator.wikimedia.org/T273563 (10jeena) [00:50:34] 10phan-taint-check-plugin, 10Patch-For-Review: Hardcode hashing functions as removing taintedness? - https://phabricator.wikimedia.org/T272492 (10Daimona) a:03Daimona [01:27:31] 10phan-taint-check-plugin, 10Patch-For-Review: Support additional characters after @param-taint and @return-taint annotations - https://phabricator.wikimedia.org/T272293 (10Daimona) a:03Daimona [01:28:47] (03PS1) 10Wolfgang Kandek: Readme type edit [blubber-doc/example/calculator-service] - 10https://gerrit.wikimedia.org/r/660970 [01:36:42] 10phan-taint-check-plugin: Add support for running on and analyzing PHP 8 - https://phabricator.wikimedia.org/T269263 (10Daimona) Full list of node kinds not handled by the plugin but should probably be (not all of these are PHP8-specific): - AST_ARROW_FUNC - Probably similar to other function cases - AST_MA... [01:39:14] (03PS2) 10Wolfgang Kandek: Readme typo edit Add Entrypoint and change CI config [blubber-doc/example/calculator-service] - 10https://gerrit.wikimedia.org/r/660970 [01:39:53] (03CR) 10jerkins-bot: [V: 04-1] Readme typo edit Add Entrypoint and change CI config [blubber-doc/example/calculator-service] - 10https://gerrit.wikimedia.org/r/660970 (owner: 10Wolfgang Kandek) [01:42:36] (03PS3) 10Wolfgang Kandek: Readme typo edit Add Entrypoint and change CI config add pkgs to build for test and requests [blubber-doc/example/calculator-service] - 10https://gerrit.wikimedia.org/r/660970 [01:46:09] 10phan-taint-check-plugin, 10Patch-For-Review: Annotate array_values as adding the NUMKEY bit - https://phabricator.wikimedia.org/T272888 (10Daimona) a:03Daimona [01:54:33] https://logstash-beta.wmflabs.org/ is not working ("Elasticsearch plugin is red") [02:08:00] it doesn't like /srv being 100% full, i guess more logs than expected? [02:20:21] "fixed" it. Seems curator isn't properly cleaning up old logs, i deleted may-sept 2020 and half of /srv is now available. Had to close/reopen .kibana, it probably lost the last write attempt [02:30:16] 10MediaWiki-Codesniffer: MediaWiki.Usage.DirUsageSniff false negative - https://phabricator.wikimedia.org/T273574 (10Anomie) [02:51:43] 10MediaWiki-Codesniffer: Improve MediaWiki\Sniffs\PHPUnit\PHPUnitTestTrait::isTestClass() - https://phabricator.wikimedia.org/T273575 (10Anomie) [02:56:07] 10MediaWiki-Codesniffer: Improve MediaWiki\Sniffs\PHPUnit\PHPUnitTestTrait::isTestClass() - https://phabricator.wikimedia.org/T273575 (10Reedy) [02:57:24] 10MediaWiki-Codesniffer, 10Regression: MediaWiki.Usage.DirUsageSniff false negative - https://phabricator.wikimedia.org/T273574 (10Reedy) [04:25:56] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Wikimedia-General-or-Unknown, 10Performance-Team (Radar): Document and automate updating of static/project-logos in mediawiki-config - https://phabricator.wikimedia.org/T98640 (10Legoktm) >>! In T98640#678... [05:17:42] (03Abandoned) 10Wolfgang Kandek: inext try http test [blubber-doc/example/calculator-service] - 10https://gerrit.wikimedia.org/r/660172 (owner: 10Wolfgang Kandek) [05:17:48] (03Abandoned) 10Wolfgang Kandek: sys access deleted [blubber-doc/example/calculator-service] - 10https://gerrit.wikimedia.org/r/660175 (owner: 10Wolfgang Kandek) [05:17:56] (03Abandoned) 10Wolfgang Kandek: http testing added [blubber-doc/example/calculator-service] - 10https://gerrit.wikimedia.org/r/660171 (owner: 10Wolfgang Kandek) [05:18:01] (03Abandoned) 10Wolfgang Kandek: test run added [blubber-doc/example/calculator-service] - 10https://gerrit.wikimedia.org/r/659463 (owner: 10Wolfgang Kandek) [05:18:06] (03Abandoned) 10Wolfgang Kandek: auto end deleted [blubber-doc/example/calculator-service] - 10https://gerrit.wikimedia.org/r/660174 (owner: 10Wolfgang Kandek) [05:18:44] (03Abandoned) 10Wolfgang Kandek: test added [blubber-doc/example/calculator-service] - 10https://gerrit.wikimedia.org/r/659459 (owner: 10Wolfgang Kandek) [05:18:54] (03Abandoned) 10Wolfgang Kandek: do not run test build [blubber-doc/example/calculator-service] - 10https://gerrit.wikimedia.org/r/659411 (owner: 10Wolfgang Kandek) [05:38:52] twentyafterfour: if you only need edit mw.o I can grant you local +ipblock-exempt which lets you bypass global blocks until that situation is resolved [06:36:09] Project beta-code-update-eqiad build #331676: 04FAILURE in 2 min 44 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/331676/ [06:44:26] Yippee, build fixed! [06:44:26] Project beta-code-update-eqiad build #331677: 09FIXED in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/331677/ [06:44:52] Majavah: that would be helpful and I appreciate the offer, but it's not too big of a deal, I can wait it out, I think the stewards are already on it! [06:47:41] twentyafterfour: which account are you using? MModell (WMF) has sysop group which already bypasses global blocks [06:59:26] Project beta-scap-eqiad build #336553: 04FAILURE in 5 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/336553/ [07:01:58] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10Joe) @Legoktm ran some tests, analogous to the o... [07:06:49] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10Legoktm) >>! In T273312#6795003, @Joe wrote: > *... [07:17:29] Yippee, build fixed! [07:17:30] Project beta-scap-eqiad build #336554: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/336554/ [07:34:09] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10MoritzMuehlenhoff) All architectual mitigations... [07:39:40] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10MoritzMuehlenhoff) In fact, we still have an API... [08:23:18] (03PS1) 10Hashar: Revert "Bring back jsonevent-layout library" [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/661048 (https://phabricator.wikimedia.org/T268020) [08:26:39] (03PS1) 10Hashar: Merge tag 'v3.2.7' into wmf/stable-3.2 [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/661050 [08:31:53] (03PS2) 10JMeybohm: Use the UID instead of username in USER instructions [blubber] - 10https://gerrit.wikimedia.org/r/660771 (https://phabricator.wikimedia.org/T228967) [08:33:23] (03CR) 10JMeybohm: "> Patch Set 1: Code-Review-1" (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/660771 (https://phabricator.wikimedia.org/T228967) (owner: 10JMeybohm) [08:34:20] (03CR) 10Hashar: [C: 03+2] Revert "Bring back jsonevent-layout library" [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/661048 (https://phabricator.wikimedia.org/T268020) (owner: 10Hashar) [08:34:50] (03CR) 10Hashar: [C: 03+2] Merge tag 'v3.2.7' into wmf/stable-3.2 [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/661050 (owner: 10Hashar) [08:40:05] (03Merged) 10jenkins-bot: Revert "Bring back jsonevent-layout library" [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/661048 (https://phabricator.wikimedia.org/T268020) (owner: 10Hashar) [08:40:08] (03Merged) 10jenkins-bot: Merge tag 'v3.2.7' into wmf/stable-3.2 [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/661050 (owner: 10Hashar) [08:42:10] 10Gerrit, 10Release-Engineering-Team (Development services): Use upstream Gerrit.war instead of building our own - https://phabricator.wikimedia.org/T268019 (10hashar) [08:42:34] 10Gerrit, 10Release-Engineering-Team (Development services), 10Patch-For-Review: Remove jsonevent-layout library from our Gerrit fork - https://phabricator.wikimedia.org/T268020 (10hashar) 05Open→03Resolved There are some more details in the commit message of https://gerrit.wikimedia.org/r/c/operations/p... [08:44:28] 10Gerrit, 10Release-Engineering-Team (Development services): Use upstream Gerrit.war instead of building our own - https://phabricator.wikimedia.org/T268019 (10hashar) With T268020 solved, we no more need to maintain our own fork and can just borrow upstream war file! What is left to do is to update https:/... [08:47:11] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Security, 10Upstream: Gerrit out of heap - https://phabricator.wikimedia.org/T263008 (10hashar) [08:47:41] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Security, 10Upstream: Gerrit out of heap - https://phabricator.wikimedia.org/T263008 (10hashar) 05Stalled→03Open Back to public. The fix has been released: https://www.gerri... [08:47:53] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Upgrade Gerrit to 3.2.7 - https://phabricator.wikimedia.org/T273223 (10hashar) [08:48:01] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Upgrade Gerrit to 3.2.7 - https://phabricator.wikimedia.org/T273223 (10hashar) [08:48:03] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Security, 10Upstream: Gerrit out of heap - https://phabricator.wikimedia.org/T263008 (10hashar) [08:48:12] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Upgrade Gerrit to 3.2.7 - https://phabricator.wikimedia.org/T273223 (10hashar) [08:48:15] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Security, 10Upstream: Gerrit out of heap - https://phabricator.wikimedia.org/T263008 (10hashar) [08:48:22] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Upgrade Gerrit to 3.2.7 - https://phabricator.wikimedia.org/T273223 (10hashar) [08:48:24] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Security, 10Upstream: Gerrit out of heap - https://phabricator.wikimedia.org/T263008 (10hashar) [09:00:04] Project mwcore-phpunit-coverage-master build #1192: 04STILL FAILING in 6 hr 0 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1192/ [09:03:03] 10Release-Engineering-Team (Pipeline), 10Prod-Kubernetes, 10serviceops, 10Kubernetes: Helm install fails in CI namespace: apparmor failed to apply profile - https://phabricator.wikimedia.org/T273563 (10JMeybohm) a:03JMeybohm Most likely my fault as I installed the apparmor package as part of T228967. Wil... [09:10:54] 10Release-Engineering-Team (Pipeline), 10Prod-Kubernetes, 10serviceops, 10Kubernetes: Helm install fails in CI namespace: apparmor failed to apply profile - https://phabricator.wikimedia.org/T273563 (10Joe) p:05Triage→03High [09:27:05] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10Legoktm) >>! In T273312#6795045, @MoritzMuehlenh... [09:40:33] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10Joe) This kind-of seals the deal. Upgrading the... [09:43:39] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10Ladsgroup) Maybe mw devs (including yours truly)... [09:51:02] (03PS1) 10Hashar: Gerrit v3.2.7 [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/661056 (https://phabricator.wikimedia.org/T273223) [09:53:00] (03CR) 10Hashar: [C: 03+2] "Uploaded to Archiva" [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/661056 (https://phabricator.wikimedia.org/T273223) (owner: 10Hashar) [09:53:17] (03CR) 10Hashar: [V: 03+2 C: 03+2] Gerrit v3.2.7 [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/661056 (https://phabricator.wikimedia.org/T273223) (owner: 10Hashar) [09:56:12] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10Joe) >>! In T273312#6795425, @Ladsgroup wrote: >... [10:07:30] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Security, 10Upstream: Gerrit out of heap - https://phabricator.wikimedia.org/T263008 (10hashar) [10:07:41] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review: Upgrade Gerrit to 3.2.7 - https://phabricator.wikimedia.org/T273223 (10hashar) 05Open→03Resolved Upgraded! [10:08:19] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Security, 10Upstream: Gerrit out of heap - https://phabricator.wikimedia.org/T263008 (10hashar) 05Open→03Resolved That was definitely epic and has been kindly addressed by u... [10:26:45] 10Gerrit: Gerrit serves unparsed HTML in error message - https://phabricator.wikimedia.org/T273595 (10Ammarpad) [10:50:35] hashar: hey, I don't know how to bug but I got "no space left on device" for puppet compiler twice in a row now: https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler-test/671/console [10:50:43] *who [10:52:10] Amir1: as usual file a task :] For that one against #sre / #continuous-integration-infrastructure and I guess #puppet-compiler ;) [10:52:31] the puppet compiler instances are maintained by SRE [10:52:46] but I guess I should have access to them as well ;) [10:53:05] Sure [10:54:45] Done [10:54:45] 10Continuous-Integration-Infrastructure, 10SRE, 10puppet-compiler: Puppet compiler running out of space - https://phabricator.wikimedia.org/T273599 (10Ladsgroup) [10:57:17] 10Continuous-Integration-Infrastructure, 10SRE, 10puppet-compiler: Puppet compiler running out of space on compiler1002.puppet-diffs.eqiad.wmflabs - https://phabricator.wikimedia.org/T273599 (10hashar) [10:59:09] !log Marking https://integration.wikimedia.org/ci/computer/compiler1002.puppet-diffs.eqiad.wmflabs/ as offline due to disk space issue # T273599 [10:59:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:59:13] T273599: Puppet compiler running out of space on compiler1002.puppet-diffs.eqiad.wmflabs - https://phabricator.wikimedia.org/T273599 [11:03:24] Amir1: I have disabled that host and will clean it up ;) [11:03:36] Thank you so much [11:03:41] 10Continuous-Integration-Infrastructure, 10SRE, 10puppet-compiler: Puppet compiler running out of space on compiler1002.puppet-diffs.eqiad.wmflabs - https://phabricator.wikimedia.org/T273599 (10hashar) The instance `compiler1002.puppet-diffs.eqiad.wmflabs` is full :/ ` name=df -h -t ext4 Filesystem... [11:13:47] Amir1: yeah well it is all uncompressed :/ [11:17:27] 10Continuous-Integration-Infrastructure, 10SRE, 10puppet-compiler: Puppet compiler running out of space on compiler1002.puppet-diffs.eqiad.wmflabs - https://phabricator.wikimedia.org/T273599 (10hashar) 05Open→03Resolved a:03hashar I have deleted bunch of old builds under `/srv/jenkins-workspace/puppet-... [11:18:37] Thanks! Maybe we should scale it up a bit. We have lots of new SREs these days [11:19:23] Amir1: it is more like each build has a couple full copies of puppet.git ;D [11:19:47] oh boy [11:19:57] shallow clone maybe then? [11:20:47] maybe [11:27:36] !log gerrit: fixed notifications queries having single quotes instead of double quotes for qchris, arturo and twentyafterfour [11:27:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:39:13] 10Release-Engineering-Team (Pipeline), 10Prod-Kubernetes, 10serviceops, 10Kubernetes, 10Patch-For-Review: Helm install fails in CI namespace: apparmor failed to apply profile - https://phabricator.wikimedia.org/T273563 (10JMeybohm) Fortunately, this issue had not effected prod clusters. So it must be som... [11:44:28] 10phan-taint-check-plugin, 10Patch-For-Review: Add support for PDO - https://phabricator.wikimedia.org/T269258 (10Daimona) a:03Daimona [11:48:36] 10Gerrit: org.apache.http.client.protocol.ResponseProcessCookies : Invalid cookie header for WMF-Last-Access - https://phabricator.wikimedia.org/T273605 (10hashar) [11:52:29] 10Gerrit: org.apache.http.client.protocol.ResponseProcessCookies : Invalid cookie header for WMF-Last-Access - https://phabricator.wikimedia.org/T273605 (10hashar) [11:56:17] 10Release-Engineering-Team (Pipeline), 10Prod-Kubernetes, 10serviceops, 10Kubernetes, 10Patch-For-Review: Helm install fails in CI namespace: apparmor failed to apply profile - https://phabricator.wikimedia.org/T273563 (10JMeybohm) 05Open→03Resolved Did a puppet run on the affected nodes and scaled b... [11:57:12] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10Daimona) >>! In T273312#6795006, @Legoktm wrote:... [12:29:03] 10Gerrit: org.apache.http.client.protocol.ResponseProcessCookies : Invalid cookie header for WMF-Last-Access - https://phabricator.wikimedia.org/T273605 (10hashar) https://issues.apache.org/jira/browse/HTTPCLIENT-1763 states the default is the Netscape cookie draft which is: Wdy, DD-Mon-YYYY HH:MM:SS GMT... [12:31:19] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T271342 (10hashar) Cool thank you for the rollback. I will follow up on T271343 :] [13:45:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Wikidata, and 3 others: Create docker based CI job to build the wikidata-query-gui - https://phabricator.wikimedia.org/T210286 (10Lucas_Werkmeister_WMDE) > I can go to Jenkins to... [13:48:14] PROBLEM - Check systemd state on doc2001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [14:11:24] 10Gerrit: Gerrit 3.2 UI does not allow to remove reviewers with long names - https://phabricator.wikimedia.org/T257105 (10Kizule) Can we reproduce this still? [14:15:37] 10MediaWiki-Codesniffer: Improve MediaWiki\Sniffs\PHPUnit\PHPUnitTestTrait::isTestClass() - https://phabricator.wikimedia.org/T273575 (10Anomie) Another option might be to make the trait stuff protected rather than private, so we can subclass the relevant tests and override them. [14:41:51] 10MediaWiki-Codesniffer: Incorrect fix for MediaWiki.PHPUnit.SpecificAssertions - https://phabricator.wikimedia.org/T273624 (10Anomie) [14:50:37] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.29 deployment blockers - https://phabricator.wikimedia.org/T271343 (10hashar) 1.36.0-wmf.28 got skipped. 1.36.0-wmf.29 is on testwikis and halted there for now given all the magi... [15:04:34] (03PS1) 10Lars Wirzenius: test: have tox run scripts/check, instead of the other way around [tools/scap] - 10https://gerrit.wikimedia.org/r/661129 [15:06:00] (03CR) 10jerkins-bot: [V: 04-1] test: have tox run scripts/check, instead of the other way around [tools/scap] - 10https://gerrit.wikimedia.org/r/661129 (owner: 10Lars Wirzenius) [15:06:33] (03CR) 10Lars Wirzenius: "This is a small change, but with a big impact: CI will the full Scap test suite via tox, in CI. This should avoid us failing to notice mis" [tools/scap] - 10https://gerrit.wikimedia.org/r/661129 (owner: 10Lars Wirzenius) [15:07:22] (03CR) 10Hashar: "This change is no more needed, we have since removed our custom log4j configuration that required jsonevent-layout and have switched to us" [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/617206 (owner: 10QChris) [15:10:21] (03PS2) 10Lars Wirzenius: test: have tox run scripts/check, instead of the other way around [tools/scap] - 10https://gerrit.wikimedia.org/r/661129 [15:13:14] (03PS3) 10Lars Wirzenius: test: have tox run scripts/check, instead of the other way around [tools/scap] - 10https://gerrit.wikimedia.org/r/661129 [15:16:25] (03CR) 10Lars Wirzenius: "A small change to get CI to run scripts/check, which runs Black and Subplot, if available." [tools/scap] - 10https://gerrit.wikimedia.org/r/661129 (owner: 10Lars Wirzenius) [15:31:19] 10MediaWiki-Codesniffer: Incorrect fix for MediaWiki.PHPUnit.SpecificAssertions - https://phabricator.wikimedia.org/T273624 (10DannyS712) p:05Triage→03High [15:35:54] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.29 deployment blockers - https://phabricator.wikimedia.org/T271343 (10Lucas_Werkmeister_WMDE) [16:08:43] 10Gerrit: Gerrit serves unparsed HTML in error message - https://phabricator.wikimedia.org/T273595 (10hashar) 05Open→03Resolved a:03hashar Pretty sure that was related to Gerrit being upgraded (T273223): ` lang=irc,name="SAL (UTC time)" 10:02 Restarted Gerrit primary on gerrit1001 # T273223 ` [16:24:19] 10MediaWiki-Codesniffer, 10MW-1.36-notes (1.36.0-wmf.29; 2021-02-02): How to handle test functions with MediaWiki.Commenting.FunctionComment.MissingParamTag sniff - https://phabricator.wikimedia.org/T238572 (10Umherirrender) 05Open→03Resolved [17:03:23] (03CR) 10Ahmon Dancy: [C: 04-1] "I cherry-picked this into my tree and ran 'tox'. One of the phases generated error messages like this (one of many):" [tools/scap] - 10https://gerrit.wikimedia.org/r/661129 (owner: 10Lars Wirzenius) [17:24:21] (03CR) 10Lars Wirzenius: "> I cherry-picked this into my tree and ran 'tox'. One of the phases generated error messages like this (one of many):" [tools/scap] - 10https://gerrit.wikimedia.org/r/661129 (owner: 10Lars Wirzenius) [17:47:10] (03CR) 10Lars Wirzenius: "Can I have a look at your changes, to see if I can reproduce them, and maybe debug?" [tools/scap] - 10https://gerrit.wikimedia.org/r/661129 (owner: 10Lars Wirzenius) [17:49:21] (03PS15) 10DannyS712: Create PHPUnitAssertEmpty sniff to warn against using assertEmpty [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/591242 (https://phabricator.wikimedia.org/T246674) [17:51:34] (03PS12) 10DannyS712: Detect problems with @codingStandardsIgnore(Start|End) tags [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/632011 (https://phabricator.wikimedia.org/T127241) [17:51:58] (03Abandoned) 10DannyS712: ExtendClassUsageSniff: $wgMemc was removed [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/626029 (owner: 10DannyS712) [18:01:23] (03CR) 10Ahmon Dancy: [C: 04-1] "Here's what I did to test:" [tools/scap] - 10https://gerrit.wikimedia.org/r/661129 (owner: 10Lars Wirzenius) [18:01:36] (03CR) 10Wolfgang Kandek: [C: 03+2] Readme typo edit Add Entrypoint and change CI config add pkgs to build for test and requests [blubber-doc/example/calculator-service] - 10https://gerrit.wikimedia.org/r/660970 (owner: 10Wolfgang Kandek) [18:01:40] (03Merged) 10jenkins-bot: Readme typo edit Add Entrypoint and change CI config add pkgs to build for test and requests [blubber-doc/example/calculator-service] - 10https://gerrit.wikimedia.org/r/660970 (owner: 10Wolfgang Kandek) [18:07:44] 10Gerrit: Gerrit 3.2 UI does not allow to remove reviewers with long names - https://phabricator.wikimedia.org/T257105 (10Reedy) On the same patch... {F34086000} If the user can't be removed because they're deactivated... That's kind of a different (but still valid) bug [18:32:27] (03PS1) 10Reedy: Start branching PageNotice [tools/release] - 10https://gerrit.wikimedia.org/r/661168 (https://phabricator.wikimedia.org/T61245) [18:33:42] (03CR) 10Jforrester: [C: 03+2] "Low risk per Security team's review. Let's get this branched." [tools/release] - 10https://gerrit.wikimedia.org/r/661168 (https://phabricator.wikimedia.org/T61245) (owner: 10Reedy) [18:36:20] (03Merged) 10jenkins-bot: Start branching PageNotice [tools/release] - 10https://gerrit.wikimedia.org/r/661168 (https://phabricator.wikimedia.org/T61245) (owner: 10Reedy) [19:04:29] (03PS1) 10Kosta Harlan: Install Helm v2 [releng/local-charts] - 10https://gerrit.wikimedia.org/r/661175 (https://phabricator.wikimedia.org/T273525) [19:12:27] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T271344 (10Jdforrester-WMF) Note: New extension (PageNotice) will be branched starting with this train; it won't be configured or anything, so... [19:16:35] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/PageNotice] Tag as in-wikimedia-production, move [integration/config] - 10https://gerrit.wikimedia.org/r/661177 (https://phabricator.wikimedia.org/T61245) [19:25:01] Project mediawiki-core-doxygen-docker build #22353: 04FAILURE in 20 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/22353/ [19:25:14] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Analytics, 10Better Use Of Data, and 5 others: EventLogging PHP EventServiceClient should use EventBus->send(). - https://phabricator.wikimedia.org/T272863 (10Ottomata) @hashar looks like that bug is fixed? What is need... [19:26:22] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/PageNotice] Tag as in-wikimedia-production, move [integration/config] - 10https://gerrit.wikimedia.org/r/661177 (https://phabricator.wikimedia.org/T61245) (owner: 10Jforrester) [19:28:03] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/PageNotice] Tag as in-wikimedia-production, move [integration/config] - 10https://gerrit.wikimedia.org/r/661177 (https://phabricator.wikimedia.org/T61245) (owner: 10Jforrester) [19:28:58] !log Zuul: [mediawiki/extensions/PageNotice] Tag as in-wikimedia-production, move T61245 [19:29:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:29:01] T61245: Review the PageNotice extension for deployment - https://phabricator.wikimedia.org/T61245 [19:39:02] !log Pruning dangling Docker images on contint1001 [19:39:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:43:46] !log Pruning dangling Docker images on contint2001 [19:43:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:01:18] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.29 deployment blockers - https://phabricator.wikimedia.org/T271343 (10Lucas_Werkmeister_WMDE) [20:02:44] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10Legoktm) >>! In T273312#6795794, @Daimona wrote:... [20:13:14] (03CR) 10Jeena Huneidi: [C: 03+2] "recheck" [blubber] - 10https://gerrit.wikimedia.org/r/658432 (https://phabricator.wikimedia.org/T272759) (owner: 10Jeena Huneidi) [20:17:05] (03Merged) 10jenkins-bot: apt.go: Add ability to target releases [blubber] - 10https://gerrit.wikimedia.org/r/658432 (https://phabricator.wikimedia.org/T272759) (owner: 10Jeena Huneidi) [20:20:32] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.29 deployment blockers - https://phabricator.wikimedia.org/T271343 (10hashar) [20:20:34] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T271342 (10hashar) [20:21:07] Yippee, build fixed! [20:21:08] Project mediawiki-core-doxygen-docker build #22354: 09FIXED in 16 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/22354/ [20:36:48] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:37:47] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:49:32] Yippee, build fixed! [20:49:33] Project mwcore-phpunit-coverage-master build #1193: 09FIXED in 5 hr 49 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1193/ [21:01:03] (03CR) 10Dduvall: [C: 03+2] Namespace build parameters to `setup.params.[name]` [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/658746 (owner: 10Dduvall) [21:02:34] (03Merged) 10jenkins-bot: Namespace build parameters to `setup.params.[name]` [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/658746 (owner: 10Dduvall) [21:09:42] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10MW-on-K8s: Security patch workflow for MediaWiki on k8s - https://phabricator.wikimedia.org/T271274 (10thcipriani) [21:09:44] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10Release Pipeline: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10thcipriani) [21:10:02] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10MW-on-K8s: Security patch workflow for MediaWiki on k8s - https://phabricator.wikimedia.org/T271274 (10thcipriani) a:03dduvall [21:13:04] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10Release Pipeline, 10serviceops: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10thcipriani) [21:17:30] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10MW-on-K8s: Jenkins job exists for building patched MediaWiki image - https://phabricator.wikimedia.org/T273676 (10thcipriani) [21:18:46] 10Phabricator (Upstream), 10Upstream: Add a close button to "Your browser timezone setting differs from the timezone setting in your profile, click to reconcile." popup - https://phabricator.wikimedia.org/T160866 (10Whatamidoing-WMF) Why is "ignore conflict" a temporary state? Imagine that you're checking P... [21:25:18] 10Continuous-Integration-Config, 10Parsoid: Set up CI for new PHP libraries: zest-css, dodo, webidl - https://phabricator.wikimedia.org/T273295 (10Jdforrester-WMF) [21:26:38] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10MW-on-K8s, 10Release Pipeline: Ensure m7i private settings are deployed as secrets to k8s - https://phabricator.wikimedia.org/T271475 (10thcipriani) [21:27:33] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10MW-on-K8s, 10Release Pipeline: Ensure m7i private settings are deployed as secrets to k8s - https://phabricator.wikimedia.org/T271475 (10thcipriani) Sounds like these can stay where they are and the wo... [21:28:14] (03PS1) 10C. Scott Ananian: Zuul: [mediawiki/libs/Dodo] Turn on jenkins CI for the `dodo` library [integration/config] - 10https://gerrit.wikimedia.org/r/661196 (https://phabricator.wikimedia.org/T257669) [21:28:17] (03PS1) 10C. Scott Ananian: Zuul: [mediawiki/libs/WebIDL] Turn on jenkins CI for the `webidl` library [integration/config] - 10https://gerrit.wikimedia.org/r/661197 [21:29:13] cscott: Want those merging/deploying if jerkins +2s? [21:30:16] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10Joe) >>! In T273312#6797450, @Legoktm wrote: > N... [21:31:19] (03CR) 10Jforrester: "Presumably this doesn't actually depend on the new release of quibble and injecting the Disambiguator extension into the Parsoid job? :-)" [integration/config] - 10https://gerrit.wikimedia.org/r/661196 (https://phabricator.wikimedia.org/T257669) (owner: 10C. Scott Ananian) [21:34:58] (03PS2) 10Jforrester: Zuul: [mediawiki/libs/Dodo] Enable CI for new library repo [integration/config] - 10https://gerrit.wikimedia.org/r/661196 (https://phabricator.wikimedia.org/T257669) (owner: 10C. Scott Ananian) [21:35:00] (03PS2) 10Jforrester: Zuul: [mediawiki/libs/WebIDL] Enable CI for new library repo [integration/config] - 10https://gerrit.wikimedia.org/r/661197 (owner: 10C. Scott Ananian) [21:35:59] 10LibUp: libup fails on update eslint-config-wikimedia from 0.17.0 -> 0.18.1 with no-nonoctal-decimal-escape - https://phabricator.wikimedia.org/T273680 (10Umherirrender) [21:37:55] 10LibUp: libup fails on update eslint-config-wikimedia from 0.17.0 -> 0.18.1 with no-nonoctal-decimal-escape - https://phabricator.wikimedia.org/T273680 (10Jdforrester-WMF) Which repos? It generally would mean that the eslint is more than 10 items deep. We could bump to 20 or whatever but the individual repos mi... [21:39:49] 10LibUp: libup fails on update eslint-config-wikimedia from 0.17.0 -> 0.18.1 with no-nonoctal-decimal-escape - https://phabricator.wikimedia.org/T273680 (10Umherirrender) Many, for example PageCreationNotif - https://libraryupgrader2.wmcloud.org/logs2/84041 [21:45:46] 10Release-Engineering-Team-TODO, 10MW-on-K8s: Puppet failing on releases hosts due to missing profile::ci::kubernetes_config::token - https://phabricator.wikimedia.org/T273681 (10dduvall) [21:47:20] 10Release-Engineering-Team-TODO, 10MW-on-K8s: Puppet failing on releases hosts due to missing profile::ci::kubernetes_config::token - https://phabricator.wikimedia.org/T273681 (10dduvall) p:05Triage→03High Marking high priority since Puppet is currently failing on two production hosts (`releases1002` and `... [21:49:25] 10LibUp: libup fails on update eslint-config-wikimedia from 0.17.0 -> 0.18.1 with no-nonoctal-decimal-escape - https://phabricator.wikimedia.org/T273680 (10Legoktm) I spent about 2 hours investigating this last week and am convinced that `npm update eslint -depth 10` is just broken. Despite eslint being at depth... [21:54:26] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1335.eqiad.wmnet'] ` an... [21:54:59] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1336.eqiad.wmnet'] ` an... [21:57:22] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10SRE, 10serviceops: Puppet failing on releases hosts due to missing profile::ci::kubernetes_config::token - https://phabricator.wikimedia.org/T273681 (10dduvall) [21:57:49] 10LibUp: libup fails on update eslint-config-wikimedia from 0.17.0 -> 0.18.1 with no-nonoctal-decimal-escape - https://phabricator.wikimedia.org/T273680 (10Umherirrender) After `npm install grunt-eslint` the dependency is "deduped" and the version mismatch is resolved. Than It looks like other extensions. ` # n... [21:58:54] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10SRE, 10serviceops: Puppet failing on releases hosts due to missing profile::ci::kubernetes_config::token - https://phabricator.wikimedia.org/T273681 (10Dzahn) a:03Dzahn [22:00:20] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10SRE, 10serviceops: Puppet failing on releases hosts due to missing profile::ci::kubernetes_config::token - https://phabricator.wikimedia.org/T273681 (10Dzahn) 21:56 <+icinga-wm> PROBLEM - Widespread puppet agent failures on alert1001 is CRITICAL: 0.01208 ge 0... [22:08:38] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10SRE, 10serviceops: Puppet failing on releases hosts due to missing profile::ci::kubernetes_config::token - https://phabricator.wikimedia.org/T273681 (10Dzahn) I fixed the first issue by adding the "kubernetes_config::token" to the releases role in the private... [22:09:02] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10SRE, 10serviceops: Puppet failing on releases hosts due to missing profile::ci::kubernetes_config::token - https://phabricator.wikimedia.org/T273681 (10Dzahn) ` Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Could not... [22:12:08] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10SRE, 10serviceops: Puppet failing on releases hosts due to missing profile::ci::kubernetes_config::token, dependency issue in kubeconfig.pp - https://phabricator.wikimedia.org/T273681 (10Dzahn) [22:45:32] 10LibUp: libup fails on update eslint-config-wikimedia from 0.17.0 -> 0.18.1 with no-nonoctal-decimal-escape - https://phabricator.wikimedia.org/T273680 (10Legoktm) a:03Legoktm >>! In T273680#6797882, @Umherirrender wrote: > After `npm install grunt-eslint` the dependency is "deduped" and the version mismatch... [22:59:01] 10LibUp: Move libup to a bigger server - https://phabricator.wikimedia.org/T273690 (10Legoktm) [23:09:52] 10LibUp, 10Patch-For-Review: libup fails on update eslint-config-wikimedia from 0.17.0 -> 0.18.1 with no-nonoctal-decimal-escape - https://phabricator.wikimedia.org/T273680 (10Legoktm) I triggered runs for all currently failing repositories [23:33:21] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10SRE, 10serviceops, 10Patch-For-Review: Puppet failing on releases hosts due to missing profile::ci::kubernetes_config::token, dependency issue in kubeconfig.pp - https://phabricator.wikimedia.org/T273681 (10Dzahn) ^ Mostly fixed the puppet runs on releases*... [23:35:03] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10SRE, 10serviceops, 10Patch-For-Review: Puppet failing on releases hosts due to missing profile::ci::kubernetes_config::token, dependency issue in kubeconfig.pp - https://phabricator.wikimedia.org/T273681 (10Dzahn) a:05Dzahn→03None At this point it is ge... [23:41:43] reedy [23:46:21] I was hoping I could convince someone more familiar with Jenkins to merge and deploy those, yes [23:50:12] cscott: I guess just James' question ousttanding