[00:22:55] 10Continuous-Integration-Config, 10MediaWiki-extensions-StringFunctionsEscaped, 10Patch-For-Review: Extension:StringFunctionsEscaped should depend on ParserFunctions in CI - https://phabricator.wikimedia.org/T275173 (10Reedy) [00:31:07] Project mediawiki-core-doxygen-docker build #22748: 04FAILURE in 18 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/22748/ [01:03:34] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO, 10Wikimedia-Site-requests, 10Patch-For-Review, and 3 others: [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10Jdlrobson) [01:04:44] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO, 10Wikimedia-Site-requests, 10Patch-For-Review, and 3 others: [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10Jdlrobson) 05Open→03Resolved a:03Jdlrobson This should all be fixed. Pl... [01:18:28] Yippee, build fixed! [01:18:29] Project mediawiki-core-doxygen-docker build #22749: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/22749/ [03:19:59] 10MediaWiki-Releasing, 10MediaWiki-extensions-FlaggedRevs: FlaggedRevs affected by new mediawiki release (prolly MediaWiki 1.36/wmf.29-31) - https://phabricator.wikimedia.org/T275181 (10Acagastya) [04:26:03] (03CR) 10Jforrester: [C: 04-2] "> Patch Set 1:" [tools/release] - 10https://gerrit.wikimedia.org/r/663892 (https://phabricator.wikimedia.org/T237809) (owner: 10Prod) [05:42:00] James_F: I see you doig the backport of the fix for the train blocker, is there a plan to roll it back out? [05:42:04] today I mean [05:44:51] apergos: Not by me; that's a RelEng call, and on a Friday it's pretty rare (though it'd be really nice to get the train fully done in its week for the first time in over a month). [05:45:03] * James_F is just gnoming. [05:45:41] James_F: there was some discussion about "don't leave the train blocked through the weekend" that I saw recently but I didn't get any clear outcome of that, is why I ask. thanks [05:45:55] and thaks forthe gnoming [05:46:03] Yeah, officially we're meant to either roll wmf.31 or rollback to wmf.30. [05:46:07] But we'll see. [05:46:10] k [05:46:20] Also, it's of course still Thursday for me. ;-) [05:46:24] same :-) [05:46:27] well not really [05:46:43] it's 7 hours of sleep and the us hasn't gone to beed so it's thurs-fri for me :-D [05:47:10] when sf goes to bed then I consider it next day for real! [06:43:32] (03PS6) 10Dduvall: Export container process output from run step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/663905 [07:04:49] (03PS7) 10Dduvall: Export container process output from run step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/663905 [07:05:13] (03PS5) 10Dduvall: Support list comprehensions as configuration values [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664957 [07:09:47] (03CR) 10Dduvall: Allow per-stage blubberfile as path or inline config (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664992 (owner: 10Dduvall) [08:14:39] (03PS8) 10Dduvall: Export container process output from run step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/663905 [08:19:26] 10Continuous-Integration-Config, 10MediaWiki-extensions-StringFunctionsEscaped, 10Patch-For-Review: Extension:StringFunctionsEscaped should depend on ParserFunctions in CI - https://phabricator.wikimedia.org/T275173 (10Peachey88) > The (old, unmaintained) extension Might be easier to archive the extension then [08:22:29] (03PS6) 10Dduvall: Support list comprehensions as configuration values [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664957 [08:22:31] (03PS8) 10Dduvall: Allow per-stage blubberfile as path or inline config [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664992 [08:24:25] (03PS9) 10Dduvall: Export container process output from run step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/663905 [08:24:27] (03PS7) 10Dduvall: Support list comprehensions as configuration values [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664957 [08:24:29] (03PS9) 10Dduvall: Allow per-stage blubberfile as path or inline config [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664992 [08:28:47] (03PS8) 10Dduvall: Support list comprehensions as configuration values [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664957 [08:28:49] (03PS10) 10Dduvall: Allow per-stage blubberfile as path or inline config [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664992 [08:29:22] (03CR) 10Dduvall: Support list comprehensions as configuration values (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664957 (owner: 10Dduvall) [08:34:06] (03PS10) 10Gehel: Remove dependency on Maven binaries and wrapper script. [integration/config] - 10https://gerrit.wikimedia.org/r/651791 (https://phabricator.wikimedia.org/T271541) [08:38:25] hashar: gearman depends on a custom script, which is getting in the way of my refactoring. Ping me if you have a few minutes to discuss how to manage that. [08:38:36] (might be a bit more than "a few") [08:39:49] gehel: german? [08:39:54] yeah [08:39:57] had my coffee [08:42:41] 10Release-Engineering-Team: deploy-1002.devtools.eqiad.wmflabs fails puppet - https://phabricator.wikimedia.org/T275186 (10hashar) [08:43:17] 10Beta-Cluster-Infrastructure: deployment-memc08.deployment-prep.eqiad.wmflabs fails puppet - https://phabricator.wikimedia.org/T275187 (10hashar) [08:50:20] 10Release-Engineering-Team: deploy-1002.devtools.eqiad.wmflabs fails puppet - https://phabricator.wikimedia.org/T275186 (10hashar) [08:52:43] (03CR) 10Jbond: [C: 04-1] "please hold off on this it may not be required?" [integration/config] - 10https://gerrit.wikimedia.org/r/665133 (owner: 10Jbond) [08:52:47] 10Beta-Cluster-Infrastructure: deployment-memc08.deployment-prep.eqiad.wmflabs fails puppet - https://phabricator.wikimedia.org/T275187 (10hashar) [09:00:05] Project mwcore-phpunit-coverage-master build #1226: 04STILL FAILING in 6 hr 0 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1226/ [09:02:56] 10Gerrit, 10MediaWiki-extensions-Examples: Rename repo name on Gerrit from "examples" to "Examples" - https://phabricator.wikimedia.org/T261199 (10Aklapper) p:05Triage→03Lowest [09:03:03] 10Gerrit, 10MediaWiki-extensions-Examples: Rename repo name on Gerrit from "examples" to "Examples" - https://phabricator.wikimedia.org/T261199 (10Aklapper) 05Open→03Stalled [09:07:53] 10Phabricator, 10Technical-Debt: Kill Phab's sprint.phragile-uri config setting - https://phabricator.wikimedia.org/T275188 (10Aklapper) p:05Triage→03Low [09:08:10] 10Phabricator, 10Technical-Debt: Kill Phab's sprint.phragile-uri config setting - https://phabricator.wikimedia.org/T275188 (10Aklapper) [09:08:48] 10Phabricator, 10Technical-Debt: Kill Phab's sprint.phragile-uri config setting - https://phabricator.wikimedia.org/T275188 (10Aklapper) [09:46:31] (03PS11) 10Gehel: Remove dependency on Maven binaries and wrapper script. [integration/config] - 10https://gerrit.wikimedia.org/r/651791 (https://phabricator.wikimedia.org/T271541) [11:51:41] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Patch-For-Review, 10User-zeljkofilipin, and 2 others: Run browser tests as part of "npm test" for wikidata/query/gui in CI - https://phabricator.wikimedia.org/T222200 (10Lucas_Werkmeister_WMDE) Wait, nevermind, the job already runs on `node10-test-brow... [11:58:49] 10phan, 10phan-taint-check-plugin: Audit WM maintained libraries for lack of phan - https://phabricator.wikimedia.org/T274475 (10Aklapper) I guess the fix was https://gerrit.wikimedia.org/r/c/CLDRPluralRuleParser/+/663381 (if anyone else wants to contribute and understand what's there to do). Do we have any f... [12:16:15] (03PS1) 10Lars Wirzenius: test: use Debian packages for dependencies, not tox/pip [tools/scap] - 10https://gerrit.wikimedia.org/r/665316 [12:27:40] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10Release Pipeline, 10serviceops, 10Patch-For-Review: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10akosiaris) >>! In T273521#6841748, @Legoktm wrote: >>>! In T273521#6841527, @dancy wro... [12:37:05] (03CR) 10Hashar: [C: 03+1] "I guess we can look at progressively updating the jobs and check they run fine." [integration/config] - 10https://gerrit.wikimedia.org/r/651791 (https://phabricator.wikimedia.org/T271541) (owner: 10Gehel) [12:37:33] (03PS2) 10Lars Wirzenius: test: use Debian packages for dependencies, not tox/pip [tools/scap] - 10https://gerrit.wikimedia.org/r/665316 [12:45:24] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "Fine for me." (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/665198 (owner: 10Umherirrender) [12:46:02] (03PS3) 10Lars Wirzenius: test: use Debian packages for dependencies, not tox/pip [tools/scap] - 10https://gerrit.wikimedia.org/r/665316 [12:52:24] (03CR) 10Lars Wirzenius: "This implements the anti-zombie tactics I outlined in my email to releng yesterday." [tools/scap] - 10https://gerrit.wikimedia.org/r/665316 (owner: 10Lars Wirzenius) [13:16:29] (03CR) 10Ammarpad: [C: 04-1] "I do think this decreases readability of this code. Unless there are other gains (not explained in the commit summary) there's no reason f" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/665198 (owner: 10Umherirrender) [13:26:21] (03PS12) 10Gehel: Remove dependency on Maven binaries and wrapper script. [integration/config] - 10https://gerrit.wikimedia.org/r/651791 (https://phabricator.wikimedia.org/T271541) [13:28:06] (03PS13) 10Gehel: Remove dependency on Maven binaries and wrapper script. [integration/config] - 10https://gerrit.wikimedia.org/r/651791 (https://phabricator.wikimedia.org/T271541) [13:28:08] hashar: I think I'm good with https://gerrit.wikimedia.org/r/c/integration/config/+/651791 [13:44:48] 10Gerrit, 10observability: Enhance Gerrit logs ingested by Logstash - https://phabricator.wikimedia.org/T274661 (10hashar) That is being done via https://gerrit.wikimedia.org/r/c/operations/puppet/+/663876 I guess I can mark this task as a dupe of the ECS migration task so? [13:45:23] gehel: should we update all those jobs? [13:45:48] hashar: what do you mean? something besides merging that patch? [13:45:58] yeah [13:45:58] * gehel does not really know how things work behind the scene [13:46:24] we run a script (jjb) from our local machine to update the jenkins jobs [13:46:33] so there is a tiny manual step involved [13:46:47] I think the gearman job will end up slightly broken since it invokes ci-src-setup.sh [13:46:47] so yeah, I think we should update all those jobs! [13:46:56] but it is probably not a big deal, it can be fixed later [13:46:58] if at all [13:47:01] ok [13:47:02] We can start with a single one and check if it does not break horribly [13:47:20] as discussed, I haven't focused on gearman at all [13:47:29] +1 [13:48:36] open CRs that we could use for a check: [13:48:37] https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/665082 [13:48:39] (03CR) 10Hashar: [C: 03+2] "That might slightly break the gearman-java job cause ci-src-setup.sh will be invoked twice. But it is not a big deal and it can always be " [integration/config] - 10https://gerrit.wikimedia.org/r/651791 (https://phabricator.wikimedia.org/T271541) (owner: 10Gehel) [13:48:55] and https://gerrit.wikimedia.org/r/c/wikimedia/discovery/discovery-parent-pom/+/662914 [13:49:03] !log Updating Jenkins jobs for "Remove dependency on Maven binaries and wrapper script." | https://gerrit.wikimedia.org/r/c/integration/config/+/651791/ [13:49:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:49:15] if those 2 work, I would be reasonably confident that I haven't broken anything terribly [13:49:29] some how I am updating all tox jobs bah [13:50:33] (03Merged) 10jenkins-bot: Remove dependency on Maven binaries and wrapper script. [integration/config] - 10https://gerrit.wikimedia.org/r/651791 (https://phabricator.wikimedia.org/T271541) (owner: 10Gehel) [13:50:57] (03CR) 10Hashar: [C: 03+2] "That got deployed but I have forgot to +2 it :-\" [integration/config] - 10https://gerrit.wikimedia.org/r/664897 (https://phabricator.wikimedia.org/T275049) (owner: 10Hashar) [13:51:21] !log Reupdating tox jobs since https://gerrit.wikimedia.org/r/c/integration/config/+/664897 did not get merged [13:51:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:51:33] gehel: they can be tested now [13:52:19] "recheck" should do just that, right? [13:52:26] (03Merged) 10jenkins-bot: jjb: update tox job to use pip from Debian [integration/config] - 10https://gerrit.wikimedia.org/r/664897 (https://phabricator.wikimedia.org/T275049) (owner: 10Hashar) [13:54:40] gehel: yeah [13:54:45] gehel: though one of the job is merged [13:54:54] so if the job fails, I don't think it will be reported back to gerrit [13:55:12] yeah, just realized that, strange, it still shows up in my review queue [13:55:16] but one can send a dummy empty change [13:55:22] other one failed :/ [13:55:58] link? ;) [13:56:10] https://integration.wikimedia.org/ci/job/wikidata-query-rdf-maven-java8-docker/2334/console [13:56:32] 00:00:24.079 /src/.git: Permission denied [13:56:33] nice [13:57:10] should /src be mounted as a volume? [13:57:59] docker-ci-src-setup-simple [13:58:01] should do it [13:58:20] docker-src-dir [13:58:20] bah [13:59:10] gehel: je fix ;) [13:59:25] ok, j'ai pas compris le problème :( [13:59:56] (03PS1) 10Hashar: jjb: do create src directory in maven jobs [integration/config] - 10https://gerrit.wikimedia.org/r/665327 (https://phabricator.wikimedia.org/T271541) [14:00:08] gehel: ^ I am deploying that fix [14:00:15] too many macros [14:00:24] :( [14:01:02] updated [14:01:35] https://integration.wikimedia.org/ci/job/wikidata-query-rdf-maven-java8-docker/2335/console :] [14:02:07] 00:00:26.594 [FATAL tini (7)] exec /src/mvnw failed: No such file or directory [14:02:10] c'est pas gagné [14:03:37] bizarre [14:04:11] (03PS1) 10Hashar: jjb: do mount src directory in maven jobs [integration/config] - 10https://gerrit.wikimedia.org/r/665329 (https://phabricator.wikimedia.org/T271541) [14:04:15] pas l'impression que ce soit un typo de ma part [14:04:24] nop [14:04:27] j'ai juste pas fait gaffe [14:04:36] et il y a trop de macros jjb specialisées [14:04:41] Ah tiens, je me suis posé la question sur celui là! [14:04:57] il faut tout normaliser [14:05:10] pour que les jobs fassent tous à peu pres la meme chose [14:05:20] gehel: https://integration.wikimedia.org/ci/job/wikidata-query-rdf-maven-java8-docker/2336/console !!!!! [14:05:58] super cool! bravo! [14:06:09] and we have a task regarding usage of ci-src-setup.sh script [14:06:25] cause some images do invoke it (like maven before your change) or the tox images for python [14:06:33] but that makes it a bit harder to reuse them locally [14:06:40] and most images do not automatically invoke ci-src-setup [14:06:46] tldr: there is a task for it https://phabricator.wikimedia.org/T274812 [14:08:17] stupid CR to test another project: https://gerrit.wikimedia.org/r/c/wikimedia/discovery/discovery-parent-pom/+/665330 [14:09:00] success! [14:09:20] this might have broken some corner cases, but I'm reasonably confident that it works [14:09:32] thanks for the help ! [14:28:50] gehel: great ;) [14:28:54] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services): Make all CI images to use ci-src-setup-simple when running on CI - https://phabricator.wikimedia.org/T274812 (10hashar) **Context** The way I have originally envisioned the CI images was to have much of the logic embedded in... [14:28:55] gehel: kids time! be back later [14:29:04] have fun! [14:29:10] if something is broken poke some task and I will catch up later this evening [15:06:52] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Discovery-Search (Current work), 10Patch-For-Review: Remove dependency on Maven for CI for Java projects and align Jenkins job configuration. - https://phabricator.wikimedia.org/T271541 (... [15:09:30] (03PS3) 10Jbond: tox-ci: Improve abbility to run localy [integration/config] - 10https://gerrit.wikimedia.org/r/665133 (https://phabricator.wikimedia.org/T274338) [15:10:44] (03CR) 10jerkins-bot: [V: 04-1] tox-ci: Improve abbility to run localy [integration/config] - 10https://gerrit.wikimedia.org/r/665133 (https://phabricator.wikimedia.org/T274338) (owner: 10Jbond) [15:23:24] (03CR) 10Umherirrender: "> Patch Set 1: Code-Review-1" (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/665198 (owner: 10Umherirrender) [15:39:20] (03CR) 10Hashar: [C: 03+2] "Would be nice to have UploadWizard improved to match other use cases, then I understand the reasolning behind the fork (listed on the page" [integration/config] - 10https://gerrit.wikimedia.org/r/665141 (owner: 10Ostrzyciel) [15:41:00] (03Merged) 10jenkins-bot: Add MediaUploader extension [integration/config] - 10https://gerrit.wikimedia.org/r/665141 (owner: 10Ostrzyciel) [15:43:49] hashar: didn't you merge https://gerrit.wikimedia.org/r/c/integration/config/+/665329/ ? [15:44:09] my bad [15:44:15] and probably https://gerrit.wikimedia.org/r/c/integration/config/+/665327/ as well [15:44:16] (03CR) 10Hashar: [C: 03+2] jjb: do mount src directory in maven jobs [integration/config] - 10https://gerrit.wikimedia.org/r/665329 (https://phabricator.wikimedia.org/T271541) (owner: 10Hashar) [15:44:20] (03CR) 10Hashar: [C: 03+2] jjb: do create src directory in maven jobs [integration/config] - 10https://gerrit.wikimedia.org/r/665327 (https://phabricator.wikimedia.org/T271541) (owner: 10Hashar) [15:44:31] my fault for quick prototyping directly to prod :/ [15:44:36] :) [15:45:14] and thanks! [15:46:36] :] [15:48:03] (03Merged) 10jenkins-bot: jjb: do create src directory in maven jobs [integration/config] - 10https://gerrit.wikimedia.org/r/665327 (https://phabricator.wikimedia.org/T271541) (owner: 10Hashar) [15:48:14] (03Merged) 10jenkins-bot: jjb: do mount src directory in maven jobs [integration/config] - 10https://gerrit.wikimedia.org/r/665329 (https://phabricator.wikimedia.org/T271541) (owner: 10Hashar) [15:49:51] kostajh: are you around to review/merge https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Echo/+/665177/ ? [15:50:16] i'm thinking of re-rolling the train soon if we can get that merged into wmf.31 [15:53:12] it's already been +2ed into master, so shouldn't the backport just go? [15:54:25] marxarelli: it looks fine to me. I don’t think I can merge the back port patch, at least that’s not how I’ve done it in the past [15:55:16] no problem. i can merge it. i just wanted your eyes on it [15:56:02] I can be around to help test again if it goes any time in the next few hours [15:56:12] apergos: awesome. ty! [15:56:44] i'll head into -operations [15:56:55] and thanks for the fix kostajh [16:03:32] (03CR) 10Ahmon Dancy: test: use Debian packages for dependencies, not tox/pip (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/665316 (owner: 10Lars Wirzenius) [16:12:44] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10Release Pipeline, 10serviceops, 10Patch-For-Review: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10dancy) Thanks for the reference @akosiaris. That was a good read. [16:24:54] 10Release-Engineering-Team, 10MediaWiki-Core-Testing, 10Readers-Web-Backlog (Tracking), 10Technical-Debt: Upgrade Sinon.JS - https://phabricator.wikimedia.org/T239271 (10Lucas_Werkmeister_WMDE) [16:26:17] (03CR) 10Ahmon Dancy: [C: 03+1] Export container process output from run step (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/663905 (owner: 10Dduvall) [16:27:35] (03CR) 10Ahmon Dancy: [C: 03+1] Support list comprehensions as configuration values [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664957 (owner: 10Dduvall) [16:32:03] (03CR) 10Ahmon Dancy: [C: 03+1] Allow per-stage blubberfile as path or inline config (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664992 (owner: 10Dduvall) [16:56:40] (03CR) 10Dduvall: Export container process output from run step (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/663905 (owner: 10Dduvall) [17:03:48] (03CR) 10Ahmon Dancy: [C: 03+1] Export container process output from run step (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/663905 (owner: 10Dduvall) [17:06:25] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.w... [17:13:44] (03CR) 10Ammarpad: [C: 04-1] "> Patch Set 1:" (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/665198 (owner: 10Umherirrender) [17:14:30] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [17:32:48] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Patch-For-Review, 10User-zeljkofilipin, and 2 others: Run browser tests as part of "npm test" for wikidata/query/gui in CI - https://phabricator.wikimedia.org/T222200 (10Lucas_Werkmeister_WMDE) a:05Lucas_Werkmeister_WMDE→03None I’m at a loss. I tri... [18:01:01] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2272.codfw.wmnet'] ` an... [18:12:55] (03PS1) 10Dduvall: copies: Allow copying directly from other images [blubber] - 10https://gerrit.wikimedia.org/r/665371 [18:15:38] (03CR) 10Dduvall: copies: Allow copying directly from other images (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/665371 (owner: 10Dduvall) [18:16:30] any ideas why grafana won't let me sign in? Clicking sign in just refreshes the page [18:20:28] WFM [18:23:36] DannyS712: try going directly to https://grafana-rw.wikimedia.org/ [18:24:21] (that would be T269272, if my guess is correct) [18:24:21] T269272: Sign-in links from Grafana dashboards don't work when not signed into SSO - https://phabricator.wikimedia.org/T269272 [18:27:34] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [18:30:13] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1341.eqiad.wmnet'] ` an... [18:32:02] (03CR) 10Ahmon Dancy: [C: 03+1] copies: Allow copying directly from other images [blubber] - 10https://gerrit.wikimedia.org/r/665371 (owner: 10Dduvall) [18:32:07] @lucas thanks, it worked [18:34:51] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1367.eqiad.wmnet'] ` an... [18:46:07] Quick question about Blubber. Does anyone know of a way to set WORKDIR in the generated Dockerfile? [18:46:32] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [18:50:46] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [18:52:20] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [18:53:26] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [18:55:44] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:43:41] 10Release-Engineering-Team (Logspam), 10wikitech.wikimedia.org, 10Wikimedia-production-error: labswiki: Field 'notification_bundle_display_hash' doesn't have a default value (10.64.0.98) - https://phabricator.wikimedia.org/T262033 (10dduvall) 05Resolved→03Open Reopening and categorizing as logspam since... [19:50:09] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.32 deployment blockers - https://phabricator.wikimedia.org/T274936 (10dduvall) [19:50:11] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.31 deployment blockers - https://phabricator.wikimedia.org/T271345 (10dduvall) [19:59:59] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.32 deployment blockers - https://phabricator.wikimedia.org/T274936 (10dduvall) [20:12:03] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1261.eqiad.wmnet'] ` an... [20:12:26] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2257.codfw.wmnet'] ` an... [20:13:49] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1270.eqiad.wmnet'] ` an... [20:14:32] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1287.eqiad.wmnet'] ` an... [20:35:35] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review: Upgrade Jenkins Gearman plugin from a forked repo - https://phabricator.wikimedia.org/T271683 (10hashar) I have tagg... [20:55:19] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:56:53] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:00:05] Project mwcore-phpunit-coverage-master build #1227: 04STILL FAILING in 6 hr 0 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1227/ [21:01:48] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:04:21] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:18:22] 10Project-Admins: Create project tag for UseResource - https://phabricator.wikimedia.org/T275247 (10BrandonXLF) [21:21:14] 10Release-Engineering-Team, 10Page-Previews, 10Readers-Web-Backlog (Tracking), 10patch-welcome: Popups should not use github.com for documentation generation - https://phabricator.wikimedia.org/T243263 (10Jdlrobson) [21:22:25] 10Project-Admins: Create project tag for UseResource - https://phabricator.wikimedia.org/T275247 (10Peachey88) 05Open→03Resolved a:03Peachey88 Created, Go forth and encourage users to become members or watches etc. [21:28:51] 10Beta-Cluster-Infrastructure: deployment-memc08.deployment-prep.eqiad.wmflabs fails puppet - https://phabricator.wikimedia.org/T275187 (10hashar) That is related to T270315 , there is a patch ongoing. [21:28:56] 10Beta-Cluster-Infrastructure: deployment-memc08.deployment-prep.eqiad.wmflabs fails puppet - https://phabricator.wikimedia.org/T275187 (10hashar) [21:33:17] (03PS1) 10Cory Massaro: Adds CI for function-schemata project. [integration/config] - 10https://gerrit.wikimedia.org/r/665419 [21:35:53] (03CR) 10jerkins-bot: [V: 04-1] Adds CI for function-schemata project. [integration/config] - 10https://gerrit.wikimedia.org/r/665419 (owner: 10Cory Massaro) [21:40:52] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2336.codfw.wmnet'] ` an... [21:48:15] (03PS2) 10Cory Massaro: Adds CI for function-schemata project. [integration/config] - 10https://gerrit.wikimedia.org/r/665419 [21:49:45] (03CR) 10jerkins-bot: [V: 04-1] Adds CI for function-schemata project. [integration/config] - 10https://gerrit.wikimedia.org/r/665419 (owner: 10Cory Massaro) [21:56:00] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:01:34] (03CR) 10Jeena Huneidi: Adds CI for function-schemata project. (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/665419 (owner: 10Cory Massaro) [22:07:53] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1262.eqiad.wmnet'] ` an... [22:09:43] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1320.eqiad.wmnet'] ` an... [22:14:04] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:17:18] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1340.eqiad.wmnet'] ` an... [22:19:06] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:28:55] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.31 deployment blockers - https://phabricator.wikimedia.org/T271345 (10dduvall) [22:29:07] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.31 deployment blockers - https://phabricator.wikimedia.org/T271345 (10dduvall) 05Open→03Resolved [22:34:09] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [23:08:44] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [23:09:23] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1339.eqiad.wmnet'] ` an... [23:27:40] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1333.eqiad.wmnet'] ` an... [23:29:25] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [23:32:20] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1342.eqiad.wmnet'] ` an... [23:48:53] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1317.eqiad.wmnet'] ` an...