[00:05:29] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.37 deployment blockers - https://phabricator.wikimedia.org/T278343 (10DannyS712) [00:08:25] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.37 deployment blockers - https://phabricator.wikimedia.org/T278343 (10DannyS712) {T278904} is an issue with .36 that wasn't discovered until now, tagging this week's .37 task sinc... [02:24:56] (03PS1) 10Cory Massaro: Add .pipeline/config.yaml-style CI to mediawiki/services/function-schemata. [integration/config] - 10https://gerrit.wikimedia.org/r/675937 [06:23:19] (03CR) 10Hashar: [C: 03+2] zuul: setup Inbox extension [integration/config] - 10https://gerrit.wikimedia.org/r/675742 (owner: 10DannyS712) [06:24:34] (03Merged) 10jenkins-bot: zuul: setup Inbox extension [integration/config] - 10https://gerrit.wikimedia.org/r/675742 (owner: 10DannyS712) [06:31:25] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10GerritBot, 10Patch-For-Review, 10Technical-Debt: gerritbot doesnt escape sequences from commitmsg - https://phabricator.wikimedia.org/T93331 (10hashar) 05Open→03Resolved Solved by wrapping the subject and the abando... [07:06:24] (03CR) 10Hashar: [C: 03+2] "Thank you ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/675340 (owner: 10Zabe) [07:06:33] (03CR) 10Hashar: [C: 03+2] Add Func in the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/675311 (owner: 10Zabe) [07:06:42] (03CR) 10Hashar: [C: 03+2] Add Sandeep Kumar Patel in the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/673770 (owner: 10Zabe) [07:07:51] (03Merged) 10jenkins-bot: Add Rubin in the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/675340 (owner: 10Zabe) [07:07:53] (03Merged) 10jenkins-bot: Add Func in the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/675311 (owner: 10Zabe) [07:07:58] (03Merged) 10jenkins-bot: Add Sandeep Kumar Patel in the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/673770 (owner: 10Zabe) [07:13:59] (03CR) 10Hashar: [C: 03+2] "I have deployed the two Jenkins jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/675937 (owner: 10Cory Massaro) [07:15:12] (03Merged) 10jenkins-bot: Add .pipeline/config.yaml-style CI to mediawiki/services/function-schemata. [integration/config] - 10https://gerrit.wikimedia.org/r/675937 (owner: 10Cory Massaro) [07:19:54] (03PS2) 10Thiemo Kreuz (WMDE): UnsortedUseStatementsSniff can loop on live coding/incomplete code [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/675905 (owner: 10Umherirrender) [07:22:37] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "Uh, thanks a lot for digging into this. I replayed the issue locally and ended with pretty much the exact same code. The only change I mad" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/675905 (owner: 10Umherirrender) [07:55:43] 10Project-Admins: Archive #WikiProject-X in Phabricator? Wiki page says it's defunct. - https://phabricator.wikimedia.org/T274351 (10Aklapper) 05Open→03Resolved a:03Aklapper * Archived https://phabricator.wikimedia.org/project/profile/1370/ * Declined https://phabricator.wikimedia.org/maniphest/?ids=132927... [08:37:12] (03CR) 10Hashar: [C: 03+2] "Job updated:" [integration/config] - 10https://gerrit.wikimedia.org/r/673803 (owner: 10Gehel) [08:37:37] (03Abandoned) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/667873 (owner: 10Hashar) [08:38:50] (03Merged) 10jenkins-bot: Revert "search/python: add proper options for sonar bot commenting" [integration/config] - 10https://gerrit.wikimedia.org/r/673803 (owner: 10Gehel) [09:13:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Patch-For-Review: Reduce size of artifacts stored on the CI Jenkins master - https://phabricator.wikimedia.org/T249268 (10hashar) I have live hacke... [09:29:23] 10Continuous-Integration-Infrastructure, 10Sustainability (Incident Followup): Create a controlled and ongoing CI pipeline test job that we can alert on - https://phabricator.wikimedia.org/T158054 (10hashar) 05Open→03Declined This was notably to ensure Nodepool was working reliably. At the time Chase went... [09:37:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10SRE: debian-glue jobs ignored error messages about libeatmydata.so in LD_PRELOAD - https://phabricator.wikimedia.org/T240430 (10hashar) That can be addressed by having the `eatmyd... [09:52:08] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10SRE, and 2 others: debian-glue jobs ignored error messages about libeatmydata.so in LD_PRELOAD - https://phabricator.wikimedia.org/T240430 (10hashar) a:03hashar [10:54:29] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [11:15:35] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [15:09:48] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10SRE, 10Patch-For-Review, and 2 others: Shell/Python/other scripts should not be generated by ERB files; dynamic parts should be a simple ERB config file - https://phabricator.wikimedia.org/T254480 (10crusnov) [15:25:02] (03PS8) 10Jgleeson: WIP: Added image for civiproxy [releng/dev-images] - 10https://gerrit.wikimedia.org/r/664919 (https://phabricator.wikimedia.org/T268683) [15:47:52] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.37 deployment blockers - https://phabricator.wikimedia.org/T278343 (10Jdforrester-WMF) [15:58:55] 10Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)), 10Release, 10Train Deployments: 1.36.0-wmf.40 deployment blockers - https://phabricator.wikimedia.org/T278346 (10thcipriani) 05Open→03Declined a:05LarsWirzenius→03None [15:59:20] 10Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)), 10Release, 10Train Deployments: 1.36.0-wmf.40 deployment blockers - https://phabricator.wikimedia.org/T278346 (10thcipriani) Declined due to Earth day holiday falling on Thursday; see: https://wikitech.wikimedia.org/wiki/Deployments/Yearly_cale... [16:00:09] 10Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)), 10Release, 10Train Deployments: 1.36.0-wmf.41 deployment blockers - https://phabricator.wikimedia.org/T278347 (10thcipriani) a:05brennen→03LarsWirzenius [16:10:02] (03CR) 10Umherirrender: [C: 03+1] "Thanks for your changes" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/675905 (owner: 10Umherirrender) [16:22:01] (03CR) 10Hashar: "Gave it a try using on contint2001:" [integration/config] - 10https://gerrit.wikimedia.org/r/674173 (https://phabricator.wikimedia.org/T252434) (owner: 10Jforrester) [16:55:23] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.37 deployment blockers - https://phabricator.wikimedia.org/T278343 (10ssastry) [17:02:32] (03PS1) 10Hashar: Wikibase selenium [integration/config] - 10https://gerrit.wikimedia.org/r/676107 [17:10:34] 10Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)), 10MW-on-K8s: Automate MediaWiki wmf branch image builds - https://phabricator.wikimedia.org/T278988 (10dduvall) [17:35:17] (03CR) 1020after4: [C: 03+1] "I haven't actually tested this but the changes look good from what I can see. Feel free to merge if you're comfortable with it but I woul" [tools/scap] - 10https://gerrit.wikimedia.org/r/674871 (https://phabricator.wikimedia.org/T269153) (owner: 10Lars Wirzenius) [17:37:32] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)): Upgrade Gerrit to 3.2.8 - https://phabricator.wikimedia.org/T278990 (10hashar) [17:37:58] 10Continuous-Integration-Config, 10Gerrit, 10Release-Engineering-Team-TODO, 10Developer Productivity: Jenkins comments no longer formatted on Gerrit 3 - https://phabricator.wikimedia.org/T256575 (10hashar) The `--gr-formatted-text-prose-max-width` get bumped from 80 characters to 120 characters with Gerrit... [17:38:39] (03PS2) 10Hashar: Wikibase selenium [integration/config] - 10https://gerrit.wikimedia.org/r/676107 [17:58:13] (03PS3) 10Hashar: Wikibase selenium [integration/config] - 10https://gerrit.wikimedia.org/r/676107 [17:59:37] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (Yak Shaving 🐃🪒 ), 10Release Pipeline: Implement pipeline config validation - https://phabricator.wikimedia.org/T225335 (10dduvall) Reviving this as a Yak Shaving task involving 2-3 people over the course of a full day. Implementation ide... [18:05:33] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO: Automate the merge of the weekly branch cut - https://phabricator.wikimedia.org/T278993 (10Jdforrester-WMF) [18:06:52] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO: Automatically tag the release task in the commit message of the weekly branch cut - https://phabricator.wikimedia.org/T278994 (10Jdforrester-WMF) [18:10:51] (03PS3) 10Umherirrender: internal: Add @codeCoverageIgnore to untestable code [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/673690 [18:11:30] (03CR) 10Umherirrender: "Patch Set 3: Removed the codeCoverageIgnore from now covered code (as of master)" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/673690 (owner: 10Umherirrender) [18:12:15] (03CR) 10Umherirrender: "Removed codeCoverageIgnore from code which is now covered (as of master)" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/673690 (owner: 10Umherirrender) [18:18:00] (03CR) 10DannyS712: internal: Add @codeCoverageIgnore to untestable code (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/673690 (owner: 10Umherirrender) [18:24:39] 10Release-Engineering-Team (Logspam), 10VisualEditor, 10Editing-team (Tracking), 10JavaScript, 10Wikimedia-production-error: Uncaught Error: Cannot add a child to text node in checkCanInsertNodeType - https://phabricator.wikimedia.org/T261785 (10egardner) This error is still occurring at a similar rate a... [18:28:43] 10Phabricator (Upstream): Phab calendar due date bug - https://phabricator.wikimedia.org/T278996 (10Reedy) [18:30:35] 10Phabricator (Upstream): Phab calendar due date bug - https://phabricator.wikimedia.org/T278996 (10Reedy) [18:31:49] 10Phabricator (Upstream): Phab calendar due date bug - https://phabricator.wikimedia.org/T278996 (10Reedy) [18:35:06] 10Phabricator (Upstream): Phab calendar due date bug - https://phabricator.wikimedia.org/T278996 (10Reedy) [18:35:49] (03PS4) 10Umherirrender: internal: Add @codeCoverageIgnore to untestable code [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/673690 [18:36:01] (03CR) 10Umherirrender: internal: Add @codeCoverageIgnore to untestable code (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/673690 (owner: 10Umherirrender) [19:44:01] Phabricator seems really slow at the moment [19:44:14] each navigation, search or type-ahead takes 1-2 seconds [19:48:52] yeah, just noticing same. [19:55:08] (03PS1) 10Jeena Huneidi: Make sure workspace is clean on checkout [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/676141 [20:01:51] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.37 deployment blockers - https://phabricator.wikimedia.org/T278343 (10Krinkle) [20:04:00] Krinkle: things seem faster now, but i do see some db connection errors in phab logstash logs. [20:04:50] (i may have spoken too soon re: speed.) [20:10:56] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.37 deployment blockers - https://phabricator.wikimedia.org/T278343 (10mmodell) [20:23:16] 10Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)), 10Release, 10Train Deployments: 1.36.0-wmf.38 deployment blockers - https://phabricator.wikimedia.org/T278344 (10Jdlrobson) [20:23:18] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.37 deployment blockers - https://phabricator.wikimedia.org/T278343 (10Jdlrobson) [20:27:55] 10Phabricator, 10User-Zabe: Add Zabe to #Triagers - https://phabricator.wikimedia.org/T279012 (10Zabe) [20:28:49] brennen: yeah it's definitely slow [20:28:55] (UK) [20:29:08] Need me to do any connectivity checks [20:42:31] RhinosF1: thanks for the report - i think it's slow overall. investigating. [20:43:00] I'll be awake for another 45 minutes or so at least if you need anything [20:44:45] ta. [20:50:06] 10Phabricator, 10DBA, 10User-brennen: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution" - https://phabricator.wikimedia.org/T279013 (10brennen) [20:50:16] filed above ^ [20:50:42] (03PS1) 10Umherirrender: Support @phan annotation and @template in UnusedUseStatementSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/676148 [20:53:44] 10Phabricator, 10DBA, 10serviceops, 10User-brennen: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution" - https://phabricator.wikimedia.org/T279013 (10brennen) [20:56:52] 10Phabricator, 10DBA, 10serviceops, 10User-brennen: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution" - https://phabricator.wikimedia.org/T279013 (10Reedy) Isn't this why in MW land we tend to use IP addresses rather than hostnames... [20:58:21] 10Phabricator, 10User-Zabe: Add Zabe to #Triagers - https://phabricator.wikimedia.org/T279012 (10Peachey88) @Zabe Can you give an example of some of the batch edits you would want to be doing? [21:05:12] 10Phabricator, 10DBA, 10serviceops, 10User-brennen: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution" - https://phabricator.wikimedia.org/T279013 (10Legoktm) For reference: ` $ host m3-master.eqiad.wmnet m3-master.eqiad.wmnet is an... [21:05:40] 10Phabricator, 10DBA, 10serviceops, 10User-brennen: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution" - https://phabricator.wikimedia.org/T279013 (10mmodell) @reedy: perhaps? but we've had it configured that way forever. [21:07:47] 10Phabricator, 10DBA, 10serviceops, 10User-brennen: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution" - https://phabricator.wikimedia.org/T279013 (10Reedy) Just because something has worked fine for a long time, doesn't mean it alwa... [21:09:23] 10Phabricator, 10DBA, 10serviceops, 10User-brennen: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution" - https://phabricator.wikimedia.org/T279013 (10mmodell) I'm certainly not against doing it that way, I presume we could have puppe... [21:09:53] (03CR) 10DannyS712: [C: 03+1] "Makes sense to me, not merging yet to give others a chance to review though" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/676148 (owner: 10Umherirrender) [21:17:24] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.37 deployment blockers - https://phabricator.wikimedia.org/T278343 (10Krinkle) [21:25:31] 10Phabricator, 10DBA, 10serviceops, 10Patch-For-Review, 10User-brennen: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution" - https://phabricator.wikimedia.org/T279013 (10CDanis) I'm not sure if we tend to use IP addresses directly... [21:35:53] 10Phabricator, 10User-Zabe: Add Zabe to #Triagers - https://phabricator.wikimedia.org/T279012 (10Zabe) For example for moving tasks from the 'Backlog' to the 'To Upload' column on the #video2commons workboard. Small, rather unimportant stuff, which is annoying to perform by hand. [21:53:27] (03CR) 10Dduvall: [C: 04-1] Make sure workspace is clean on checkout (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/676141 (owner: 10Jeena Huneidi) [22:17:31] (03PS1) 10Dduvall: Always copy the contents of source directories [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/676163 [22:22:02] (03PS2) 10Dduvall: Always copy the contents of source directories [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/676163 [22:23:38] (03PS2) 10Jeena Huneidi: Make sure workspace is clean on checkout [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/676141 [22:24:05] (03PS3) 10Dduvall: Always copy the contents of source directories [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/676163 [22:25:15] (03CR) 10Jeena Huneidi: Make sure workspace is clean on checkout (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/676141 (owner: 10Jeena Huneidi) [22:28:03] (03CR) 10Jeena Huneidi: [C: 03+2] "LGTM" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/676163 (owner: 10Dduvall) [22:28:40] (03Merged) 10jenkins-bot: Always copy the contents of source directories [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/676163 (owner: 10Dduvall) [22:41:10] (03CR) 10Dduvall: [C: 03+2] Make sure workspace is clean on checkout [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/676141 (owner: 10Jeena Huneidi) [22:41:46] (03Merged) 10jenkins-bot: Make sure workspace is clean on checkout [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/676141 (owner: 10Jeena Huneidi) [22:43:57] (03CR) 10Dduvall: Support default values in variable expressions (032 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/674689 (owner: 10Dduvall)