[00:12:15] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.34.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T220734 (10greg) 05Open→03Declined a:05dduvall→03None Week of the SRE Summit. No train. [00:15:06] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.34.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T220735 (10greg) a:05dduvall→03mmodell [00:21:22] 10Phabricator (Upstream), 10Upstream: Replace "pronoun" in Phabricator Account Settings page by "adjective" - https://phabricator.wikimedia.org/T77018 (10greg) 05Open→03Declined Thanks Evan. Declining as this is sufficiently clear (very low incidence of people complaining about it, and it currently resona... [00:55:25] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10serviceops, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['phab2001.codfw.wmnet'] ` Of which those *... [02:33:29] (03CR) 10Legoktm: [C: 03+2] Update my zuul email address to match my Gerrit primary email [integration/config] - 10https://gerrit.wikimedia.org/r/513129 (owner: 10TK-999) [02:33:42] (03CR) 10Legoktm: [C: 03+2] Add BlueSpiceExtendedSeach as dependency for BlueSpiceSocial [integration/config] - 10https://gerrit.wikimedia.org/r/513071 (owner: 10Pwirth) [02:34:59] (03Merged) 10jenkins-bot: Update my zuul email address to match my Gerrit primary email [integration/config] - 10https://gerrit.wikimedia.org/r/513129 (owner: 10TK-999) [02:35:13] (03Merged) 10jenkins-bot: Add BlueSpiceExtendedSeach as dependency for BlueSpiceSocial [integration/config] - 10https://gerrit.wikimedia.org/r/513071 (owner: 10Pwirth) [02:35:43] !log deployed https://gerrit.wikimedia.org/r/513129 https://gerrit.wikimedia.org/r/513071 [02:35:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:43:52] (03CR) 10Legoktm: [C: 03+2] [CommonsMetadata] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/512474 (owner: 10Umherirrender) [02:43:54] (03CR) 10Legoktm: [C: 03+2] [PagedTiffHandler] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/512507 (owner: 10Umherirrender) [02:43:58] (03CR) 10Legoktm: [C: 03+2] [ConfirmEdit] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/513153 (owner: 10Umherirrender) [02:44:00] (03CR) 10Legoktm: [C: 03+2] [Kartographer] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/513157 (owner: 10Umherirrender) [02:44:03] (03CR) 10Legoktm: [C: 03+2] [Passwordless Login] Jobs for extension [integration/config] - 10https://gerrit.wikimedia.org/r/512729 (https://phabricator.wikimedia.org/T224371) (owner: 10Florianschmidtwelzow) [02:44:24] (03CR) 10Legoktm: [C: 03+2] [CleanChanges] Add cldr as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/513206 (owner: 10Umherirrender) [02:44:27] (03CR) 10Legoktm: [C: 03+2] [LiquidThreads] Add dependency [integration/config] - 10https://gerrit.wikimedia.org/r/513214 (owner: 10Umherirrender) [02:44:32] (03CR) 10Legoktm: [C: 03+2] [Translate] Add dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/513217 (owner: 10Umherirrender) [02:44:34] (03CR) 10Legoktm: [C: 03+2] [CodeReview] Add dependency [integration/config] - 10https://gerrit.wikimedia.org/r/513218 (owner: 10Umherirrender) [02:44:38] (03CR) 10Legoktm: [C: 03+2] [CentralNotice] Add dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/513211 (owner: 10Umherirrender) [02:45:25] (03Merged) 10jenkins-bot: [CommonsMetadata] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/512474 (owner: 10Umherirrender) [02:45:30] (03Merged) 10jenkins-bot: [PagedTiffHandler] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/512507 (owner: 10Umherirrender) [02:45:33] (03Merged) 10jenkins-bot: [ConfirmEdit] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/513153 (owner: 10Umherirrender) [02:45:36] (03Merged) 10jenkins-bot: [Kartographer] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/513157 (owner: 10Umherirrender) [02:46:04] (03Merged) 10jenkins-bot: [Passwordless Login] Jobs for extension [integration/config] - 10https://gerrit.wikimedia.org/r/512729 (https://phabricator.wikimedia.org/T224371) (owner: 10Florianschmidtwelzow) [02:46:52] (03Merged) 10jenkins-bot: [CleanChanges] Add cldr as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/513206 (owner: 10Umherirrender) [02:47:01] (03Merged) 10jenkins-bot: [LiquidThreads] Add dependency [integration/config] - 10https://gerrit.wikimedia.org/r/513214 (owner: 10Umherirrender) [02:47:10] (03Merged) 10jenkins-bot: [Translate] Add dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/513217 (owner: 10Umherirrender) [02:47:34] (03Merged) 10jenkins-bot: [CodeReview] Add dependency [integration/config] - 10https://gerrit.wikimedia.org/r/513218 (owner: 10Umherirrender) [02:47:37] (03Merged) 10jenkins-bot: [CentralNotice] Add dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/513211 (owner: 10Umherirrender) [02:48:46] !log deploying https://gerrit.wikimedia.org/r/512474 https://gerrit.wikimedia.org/r/512507 https://gerrit.wikimedia.org/r/513153 https://gerrit.wikimedia.org/r/513157 https://gerrit.wikimedia.org/r/512729 https://gerrit.wikimedia.org/r/513206 https://gerrit.wikimedia.org/r/513214 https://gerrit.wikimedia.org/r/513217 https://gerrit.wikimedia.org/r/513218 https://gerrit.wikimedia.org/r/513211 [02:48:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:08:37] (03CR) 10Legoktm: [C: 03+2] [Newsletter] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/511057 (owner: 10Umherirrender) [03:10:16] (03Merged) 10jenkins-bot: [Newsletter] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/511057 (owner: 10Umherirrender) [04:10:43] awight: hmm, I don't see the depenency planning/optimization stuff we briefly talked about on IRC in your patch [04:15:22] (03PS1) 10Legoktm: backend: Fix likely typo in variable name [integration/quibble] - 10https://gerrit.wikimedia.org/r/513254 [04:22:11] (03CR) 10Legoktm: [C: 03+1] "LGTM, but this is large enough that someone else should take a look :)" [integration/quibble] - 10https://gerrit.wikimedia.org/r/511749 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight) [06:25:57] PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: (Service Check Timed Out) [11:49:48] 10Phabricator (Upstream), 10Upstream: After renaming a project in Phabricator, its /tag/projectname URL ends up as a 404 - https://phabricator.wikimedia.org/T224662 (10Aklapper) p:05Triage→03High [11:53:14] 10Phabricator (Upstream), 10Upstream: After renaming a project in Phabricator, its /tag/projectname URL ends up as a 404 - https://phabricator.wikimedia.org/T224662 (10Reedy) [13:18:31] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:24:56] (03PS1) 10Kosta Harlan: Add CentralAuth, MassMessage and WikibaseLexeme to gated extensions [integration/config] - 10https://gerrit.wikimedia.org/r/513283 (https://phabricator.wikimedia.org/T224607) [13:26:26] (03CR) 10jerkins-bot: [V: 04-1] Add CentralAuth, MassMessage and WikibaseLexeme to gated extensions [integration/config] - 10https://gerrit.wikimedia.org/r/513283 (https://phabricator.wikimedia.org/T224607) (owner: 10Kosta Harlan) [13:28:32] (03PS2) 10Kosta Harlan: Add CentralAuth, MassMessage and WikibaseLexeme to gated extensions [integration/config] - 10https://gerrit.wikimedia.org/r/513283 (https://phabricator.wikimedia.org/T224607) [13:28:50] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T220732 (10Fomafix) [13:29:18] (03PS3) 10Kosta Harlan: Add CentralAuth, MassMessage and WikibaseLexeme to gated extensions [integration/config] - 10https://gerrit.wikimedia.org/r/513283 (https://phabricator.wikimedia.org/T224607) [13:32:36] (03CR) 10Daniel Kinzler: [C: 03+1] "Agree with intent. Don't know about implementation or implications." [integration/config] - 10https://gerrit.wikimedia.org/r/513283 (https://phabricator.wikimedia.org/T224607) (owner: 10Kosta Harlan) [13:34:51] 10Phabricator (Upstream), 10Upstream: After renaming a project in Phabricator, its /tag/projectname URL ends up as a 404 - https://phabricator.wikimedia.org/T224662 (10mmodell) For whatever reason, it seems that the project name needs to be repeated in the hashtag field. Perhaps the "additional hashtags" field... [13:42:09] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T220732 (10zeljkofilipin) Group2 at 1.34.0-wmf.7 ([[ https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/513284 | gerrit ]], [[ https://... [13:48:09] (03PS1) 10Kosta Harlan: Document how to debug CLI scripts with Xdebug [releng/local-charts] - 10https://gerrit.wikimedia.org/r/513287 [14:15:12] 10Phabricator (Upstream), 10Upstream: After renaming a project in Phabricator, its /tag/projectname URL ends up as a 404 - https://phabricator.wikimedia.org/T224662 (10epriestley) I think this is likely an upstream problem. I think we try to add the "natural" hashtag on a rename, but maybe we're looking at the... [14:36:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10Cmjohnson) [14:38:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10Cmjohnson) @greg @robh I am just plugging these disks into the server correct? nothing else? this will not r... [14:38:38] 10Continuous-Integration-Infrastructure, 10Browser-Tests, 10Code-Health: Automate identifying flaky tests - https://phabricator.wikimedia.org/T224673 (10kostajh) [14:42:22] 10Continuous-Integration-Infrastructure, 10Browser-Tests, 10Code-Health: Automate identifying flaky tests - https://phabricator.wikimedia.org/T224673 (10kostajh) It looks like we have +16,500 patches with "recheck" in the comments since January 2014 https://gerrit.wikimedia.org/r/q/recheck,16500 [14:56:01] 10Phabricator (Upstream), 10WMSE-Bug-Reporting-and-Translation-2019, 10Upstream: "Move Tasks to Column..." opens dialogue for moving to project - https://phabricator.wikimedia.org/T220585 (10epriestley) I think we can do something to make this less confusing, this isn't the first time I've seen someone run i... [15:16:49] (03CR) 10Jforrester: [C: 04-1] "The rough rule of thumb is that we don't add new extensions to the gate unless their test suites (particularly PHPunit, which is run the m" [integration/config] - 10https://gerrit.wikimedia.org/r/513283 (https://phabricator.wikimedia.org/T224607) (owner: 10Kosta Harlan) [15:20:32] 10Phabricator (Upstream), 10Upstream: Time zone drop-down menu in Phabricator uses underscores in entries - https://phabricator.wikimedia.org/T902 (10epriestley) This should be resolved upstream by . [15:51:31] 10Phabricator (Upstream), 10Upstream: In Phabricator forms, text fields made visible by an option should become focused when they appear - https://phabricator.wikimedia.org/T78196 (10epriestley) I think there are at least a few different flows here: In one flow, you want to take exactly one action (like "Add... [15:56:57] (03CR) 10Kosta Harlan: "> Patch Set 3: Code-Review-1" [integration/config] - 10https://gerrit.wikimedia.org/r/513283 (https://phabricator.wikimedia.org/T224607) (owner: 10Kosta Harlan) [15:59:40] 10Phabricator (Upstream), 10Upstream: Phabricator bots should not trigger notifications - https://phabricator.wikimedia.org/T198574 (10epriestley) It is intentional that bots trigger notifications and can not be silenced. This serves to prevent a rogue administrator from creating a bot, grabbing a bot API key,... [16:01:55] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Backlog), 10Striker, 10Tools: Add Gerrit support to Striker (toolsadmin) - https://phabricator.wikimedia.org/T224676 (10LucasWerkmeister) [16:04:38] (03CR) 10C. Scott Ananian: [C: 03+1] "LGTM" [integration/config] - 10https://gerrit.wikimedia.org/r/513168 (https://phabricator.wikimedia.org/T221872) (owner: 10Hashar) [16:07:06] (03CR) 10C. Scott Ananian: [C: 03+1] Remove unused mwgate composer-package jobs [integration/config] - 10https://gerrit.wikimedia.org/r/513170 (owner: 10Hashar) [16:08:53] (03CR) 10C. Scott Ananian: [C: 03+1] Run repo specific composer jobs with more memory [integration/config] - 10https://gerrit.wikimedia.org/r/513171 (https://phabricator.wikimedia.org/T221872) (owner: 10Hashar) [16:09:55] (03CR) 10Jforrester: [C: 04-1] "> Patch Set 3:" [integration/config] - 10https://gerrit.wikimedia.org/r/513283 (https://phabricator.wikimedia.org/T224607) (owner: 10Kosta Harlan) [16:11:24] (03CR) 10Kosta Harlan: "> Sure. There are ~190 extensions in production. Adding all of them to the gate (which wouldn't work as some of their test suites interfer" [integration/config] - 10https://gerrit.wikimedia.org/r/513283 (https://phabricator.wikimedia.org/T224607) (owner: 10Kosta Harlan) [16:11:29] (03Abandoned) 10Kosta Harlan: Add CentralAuth, MassMessage and WikibaseLexeme to gated extensions [integration/config] - 10https://gerrit.wikimedia.org/r/513283 (https://phabricator.wikimedia.org/T224607) (owner: 10Kosta Harlan) [16:21:32] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Backlog), 10Striker, 10Tools: Add Gerrit support to Striker (toolsadmin) - https://phabricator.wikimedia.org/T224676 (10Paladox) a:03Paladox [16:22:00] 10Diffusion: Cannot connect to vcs@git-ssh.wikimedia.org - https://phabricator.wikimedia.org/T224677 (10LucasWerkmeister) [16:23:44] 10Continuous-Integration-Infrastructure, 10Fresnel, 10Performance-Team, 10Quibble: Enable compression for MW web responses in Jenkins Jenkins (e.g. Quibble, Fresnel) - https://phabricator.wikimedia.org/T219694 (10Krinkle) a:05Krinkle→03None [16:24:43] 10Continuous-Integration-Infrastructure, 10Fresnel, 10Performance-Team, 10Quibble, 10patch-welcome: Enable compression for MW web responses in Jenkins Jenkins (e.g. Quibble, Fresnel) - https://phabricator.wikimedia.org/T219694 (10Krinkle) //(Still interested as deliverable in a future quarter as part of... [16:31:34] 10Phabricator (Upstream), 10Upstream: Shortcut keys for description fields text formatting are needed. - https://phabricator.wikimedia.org/T816 (10epriestley) This looks like a couple of different requests: - Add text formatting shortcuts like {key command B} (or whatever) to make selected text bold when ed... [16:38:00] 10Diffusion, 10Phabricator (Upstream), 10Upstream: Big/large changesets in review are not displayed in Phabricator - https://phabricator.wikimedia.org/T128760 (10epriestley) Based on this request, I don't currently plan to change this behavior upstream. This request lacks a use case, and we will essentially... [16:39:29] 10Diffusion, 10Phabricator (Upstream), 10Upstream: Big/large changesets in review are not displayed in Phabricator - https://phabricator.wikimedia.org/T128760 (10mmodell) 05Open→03Declined [16:56:41] 10Phabricator (Upstream), 10Upstream: Private tasks should have a placeholder on the workboard - https://phabricator.wikimedia.org/T90833 (10epriestley) I think we're very unlikely to implement this specific change (placeholder cards) in the upstream. See also T187051#5191464 for a similar issue. If we show a... [17:02:50] 10Phabricator (Upstream), 10Upstream: Connect Pastes to their related tasks - https://phabricator.wikimedia.org/T98899 (10epriestley) After , commenting with the full URI of a paste or task on an object will count as mentioning that paste or task. That is, commenting `P12... [17:03:33] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10JavaScript, 10Patch-For-Review: Switch quibble-based CI jobs from node6 to node10 - https://phabricator.wikimedia.org/T222406 (10Jdforrester-WMF) a:03Jdforrester-WMF [17:09:40] 10Gerrit, 10Security: Cannot assign user name "XXX" to account ####; name already in use. - https://phabricator.wikimedia.org/T216605 (10mmodell) [17:11:14] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10serviceops, 10Epic: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) [17:11:30] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10JavaScript, 10Patch-For-Review: Switch quibble-based CI jobs from node6 to node10 - https://phabricator.wikimedia.org/T222406 (10Jdforrester-WMF) [17:11:49] 10Release-Engineering-Team (Kanban), 10Reading-Infrastructure-Team-Backlog, 10Wikimedia-Site-requests, 10Epic, 10Patch-For-Review: Sunset Wikipedia Zero - https://phabricator.wikimedia.org/T187716 (10Jdforrester-WMF) [17:12:12] 10Release-Engineering-Team (Kanban), 10Wikimedia-Site-requests, 10foundation.wikimedia.org: Enable SUL accounts on Governance wiki - https://phabricator.wikimedia.org/T205347 (10Jdforrester-WMF) [17:12:26] 10Release-Engineering-Team (Kanban), 10Wikimedia-Site-requests, 10foundation.wikimedia.org: Enable SUL accounts on Governance wiki - https://phabricator.wikimedia.org/T205347 (10Jdforrester-WMF) a:03Jdforrester-WMF [17:12:52] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10serviceops, 10Epic: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) a:03Jdforrester-WMF [17:13:57] 10Gerrit, 10Security: Cannot assign user name "XXX" to account ####; name already in use. - https://phabricator.wikimedia.org/T216605 (10Mhurd) Happening to me too: {F29280815} [17:25:32] 10Phabricator (Upstream), 10Upstream: Link to task should appear before message in phabricator emails - https://phabricator.wikimedia.org/T1050 (10epriestley) Upstream, will add a "View Task" button to the upper-right corner of HTML email from Maniphest. The underlying... [17:27:35] 10Diffusion: Cannot connect to vcs@git-ssh.wikimedia.org - https://phabricator.wikimedia.org/T224677 (10mmodell) I see this in the logs: `User vcs not allowed because account is locked` So something is wrong with the vcs user on `phab1003`. [18:53:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10ayounsi) [18:53:03] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): - https://phabricator.wikimedia.org/T219850 (10ayounsi) 05Resolved→03Open Reopening as it alerted again: https://icinga.wikimedia.org/cgi-bin/... [18:53:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): - https://phabricator.wikimedia.org/T219850 (10ayounsi) p:05Unbreak!→03High [18:59:30] 10Phabricator (Upstream), 10Browser-Support-Opera, 10Upstream: "Choose subtype" box does not disappear in Opera 58 after choosing an option when creating new subtask - https://phabricator.wikimedia.org/T220897 (10epriestley) This reproduces for me in browsers other than Opera, but requires the "Persistent Ch... [18:59:52] 10Phabricator (Upstream), 10Upstream: "Choose subtype" box does not disappear after choosing an option when creating new subtask with "Persistent Chat" option in Conpherence enabled - https://phabricator.wikimedia.org/T220897 (10epriestley) [18:59:55] legoktm: Just got your ping. I'm curious what you meant re. optimizations. My plan was to do a straight port with equivalent functionality before trying to get fancy... [19:02:03] 10Phabricator (Upstream), 10Accessibility, 10Upstream: Localizable tooltips, for html form fields - guidelines? - https://phabricator.wikimedia.org/T95076 (10epriestley) I suspect this is largely obsolete nowadays with the option to add instructions to custom forms? The {nav New Bug Report} form now has a bl... [19:03:25] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [19:11:53] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10akosiaris) >>! In T198901#5220959, @Krenair wrote: > Is apertium part o... [19:11:59] 10Diffusion: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10Aklapper) [19:12:55] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Operations, 10Goal, 10User-fgiunchedi: Add Prometheus exporter to Jenkins instances - https://phabricator.wikimedia.org/T182759 (10greg) [19:13:16] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10Krenair) Okay. Is it potentially in-scope i.e. should it appear on one... [19:15:22] awight: gotcha. I just (mistakenly) assumed that the performance refactoring you suggested would also be in that patch :p [19:15:29] got too excited haha [19:16:58] 10Phabricator (Upstream), 10Upstream: Access-restricted Phab applications appear in quick search results for all users, give "Access Denied" - https://phabricator.wikimedia.org/T94268 (10epriestley) I can't immediately reproduce this in modern Phabricator. Here's a user who can use the "Paste" application: {F... [19:18:42] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10akosiaris) [19:19:09] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10akosiaris) [19:19:41] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10akosiaris) Indeed. I added it under TBD. The exact way this will be don... [19:20:58] legoktm: :D I was thinking I might try to cut that patch down into even smaller pieces, e.g. converting small groups of functions, just calling directly, before the final patch which would build the command queue. [19:21:24] At least it makes each operation testable, so delivers some initial value. [19:21:56] merging the whole thing is probably better for me at this point, though. Limited time to play around in quibble, obviously. :-/ [19:22:26] I started working on the dependency thing last night and have a weak WIP locally [19:22:28] tangentially, we also need to move the test suite to use pytest instead of nose at some point [19:36:38] 10Phabricator (Upstream), 10Upstream: Emails about users adding blank comments via email which don't show in the web UI - https://phabricator.wikimedia.org/T125834 (10epriestley) I can't figure out how to reproduce this in modern Phabricator. I suspect the original bug was related to inbound email processing o... [19:39:13] 10Phabricator (Upstream), 10Upstream: Phabricator comment input should trim whitespace - https://phabricator.wikimedia.org/T88655 (10epriestley) This should be resolved upstream by . I'm going to tweak the title here to align it slightly better to the desired/implemented... [19:39:53] 10Phabricator (Upstream), 10Upstream: Phabricator should treat comment inputs containing only whitespace characters as empty when deciding whether the user made a comment or not - https://phabricator.wikimedia.org/T88655 (10epriestley) [19:45:01] legoktm: Cool! Looking forward to seeing what you've done. [19:45:16] porting the tests should be quick... [19:45:49] (I shouldn't say that without looking first) [19:50:24] Did you happen to base the dependency work on the command pattern version, or the current code? Mostly wondering whether the refactor made this work easier. [19:50:25] 10Continuous-Integration-Config, 10Wikidata, 10Patch-For-Review, 10User-Addshore: Install cache/integration-tests with Wikibase CI tests - https://phabricator.wikimedia.org/T199440 (10Legoktm) [19:50:33] 10Release-Engineering-Team, 10Librarization, 10Quibble, 10Patch-For-Review: Install extension require-dev dependencies in wmf-quibble-vendor-mysql-hhvm-docker - https://phabricator.wikimedia.org/T220723 (10Legoktm) This was previously discussed at {T199440} FWIW. [19:58:04] awight: on top of your refactor [19:58:41] (03PS1) 10Legoktm: [WIP] Magic dependency thing [integration/quibble] - 10https://gerrit.wikimedia.org/r/513374 [19:59:01] Lmk if you noticed anything annoying, cos I'm steeling my resolve to split that patch into more palatable chunks. [19:59:09] awight: ^ that's how far I got. I have the actual dependency sorter planned out in my head, just not in paper yet [20:05:34] 10Phabricator (Upstream), 10Accessibility, 10Upstream: Localizable tooltips, for html form fields - guidelines? - https://phabricator.wikimedia.org/T95076 (10mmodell) 05Open→03Resolved a:03mmodell [20:09:17] (03CR) 10Awight: "This is helpful to look at, thanks. I'm not sure where we should code the dependency knowledge, but it feels slightly wrong to have it in" [integration/quibble] - 10https://gerrit.wikimedia.org/r/513374 (owner: 10Legoktm) [20:10:49] (03CR) 10Awight: "I think the commands are the nodes of a dependency/process graph, and the dependencies are the lines. Nodes shouldn't have to know anythi" [integration/quibble] - 10https://gerrit.wikimedia.org/r/513374 (owner: 10Legoktm) [20:14:11] 10Diffusion: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10mmodell) Ok at least the ssh layer seems to be working now: ` debug1: Offering RSA public key: *** debug1: Authentication succeeded (publickey). Authenticated to git-ssh... [20:22:28] (03PS1) 10Awight: Extract zuul clone into a command object [integration/quibble] - 10https://gerrit.wikimedia.org/r/513377 (https://phabricator.wikimedia.org/T223752) [20:22:30] legoktm: baby steps ^ [20:23:27] :D [20:26:36] 10Release-Engineering-Team (Kanban), 10User-MModell: Talk with Timo and Fillipo about grafana and sentury for LM ("logging, monitoring, metrics") - https://phabricator.wikimedia.org/T222638 (10mmodell) 05Open→03Resolved [20:28:18] (03CR) 10Dduvall: [C: 03+2] pipeline: Test more of the stage steps implementation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/513236 (owner: 10Dduvall) [20:28:31] (03CR) 10Jforrester: [C: 03+1] Extract zuul clone into a command object [integration/quibble] - 10https://gerrit.wikimedia.org/r/513377 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight) [20:29:00] (03Merged) 10jenkins-bot: pipeline: Test more of the stage steps implementation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/513236 (owner: 10Dduvall) [20:29:58] (03CR) 10jenkins-bot: pipeline: Test more of the stage steps implementation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/513236 (owner: 10Dduvall) [20:30:35] (03PS1) 10Awight: Extension and skin submodule update as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513406 (https://phabricator.wikimedia.org/T223752) [20:37:50] 10Continuous-Integration-Infrastructure (Slipway), 10Wikimedia-Fundraising-CiviCRM: Migrate wikimedia-fundraising-civicrm to a Docker container - https://phabricator.wikimedia.org/T210287 (10Ejegg) Thanks @hashar, that sounds great! @Eileenmcnaughton is the local subject matter expert on CiviCRM and its build... [20:43:11] hashar left me some integration/config patches in T221872 -- could someone here help me get them merged and deployed? [20:43:17] T221872: composer-package-php73-docker seems to fail often on Parsoid builds - https://phabricator.wikimedia.org/T221872 [20:44:41] cscott: yes [20:45:35] legoktm: my hero! [20:46:04] legoktm: looks like https://gerrit.wikimedia.org/r/513168 is the first in the stack [20:47:57] https://gerrit.wikimedia.org/r/513170 is just clean up, and https://gerrit.wikimedia.org/r/513171 depends on 513168 but is the patch that actually improves this for us. [20:48:33] (03CR) 10Legoktm: [C: 03+2] "INFO:jenkins_jobs.builder:Creating jenkins job parsoidsvc-composer-package-php72-docker" [integration/config] - 10https://gerrit.wikimedia.org/r/513168 (https://phabricator.wikimedia.org/T221872) (owner: 10Hashar) [20:50:13] (03CR) 10Legoktm: [C: 03+2] "INFO:jenkins_jobs.builder:Deleting jenkins job mwgate-composer-package-hhvm-docker" [integration/config] - 10https://gerrit.wikimedia.org/r/513170 (owner: 10Hashar) [20:50:44] (03Merged) 10jenkins-bot: Use more explicit jobs for Parsoid service [integration/config] - 10https://gerrit.wikimedia.org/r/513168 (https://phabricator.wikimedia.org/T221872) (owner: 10Hashar) [20:51:00] (03CR) 10Legoktm: [C: 03+2] "INFO:jenkins_jobs.builder:Reconfiguring jenkins job parsoidsvc-composer-package-php72-docker" [integration/config] - 10https://gerrit.wikimedia.org/r/513171 (https://phabricator.wikimedia.org/T221872) (owner: 10Hashar) [20:51:10] !log deploying https://gerrit.wikimedia.org/r/513168 [20:51:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:51:29] cscott: ok, should be all deployed [20:51:42] ok, let me recheck some stuff and see how we're doing [20:53:17] (03Merged) 10jenkins-bot: Remove unused mwgate composer-package jobs [integration/config] - 10https://gerrit.wikimedia.org/r/513170 (owner: 10Hashar) [20:54:00] (03Merged) 10jenkins-bot: Run repo specific composer jobs with more memory [integration/config] - 10https://gerrit.wikimedia.org/r/513171 (https://phabricator.wikimedia.org/T221872) (owner: 10Hashar) [20:56:45] (03PS1) 10Jforrester: WIP: dockerfiles: quibble-fresnel: Upgrade fresnel to 0.3.0 [integration/config] - 10https://gerrit.wikimedia.org/r/513493 [21:05:10] cscott: I'm gonna head out in a few - are things looking ok? [21:05:31] note that because the job was just renamed, the composer/npm/etc. caches are going to be cold [21:05:35] yup. i'm just watching 512769,4 complete on zuul, but it's looking fine. [21:06:02] jobs have the right name, were triggered ok, seem to be running fine. [21:06:38] they're on m4executor instances, etc. basically everything i can check seems to be correct. [21:07:39] (03CR) 10Jforrester: "Half-done, as docker-pkg doesn't work for me locally." [integration/config] - 10https://gerrit.wikimedia.org/r/513493 (owner: 10Jforrester) [21:14:04] legoktm: the post-merge queue for operations/mediawiki-config is quite backed up though: oldest jobs have been in there 22 hours. [21:15:14] Ugh [21:15:29] not your fault of course [21:15:30] The executor probably needs to be readded [21:15:47] Unfortunately I just left without my laptop [21:18:18] 10Continuous-Integration-Config, 10Parsoid, 10phan: composer-package-php73-docker seems to fail often on Parsoid builds - https://phabricator.wikimedia.org/T221872 (10cscott) This is now *mostly* fixed. Getting spurious failures on other jobs, eg ENOMEM on https://integration.wikimedia.org/ci/job/parsoidsvc... [21:21:02] !log doing the zuul deadlock dance again [21:21:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:27:19] !log back in business (ugh) [21:27:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:33:29] (03PS1) 10Awight: Composer dependencies as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513498 (https://phabricator.wikimedia.org/T223752) [21:33:35] !log Jenkins admin says ""Your Jenkins data directory /var/lib/jenkins (AKA JENKINS_HOME) is almost full. You should act on it before it gets completely full."" [21:33:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:33:37] greg-g: ^ [21:34:03] geebus [21:36:22] (03PS2) 10Brennen Bearnes: Add .gitreview for use with git-review [releng/local-charts] - 10https://gerrit.wikimedia.org/r/511114 (owner: 10Tarrow) [21:36:35] (03CR) 10Jforrester: [C: 03+1] Add .gitreview for use with git-review [releng/local-charts] - 10https://gerrit.wikimedia.org/r/511114 (owner: 10Tarrow) [21:36:43] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Striker, 10Tools: Add Gerrit support to Striker (toolsadmin) - https://phabricator.wikimedia.org/T224676 (10greg) [21:38:22] (03CR) 10Brennen Bearnes: [V: 03+2 C: 03+2] "Tested with `git review` by making a minor change (the .git extension / commit message) and pushing. Looks good to me; merging." [releng/local-charts] - 10https://gerrit.wikimedia.org/r/511114 (owner: 10Tarrow) [21:43:59] PROBLEM - Free space - all mounts on deployment-mwmaint01 is CRITICAL: CRITICAL: deployment-prep.deployment-mwmaint01.diskspace.root.byte_percentfree (<10.00%) [21:45:32] (03PS1) 10Awight: Vendor dependencies as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513503 (https://phabricator.wikimedia.org/T223752) [21:49:46] (03PS1) 10Awight: Composer and npm test as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513504 (https://phabricator.wikimedia.org/T223752) [21:52:22] (03PS1) 10Awight: Npm install as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513510 (https://phabricator.wikimedia.org/T223752) [21:52:58] (03PS1) 10Krinkle: zuul: Remove link to 'nodepool' Grafana dash (no longer exists) [integration/docroot] - 10https://gerrit.wikimedia.org/r/513511 [21:53:12] (03CR) 10Krinkle: [C: 03+2] zuul: Remove link to 'nodepool' Grafana dash (no longer exists) [integration/docroot] - 10https://gerrit.wikimedia.org/r/513511 (owner: 10Krinkle) [21:53:49] (03Merged) 10jenkins-bot: zuul: Remove link to 'nodepool' Grafana dash (no longer exists) [integration/docroot] - 10https://gerrit.wikimedia.org/r/513511 (owner: 10Krinkle) [21:53:57] (03CR) 10jenkins-bot: zuul: Remove link to 'nodepool' Grafana dash (no longer exists) [integration/docroot] - 10https://gerrit.wikimedia.org/r/513511 (owner: 10Krinkle) [21:54:00] RECOVERY - Free space - all mounts on deployment-mwmaint01 is OK: OK: All targets OK [21:55:36] (03PS1) 10Awight: phpunit tests as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513513 (https://phabricator.wikimedia.org/T223752) [21:58:34] (03PS1) 10Awight: Core npm composer test as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513515 (https://phabricator.wikimedia.org/T223752) [22:03:50] (03PS1) 10Awight: browser tests as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513518 (https://phabricator.wikimedia.org/T223752) [22:09:23] (03PS1) 10Awight: MediaWiki installation... as command [integration/quibble] - 10https://gerrit.wikimedia.org/r/513520 (https://phabricator.wikimedia.org/T223752) [22:11:03] twentyafterfour: it's known that aphlict doesn't run on the passive server, but the actual issue is: [22:11:08] No such file or directory @ dir_chdir - /srv/phab/phabricator/support [22:11:18] Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /srv/phab/phabricator/support/aphlict/server [22:11:31] where would the "support" directory come from? [22:11:37] paladox: ^ [22:11:51] i applied your patch paladox, but that did not fix this issue [22:12:03] since the issue was actually not "failed to start aphlict service" [22:12:10] but instead this missing dir [22:12:35] redirector.pp: file { "${rootdir}/phabricator/support/preamble.php": [22:12:38] also this [22:12:47] has the same issue since it does stuff inside ./support/ [22:12:56] but ./support/ itself isn't there [22:13:23] guess we should puppetize it [22:18:29] (03PS8) 10Awight: Separate planning and execution phases [integration/quibble] - 10https://gerrit.wikimedia.org/r/511749 (https://phabricator.wikimedia.org/T223752) [22:20:55] (03CR) 10Awight: "PS8: All the heavy lifting is done in smaller, precursor patches." [integration/quibble] - 10https://gerrit.wikimedia.org/r/511749 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight) [22:22:53] (03CR) 10Jforrester: "In future we should split this into different commands, one for unit tests and one for integration tests, but that's not for now. :-)" [integration/quibble] - 10https://gerrit.wikimedia.org/r/513513 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight) [22:22:58] legoktm: ^ done! Thanks for the encouragement. [22:24:00] (03CR) 10Jforrester: "Does this need to be re-done on top of Iee50109a2e?" [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116) (owner: 10Awight) [22:24:17] (03CR) 10Jforrester: [C: 03+1] Separate planning and execution phases [integration/quibble] - 10https://gerrit.wikimedia.org/r/511749 (https://phabricator.wikimedia.org/T223752) (owner: 10Awight) [22:24:43] James_F: Thanks for all the review! [22:24:53] awight: Thanks for all the work. :-) [22:25:49] Making that last chain of patches was oddly satisfying, sort of like butchering a hock... [22:27:51] (03CR) 10Awight: "> Does this need to be re-done on top of Iee50109a2e?" [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116) (owner: 10Awight) [22:35:21] (03CR) 10Dzahn: "thanks for doing this! actually we could use "deployment.eqiad.wmnet". That is currently an alias for deploy1001 which is correct but it w" [tools/scap] - 10https://gerrit.wikimedia.org/r/513238 (owner: 10Paladox) [22:40:12] (03CR) 10Krinkle: WIP: dockerfiles: quibble-fresnel: Upgrade fresnel to 0.3.0 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/513493 (owner: 10Jforrester) [22:42:41] (03CR) 10Krinkle: "most of these can also say "deployment server" instead, which will both be easier to read and understand for the consumers, and less hassl" [tools/scap] - 10https://gerrit.wikimedia.org/r/513238 (owner: 10Paladox) [22:43:12] mutante: that directory should be part of the phabricator source checkout [22:44:05] twentyafterfour: i don't really understand how there are _some_ files there but not all [22:44:23] and when i do git status it is unclean on both servers.. but also different [22:44:29] is this on phab2001? [22:44:36] yes, and phab1003 [22:44:59] i tried "scap pull" too btw.. and that fails horribly because it wants /srv/mediawiki [22:45:35] i feel like a simple rsync from deploy server would make it all so much easier [22:46:04] since scap just wraps around that but we only have like 1 or 2 servers anyways [22:50:55] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): - https://phabricator.wikimedia.org/T219850 (10dduvall) a:05hashar→03dduvall >>! In T219850#5224558, @ayounsi wrote: > Reopening as it alerted... [22:56:35] (03PS2) 10Jforrester: dockerfiles: quibble-fresnel: Upgrade fresnel to 0.3.0 [integration/config] - 10https://gerrit.wikimedia.org/r/513493 [22:57:54] (03CR) 10Jforrester: dockerfiles: quibble-fresnel: Upgrade fresnel to 0.3.0 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/513493 (owner: 10Jforrester) [23:00:31] (03CR) 10Jforrester: [C: 03+2] "Build works locally." [integration/config] - 10https://gerrit.wikimedia.org/r/513493 (owner: 10Jforrester) [23:02:07] (03Merged) 10jenkins-bot: dockerfiles: quibble-fresnel: Upgrade fresnel to 0.3.0 [integration/config] - 10https://gerrit.wikimedia.org/r/513493 (owner: 10Jforrester) [23:02:40] !log Updating docker-pkg files on contint1001 for +quibble-fresnel (0.0.31-4) [23:02:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:03:20] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10dduvall) [23:03:23] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): - https://phabricator.wikimedia.org/T219850 (10dduvall) 05Open→03Resolved Alert is back to OK. [23:09:00] (03PS1) 10Jforrester: jjb: Update fresnel job to Fresnel 0.3.0 [integration/config] - 10https://gerrit.wikimedia.org/r/513528 [23:09:55] (03CR) 10Jforrester: [C: 03+2] jjb: Update fresnel job to Fresnel 0.3.0 [integration/config] - 10https://gerrit.wikimedia.org/r/513528 (owner: 10Jforrester) [23:10:07] (03CR) 10Jforrester: jjb: Update fresnel job to Fresnel 0.3.0 [integration/config] - 10https://gerrit.wikimedia.org/r/513528 (owner: 10Jforrester) [23:13:25] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/513528 (owner: 10Jforrester) [23:15:36] (03Merged) 10jenkins-bot: jjb: Update fresnel job to Fresnel 0.3.0 [integration/config] - 10https://gerrit.wikimedia.org/r/513528 (owner: 10Jforrester) [23:17:12] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog): Fix partitions on CI slaves , some are missing /var/lib/docker - https://phabricator.wikimedia.org/T221040 (10greg) p:05Triage→03Normal [23:17:23] (03CR) 10Jforrester: "Working: https://integration.wikimedia.org/ci/job/mediawiki-fresnel-patch-docker/4432/console" [integration/config] - 10https://gerrit.wikimedia.org/r/513528 (owner: 10Jforrester) [23:19:57] !log Pruned releng/quibble-fresnel:0.0.31-3 8ca03484a2c3 from contint1001 to save space [23:19:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:22:00] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10local-charts: Fix Install Script For Some Versions of Ubuntu - https://phabricator.wikimedia.org/T223715 (10brennen) Taking a step back, I now wonder if we even need to install docker on host machines by default? Assuming people don't have to...