[01:36:32] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Upgrade to Gerrit 2.16.8 - https://phabricator.wikimedia.org/T200739 (10Paladox) Grrrit 2.15 becomes EOL this year after gerrit 3.1 is released (gerrits ESC ( Engineering Steering Committee)) https://groups.google.com/forum/m/#!topic/repo-di... [02:27:28] 10Gerrit: "Server error: line 1:33 no viable alternative at character '['" when visiting particular patch - https://phabricator.wikimedia.org/T223278 (10Paladox) For the record this is fixed in https://github.com/GerritCodeReview/gerrit/commit/efcce2f7ee6f5e50f54fd6371e27d661c392135d#diff-7ec4adef9ea656b2ea4ec7a... [04:31:16] 10Phabricator, 10Developer-Advocacy (Apr-Jun 2019): Re-evaluate our use of Phabricator Conpherence chat - https://phabricator.wikimedia.org/T127640 (10revi) @aklapper I've started a discussion on stewards-l, it might take a bit of time. (Maybe worth a new task, indeed.) [06:25:58] PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: (Service Check Timed Out) [07:03:23] 10Phabricator, 10Developer-Advocacy (Apr-Jun 2019): Re-evaluate our use of Phabricator Conpherence chat - https://phabricator.wikimedia.org/T127640 (10Aklapper) Thank you! [07:13:22] (03CR) 10Hashar: "It feels somehow wrong to have VisualEditor depend on ConfirmEdit andFlaggedRevs. That should really be the other way around. I am tempte" [integration/config] - 10https://gerrit.wikimedia.org/r/512972 (owner: 10Umherirrender) [07:32:40] (03CR) 10Hashar: [C: 03+2] [Kartographer] Add GeoData as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/512958 (owner: 10Umherirrender) [07:33:03] (03CR) 10Hashar: [C: 03+2] [ConfirmEdit] Add Math as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/512957 (owner: 10Umherirrender) [07:35:41] (03Merged) 10jenkins-bot: [Kartographer] Add GeoData as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/512958 (owner: 10Umherirrender) [07:36:22] (03Merged) 10jenkins-bot: [ConfirmEdit] Add Math as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/512957 (owner: 10Umherirrender) [07:50:17] (03CR) 10Hashar: [C: 03+2] Make it more clear that the codehealth patch job is non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/512443 (owner: 10Kosta Harlan) [07:50:44] (03CR) 10Hashar: [C: 03+2] Add BlueSpiceSocial [integration/config] - 10https://gerrit.wikimedia.org/r/512661 (owner: 10Pwirth) [07:51:53] (03Merged) 10jenkins-bot: Make it more clear that the codehealth patch job is non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/512443 (owner: 10Kosta Harlan) [07:52:22] (03Merged) 10jenkins-bot: Add BlueSpiceSocial [integration/config] - 10https://gerrit.wikimedia.org/r/512661 (owner: 10Pwirth) [08:00:17] (03Abandoned) 10Hashar: Allow project owners/wmde to rebase changes [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/498049 (https://phabricator.wikimedia.org/T218869) (owner: 10Hashar) [08:12:01] 10Beta-Cluster-Infrastructure, 10RelEng-Archive-FY201718-Q1, 10Release-Engineering-Team (Kanban), 10media-storage, 10Patch-For-Review: deployment-ms-be03.deployment-prep and deployment-ms-be04.deployment-prep have high load / system CPU - https://phabricator.wikimedia.org/T160990 (10hashar) 05Resolved→... [08:57:49] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10serviceops, 10Epic: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Joe) I don't think we can really read a yaml file at runtime. As php has no way to share information between... [08:58:54] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10serviceops, 10Epic: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Joe) we could even think of just keeping these files served statically by http from a centralized service, i... [09:39:28] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T220732 (10zeljkofilipin) @Tgr T224493 was closed as duplicate of T224491, and it's not added to train blockers. [09:47:28] (03PS1) 10Pwirth: Add BlueSpiceExtendedSeach as dependency for BlueSpiceSocial [integration/config] - 10https://gerrit.wikimedia.org/r/513071 [09:48:07] (03PS2) 10Pwirth: Add BlueSpiceExtendedSeach as dependency for BlueSpiceSocial [integration/config] - 10https://gerrit.wikimedia.org/r/513071 [09:48:26] (03CR) 10Pwirth: "I forgot to add 'BlueSpiceExtendedSearch' as a dependency, sorry" [integration/config] - 10https://gerrit.wikimedia.org/r/513071 (owner: 10Pwirth) [09:50:07] (03CR) 10jerkins-bot: [V: 04-1] Add BlueSpiceExtendedSeach as dependency for BlueSpiceSocial [integration/config] - 10https://gerrit.wikimedia.org/r/513071 (owner: 10Pwirth) [10:18:33] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Quibble, 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28), and 2 others: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests - https://phabricator.wikimedia.org/T199116 (10awight) a:03awight [10:19:30] zeljkof: hi! I just wanted to let you know that I'm interested in solving ^, there's a patch ready for CR--and I'm in CEST if you want to ping me any time. [10:20:14] awight: great! you should coordinate with hashar, I'm running train this and the next week, I won't be able to help much [10:20:37] ooh good luck with that. Hopefully things have settled down since the hackathon. [10:33:35] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair) [10:37:55] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10serviceops, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10MoritzMuehlenhoff) [11:05:29] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [11:24:23] Hello! Where do I get logstash beta credentials? [11:28:23] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [11:28:58] (03CR) 10Pwirth: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/513071 (owner: 10Pwirth) [11:30:22] (03CR) 10jerkins-bot: [V: 04-1] Add BlueSpiceExtendedSeach as dependency for BlueSpiceSocial [integration/config] - 10https://gerrit.wikimedia.org/r/513071 (owner: 10Pwirth) [11:30:28] (03PS3) 10Pwirth: Add BlueSpiceExtendedSeach as dependency for BlueSpiceSocial [integration/config] - 10https://gerrit.wikimedia.org/r/513071 [11:33:12] FYI, I found it. Turns out chrome doesn't show the login message so I added the hint for the location to the pages on Wikitech [12:33:00] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair) [13:09:09] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 35.71% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [13:09:26] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Operations: Reimage cobalt as stretch - https://phabricator.wikimedia.org/T176774 (10MoritzMuehlenhoff) [13:16:11] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [13:27:26] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T220732 (10zeljkofilipin) Group1 at 1.34.0-wmf.7 ([[ https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/513115 | gerrit ]], [[ https://... [13:35:02] 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, 10serviceops, and 3 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) [13:35:47] 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, 10serviceops, and 3 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) [13:39:30] 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, 10serviceops, and 3 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) If we're content to stick with simple Docker instances due to beta's relatively small scale, then I suggest... [13:47:22] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Test-Coverage: generate Wikibase.git code coverage on Jenkins - https://phabricator.wikimedia.org/T88434 (10Lucas_Werkmeister_WMDE) > There's some precedent with respect to how ConfirmEdit has "sub-extensions"... [13:54:32] 10Continuous-Integration-Infrastructure, 10Operations: Migrate contint* hosts to Stretch/Buster - https://phabricator.wikimedia.org/T224591 (10MoritzMuehlenhoff) [13:54:59] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair) [13:56:13] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Test-Coverage: generate Wikibase.git code coverage on Jenkins - https://phabricator.wikimedia.org/T88434 (10Lucas_Werkmeister_WMDE) I’ve uploaded a [WIP patch](https://gerrit.wikimedia.org/r/513126) moving `ex... [13:59:00] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10Krenair) Is apertium part of the cxserver migration? [14:00:44] (03PS1) 10TK-999: Update my zuul email address to match my Gerrit primary email [integration/config] - 10https://gerrit.wikimedia.org/r/513129 [14:03:42] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [14:24:18] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [14:49:42] (03CR) 10Jforrester: [C: 03+1] "We can have both if you're going to trigger this often, BTW." [integration/config] - 10https://gerrit.wikimedia.org/r/513129 (owner: 10TK-999) [14:52:30] (03CR) 10TK-999: "> Patch Set 1: Code-Review+1" [integration/config] - 10https://gerrit.wikimedia.org/r/513129 (owner: 10TK-999) [14:59:01] File "/usr/lib/python2.7/dist-packages/scap/opcache_manager.py", line 3, in [14:59:01] from concurrent.futures import ThreadPoolExecutor [14:59:01] ImportError: No module named concurrent.futures [14:59:12] (03CR) 10Krinkle: "As I understand it, the Node 10 upgrade is blocked on this, not the other way around, because we cannot atomically upgrade all tests at on" [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116) (owner: 10Awight) [14:59:57] exists in python3 but not 2.7 [15:02:36] (03CR) 10Krinkle: Run node browser tests in each repo (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116) (owner: 10Awight) [15:05:17] (03CR) 10Jforrester: "> Patch Set 3:" [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116) (owner: 10Awight) [15:11:33] https://gerrit.wikimedia.org/r/#/c/mediawiki/tools/scap/+/490888/ makes it look like there was supposed to be polyfill for this [15:12:56] installed the python-concurrent.futures apt package [15:17:11] 10Gerrit: Cannot log into gerrit: 'Cannot assign user name "want" to account 7082; name already in use.' - https://phabricator.wikimedia.org/T222336 (10Want) 05Open→03Resolved a:03Want The problem was solved during Hackaton 2019. [15:35:08] 10Beta-Cluster-Infrastructure: Figure out future for newly created deployment-prep jessie instances - https://phabricator.wikimedia.org/T218609 (10Krenair) I made deployment-sessionstore02 for it but couldn't get cassandra to work yet. [15:40:31] 10Beta-Cluster-Infrastructure: Figure out future for newly created deployment-prep jessie instances - https://phabricator.wikimedia.org/T218609 (10Krenair) I pressed some buttons and it started. Success? [15:58:10] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: deployment-mediawiki-09 PHP7 has broken cache of codebase? - https://phabricator.wikimedia.org/T219242 (10Krinkle) [16:02:03] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10serviceops, 10Epic: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) >>! In T223602#5219845, @Joe wrote: > I don't think we can really read a yaml file at runti... [16:04:26] (03CR) 10Krinkle: "Good point. Right now the deps are only effective for CI in one place (core), however the tests are not in one place, they are everywhere," [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116) (owner: 10Awight) [16:19:45] (03PS1) 10Umherirrender: [ConfirmEdit] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/513153 [16:26:30] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair) [16:31:25] (03CR) 10Jforrester: "> Patch Set 3:" [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116) (owner: 10Awight) [16:32:49] (03PS1) 10Umherirrender: [Kartographer] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/513157 [16:48:27] 10Release-Engineering-Team (Kanban), 10Repository-Admins: WebAuthn repo isn't in extensions.git - https://phabricator.wikimedia.org/T224191 (10Jdforrester-WMF) [16:48:38] 10Release-Engineering-Team (Kanban), 10TimedMediaHandler, 10MW-1.34-notes (1.34.0-wmf.5; 2019-05-14), 10Wikimedia-production-error: commonswiki: id3handler ErrorException from line 16 of WebMHandler.php: PHP Notice: Undefined index: comments - https://phabricator.wikimedia.org/T223445 (10Jdforrester-WMF) [16:48:53] (03CR) 10Florianschmidtwelzow: "Anything else that needs to be done here? :)" [integration/config] - 10https://gerrit.wikimedia.org/r/512729 (https://phabricator.wikimedia.org/T224371) (owner: 10Florianschmidtwelzow) [16:49:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10PHP 7.2 support: Create composer test containers for php7.2 - https://phabricator.wikimedia.org/T223428 (10Jdforrester-WMF) [16:49:55] 10Release-Engineering-Team (Kanban), 10Growth-Team, 10Notifications, 10MW-1.34-notes (1.34.0-wmf.5; 2019-05-14): Echo Selenium tests - Build failure - Cannot find module wdio-mediawiki/Util - https://phabricator.wikimedia.org/T223150 (10Jdforrester-WMF) [16:50:15] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-WikibaseRepository, 10Performance-Team, 10Wikidata, and 2 others: ErrorException from line 155 of /srv/mediawiki/php-1.34.0-wmf.4/extensions/Wikibase/lib/includes/Formatters/CachingKartographerEmbeddingHan... - https://phabricator.wikimedia.org/T223085 [16:50:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10PHP 7.2 support: Create composer test containers for php7.2 - https://phabricator.wikimedia.org/T223428 (10Jdforrester-WMF) p:05Triage→03Normal [16:52:27] 10Release-Engineering-Team (Kanban), 10GitHub-Mirrors: Request wikimedia team membership at github - https://phabricator.wikimedia.org/T222095 (10Jdforrester-WMF) [16:52:38] 10Release-Engineering-Team (Kanban), 10Wikibase-Containers, 10Wikidata, 10Wikimedia-production-error (Shared Build Failure): Wikibase's ItemMoveTest blocking merge in MediaWiki core, other places? - https://phabricator.wikimedia.org/T222389 (10Jdforrester-WMF) [17:14:59] 10Continuous-Integration-Config, 10Patch-For-Review, 10User-Addshore: Allow use of phan 0.8.5+ in wikimedia CI - https://phabricator.wikimedia.org/T174339 (10Umherirrender) 05Open→03Resolved Now on 1.3.4 [17:20:02] (03PS1) 10Hashar: Use more explicit jobs for Parsoid service [integration/config] - 10https://gerrit.wikimedia.org/r/513168 (https://phabricator.wikimedia.org/T221872) [17:20:04] 10Continuous-Integration-Config, 10Parsoid, 10phan: composer-package-php73-docker seems to fail often on Parsoid builds - https://phabricator.wikimedia.org/T221872 (10hashar) The job roams on all CI instances we have, and some are rather small / low memory ones (2GB instances iirc). For composer test it is u... [17:29:43] (03PS1) 10Hashar: Remove unused mwgate composer-package jobs [integration/config] - 10https://gerrit.wikimedia.org/r/513170 [17:29:45] (03PS1) 10Hashar: Run repo specific composer jobs with more memory [integration/config] - 10https://gerrit.wikimedia.org/r/513171 (https://phabricator.wikimedia.org/T221872) [17:31:14] 10Release-Engineering-Team (Kanban), 10MediaWiki-Release-Tools: merge branch.py and make-wmf-branch - https://phabricator.wikimedia.org/T222829 (10MarkAHershberger) Working on this [[ https://github.com/hexmode/mw-release | here ]]. See especially [[ https://github.com/hexmode/mw-release/blob/master/composer.... [17:33:04] 10Continuous-Integration-Config, 10Parsoid, 10Patch-For-Review, 10phan: composer-package-php73-docker seems to fail often on Parsoid builds - https://phabricator.wikimedia.org/T221872 (10hashar) I am in holidays time for the next few days but the patches should fix it: https://gerrit.wikimedia.org/r/5131... [17:34:26] 10Continuous-Integration-Config, 10Fundraising-Backlog: Run wikimedia/fundraising/crm CI jobs on PHP7x, not PHP5x - https://phabricator.wikimedia.org/T223348 (10hashar) That is blocked on migrating the Jenkins job toward a Docker container (T210287). I haven't had the opportunity to jump on that horse yet unfo... [17:36:54] 10Release-Engineering-Team (Kanban), 10Scap, 10MediaWiki-ResourceLoader, 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28), and 3 others: Scap deployments are not purging MessageBlobStore (was: Stale localized messages) - https://phabricator.wikimedia.org/T222539 (10Catrope) Thanks for cleaning this up, @Krinkle... [17:39:02] 10Continuous-Integration-Infrastructure (Slipway), 10Wikimedia-Fundraising-CiviCRM: Migrate wikimedia-fundraising-civicrm to a Docker container - https://phabricator.wikimedia.org/T210287 (10hashar) The job is defined in integration/config.git in `jjb/wm-fundraising.yaml`. It got written by/with @awight and @... [17:45:03] (03CR) 10Catrope: clear_message_blobs: call once (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/512219 (https://phabricator.wikimedia.org/T222539) (owner: 10Thcipriani) [17:45:31] 10Release-Engineering-Team (Kanban), 10MediaWiki-Release-Tools, 10Core Platform Team (Release Automation (TEC1)), 10Core Platform Team Kanban (Contractor - Doing), 10MediaWiki-Releasing (Workflow Improvements): merge branch.py and make-wmf-branch - https://phabricator.wikimedia.org/T222829 (10CCicalese_WM... [17:45:48] 10Release-Engineering-Team (Kanban), 10MediaWiki-Release-Tools, 10Core Platform Team (Release Automation (TEC1)), 10Core Platform Team Kanban (Contractor - Doing), 10MediaWiki-Releasing (Workflow Improvements): merge branch.py and make-wmf-branch - https://phabricator.wikimedia.org/T222829 (10CCicalese_WM... [17:48:56] 10Release-Engineering-Team (Kanban), 10MediaWiki-Release-Tools, 10Core Platform Team (Release Automation (TEC1)), 10Core Platform Team Kanban (Contractor - Doing), 10MediaWiki-Releasing (Workflow Improvements): merge branch.py and make-wmf-branch - https://phabricator.wikimedia.org/T222829 (10mmodell) I... [17:50:53] 10Release-Engineering-Team (Kanban), 10MediaWiki-Release-Tools, 10Core Platform Team (Release Automation (TEC1)), 10Core Platform Team Kanban (Contractor - Doing), 10MediaWiki-Releasing (Workflow Improvements): merge branch.py and make-wmf-branch - https://phabricator.wikimedia.org/T222829 (10mmodell) I'... [17:58:59] (03PS5) 10Krinkle: clear_message_blobs: call once [tools/scap] - 10https://gerrit.wikimedia.org/r/512219 (https://phabricator.wikimedia.org/T222539) (owner: 10Thcipriani) [17:59:06] (03CR) 10Krinkle: "thx, amended accordingly" [tools/scap] - 10https://gerrit.wikimedia.org/r/512219 (https://phabricator.wikimedia.org/T222539) (owner: 10Thcipriani) [18:49:57] 10Release-Engineering-Team (Kanban), 10MediaWiki-Release-Tools, 10Core Platform Team (Release Automation (TEC1)), 10Core Platform Team Kanban (Contractor - Doing), 10MediaWiki-Releasing (Workflow Improvements): merge branch.py and make-wmf-branch - https://phabricator.wikimedia.org/T222829 (10MarkAHershbe... [18:58:35] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10WDoranWMF) I am an engineering manager for Core Platform Team, I'd like to get permission to create projects [19:00:18] (03CR) 10Umherirrender: "you have to use phan ;-)" [integration/config] - 10https://gerrit.wikimedia.org/r/512972 (owner: 10Umherirrender) [19:32:32] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10serviceops, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [19:34:31] 10Continuous-Integration-Config, 10Parsoid, 10Patch-For-Review, 10phan: composer-package-php73-docker seems to fail often on Parsoid builds - https://phabricator.wikimedia.org/T221872 (10cscott) We could probably also tweak the `.phan/config.php` to exclude more stuff in `vendor/`, including phan itself. [19:44:26] (03PS1) 10Brennen Bearnes: Allow project owners to push to refs/tags/* [unicodejs] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/513194 (https://phabricator.wikimedia.org/T224618) [19:45:22] (03CR) 10Paladox: [C: 03+2] Allow project owners to push to refs/tags/* [unicodejs] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/513194 (https://phabricator.wikimedia.org/T224618) (owner: 10Brennen Bearnes) [19:50:16] 10Phabricator, 10CommRel-Design, 10CommRel-Internals: Create Phabricator form for CommRel-Design and Comms requests and add a link to it in the "Star" dropdown - https://phabricator.wikimedia.org/T223102 (10hdothiduc) Thank you @Aklapper for the heads up. For the Phab Herald rule, should I simply create a ta... [20:02:40] (03PS1) 10Kosta Harlan: Add configuration for logging to stderr [releng/local-charts] - 10https://gerrit.wikimedia.org/r/513195 [20:31:28] (03Abandoned) 10Brennen Bearnes: Allow project owners to push to refs/tags/* [unicodejs] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/513194 (https://phabricator.wikimedia.org/T224618) (owner: 10Brennen Bearnes) [20:31:34] 10Phabricator, 10CommRel-Design, 10CommRel-Internals: Create Phabricator form for CommRel-Design and Comms requests and add a link to it in the "Star" dropdown - https://phabricator.wikimedia.org/T223102 (10Quiddity) >>! In T223102#5222173, @hdothiduc wrote: > @Quiddity would it be ok to reach out to you, in... [20:33:00] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10serviceops, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['phab2001.codfw.wmnet'] ` Of which those *... [20:34:32] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10serviceops, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [20:40:12] (03PS1) 10Brennen Bearnes: Allow owners to (force) push tags. [unicodejs] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/513204 (https://phabricator.wikimedia.org/T224618) [20:41:08] (03CR) 10Jforrester: [V: 03+2 C: 03+1] "Seems reasonable." [unicodejs] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/513204 (https://phabricator.wikimedia.org/T224618) (owner: 10Brennen Bearnes) [20:41:15] (03CR) 10Jforrester: [V: 03+2 C: 03+2] Allow owners to (force) push tags. [unicodejs] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/513204 (https://phabricator.wikimedia.org/T224618) (owner: 10Brennen Bearnes) [20:43:47] 10Release-Engineering-Team (Kanban), 10Gerrit-Privilege-Requests, 10Utilities-UnicodeJS: Add the VisualEditor group to be able to push (signed) tags on the UnicodeJS gerrit repo - https://phabricator.wikimedia.org/T224618 (10Jdforrester-WMF) 05Open→03Resolved a:03brennen Worked great: https://gerrit.wi... [20:48:05] (03PS1) 10Umherirrender: [CleanChanges] Add cldr as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/513206 [20:55:25] 10Continuous-Integration-Config, 10Fundraising-Backlog: Run wikimedia/fundraising/crm CI jobs on PHP7x, not PHP5x - https://phabricator.wikimedia.org/T223348 (10Eileenmcnaughton) @hashar do you have a sense of when - ie. is this something that might happen next quarter or is that unrealistic? [21:12:04] (03PS1) 10Umherirrender: [CentralNotice] Add dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/513211 [21:20:11] (03PS1) 10Umherirrender: [LiquidThreads] Add dependency [integration/config] - 10https://gerrit.wikimedia.org/r/513214 [21:23:01] (03PS1) 10Umherirrender: [Translate] Add dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/513217 [21:25:22] (03PS1) 10Umherirrender: [CodeReview] Add dependency [integration/config] - 10https://gerrit.wikimedia.org/r/513218 [21:37:38] Fair bit of executor starvation in CI. [21:38:57] beta-scap-eqiad say it's waiting on deployment-deploy01 but that's empty… [21:39:20] Same with beta-update-databases-eqiad. [21:40:07] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10serviceops, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['phab2001.codfw.wmnet'] ` Of which those *... [21:46:37] !log doing the executor deadlock dance: https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Jenkins_execution_lock [21:46:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:47:35] greg-g: That seems to have fixed things, thanks [21:48:51] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T220732 (10matmarex) [21:50:36] maybe... [21:51:40] there we go, *crosses fingers* [21:55:20] Project beta-scap-eqiad build #251266: 04FAILURE in 0.59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/251266/ [21:58:07] Project beta-scap-eqiad build #251267: 04STILL FAILING in 0.58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/251267/ [22:00:33] Well, that's at least a different kind of failure. [22:01:18] Project beta-scap-eqiad build #251268: 04STILL FAILING in 0.61 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/251268/ [22:04:21] Project beta-scap-eqiad build #251269: 04STILL FAILING in 0.52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/251269/ [22:14:15] 10Beta-Cluster-Infrastructure: beta-scap-eqiad failing – "Call to mwscript eval.php returned: None" - https://phabricator.wikimedia.org/T224634 (10Jdforrester-WMF) [22:14:24] Project beta-scap-eqiad build #251270: 04STILL FAILING in 0.75 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/251270/ [22:17:43] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10serviceops, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [22:20:05] Project beta-update-databases-eqiad build #34196: 04FAILURE in 3.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/34196/ [22:24:22] Project beta-scap-eqiad build #251271: 04STILL FAILING in 0.54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/251271/ [22:28:42] (03PS1) 10Dduvall: WIP: Fixing publish tag... [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/513228 [22:28:44] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Discovery-Search: beta-scap-eqiad failing – "Call to mwscript eval.php returned: None" - https://phabricator.wikimedia.org/T224634 (10Jdforrester-WMF) beta-update-databases-eqiad is also failing, with a specific error: /usr/local/bin/mwscript update.php --w... [22:28:45] (03PS1) 10Dduvall: pipeline: Work around Groovy CPS replaceAll limitations [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/513229 [22:29:31] (03Abandoned) 10Dduvall: WIP: Fixing publish tag... [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/513228 (owner: 10Dduvall) [22:34:22] Project beta-scap-eqiad build #251272: 04STILL FAILING in 0.6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/251272/ [22:43:28] Yippee, build fixed! [22:43:29] Project beta-update-databases-eqiad build #34197: 09FIXED in 1 min 21 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/34197/ [22:45:43] Tah-dah. [22:47:14] haha, we will never get rid of tin [22:47:16] Could not resolve host: tin.eqiad.wmnet [22:47:41] still happening when first deploying a new server that is pulling things [22:50:22] Yippee, build fixed! [22:50:23] Project beta-scap-eqiad build #251273: 09FIXED in 9 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/251273/ [22:51:52] 10Phabricator, 10CommRel-Design, 10CommRel-Internals: Create Phabricator form for CommRel-Design and Comms requests and add a link to it in the "Star" dropdown - https://phabricator.wikimedia.org/T223102 (10mcruzWMF) >>! In T223102#5222173, @hdothiduc wrote: > Thank you @Aklapper for the heads up. > For the... [22:52:21] mutante: Fun. [22:52:50] 10Phabricator, 10CommRel-Design, 10CommRel-Internals: Create Phabricator form for CommRel-Design and Comms requests and add a link to it in the "Star" dropdown - https://phabricator.wikimedia.org/T223102 (10mcruzWMF) >>! In T223102#5205712, @mcruzWMF wrote: >>>! In T223102#5203223, @mmodell wrote: >> I've ad... [22:53:09] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10CirrusSearch, 10Discovery-Search: beta-scap-eqiad failing – "Call to mwscript eval.php returned: None" - https://phabricator.wikimedia.org/T224634 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF [22:53:32] James_F: somehow it is in the cache dir. so manually editing something like iegreview-cache/.config to set to deploy1001 fixes it [22:54:30] New servers get the old caches from other servers copied to them? [22:55:40] (03PS2) 10Dduvall: pipeline: Work around Groovy CPS replaceAll limitations [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/513229 [22:56:01] James_F: greg-g: might be of interest - little tool to count incident reports by month/year; https://codepen.io/Krinkle/full/wbYMZK [22:56:50] Krinkle: neat, thanks! [22:57:38] (03PS1) 10Dduvall: Fixed `Utility.timestampLabel` format [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/513235 [22:58:32] (03PS1) 10Dduvall: pipeline: Test more of the stage steps implementation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/513236 [22:58:33] Very. [22:59:01] Krinkle: BTW, are you deploying https://gerrit.wikimedia.org/r/c/mediawiki/core/+/512531 to wmf.6 (you've got 12 hours before it rolls off prod), or https://gerrit.wikimedia.org/r/c/mediawiki/extensions/AbuseFilter/+/512949 to wmf.7? [22:59:07] (03CR) 10jerkins-bot: [V: 04-1] pipeline: Test more of the stage steps implementation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/513236 (owner: 10Dduvall) [22:59:53] James_F: former, probably not. latter, yes will do soon-ish. [23:00:22] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [23:00:24] (03CR) 10Dduvall: [C: 03+2] pipeline: Work around Groovy CPS replaceAll limitations [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/513229 (owner: 10Dduvall) [23:00:32] Kk. Am cleaning up open patches in deploy branches. [23:00:50] (03Merged) 10jenkins-bot: pipeline: Work around Groovy CPS replaceAll limitations [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/513229 (owner: 10Dduvall) [23:01:13] (03CR) 10Dduvall: [C: 03+2] Fixed `Utility.timestampLabel` format [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/513235 (owner: 10Dduvall) [23:01:17] (03CR) 10jenkins-bot: pipeline: Work around Groovy CPS replaceAll limitations [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/513229 (owner: 10Dduvall) [23:01:41] (03Merged) 10jenkins-bot: Fixed `Utility.timestampLabel` format [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/513235 (owner: 10Dduvall) [23:02:07] (03CR) 10jenkins-bot: Fixed `Utility.timestampLabel` format [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/513235 (owner: 10Dduvall) [23:09:10] (03PS1) 10Paladox: Replace tin.eqiad.wmnet with deploy1001.eqiad.wmnet [tools/scap] - 10https://gerrit.wikimedia.org/r/513238 [23:10:23] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [23:18:17] (03PS2) 10Paladox: Replace tin.eqiad.wmnet with deploy1001.eqiad.wmnet [tools/scap] - 10https://gerrit.wikimedia.org/r/513238 [23:19:40] (03CR) 10jerkins-bot: [V: 04-1] Replace tin.eqiad.wmnet with deploy1001.eqiad.wmnet [tools/scap] - 10https://gerrit.wikimedia.org/r/513238 (owner: 10Paladox) [23:21:11] (03PS3) 10Paladox: Replace tin.eqiad.wmnet with deploy1001.eqiad.wmnet [tools/scap] - 10https://gerrit.wikimedia.org/r/513238 [23:24:39] (03CR) 10jerkins-bot: [V: 04-1] Replace tin.eqiad.wmnet with deploy1001.eqiad.wmnet [tools/scap] - 10https://gerrit.wikimedia.org/r/513238 (owner: 10Paladox) [23:24:57] 10Release-Engineering-Team (Kanban), 10Scap: php-1.33.0-wmf.23/cache/l10n isn't cleaned up in prod - https://phabricator.wikimedia.org/T224637 (10Jdforrester-WMF) [23:26:43] (03PS4) 10Paladox: Replace tin.eqiad.wmnet with deploy1001.eqiad.wmnet [tools/scap] - 10https://gerrit.wikimedia.org/r/513238 [23:31:29] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10serviceops, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Dzahn) [23:43:52] (03PS2) 10Dduvall: pipeline: Test more of the stage steps implementation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/513236 [23:48:47] (03PS3) 10Dduvall: pipeline: Test more of the stage steps implementation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/513236