[01:06:25] 10phan, 10MediaWiki-General, 10User-DannyS712: PhanRedundantCondition is still suppressed - https://phabricator.wikimedia.org/T248438 (10DannyS712) [01:13:10] 10phan, 10MediaWiki-General, 10User-DannyS712: redundant_condition_detection is still suppressed - https://phabricator.wikimedia.org/T248438 (10DannyS712) [01:19:32] 10phan, 10MediaWiki-General, 10User-DannyS712: redundant_condition_detection is still suppressed - https://phabricator.wikimedia.org/T248438 (10DannyS712) PhanRedundantCondition: https://gerrit.wikimedia.org/r/#/q/topic:PhanRedundantCondition [01:32:47] PROBLEM - Free space - all mounts on deployment-poolcounter06 is CRITICAL: (Service Check Timed Out) [01:37:36] RECOVERY - Free space - all mounts on deployment-poolcounter06 is OK: OK: All targets OK [01:52:36] 10phan-taint-check-plugin, 10Security-Team, 10MW-1.35-notes (1.35.0-wmf.25; 2020-03-24), 10Readers-Web-Backlog (Kanbanana-2019-20-Q3), 10Security: Phan warning in "includes/specials/SpecialMobileDiff.php" - working around by disabling - https://phabricator.wikimedia.org/T248360 (10sbassett) >>! In T24836... [06:56:23] 10LibUp, 10MediaWiki-General, 10MediaWiki-Vendor, 10Patch-For-Review: jakub-onderka/php-parallel-lint has been abandoned, use php-parallel-lint/php-parallel-lint instead - https://phabricator.wikimedia.org/T247827 (10Legoktm) Also filed https://github.com/php-parallel-lint/PHP-Console-Highlighter/issues/1... [08:13:29] RECOVERY - Free space - all mounts on integration-puppetmaster-02 is OK: OK: All targets OK [09:08:45] James_F: Done https://wikitech.wikimedia.org/wiki/User:RhinosF1/ve-enable - feel free to put somewhere useful, make more standard, fix any mistakes (hopefully none) etc. [09:09:51] it (pretty much) works for any dblist change and should be simple for any newbie that knows git [09:14:15] Reedy: you were there the other day so it feel free to look [12:15:23] 10phan-taint-check-plugin, 10MediaWiki-extensions-Translate: Add phan-taint-check-plugin to Translate extension - https://phabricator.wikimedia.org/T202379 (10Nikerabbit) [12:38:28] (03PS1) 10Lars Wirzenius: Add detailed release procedure documentation [tools/scap] - 10https://gerrit.wikimedia.org/r/583319 [12:39:12] (03CR) 10Lars Wirzenius: "This is meant to expand on" [tools/scap] - 10https://gerrit.wikimedia.org/r/583319 (owner: 10Lars Wirzenius) [12:46:44] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Scap: Make scap release with --canary-wait-time and integration test changes - https://phabricator.wikimedia.org/T246455 (10LarsWirzenius) Wrote up a more detailed checklist for making a Scap release. https://gerrit.wikimedia.org/r/c/mediawiki/tools/... [12:49:34] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Scap: Scap fails if --canary-wait-time isn't used - https://phabricator.wikimedia.org/T248467 (10LarsWirzenius) [12:50:26] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Scap: Scap fails if --canary-wait-time isn't used - https://phabricator.wikimedia.org/T248467 (10LarsWirzenius) p:05Triage→03High [12:50:52] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Scap: Make scap release with --canary-wait-time and integration test changes - https://phabricator.wikimedia.org/T246455 (10LarsWirzenius) [12:50:53] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Scap: Scap fails if --canary-wait-time isn't used - https://phabricator.wikimedia.org/T248467 (10LarsWirzenius) [12:51:15] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Scap: Make scap release with --canary-wait-time and integration test changes - https://phabricator.wikimedia.org/T246455 (10LarsWirzenius) [12:51:17] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Scap: Scap fails if --canary-wait-time isn't used - https://phabricator.wikimedia.org/T248467 (10LarsWirzenius) [12:51:36] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Scap: Make scap release with --canary-wait-time and integration test changes - https://phabricator.wikimedia.org/T246455 (10LarsWirzenius) [12:51:38] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Scap: Scap fails if --canary-wait-time isn't used - https://phabricator.wikimedia.org/T248467 (10LarsWirzenius) [13:05:38] (03PS4) 10Kosta Harlan: Provide support for setting default configuration in dev image [releng/dev-images] - 10https://gerrit.wikimedia.org/r/582498 (https://phabricator.wikimedia.org/T247072) [13:06:56] (03CR) 10Kosta Harlan: [C: 04-1] "> Patch Set 3:" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/582498 (https://phabricator.wikimedia.org/T247072) (owner: 10Kosta Harlan) [14:19:36] zeljkof: should we change core's selenium tests to work without Vector? [14:20:00] kostajh: I'm not sure if that would work at all [14:20:34] I think we would just need to change some of the selectors [14:20:36] the same tests need to work in various environments (mw-docker, mw-vagrant, CI, beta cluster...) and all of them have vector [14:20:59] if we make it work in mw-docker without vector, they would fail in other environments [14:22:04] (03PS2) 10Hashar: jjb: migrate refinery release job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/583092 (https://phabricator.wikimedia.org/T210271) [14:27:24] like, one of the errors is that #firstHeading is not found [14:27:33] core provides a class, `.firstHeading` by default [14:28:07] Vector provides both an ID (`#firstHeading`) and a class (`.firstHeading`) so if we change the relevant code to target the class which is used by both core and Vector, we'd drop that dependency on Vector for the tests [14:28:58] kostajh: well, try it out :) [14:33:07] (03PS3) 10Hashar: jjb: migrate refinery release job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/583092 (https://phabricator.wikimedia.org/T210271) [14:43:20] (03PS1) 10Lars Wirzenius: Change scripts/check to be more explicit, especially about success [tools/scap] - 10https://gerrit.wikimedia.org/r/583351 [14:44:33] (03CR) 10Lars Wirzenius: "As part of my changes to make the Scap release process more easily done by a newbie (like I've been), I think it makes sense for the check" [tools/scap] - 10https://gerrit.wikimedia.org/r/583351 (owner: 10Lars Wirzenius) [14:51:56] (03PS1) 10Hashar: docker: bump java and add ssh to java8 [integration/config] - 10https://gerrit.wikimedia.org/r/583356 (https://phabricator.wikimedia.org/T210271) [14:53:06] (03PS4) 10Hashar: jjb: migrate refinery release job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/583092 (https://phabricator.wikimedia.org/T210271) [14:55:00] (03CR) 10Hashar: [C: 03+2] docker: bump java and add ssh to java8 [integration/config] - 10https://gerrit.wikimedia.org/r/583356 (https://phabricator.wikimedia.org/T210271) (owner: 10Hashar) [14:55:54] (03Merged) 10jenkins-bot: docker: bump java and add ssh to java8 [integration/config] - 10https://gerrit.wikimedia.org/r/583356 (https://phabricator.wikimedia.org/T210271) (owner: 10Hashar) [15:05:07] zeljkof: challenge accepted :) [15:19:38] kostajh: :) [15:27:00] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Patch-For-Review: Drop php56 testing support - https://phabricator.wikimedia.org/T224906 (10Dzahn) [15:27:03] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10Dzahn) 05Open→03Stalled @hashar Per your c... [15:28:17] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10MoritzMuehlenhoff) Nothing can be blocked on 5... [15:31:07] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Operations, 10serviceops: Upload docker-ce 18.06.3 upstream package for Stretch - https://phabricator.wikimedia.org/T226236 (10Dzahn) >>! In T226236#5505636, @MoritzMuehlenhoff wrote: >>>! In T226236#5505608, @hashar wrote: >> Anywa,y I am declini... [15:31:35] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10Dzahn) also see T226236#5998777 [15:33:15] (03PS1) 10Hashar: docker: in java8: use https to download maven [integration/config] - 10https://gerrit.wikimedia.org/r/583364 [15:37:16] (03CR) 10Hashar: [C: 03+2] docker: in java8: use https to download maven [integration/config] - 10https://gerrit.wikimedia.org/r/583364 (owner: 10Hashar) [15:38:06] (03Merged) 10jenkins-bot: docker: in java8: use https to download maven [integration/config] - 10https://gerrit.wikimedia.org/r/583364 (owner: 10Hashar) [15:43:50] (03CR) 10Thcipriani: "Looks like good documentation. Made some inline suggestions/nits." (035 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/583319 (owner: 10Lars Wirzenius) [15:48:26] (03CR) 10Thcipriani: [C: 03+2] Change scripts/check to be more explicit, especially about success [tools/scap] - 10https://gerrit.wikimedia.org/r/583351 (owner: 10Lars Wirzenius) [15:52:19] (03Merged) 10jenkins-bot: Change scripts/check to be more explicit, especially about success [tools/scap] - 10https://gerrit.wikimedia.org/r/583351 (owner: 10Lars Wirzenius) [16:03:33] (03PS1) 10Hashar: dockerfiles: java8: add ssh host key for Gerrit [integration/config] - 10https://gerrit.wikimedia.org/r/583374 (https://phabricator.wikimedia.org/T210271) [16:04:28] (03CR) 10Hashar: [C: 03+2] dockerfiles: java8: add ssh host key for Gerrit [integration/config] - 10https://gerrit.wikimedia.org/r/583374 (https://phabricator.wikimedia.org/T210271) (owner: 10Hashar) [16:05:20] (03Merged) 10jenkins-bot: dockerfiles: java8: add ssh host key for Gerrit [integration/config] - 10https://gerrit.wikimedia.org/r/583374 (https://phabricator.wikimedia.org/T210271) (owner: 10Hashar) [16:06:37] (03PS1) 10Lars Wirzenius: Fix scap sync without -w (--canary-wait-time) [tools/scap] - 10https://gerrit.wikimedia.org/r/583375 [16:07:21] (03CR) 10Lars Wirzenius: "This fixes https://phabricator.wikimedia.org/T248467" [tools/scap] - 10https://gerrit.wikimedia.org/r/583375 (owner: 10Lars Wirzenius) [16:07:42] 10Phabricator: "AphrontDuplicateKeyQueryException" when trying to drag a task from (sub-project) column to other column on parent workboard - https://phabricator.wikimedia.org/T139396 (10Aklapper) 05Open→03Declined No reply, hence declining. If this still happens, please set the status of this task back to "... [16:13:15] (03PS1) 10Hashar: dockerfiles: java8: add Gerrit IPv4 to ssh known hosts [integration/config] - 10https://gerrit.wikimedia.org/r/583379 [16:13:32] (03CR) 10Hashar: [C: 03+2] dockerfiles: java8: add Gerrit IPv4 to ssh known hosts [integration/config] - 10https://gerrit.wikimedia.org/r/583379 (owner: 10Hashar) [16:14:20] (03Merged) 10jenkins-bot: dockerfiles: java8: add Gerrit IPv4 to ssh known hosts [integration/config] - 10https://gerrit.wikimedia.org/r/583379 (owner: 10Hashar) [16:19:05] (03CR) 10Lars Wirzenius: "I'm making changes as suggested and pushing a new version momentarily." (035 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/583319 (owner: 10Lars Wirzenius) [16:20:11] (03PS2) 10Lars Wirzenius: Add detailed release procedure documentation [tools/scap] - 10https://gerrit.wikimedia.org/r/583319 [16:21:19] (03PS2) 10Jforrester: Fix scap sync without -w (--canary-wait-time) [tools/scap] - 10https://gerrit.wikimedia.org/r/583375 (https://phabricator.wikimedia.org/T248467) (owner: 10Lars Wirzenius) [16:22:03] (03CR) 10Jforrester: [C: 03+1] Add detailed release procedure documentation (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/583319 (owner: 10Lars Wirzenius) [16:28:02] 10phan-taint-check-plugin, 10Security-Team, 10MW-1.35-notes (1.35.0-wmf.25; 2020-03-24), 10Patch-For-Review, and 2 others: Phan warning in "includes/specials/SpecialMobileDiff.php" - working around by disabling - https://phabricator.wikimedia.org/T248360 (10Jdlrobson) @matmarex @sbassett looks like we ca... [16:33:28] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Analytics, and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10hashar) I have tried the job, a... [17:01:41] hi releng! [17:01:44] 10Release-Engineering-Team-TODO, 10Scap: Running "scap pull" on deploy1001 halts for 2 minutes, then reports a php-fpm error - https://phabricator.wikimedia.org/T246959 (10thcipriani) p:05Triage→03Medium FWIW, the pausing and the error are two different pieces. That is, the pause is cdb rebuild IIRC. The e... [17:01:50] I'm seeing spurious CI failures [17:01:54] with 'Could not authenticate against github.com [17:01:56] '' [17:02:07] for example: https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-docker/2344/console [17:02:26] Reedy took a look at a similar one yesterday and said a github API token had expired [17:02:49] and he was able to remove it somewhere to get things working again [17:03:26] would that be inside the docker image, or just in a homedir in the integration-agent-docker-* machines? [17:04:01] Ahh, I see there's a ticket for it now [17:10:55] hmm, so Reedy fixed it by deleting a file in someone's ~/.composer, but I'm not finding that dir on integration-agent-docker-1005 [17:10:59] > 19:59 < Reedy> Found the solution: rm ~/.composer/auth.json [17:11:21] Looking in the homedirs of root and jenkins-deploy at least [17:11:36] hi all, I need to request access to deployment-eventlog05.eqiad.wmflabs so I can check event logs for the iOS mobile app beta. Username is snowick. Let me know if this is the right place to request, thanks [17:13:18] snowick: https://wikitech.wikimedia.org/wiki/User:Snowick is not registered, what is your Developer Account username? [17:13:21] PROBLEM - Free space - all mounts on integration-agent-docker-1001 is CRITICAL: CRITICAL: integration.integration-agent-docker-1001.diskspace._srv.byte_percentfree (<10.00%) [17:13:37] Developer Account == wikitech account [17:14:12] nothing comes back from "root@integration-agent-docker-1005:~# find / -type d -name '.composer'" ... maybe the dir is inside a docker image? [17:14:39] ejegg: it is indeed [17:15:31] ahh, thanks greg-g - do you have the command line handy for getting into the image? [17:17:00] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Scap, 10Wikimedia-Site-requests: Move scap plugins from mediawiki-config repo to scap repo - https://phabricator.wikimedia.org/T248490 (10Jdforrester-WMF) [17:17:02] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-DannyS712, 10ci-test-error: operations/mediawiki-config master branch failing tests - https://phabricator.wikimedia.org/T248040 (10thcipriani) p:05High→03Medium We have this switched off linting for the moment, planning on moving these plu... [17:17:47] I don't, sorry, releng team is in their weekly right now doing task triage, meeting ends in 45 [17:17:57] James_F: did you see my earlier ping [17:18:03] @greg-g sorry, have to check that, my full name is Shay Nowick [17:19:26] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10MediaWiki-Docker, 10dev-images, 10User-brennen: ffmpeg in MW-Docker lacks -row-mt option for improved multithreaded VP9 encoding - https://phabricator.wikimedia.org/T247153 (10thcipriani) a:03brennen [17:19:44] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10MediaWiki-Docker, 10dev-images, 10User-brennen: ffmpeg in MW-Docker lacks -row-mt option for improved multithreaded VP9 encoding - https://phabricator.wikimedia.org/T247153 (10thcipriani) p:05Triage→03... [17:20:38] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10MediaWiki-Docker, 10dev-images, 10User-brennen: Job queue runners for MediaWiki-Docker - https://phabricator.wikimedia.org/T246935 (10thcipriani) p:05Triage→03Medium a:03brennen [17:21:24] @greg-g SNowick_WMF is most likely, this can wait until after your meeting [17:21:32] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10MediaWiki-Docker, 10User-brennen: TimedMediaHandler's ffmpeg processes get stuck when using resource limits on Docker image - https://phabricator.wikimedia.org/T246942 (10thcipriani) p:05Triage→03Medium a:03brennen [17:22:23] snowick: also not it https://wikitech.wikimedia.org/wiki/User:SNowick_WMF [17:22:28] do you have a wikitech account? :) [17:22:38] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10BlueSpice: BlueSpice related tests fails on gate-and-submit-1.31 in quibble-composer tests - https://phabricator.wikimedia.org/T235807 (10thcipriani) [17:23:22] (03PS5) 10Hashar: jjb: migrate refinery release job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/583092 (https://phabricator.wikimedia.org/T210271) [17:23:23] RECOVERY - Free space - all mounts on integration-agent-docker-1001 is OK: OK: All targets OK [17:23:37] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Scap, 10Developer Productivity, 10Patch-For-Review: Scap's link to Logstash for canary failure is not in sync with the monitoring query - https://phabricator.wikimedia.org/T247005 (10thcipriani) a:03Jdforrester-WMF [17:25:54] snowick, greg-g: We don't allow "WMF" names on wikitech, so it sounds like snowick needs to do the things from https://www.mediawiki.org/wiki/Developer_account first [17:27:27] "don't allow" == TitleBlacklist bans on those names. Developer accounts are about people, not the orgs they work for at any given time. [17:27:39] I do have a wikitech account, my login is Shay Nowick - https://wikitech.wikimedia.org/wiki/User:Shay_Nowick [17:28:16] aha, good deal [17:29:06] Sorry, they use that _WMF LDAP for access to some of the analytics servers, I get them mixed up [17:30:23] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Code-Review-Workgroup: Define code review metrics - https://phabricator.wikimedia.org/T229510 (10Jdforrester-WMF) [17:30:39] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Wikimedia-Site-requests, 10WikimediaMessages: Put "shim" code for namespaces, logs, and log i18n into WikimediaMessages so we can undeploy extensions - https://phabricator.wikimedia.org/T222918 (10Jdforrester-WMF) [17:30:55] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Epic, 10Technical-Debt, 10ci-test-error: Figure out which production extensions have conflicting tests and fix them - https://phabricator.wikimedia.org/T238492 (10Jdforrester-WMF) [17:31:36] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Growth-Team, 10MediaWiki-extensions-GettingStarted, 10User-notice: Deprecate/undeploy the GettingStarted extension - https://phabricator.wikimedia.org/T235752 (10Jdforrester-WMF) [17:31:38] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10User-brennen, 10User-greg: Underrun and CI capacity - https://phabricator.wikimedia.org/T244515 (10thcipriani) [17:31:43] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Parsoid, 10Patch-For-Review: Undeploy ParsoidBatchAPI from the Wikimedia cluster - https://phabricator.wikimedia.org/T242430 (10Jdforrester-WMF) [17:32:10] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10JavaScript: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 (10Jdforrester-WMF) [17:32:45] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10VisualEditor, 10Editing-team (Tracking): Firefox CI tests keep failing in VE - https://phabricator.wikimedia.org/T240955 (10Jdforrester-WMF) [17:34:04] 10Phabricator: move_project breaks the world when moving a subproject that already has subprojects. - https://phabricator.wikimedia.org/T242254 (10mmodell) 05Open→03Stalled [17:42:39] 10Gerrit, 10Vector, 10Readers-Web-Backlog (Kanbanana-2019-20-Q3): Default to feature branch in Vector - https://phabricator.wikimedia.org/T247890 (10ovasileva) 05Open→03Declined Declined based on conversation above. For now, we will proceed without a feature branch, with extra care for minimizing risk s... [17:43:12] snowick: you are now a member of deployment-prep, you should have the access you need. [17:43:32] thank you, appreciate it [17:44:49] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Delete Jenkins label DebianJessie - https://phabricator.wikimedia.org/T239981 (10Jdforrester-WMF) p:05Medium→03High [17:44:59] !log Added Shay_Nowick to deployment-prep [17:44:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:47:20] So I am trying to `ssh deployment-eventlog05.eqiad.wmflabs` and getting `ssh: Could not resolve hostname deployment-eventlog05.eqiad.wmflabs: Name or service not known`, am I doing something wrong? [17:48:03] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Wikimedia-General-or-Unknown, 10PHP 7.4 support: Make Wikimedia Production MediaWiki compatible with PHP 7.4 - https://phabricator.wikimedia.org/T247658 (10Jdforrester-WMF) [17:49:23] try `deployment-eventlog05.deployment-prep.eqiad.wmflabs` [17:49:32] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10User-DannyS712, 10ci-test-error: operations/mediawiki-config master branch failing tests - https://phabricator.wikimedia.org/T248040 (10Jdforrester-WMF) [17:49:45] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10User-DannyS712, 10ci-test-error: operations/mediawiki-config master branch failing tests - https://phabricator.wikimedia.org/T248040 (10Jdforrester-WMF) [17:49:47] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Scap, 10Wikimedia-Site-requests: Move scap plugins from mediawiki-config repo to scap repo - https://phabricator.wikimedia.org/T248490 (10Jdforrester-WMF) [17:50:08] 10Release-Engineering-Team (Pipeline), 10Analytics, 10Analytics-Kanban, 10Release Pipeline, and 2 others: Migrate EventStreams to k8s deployment pipeline - https://phabricator.wikimedia.org/T238658 (10Ottomata) Ah we need to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/583073 first, before th... [17:51:51] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10MediaWiki-Docker, 10dev-images, 10User-brennen: Demonstrate abilities of MediaWiki-Docker with a full TimedMediaHandler stack - https://phabricator.wikimedia.org/T247037 (10Jdforrester-WMF) [17:51:54] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10MediaWiki-Docker, 10Developer Productivity, and 3 others: Command-line wrapper for interacting with core's docker-compose stack - https://phabricator.wikimedia.org/T246111 (10Jdforrester-WMF) [17:51:57] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10MediaWiki-Docker, 10dev-images, 10User-brennen: dev-images: reorganize to reduce various image sizes - https://phabricator.wikimedia.org/T245443 (10Jdforrester-WMF) [17:52:00] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10MediaWiki-Docker, 10dev-images, 10User-brennen: Create PHP 7.3 and PHP 7.4 variants of docker-registry.wikimedia.org/dev/stretch-php72-fpm-apache2-xdebug - https://phabricator.wikimedia.org/T245444 (10Jdfo... [17:52:02] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10User-brennen, 10User-greg: Instrument Zuul/geard to gather the statistics we need to make capacity decisions - https://phabricator.wikimedia.org/T245047 (10Jdforrester-WMF) [17:52:04] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release Pipeline (Blubber): Implement golang directives for Blubber - https://phabricator.wikimedia.org/T246700 (10Jdforrester-WMF) [17:52:06] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10CPT Initiatives (API Integration Tests), 10Code-Health, 10MW-1.35-notes (1.35.0-wmf.19; 2020-02-11): Enable API integration tests in CI for MediaWiki core - https://phabricator.wikimedia.org/T236680 (10Jdforreste... [17:52:09] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Quibble: Feature request: Evaluate "require" field from "extension.json" in automated test environment - https://phabricator.wikimedia.org/T185736 (10Jdforrester-WMF) [17:58:44] getting error `ssh: Could not resolve hostname deployment-eventlog05.deployment-prep.eqiad.wmflabs: Name or service not known` [17:59:52] 10phan-taint-check-plugin, 10Security-Team, 10MW-1.35-notes (1.35.0-wmf.25; 2020-03-24), 10Patch-For-Review, and 2 others: Phan warning in "includes/specials/SpecialMobileDiff.php" - working around by disabling - https://phabricator.wikimedia.org/T248360 (10Jdlrobson) It's passing. Looks like it was fixed... [18:01:00] snowick: your ssh config might not have a proxy command to use the bastion? [18:04:30] RhinosF1: Yes, will look tomorrow. [18:11:36] snowick: https://wikitech.wikimedia.org/wiki/Help:Accessing_Cloud_VPS_instances may help with your ssh issues [18:13:49] thanks will check it out [18:13:56] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:14:26] James_F: thx! [18:28:47] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 93273 bytes in 0.888 second response time [18:39:02] releng can anyone point me where in the integration agent I need to delete the expired github tokens? [18:39:40] I tried docker exec -it bash and looking around, and didn't find any .composer/auth.json [18:40:22] https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-docker/2345/console [18:42:35] ejegg: FWIW, that was a copy paste from a GH/SO post [18:42:38] I didn't actually do anything [18:43:34] Well, beyond filing the task when other stuff was breaking [18:59:14] 10Release-Engineering-Team, 10serviceops: mw1251 down (no ssh) but still in dsh group? - https://phabricator.wikimedia.org/T248501 (10Jdforrester-WMF) p:05Triage→03High [19:01:02] oh weird, when I tried a recheck after you said that, it worked [19:23:35] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Analytics, and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10hashar) So I have played a bit... [19:24:10] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T233873 (10Jdforrester-WMF) [19:24:41] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T233873 (10Jdforrester-WMF) [19:25:00] 10MediaWiki-Releasing, 10Security-Team, 10Security: Write and send pre-release announcements for MediaWiki 1.31.7/1.33.3/1.34.1 - https://phabricator.wikimedia.org/T240394 (10Reedy) [19:29:57] 10Release-Engineering-Team, 10serviceops: mw1251 down (no ssh) but still in dsh group? - https://phabricator.wikimedia.org/T248501 (10Dzahn) a:03Dzahn [19:33:27] 10Release-Engineering-Team, 10serviceops: mw1251 down (no ssh) but still in dsh group? - https://phabricator.wikimedia.org/T248501 (10Dzahn) should have been removed by this change: https://gerrit.wikimedia.org/r/c/operations/puppet/+/583114/2/conftool-data/node/eqiad.yaml conftool generates dsh groups.. unl... [19:40:55] Reedy: I guess the recheck that worked on the earlier patch was probably just run on a different host [19:41:06] and maybe now all the tokens are expired? [19:41:09] Wouldn't surprise me if it's something like that [19:41:21] Is everything breaking? [19:41:25] Are other patches getting through fine? [19:41:39] This one fails every time doing composer install: https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/583193/ [19:42:02] lemme try rechecking a mediawiki extension patch [19:42:17] (though maybe those tests have more of the composer deps cached) [19:48:20] yeah, those are all cached... but it's definitely blocking our CiviCRM patches [19:48:37] and we're really hoping to deploy one today for an imminent test in the Netherlands [19:57:37] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Could not authenticate against github.com - https://phabricator.wikimedia.org/T248387 (10Reedy) p:05High→03Unbreak! [19:57:48] ejegg: That's for starters... [20:03:53] ejegg: It does look like some sort of rate limiting [20:04:10] "Specify your GitHub API token to avoid it." [20:05:00] I'm guessing all CI hosts present as coming from the same public IP? [20:06:01] Reedy: should be through the WMCS public IP yeah [20:06:21] the only exception is contint1001 [20:06:41] I guess we have reached some github limit by hammering it too much? :-\\ [20:06:47] Yeah... [20:07:00] OR... If we're sharing the IP with other cloud stuff... Someone else doing something to cause it? [20:07:05] It seems, we can specify an API key [20:07:13] composer config --global github-oauth.github.com [20:07:43] https://phabricator.wikimedia.org/T106452 has some infos [20:08:32] Aaha [20:08:46] X-Ratelimit-Limit: 60 [20:08:46] X-Ratelimit-Remaining: 0 [20:08:46] X-Ratelimit-Reset: 1585167385 [20:08:51] from a wmcs instance [20:09:14] heh [20:10:05] Can we just change the IP? :P [20:10:13] It would obviously be nice to know why it's suddenly become a problem again [20:11:34] cause the ip has hit the limit of 60 requests per hour [20:13:34] Well, sure [20:13:42] But what is causing extra requests [20:14:46] pretty much everything that tries to install something from api.github.com I guess [20:15:03] mediawiki/core composer.lock has 109 of them for example [20:24:03] heh [20:24:23] Could we inject an api key into CI jobs without exposing it? [20:24:39] wow, 60 per hour seems super draconian [20:25:01] I suspect for most cases, that's more than enough [20:47:28] Reedy: I think we put an auth key into mediawiki-vagrant for this ... [20:48:42] bd808: https://phabricator.wikimedia.org/T106452#1479271 says in 2015 you suggested we *could*, but not sure it ever happened.. :D [20:48:58] yeah, I jsut looked and apparently we did not [20:49:34] I would suggest making a throw away github account and using it. M$ can afford the bandwidth [20:50:00] as a matter of fact, let me be the one to violate TOS [20:50:04] heh [20:50:20] And then just put the token into CI? :P [20:52:58] that's my plan. verifying account now... [21:00:10] <3 [21:01:13] ok, so how do we put this into CI? Jerkins has a secrets store right? [21:02:56] It does, yeah [21:03:22] https://integration.wikimedia.org/ci/credentials/ [21:05:20] would potentially need JJB to inject/include it? I think [21:05:48] yeah, I guess we need `composer config -g github-oauth.github.com ` in the build steps [21:06:06] I was in the midst of writing that, yeah :) [21:06:20] I guess it needs to be somewhere before it starts running composer stuff that pull from the interwebs [21:07:33] I don't see any "[Add Credential]" links via my user login :/ [21:07:58] found it! [21:08:06] twisted ui is twisted [21:09:52] Reedy: ok, https://integration.wikimedia.org/ci/credentials/store/system/domain/_/credential/composer-github-oauthtoken/ exists now [21:11:02] https://github.com/wikimedia/integration-config/search?q=SONAR_API_KEY&unscoped_q=SONAR_API_KEY [21:11:59] cool. I guess I'm heading into this rabbit hole ;) [21:12:14] Looks like it shouldn't be much of an issue... Just maybe a lot of jobs need it adding in? [21:13:13] credentials-binding looks to be the glue then [21:18:01] yeah, I just need to remember how this twisted pile of yaml turns into actual jobs at this point [21:24:03] I think I'm going to need help from a wizard like James_F or legoktm [21:24:22] nothing works like I remember from ~5 years ago :) [21:33:29] thanks for jumping in to fix this, folks! [21:33:36] Hello, I am summoned. [21:35:56] Hmm. Sorry, no idea how to use the creds store in henkins. [21:36:11] James_F: See the SONAR_API_KEY above [21:36:16] That's in jerkins too [21:36:21] Yeah. [21:36:32] Again, dark magic stuff I veer away from. ;-) [21:36:38] The magic seems to be https://github.com/wikimedia/integration-config/blob/3c76c0458d5fac40923a500788b09c5df8719956/jjb/search.yaml#L63-L68 [21:37:07] I suspect a version of that just needs copy pasta into all the right job stanzas [21:37:22] And then a `composer config` call into the build scripts [21:37:24] That looks roughly right. [21:38:29] Hi [21:39:13] Why did we start hitting the rate limit today? What changed? [21:39:32] It was yesterday [21:39:34] But I won't have time to look until tonight sorry [21:39:35] But yeah... [21:39:51] Increase of CI jobs pulling libraries not from caches? [21:39:57] Plus shared IP used in cloud too? [22:15:07] Reedy: ugh. Ok I dug into the far enough to figure out that it is not simple. The jjb tree uses a docker container as the composer binary. There is not a central wrapper script that I can inject the env var into. I will make some notes on the bug [22:16:12] (03PS1) 10Zoranzoki21: Enable CI tests for mediawiki/extensions/UserProtect [integration/config] - 10https://gerrit.wikimedia.org/r/583465 [22:45:45] (03CR) 10Jforrester: [C: 03+2] Enable CI tests for mediawiki/extensions/UserProtect [integration/config] - 10https://gerrit.wikimedia.org/r/583465 (owner: 10Zoranzoki21) [22:46:37] (03Merged) 10jenkins-bot: Enable CI tests for mediawiki/extensions/UserProtect [integration/config] - 10https://gerrit.wikimedia.org/r/583465 (owner: 10Zoranzoki21) [22:51:29] !log Zuul: Install CI for mediawiki/extensions/UserProtect [22:51:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:52:16] (03PS1) 10Jforrester: jjb: Don't tell people about a job that doesn't exist any more [integration/config] - 10https://gerrit.wikimedia.org/r/583473 [22:53:40] James_F: is there a cool magic trick to generate changelog entries for a bunch of dockerfiles in the CI tree? For T248387 I have touched 8 of them and I'm hoping I can be lazy about adding that metadata [22:53:41] T248387: Could not authenticate against github.com - https://phabricator.wikimedia.org/T248387 [22:54:22] bd808: Are you updating a base image and then bumping its inheritors? If so, `docker-pkg` has an `update` command. [22:54:55] bd808: Example command is in the commit message of f0d551282a42dc957b60f9c6bfa78bf89b26f3b7 [22:55:24] yeah I think that would be what happens. [22:55:43] I'll try it and see what it does to my working copy :) [22:55:53] (03CR) 10Jforrester: [C: 03+2] jjb: Don't tell people about a job that doesn't exist any more [integration/config] - 10https://gerrit.wikimedia.org/r/583473 (owner: 10Jforrester) [22:55:59] * James_F grins. [22:56:50] (03Merged) 10jenkins-bot: jjb: Don't tell people about a job that doesn't exist any more [integration/config] - 10https://gerrit.wikimedia.org/r/583473 (owner: 10Jforrester) [22:57:13] why are we not publishing docker-pkg on pypi? That would make this easier... [22:57:37] bd808: I dunno. It's just the tools SRE hand me, I just work here man, etc. ;-) [22:57:55] I expect more from a Varsity letterman James_F [22:58:26] Luckily I didn't do anything for my university and they didn't give me any awards, so I'm exempt. [22:59:41] Surprise, the wiki contains lies -- https://www.mediawiki.org/wiki/Continuous_integration/Docker#Installing_docker-pkg -- "ERROR: Directory '.' is not installable. Neither 'setup.py' nor 'pyproject.toml' found." [22:59:53] Oh, huh. [22:59:56] doh my fault [23:00:02] I've not installed it locally for ages. [23:00:04] one level too deep in the tree :) [23:00:12] OK. [23:03:31] James_F: one more annoying question: what is the version number policy at work here? If I'm updating something currently versioned "0.1.1-s1" do I go "0.1.1-s2" or "0.1.2"? [23:03:58] or I guess even "0.2.0" [23:06:04] oh noes. the versions are not strictly increasing in the file either. there was a 0.1.6 and then a 0.1.0 at a later date [23:06:24] ah following a rename [23:06:30] the world is not all chaos [23:07:06] looks like I should do 0.1.2 for this change (nothing to see here except bd808 talking to himself) [23:08:46] bd808: -s1 for bumps; version for functionality changes. [23:11:34] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, and 6 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Krinkle) [23:15:08] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Discovery-Search: quibble-vendor-mysql-hhvm-docker for WikibaseCirrusSearch takes over 40 minutes - https://phabricator.wikimedia.org/T222757 (10Krinkle) 05Open→03Resolved a:... [23:16:17] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, and 6 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Krinkle) [23:22:33] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [23:30:09] 10Continuous-Integration-Config: Selenium and PHPUnit: Stop execution on failure - https://phabricator.wikimedia.org/T225871 (10Krinkle) 05Open→03Declined I'm declining this per the above, as it would imho negatively affect developer productivity to only get the first failure within a certain stage. As an ex... [23:30:14] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, and 6 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Krinkle) [23:32:41] (03PS1) 10BryanDavis: Inject GitHub OAuth token into composer runtime [integration/config] - 10https://gerrit.wikimedia.org/r/583483 (https://phabricator.wikimedia.org/T106452) [23:33:45] (03CR) 10jerkins-bot: [V: 04-1] Inject GitHub OAuth token into composer runtime [integration/config] - 10https://gerrit.wikimedia.org/r/583483 (https://phabricator.wikimedia.org/T106452) (owner: 10BryanDavis) [23:37:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10Zuul: Abort a Zuul pipeline when one job completed with failures. - https://phabricator.wikimedia.org/T248531 (10Krinkle) [23:38:41] (03PS2) 10BryanDavis: Inject GitHub OAuth token into composer runtime [integration/config] - 10https://gerrit.wikimedia.org/r/583483 (https://phabricator.wikimedia.org/T106452) [23:39:34] (03CR) 10jerkins-bot: [V: 04-1] Inject GitHub OAuth token into composer runtime [integration/config] - 10https://gerrit.wikimedia.org/r/583483 (https://phabricator.wikimedia.org/T106452) (owner: 10BryanDavis) [23:41:16] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO: quibble-selenium jobs re-downloading npm packages (castor not loading) - https://phabricator.wikimedia.org/T234738 (10Krinkle) It looks like Castor is still not working correctly for `node_modules`. The main "mediaw... [23:41:37] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO: Quibble jobs re-download npm packages every build (Castor not loading?) - https://phabricator.wikimedia.org/T234738 (10Krinkle) [23:42:36] (03CR) 10BryanDavis: "NFI what the jerkins failure is telling me. Maybe I mangled the php.yaml file?" [integration/config] - 10https://gerrit.wikimedia.org/r/583483 (https://phabricator.wikimedia.org/T106452) (owner: 10BryanDavis) [23:44:15] * bd808 blames Reedy for the rabbit hole he spent the last 3 hours crawling around in [23:46:40] heh [23:48:45] 10Release-Engineering-Team, 10serviceops: mw1251 down (no ssh) but still in dsh group? - https://phabricator.wikimedia.org/T248501 (10Catrope) This is still broken, and was causing confusion during the 4pm SWAT deployment. Thankfully scap appears to route around broken proxies, so it didn't fail to sync 1/9th... [23:48:55] Reedy: disappoint. I expected "you're welcome" ;) [23:52:10] 10Continuous-Integration-Config, 10phan-taint-check-plugin, 10MediaWiki-Vendor, 10ci-test-error: CI error on WMF branches: Cannot use the final modifier on an abstract class in vendor/microsoft/tolerant-php-parser/tests/cases/parser/abstractMethodDeclaration7.php o... - https://phabricator.wikimedia.org/T248306 [23:53:19] 10Continuous-Integration-Config, 10phan-taint-check-plugin, 10MediaWiki-Vendor, 10ci-test-error: CI error on WMF branches: Cannot use the final modifier on an abstract class in vendor/microsoft/tolerant-php-parser/tests/cases/parser/abstractMethodDeclaration7.php o... - https://phabricator.wikimedia.org/T248306