[00:23:01] (03PS1) 10Krinkle: make-deploy-notes: Simplify and prettify botclasses.php [tools/release] - 10https://gerrit.wikimedia.org/r/709851 [00:23:38] (03CR) 10jerkins-bot: [V: 04-1] make-deploy-notes: Simplify and prettify botclasses.php [tools/release] - 10https://gerrit.wikimedia.org/r/709851 (owner: 10Krinkle) [00:24:35] (03PS2) 10Krinkle: make-deploy-notes: Simplify and prettify botclasses.php [tools/release] - 10https://gerrit.wikimedia.org/r/709851 [00:38:40] 10Release-Engineering-Team (Doing), 10MediaWiki-Docker, 10dev-images, 10Patch-For-Review, 10User-brennen: dev-images: upgrade images for MediaWiki-Docker from wikimedia-stretch to wikimedia-buster - https://phabricator.wikimedia.org/T273100 (10Jdforrester-WMF) [00:38:50] 10Release-Engineering-Team (Doing), 10MediaWiki-Docker, 10dev-images, 10Patch-For-Review, 10User-brennen: dev-images: upgrade images for MediaWiki-Docker from wikimedia-stretch to wikimedia-buster - https://phabricator.wikimedia.org/T273100 (10Jdforrester-WMF) [00:40:58] 10Gerrit, 10Mobile, 10Upstream: Gerrit: issues with Responsive Web Design for mobile view - https://phabricator.wikimedia.org/T256547 (10Paladox) Resolved in the gerrit 3.3 update done by @hashar [00:41:04] 10Gerrit, 10Mobile, 10Upstream: Gerrit: issues with Responsive Web Design for mobile view - https://phabricator.wikimedia.org/T256547 (10Paladox) 05Open→03Resolved [00:55:26] 10Release-Engineering-Team (Radar), 10Performance-Team, 10serviceops: Create warmup procedure for MediaWiki app servers - https://phabricator.wikimedia.org/T230037 (10Krinkle) 05Open→03Resolved a:03Krinkle We basically have this, and used for dc-switchovers. If and when we need it elsewhere (e.g. for P... [01:12:47] 10Gerrit, 10Release-Engineering-Team (Radar): Gerrit errors on change #650396, missing blob - https://phabricator.wikimedia.org/T270472 (10Paladox) Seems this works now? At least I don't get the error. [02:27:04] Another neat thing about the "Attention set" thing in Gerrit is that you can remove yourself from the hover popover even when not on the change page. E.g. from a search query or dashboard you can hover your own name and remove with one click [02:40:22] On the other hand, it seems Gerit has killed status texts mostly. Limited to popover. No more nickname carry-ons anymore. [02:40:34] the ones we abused with emojis [03:02:02] (03PS1) 10TrainBranchBot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/709866 [03:02:04] (03CR) 10TrainBranchBot: [C: 03+2] Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/709866 (owner: 10TrainBranchBot) [03:02:29] 10Release-Engineering-Team (Logspam), 10MediaWiki-extensions-LiquidThreads, 10Platform Team Workboards (MW Expedition), 10Wikimedia-production-error: Wikimedia\Assert\ParameterTypeException: Bad value for parameter $target: must be a MediaWiki\Linker\LinkTarget|Med... - https://phabricator.wikimedia.org/T287619 [03:03:09] (03Merged) 10jenkins-bot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/709866 (owner: 10TrainBranchBot) [03:06:46] (03PS1) 10Ahmon Dancy: WIP: Make image build process work with Docker Desktop (Mac OS) [tools/release] - 10https://gerrit.wikimedia.org/r/709867 [03:09:16] (03PS1) 10TrainBranchBot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/709868 [03:09:18] (03CR) 10TrainBranchBot: [C: 03+2] Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/709868 (owner: 10TrainBranchBot) [03:10:04] (03Merged) 10jenkins-bot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/709868 (owner: 10TrainBranchBot) [03:13:10] (03PS2) 10Ahmon Dancy: WIP: Make image build process work with Docker Desktop (Mac OS) [tools/release] - 10https://gerrit.wikimedia.org/r/709867 [03:13:47] (03CR) 10jerkins-bot: [V: 04-1] WIP: Make image build process work with Docker Desktop (Mac OS) [tools/release] - 10https://gerrit.wikimedia.org/r/709867 (owner: 10Ahmon Dancy) [03:17:43] (03PS3) 10Ahmon Dancy: WIP: Make image build process work with Docker Desktop (Mac OS) [tools/release] - 10https://gerrit.wikimedia.org/r/709867 [03:17:45] (03PS1) 10TrainBranchBot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/709869 [03:17:47] (03CR) 10TrainBranchBot: [C: 03+2] Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/709869 (owner: 10TrainBranchBot) [03:19:02] (03Merged) 10jenkins-bot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/709869 (owner: 10TrainBranchBot) [03:57:12] (03PS1) 10TrainBranchBot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/709874 [03:57:14] (03CR) 10TrainBranchBot: [C: 03+2] Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/709874 (owner: 10TrainBranchBot) [03:58:09] (03Merged) 10jenkins-bot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/709874 (owner: 10TrainBranchBot) [04:04:00] (03PS1) 10TrainBranchBot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/709875 [04:04:02] (03CR) 10TrainBranchBot: [C: 03+2] Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/709875 (owner: 10TrainBranchBot) [04:04:57] (03Merged) 10jenkins-bot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/709875 (owner: 10TrainBranchBot) [04:12:00] (03PS1) 10TrainBranchBot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/709876 [04:12:02] (03CR) 10TrainBranchBot: [C: 03+2] Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/709876 (owner: 10TrainBranchBot) [04:13:05] (03Merged) 10jenkins-bot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/709876 (owner: 10TrainBranchBot) [04:40:59] 10Release-Engineering-Team, 10SRE: releases1002 /srv/docker DISK SPACE alert - https://phabricator.wikimedia.org/T288024 (10Marostegui) [05:39:12] 10Release-Engineering-Team, 10SRE: releases1002 /srv/docker DISK SPACE alert - https://phabricator.wikimedia.org/T288024 (10MoritzMuehlenhoff) p:05Triage→03Medium a:03hashar Antoine, could you please have a look whether we can free something? [05:44:21] 10Gerrit, 10Gerrit-Privilege-Requests: Give access to l10n-bot for operations/software/mailman-templates - https://phabricator.wikimedia.org/T288027 (10abi_) [05:54:08] 10Release-Engineering-Team, 10SRE: releases1002 /srv/docker DISK SPACE alert - https://phabricator.wikimedia.org/T288024 (10Joe) I've done a `docker image prune -a` on that server, but I think we will need to give it a larger docker partition given the amount of images we're building there. [06:57:21] 10Release-Engineering-Team, 10Gerrit (Gerrit 3.3): "Only comments" switch went missing with Gerrit 3.3 - https://phabricator.wikimedia.org/T288031 (10thiemowmde) [07:24:10] PROBLEM - SSH on contint2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [07:27:15] 10Release-Engineering-Team, 10SRE: releases1002 /srv/docker DISK SPACE alert - https://phabricator.wikimedia.org/T288024 (10hashar) [07:29:04] 10Release-Engineering-Team, 10SRE: releases1002 /srv/docker DISK SPACE alert - https://phabricator.wikimedia.org/T288024 (10hashar) a:05hashar→03None That is routinely filing up due to some Jenkins job creating images/containers but not reclaiming them at end of build. @dancy / @dduvall will know the detai... [07:34:20] 10Gerrit, 10Upstream: "Only comments" switch went missing with Gerrit 3.3 - https://phabricator.wikimedia.org/T288031 (10Aklapper) https://bugs.chromium.org/p/gerrit/issues/detail?id=12991 [07:42:18] 10Release-Engineering-Team, 10SRE: releases1002 /srv/docker DISK SPACE alert - https://phabricator.wikimedia.org/T288024 (10MoritzMuehlenhoff) If there's no immediate fix on the Jenkins side we should add a systemd timer to trigger a cleanup before this escalates to alerts [07:58:04] 10Gerrit, 10Regression, 10Upstream: "Only comments" switch went missing with Gerrit 3.3 - https://phabricator.wikimedia.org/T288031 (10Michael) >>! In T288031#7258018, @Aklapper wrote: > https://bugs.chromium.org/p/gerrit/issues/detail?id=12991 That link is useful insofar as it contains a further link to: h... [07:58:10] 10Gerrit, 10Regression, 10Upstream: "Only comments" switch went missing with Gerrit 3.3 - https://phabricator.wikimedia.org/T288031 (10hashar) Thanks for filing it, that got mentioned yesterday over IRC and from the quick investigation it is related to [[ https://www.gerritcodereview.com/2020-05-06-change-lo... [07:58:22] 10Gerrit, 10Release-Engineering-Team (Doing), 10Regression, 10Upstream: "Only comments" switch went missing with Gerrit 3.3 - https://phabricator.wikimedia.org/T288031 (10hashar) [08:02:49] 10Gerrit, 10Release-Engineering-Team (Doing), 10Regression, 10Upstream: "Only comments" switch went missing with Gerrit 3.3 - https://phabricator.wikimedia.org/T288031 (10hashar) [08:23:27] 10Gerrit, 10Release-Engineering-Team: Gerrit names misalignment in 3.3.5 - https://phabricator.wikimedia.org/T288035 (10fgiunchedi) [08:24:54] RECOVERY - SSH on contint2001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [08:30:13] 10Gerrit, 10Wikidata, 10Wikidata-Campsite, 10wdwb-tech: wikidata-dev instances causing git "Internal error during upload-pack" every 5 minutes - https://phabricator.wikimedia.org/T287459 (10Addshore) [09:06:38] 10Gerrit, 10Release-Engineering-Team (Doing), 10Regression, 10Upstream: "Only comments" switch went missing with Gerrit 3.3 - https://phabricator.wikimedia.org/T288031 (10thiemowmde) Hm. The (so far) only thing I slightly disagree with is that it appears to show all auto-generated `Added to reviewer:`, `Re... [09:29:28] 10Gerrit, 10Release-Engineering-Team, 10Upstream: Gerrit names misalignment in 3.3.5 - https://phabricator.wikimedia.org/T288035 (10hashar) By default there should be a user avatar in front of the name which is made to stick to the border of the ship: {F34575555 size=full} That is done by removing the left... [09:30:14] 10Gerrit, 10Release-Engineering-Team, 10Upstream: Gerrit names misalignment in 3.3.5 - https://phabricator.wikimedia.org/T288035 (10hashar) I wanted to file a bug upstream but their issue tracker no more let me submit a new issue (either on Firefox or Chromium). When it is back we can post the following: `... [09:33:30] 10Gerrit, 10Release-Engineering-Team (Radar), 10Upstream: Gerrit names misalignment in 3.3.5 - https://phabricator.wikimedia.org/T288035 (10hashar) [09:38:00] 10Gerrit, 10Upstream: No keyboard shortcut for adding reviewers in PolyGerrit - https://phabricator.wikimedia.org/T199463 (10hashar) 05Open→03Declined The new one is {key A} to //Open reply dialog to publish comments and add reviewers// and that sounds good enough. [09:56:23] 10Gerrit, 10Wikidata, 10Wikidata-Campsite, 10wdwb-tech: wikidata-dev instances causing git "Internal error during upload-pack" every 5 minutes - https://phabricator.wikimedia.org/T287459 (10Addshore) p:05Low→03Medium [10:00:11] heya, looks like you have puppet agent failures in integration-agent-pkgbuilder-1001/1002, according to our very experimental and limited prometheus instance in wmcs [10:05:14] 10Release-Engineering-Team (Doing), 10dev-images, 10mwcli, 10Patch-For-Review, and 2 others: dev PHP and apache docker images should output logs to STDOUT and STDERR - https://phabricator.wikimedia.org/T283416 (10hashar) [10:25:42] (03PS1) 10Hashar: scap: automatize plugins handling [software/gerrit] (deploy/wmf/stable-3.3) - 10https://gerrit.wikimedia.org/r/709975 [10:25:57] (03Abandoned) 10Hashar: scap: automatize plugins handling [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/706038 (owner: 10Hashar) [10:40:32] (03PS1) 10Hashar: Review access change [software/mailman-templates] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/709976 [10:40:58] (03PS2) 10Hashar: Review access change [software/mailman-templates] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/709976 (https://phabricator.wikimedia.org/T288027) [11:16:31] (03PS1) 10TrainBranchBot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/710001 [11:16:33] (03CR) 10TrainBranchBot: [C: 03+2] Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/710001 (owner: 10TrainBranchBot) [11:18:36] (03Merged) 10jenkins-bot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/710001 (owner: 10TrainBranchBot) [11:26:04] (03PS1) 10TrainBranchBot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/710003 [11:26:07] (03CR) 10TrainBranchBot: [C: 03+2] Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/710003 (owner: 10TrainBranchBot) [11:27:27] (03Merged) 10jenkins-bot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/710003 (owner: 10TrainBranchBot) [11:35:08] (03PS1) 10TrainBranchBot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/710004 [11:35:10] (03CR) 10TrainBranchBot: [C: 03+2] Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/710004 (owner: 10TrainBranchBot) [11:36:06] (03Merged) 10jenkins-bot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/710004 (owner: 10TrainBranchBot) [12:24:32] (03Abandoned) 10Hashar: [DNM] Investigating CI error [integration/quibble] - 10https://gerrit.wikimedia.org/r/693602 (owner: 10Awight) [13:15:58] (03PS1) 10Hashar: test: fix flappy test for core being cloned first [integration/quibble] - 10https://gerrit.wikimedia.org/r/710032 [13:19:02] (03CR) 10Hashar: "That one fails from time to time, specially under python3.8." [integration/quibble] - 10https://gerrit.wikimedia.org/r/710032 (owner: 10Hashar) [14:23:54] Project mediawiki-core-doxygen-docker build #26906: 04FAILURE in 19 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/26906/ [14:42:09] (03CR) 10Hashar: "The flappy test should be fixed by https://gerrit.wikimedia.org/r/c/integration/quibble/+/710032" [integration/quibble] - 10https://gerrit.wikimedia.org/r/693602 (owner: 10Awight) [15:01:32] (03CR) 10Awight: [C: 03+2] "Great!" [integration/quibble] - 10https://gerrit.wikimedia.org/r/710032 (owner: 10Hashar) [15:01:52] awight: I am sure one could have written a better test, but that one looked good enough ;] [15:03:41] awight: I have a task about settling up on a communication channel for quibble ( https://phabricator.wikimedia.org/T286770 ) [15:03:54] I guess it is usually done in this channel, which is probably fine [15:04:06] but maybe there are better alternative [15:04:16] idea is to be able to tell folks to join X if they want to ask questions [15:04:22] and add it to the doc [15:08:57] !log rebase deployment-puppetmaster04:labs/private causing deployment-deploy{01,03} failure for...¯\_(ツ)_/¯ [15:08:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:13:25] !log puppet fixed on deployment-deploy{01,03} [15:13:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:14:15] yay! [15:20:43] (03Merged) 10jenkins-bot: test: fix flappy test for core being cloned first [integration/quibble] - 10https://gerrit.wikimedia.org/r/710032 (owner: 10Hashar) [15:20:50] Yippee, build fixed! [15:20:50] Project mediawiki-core-doxygen-docker build #26907: 09FIXED in 16 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/26907/ [15:28:06] 10Continuous-Integration-Config, 10Inuka-Team: Setup standard MediaWiki extensions CI for Wikistories repo - https://phabricator.wikimedia.org/T288111 (10SBisson) [15:40:03] 10Release-Engineering-Team, 10SRE: releases1002 /srv/docker DISK SPACE alert - https://phabricator.wikimedia.org/T288024 (10thcipriani) >>! In T288024#7258016, @hashar wrote: > That is routinely filing up due to some Jenkins job creating images/containers but not reclaiming them at end of build. @dancy / @dduv... [15:41:36] 10Release-Engineering-Team, 10SRE: releases1002 /srv/docker DISK SPACE alert - https://phabricator.wikimedia.org/T288024 (10hashar) 05Open→03Resolved a:03hashar Great, and this task can be marked as resolved since immediate action have been taken earlier today to resolve the alarm. [15:53:15] (03PS1) 10Ahmon Dancy: make-container-image: Report how long it takes to push the image [tools/release] - 10https://gerrit.wikimedia.org/r/710061 [15:53:51] (03CR) 10Ahmon Dancy: [C: 03+2] make-container-image: Report how long it takes to push the image [tools/release] - 10https://gerrit.wikimedia.org/r/710061 (owner: 10Ahmon Dancy) [15:56:43] (03Merged) 10jenkins-bot: make-container-image: Report how long it takes to push the image [tools/release] - 10https://gerrit.wikimedia.org/r/710061 (owner: 10Ahmon Dancy) [15:59:24] 10Phabricator, 10Developer Productivity: Show previews for Phabricator links in Slack (and other messaging apps) - https://phabricator.wikimedia.org/T288117 (10alexhollender) [15:59:30] 10Gerrit, 10Release-Engineering-Team (Radar), 10Upstream: Gerrit names misalignment in 3.3.5 - https://phabricator.wikimedia.org/T288035 (10fgiunchedi) All makes sense to me, thank you @hashar for the quick and through investigation! [16:39:04] (03PS1) 10Sbisson: Setup CI for Wikistories [integration/config] - 10https://gerrit.wikimedia.org/r/710066 (https://phabricator.wikimedia.org/T288111) [16:39:47] (03CR) 10Hashar: [C: 03+2] "Thank you! :]" [integration/config] - 10https://gerrit.wikimedia.org/r/710066 (https://phabricator.wikimedia.org/T288111) (owner: 10Sbisson) [16:40:09] 10Continuous-Integration-Config, 10Inuka-Team (Kanban), 10Patch-For-Review: Setup standard MediaWiki extensions CI for Wikistories repo - https://phabricator.wikimedia.org/T288111 (10SBisson) p:05Triage→03Medium a:03SBisson [16:41:04] (03Merged) 10jenkins-bot: Setup CI for Wikistories [integration/config] - 10https://gerrit.wikimedia.org/r/710066 (https://phabricator.wikimedia.org/T288111) (owner: 10Sbisson) [16:41:38] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/710066 # T288111 [16:41:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:41:43] T288111: Setup standard MediaWiki extensions CI for Wikistories repo - https://phabricator.wikimedia.org/T288111 [16:48:45] 10Release-Engineering-Team (Radar), 10Performance-Team, 10Wikimedia-Rdbms, 10MW-1.37-notes (1.37.0-wmf.16; 2021-07-26), and 2 others: PHP Warning: Class __PHP_Incomplete_Class has no unserializer - https://phabricator.wikimedia.org/T287988 (10Urbanecm) a:05dduvall→03Urbanecm [17:00:28] jeena: is there a page on mw.o for the local-charts project? I wanted to put a link in a commit message, but I'm not guessing the right page names or keywords to find it. [17:06:38] 10Continuous-Integration-Config, 10Inuka-Team (Kanban), 10Patch-For-Review: Setup standard MediaWiki extensions CI for Wikistories repo - https://phabricator.wikimedia.org/T288111 (10SBisson) 05Open→03Resolved @hashar Thank you! [17:12:16] bd808: There is this: https://wikitech.wikimedia.org/wiki/Local-charts [17:12:31] ps I started reviewing your chart yesterday, hopefully will finish today [17:16:02] oh hey we passed 710k changes today https://gerrit.wikimedia.org/r/c/operations/puppet/+/710000/ [17:22:26] (03PS1) 10Ahmon Dancy: make-container-image Makefile: Set shell to /bin/bash [tools/release] - 10https://gerrit.wikimedia.org/r/710070 [17:22:54] (03CR) 10Ahmon Dancy: [C: 03+2] make-container-image Makefile: Set shell to /bin/bash [tools/release] - 10https://gerrit.wikimedia.org/r/710070 (owner: 10Ahmon Dancy) [17:24:36] (03Merged) 10jenkins-bot: make-container-image Makefile: Set shell to /bin/bash [tools/release] - 10https://gerrit.wikimedia.org/r/710070 (owner: 10Ahmon Dancy) [17:25:29] jeena: I just pushed an update to the chart that adds a whole lot of stuff for use with local-charts. I have it running fully matching my docker-compose demo environment now which is pretty cool. [17:26:15] ooh nice :D we can add it it in the local-charts requirements too [17:49:17] !log gitlab-test: testing upgrade to 13.12.9 [17:49:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:56:04] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline: Support incremental multiversion MW image builds - https://phabricator.wikimedia.org/T286505 (10dancy) 05Open→03Resolved The incremental build process is live now. [18:12:32] (03PS4) 10Ahmon Dancy: Make image build process work with Docker Desktop (Mac OS) [tools/release] - 10https://gerrit.wikimedia.org/r/709867 [18:37:14] (03PS1) 10Ahmon Dancy: Create a variant of mediawiki-multiversion which installs php-tideways-xhprof [tools/release] - 10https://gerrit.wikimedia.org/r/710077 (https://phabricator.wikimedia.org/T287495) [18:39:10] (03CR) 10Ahmon Dancy: [C: 03+2] Create a variant of mediawiki-multiversion which installs php-tideways-xhprof [tools/release] - 10https://gerrit.wikimedia.org/r/710077 (https://phabricator.wikimedia.org/T287495) (owner: 10Ahmon Dancy) [18:40:06] (03Merged) 10jenkins-bot: Create a variant of mediawiki-multiversion which installs php-tideways-xhprof [tools/release] - 10https://gerrit.wikimedia.org/r/710077 (https://phabricator.wikimedia.org/T287495) (owner: 10Ahmon Dancy) [18:49:10] 10Release-Engineering-Team, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Create a variant of mediawiki-multiversion which installs php-tideways-xhprof - https://phabricator.wikimedia.org/T287495 (10dancy) 05Open→03Resolved This is done. Now whenever a docker-registry.discovery.wmnet/restricted/medi... [18:57:08] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments, 10User-jeena (Sprint 7/28-8/10): 1.37.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T281159 (10mmodell) [19:28:52] 10MediaWiki-Releasing, 10Release-Engineering-Team (Radar), 10Security-Team: Write a script to create security release task tree - https://phabricator.wikimedia.org/T256285 (10Reedy) 05Stalled→03Resolved a:03Reedy [19:29:08] 10MediaWiki-Releasing, 10Release-Engineering-Team (Radar), 10Security-Team: Write a script to create security release task tree - https://phabricator.wikimedia.org/T256285 (10sbassett) Resolving now per conversation with @Reedy, most of this work has been completed. [19:37:51] 10Release-Engineering-Team, 10Infrastructure-Foundations, 10CAS-SSO, 10GitLab (Initialization), 10User-brennen: Open gitlab.wikimedia.org to all users with Wikimedia developer accounts - https://phabricator.wikimedia.org/T288162 (10brennen) [19:41:03] 10Release-Engineering-Team (Doing), 10Infrastructure-Foundations, 10CAS-SSO, 10GitLab (Initialization), and 2 others: Open gitlab.wikimedia.org to all users with Wikimedia developer accounts - https://phabricator.wikimedia.org/T288162 (10brennen) [19:51:10] 10Release-Engineering-Team, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Create a variant of mediawiki-multiversion which installs php-tideways-xhprof - https://phabricator.wikimedia.org/T287495 (10Krinkle) [20:06:34] (03PS1) 10Accraze: inference-services: added articlequality pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/710108 (https://phabricator.wikimedia.org/T287786) [20:07:43] (03CR) 10jerkins-bot: [V: 04-1] inference-services: added articlequality pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/710108 (https://phabricator.wikimedia.org/T287786) (owner: 10Accraze) [20:09:58] (03PS2) 10Accraze: inference-services: added articlequality pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/710108 (https://phabricator.wikimedia.org/T287786) [20:42:23] hey releng! my team and I are writing up some best practices around API Development at WMF and on the topic of deployments, wanted to get an opinion: Is there a hard and fast rule of when to use a cloud services like CloudVPS/Toolforge or taking the time to deploy it in K8s? If the developer had the time and capacity, is deploying it in K8s preferred? [20:47:42] nikkinikk_: privacy would be the biggest issue [20:48:24] Things like tools cdn aren't supposed to be used by prod wikis for loading JS etc [21:06:34] so the "rule" or guideline may be something like toolforge/cloudVPS until it becomes a production service [21:08:25] I'm not rel eng nor do I have any authority but cloud vps / toolforge shouldn't be backing production services [21:08:36] So my guess would be yes [21:09:25] Cloud Services & Security team would probably also want involving though in that [21:09:32] I can imagine they have thoughts [21:10:39] thanks for the disclaimer, i appreciate the thoughts either way they make sense to me. ill see if cloud services has any thoughts too :) [21:17:56] No problem [21:33:31] nikkinikk: Yes, that's definitely a grey area. Lots of production-ish stuff does live on toolforge/cloud, though the secteam doesn't really prefer that serve as an end-around to getting things into proper Wikimedia production. At least not anymore than it already does. [21:39:35] nikkinikk_: ^ not sure if that pinged because _ [21:43:14] thanks RhinosF1 ! And thanks sbassett in the cloud channel Bryan was saying a similar thing, that we shouldn't be putting any production services in TF/CloudVPS. So will go with that as the rule of thumb. [21:43:49] nikkinikk_: it would be nice to know more about what you're building [21:44:04] I just think sometimes even the word "production" can be a grey area. I feel like things can go from"proof of concept" to "production" quickly haha [21:44:09] As I'm sure with context you'll fine many friendly faces happy to advise on what to implement [21:44:20] Yes production can mean a lot [21:46:50] Our goal is to make a guide for API development at WMF, so actually avoiding having to ask releng/cloud team for specifics per use case every time. But the general use case as of now is any new API developed, assuming it will eventually end up in production, and basically the happy path to get there deployment wise [21:51:57] but you're right it probably depends what kind of API it is... [22:07:03] While there's general advice, one size doesn't fit all. [22:07:18] I'd aim for production to be in production [22:08:14] Telling people to file a task and get security/performance/serviceops/legal if needed reviews isnt a bad thing [22:10:51] Getting a new thing deployed to production is a lot of steps and checks and cross checks, so I totally understand why folks are looking for both "best practices" and also less complex paths. [22:12:04] Oh it's definately not an easy thing at the moment [22:12:06] I'm excited and hopeful for changes to come in getting things into the production kubernetes cluster once mw-on-k8s is mostly {{Done}} [22:12:46] mw-on-k8s simultaneously excites and scares me at the same time [22:13:17] Do you have a tangible fear or mostly just change can be scary thoughts? [22:14:11] k8s scares me [22:14:24] It's more new stuff to think about [22:14:25] As the person who mostly did the rewrite of scap into python many years ago, anythign that gets rid of scap is a good thing IMO. :) [22:15:04] I'm still trying to replace pulling from git on every appserver and running various scripts [22:15:25] kubernetes is a lot for sure. I feel like I'm pretty ok at being a user in a k8s cluster, but I get lots pretty fast when trying to troubleshoot the underlying service [22:15:38] *get lost [22:16:06] Kubernetes will take me a lot of learning if I decide to steal anything from the WMF [22:16:50] I'm stealing the concept of scap at the moment or in all honesty the concept of deploys not being 50 commands [22:17:35] turning N scripts into 1 script is a good first step at cleaning up a deployment tool [22:18:47] scap cheats at that a bit by making it one command with a lot of sub commands, but it did at least move from being part bash, part perl, part php into just python [22:20:33] bd808: if you think scap cheats at stuff, read the first half of https://github.com/miraheze/puppet/blob/master/modules/mediawiki/files/bin/deploy-mediawiki [22:20:45] (It doesn't do any deploying because I can't get ssh agent working) [22:21:04] Leaving Kubernetes out of the picture though, there is a big win in the mw-on-k8s project by getting all the code to deploy smashed into a Docker container. That can be used in a lot of ways other than as part of a Kubernetes pod. [22:21:12] RhinosF1: heh. the "python as bash" model [22:21:33] bd808: yep [22:21:54] Most of our 'scripts' I've invented majav.ah keeps telling me could be bash [22:22:14] They're mostly if statements and os.system calls [22:22:27] yeah.. that's what bask is great at [22:22:30] *bash [22:22:56] There's all sorts of nonsense in that folder [22:23:04] None of it with any linting or tests too [22:23:07] Which is scary [22:24:31] The firejail one is stolen from the WMF though so that got tested in WMF prod first to switch to python3 because I knew no one would merge it in WMF puppet if it wouldn't work and I'd actually get code review for adding a single number [22:25:54] not hoping to shame, but https://github.com/miraheze/puppet/blob/master/modules/mediawiki/files/mediawiki-firejail-ghostscript.py would be both simpler and faster as shell script [22:26:13] python isn't doing anything to help you there [22:26:44] That's stolen from WMF prod [22:27:29] * bd808 goes to see if there is any sign of why python in that history [22:27:29] https://github.com/wikimedia/puppet/blob/production/modules/mediawiki/files/mediawiki-firejail-convert.py [22:28:00] That's the one I tested in WMF prod switch to py3 as it would have meant actual code review [22:29:14] https://github.com/wikimedia/puppet/commit/01a8a274c7eeca87e25ec4f0ee6c54ace8711fce#diff-4b1b571eb39204d60349dc332019b864d15972b45969cdae948d86a6e8a55226 -- no reason given for python there. shrug [22:30:02] shell out from php to python to shell out to firejail which will shell out to convert [22:31:22] Might be a question to raise [22:31:40] That's a lot of shelling out though [22:32:17] fork(2) gets a lot of use in a unix system :) [22:32:56] * RhinosF1 should be asleep but would quite happily chat for hours about better ways of scripting stuff [22:33:13] My brain will complain at me all tommorow if I don't sleep though [22:33:44] the problems will still be here to think about when you wake up 💤 [22:34:25] :) [22:38:47] the firejail wrapper is in Python because M.oritz doesn't like shell scripts [22:39:01] shell script racism! [22:39:27] for that specific case it's overkill but I agree with him on the general principle [22:43:20] I have lost my former love of writing complex software applications in posix shell, but for scripting a pipeline of binaries it feels like the correct tool. (and an `exec ...` also removes the calling script from the process stack which is often a very nice thing to do) [22:53:17] !log Updated composer-github-oauthtoken in Jenkins config to us a newer personal access token from GitHub per notices about https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/ [22:53:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:54:06] legoktm, thcipriani: ^ If y'all see things complaining about composer pulls in CI that would be a likely cause [22:55:03] * legoktm nods [22:55:27] ci broken, blame bd808 : got it :) [22:55:41] sounds about right :) [23:03:44] thank you for updating the token [23:09:13] reading scrollback: someday I'd like for it to be a single line of config to deploy to WMCS vs Wikiprod [23:09:41] that isn't really entirely up to me, but I think that'd be a nice target [23:10:33] 10Release-Engineering-Team (Logspam), 10MediaWiki-extensions-CategoryTree, 10MediaWiki-extensions-DoubleWiki, 10Wikimedia-production-error: Error: Call to a member function getInterwiki() on null - https://phabricator.wikimedia.org/T286672 (10Umherirrender) Fresh parse of the langlinks with action=parse -... [23:36:50] hashar: found an even better fix! (doing it in the plugin) https://gerrit-review.googlesource.com/c/plugins/codemirror-editor/+/313722 [23:40:11] My performance change was merged for the plugin too :D