[00:03:06] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.29 deployment blockers - https://phabricator.wikimedia.org/T271343 (10thcipriani) 05Open→03Declined We're rolling out wmf.30 from wmf.27 so that we have a stable base in case... [00:04:12] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T271344 (10mmodell) [00:04:18] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T271344 (10thcipriani) **End of Tuesday status**: wmf.30 on test wikis; wmf.27 on all other wikis **Needs attention**: https://gerrit.wikimedi... [00:05:02] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T271344 (10mmodell) [00:10:17] arrrrg! The order of Blubber operations strikes again on my hack. environment settings are injected after the command where I need them to be set. [00:10:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) changes to add the new backends to scap... [00:16:32] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) @Krinkle also see T149924#6699330 and... [00:17:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) >>! In T247653#6814030, @Krinkle wrote:... [00:19:12] bd808: shoot. the pythonpath env issue? I'm afk for the day but I can look into it tomorrow [00:21:59] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Krinkle) >>! In T247653#6817122, @Dzahn wrote:... [00:24:10] https://cryptography.io/en/latest/installation.html#static-wheels "Cryptography ships statically-linked wheels for macOS, Windows, and Linux (via manylinux). This allows compatible environments to use the most recent OpenSSL, regardless of what is shipped by default on those platforms." sigh [00:59:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) There is definitely only doc1001 in the... [01:19:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) ` Got error 'PHP message: PHP Parse er... [01:25:10] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) This is a stretch server with PHP 7.0.... [01:26:16] (03PS1) 10Krinkle: tests: Fix xml attribute warning on PHPUnit 8 [integration/docroot] - 10https://gerrit.wikimedia.org/r/663092 [01:27:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Krinkle) >>! In T247653#6817226, @Dzahn wrote:... [01:39:13] (03PS1) 10Krinkle: shared: Unbreak Page.php for old php 7.0 doc1001 server [integration/docroot] - 10https://gerrit.wikimedia.org/r/663093 (https://phabricator.wikimedia.org/T247653) [01:39:58] (03CR) 10Krinkle: [C: 03+2] tests: Fix xml attribute warning on PHPUnit 8 [integration/docroot] - 10https://gerrit.wikimedia.org/r/663092 (owner: 10Krinkle) [01:40:36] (03CR) 10Dzahn: [C: 03+1] shared: Unbreak Page.php for old php 7.0 doc1001 server [integration/docroot] - 10https://gerrit.wikimedia.org/r/663093 (https://phabricator.wikimedia.org/T247653) (owner: 10Krinkle) [01:40:55] (03Merged) 10jenkins-bot: tests: Fix xml attribute warning on PHPUnit 8 [integration/docroot] - 10https://gerrit.wikimedia.org/r/663092 (owner: 10Krinkle) [01:42:10] (03CR) 10Krinkle: [C: 03+2] shared: Unbreak Page.php for old php 7.0 doc1001 server [integration/docroot] - 10https://gerrit.wikimedia.org/r/663093 (https://phabricator.wikimedia.org/T247653) (owner: 10Krinkle) [01:42:55] (03Merged) 10jenkins-bot: shared: Unbreak Page.php for old php 7.0 doc1001 server [integration/docroot] - 10https://gerrit.wikimedia.org/r/663093 (https://phabricator.wikimedia.org/T247653) (owner: 10Krinkle) [01:52:47] (03PS1) 10Krinkle: shared: ... and another php 7.0 fix in CoveragePage.php [integration/docroot] - 10https://gerrit.wikimedia.org/r/663095 [01:52:55] (03CR) 10Krinkle: [C: 03+2] shared: ... and another php 7.0 fix in CoveragePage.php [integration/docroot] - 10https://gerrit.wikimedia.org/r/663095 (owner: 10Krinkle) [01:53:40] (03Merged) 10jenkins-bot: shared: ... and another php 7.0 fix in CoveragePage.php [integration/docroot] - 10https://gerrit.wikimedia.org/r/663095 (owner: 10Krinkle) [01:57:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) after the changes above and rebooting th... [08:14:30] (03CR) 10Hashar: "I had wmfmariadbpy changed to use tox-mysqld in order to run the integration suite and even added the percona-toolkit package to the image" [integration/config] - 10https://gerrit.wikimedia.org/r/662992 (owner: 10Kormat) [08:21:26] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10DBA, 10Patch-For-Review, 10User-Kormat: Run wmfmariadbpy integration test suite on CI - https://phabricator.wikimedia.org/T261098 (10hashar) The CI configuration has been changed and the job Docker image no more contai... [08:32:17] 10Continuous-Integration-Infrastructure, 10SRE-tools, 10Patch-For-Review, 10ci-test-error: spicerack: tox creates py39-tests env even when python 3.9 isn't available - https://phabricator.wikimedia.org/T274232 (10hashar) Seems good to me, thank you for the patches, updates etc. I have announced the tox up... [08:32:41] 10Continuous-Integration-Config, 10SRE-tools, 10ci-test-error: spicerack: tox creates py39-tests env even when python 3.9 isn't available - https://phabricator.wikimedia.org/T274232 (10hashar) [08:36:07] Yippee, build fixed! [08:36:08] Project mwcore-phpunit-coverage-master build #1208: 09FIXED in 5 hr 36 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1208/ [08:44:25] 10Project-Admins: Create three Phab Projects for Machine Learning: Lift Wing, Pilot Flag, Test Grounds - https://phabricator.wikimedia.org/T264774 (10Aklapper) [08:46:11] 10Project-Admins: Create three Phab Projects for Machine Learning: Lift Wing, Pilot Flag, Test Grounds - https://phabricator.wikimedia.org/T264774 (10Aklapper) Are there any code base repositories to link to already (or a ticket tracking to create #documentation on-wiki for these code bases)? [08:49:19] 10Project-Admins: Create Github-notif-bot project - https://phabricator.wikimedia.org/T165060 (10Aklapper) @Framawiki: Hi, there is both https://github.com/framawiki/Github-notif-bot/issues and https://phabricator.wikimedia.org/maniphest/?project=PHID-PROJ-qivzb7ypl2ernunlcft6 . Where is the canonical place for... [09:47:23] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10DBA, 10Patch-For-Review, 10User-Kormat: Run wmfmariadbpy integration test suite on CI - https://phabricator.wikimedia.org/T261098 (10Kormat) @hashar: i've been working on a new integration framework for wmfmariadbpy ({... [09:50:51] (03PS1) 10AndyRussG: Fundraising images: minor improvements for running civibuild [releng/dev-images] - 10https://gerrit.wikimedia.org/r/663159 (https://phabricator.wikimedia.org/T262975) [09:52:01] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10DBA, 10Patch-For-Review, 10User-Kormat: Run wmfmariadbpy integration test suite on CI - https://phabricator.wikimedia.org/T261098 (10jcrespo) I am very happy with @Kormat proposals and the direction she is taking the r... [10:32:14] 10Continuous-Integration-Config, 10SRE-tools, 10ci-test-error: spicerack: tox creates py39-tests env even when python 3.9 isn't available - https://phabricator.wikimedia.org/T274232 (10dcaro) 05Open→03Resolved Closing this as it's already deployed and announced :) [10:45:03] 10Continuous-Integration-Config, 10Release-Engineering-Team: Add a bypass for the ci-src-setup script - https://phabricator.wikimedia.org/T274347 (10hashar) [10:47:29] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Patch-For-Review: Drop jessie testing support - https://phabricator.wikimedia.org/T224908 (10hashar) @Jdforrester-WMF oops sorry to have ignored your changes to... [11:02:03] 10Continuous-Integration-Config, 10Release-Engineering-Team: Add a bypass for the ci-src-setup script - https://phabricator.wikimedia.org/T274347 (10dcaro) a:03dcaro [12:02:03] 10Project-Admins, 10WikiProject-X: Archive #WikiProject-X in Phabricator? Wiki page says it's defunct. - https://phabricator.wikimedia.org/T274351 (10Aklapper) [12:05:18] 10Project-Admins, 10wb_vandalism: Archive #wb_vandalism? (Its Git repository is archived.) - https://phabricator.wikimedia.org/T274352 (10Aklapper) [12:13:44] 10Continuous-Integration-Config, 10Release-Engineering-Team: Add a bypass for the ci-src-setup script - https://phabricator.wikimedia.org/T274347 (10dcaro) [13:14:42] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Patch-For-Review, 10User-zeljkofilipin, and 2 others: Run browser tests as part of "npm test" for wikidata/query/gui in CI - https://phabricator.wikimedia.org/T222200 (10darthmon_wmde) [13:15:06] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Patch-For-Review, 10User-zeljkofilipin, and 2 others: Run browser tests as part of "npm test" for wikidata/query/gui in CI - https://phabricator.wikimedia.org/T222200 (10darthmon_wmde) [13:29:25] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO, 10Code-Stewardship-Reviews: deployment-prep: Code stewardship request - https://phabricator.wikimedia.org/T215217 (10ArielGlenn) Apologies for posting on this closed task, but is there any news on the... [13:55:59] (03PS1) 10C. Scott Ananian: Zuul: [mediawiki/libs/IDLeDOM] Turn on jenkins CI for the `idle-dom` library [integration/config] - 10https://gerrit.wikimedia.org/r/663198 [14:15:18] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Patch-For-Review, 10User-zeljkofilipin, and 2 others: Run browser tests as part of "npm test" for wikidata/query/gui in CI - https://phabricator.wikimedia.org/T222200 (10noarave) Task inspection note: we probably want to start with adding the browser_t... [14:19:29] (03PS1) 10David Caro: ci-common: Add a bypass for the ci-src-setup script [integration/config] - 10https://gerrit.wikimedia.org/r/663202 (https://phabricator.wikimedia.org/T274347) [14:19:34] (03PS1) 10David Caro: tox: Add a bypass for the ci-src-setup script [integration/config] - 10https://gerrit.wikimedia.org/r/663203 (https://phabricator.wikimedia.org/T274347) [14:20:55] (03CR) 10jerkins-bot: [V: 04-1] ci-common: Add a bypass for the ci-src-setup script [integration/config] - 10https://gerrit.wikimedia.org/r/663202 (https://phabricator.wikimedia.org/T274347) (owner: 10David Caro) [14:21:17] (03CR) 10jerkins-bot: [V: 04-1] tox: Add a bypass for the ci-src-setup script [integration/config] - 10https://gerrit.wikimedia.org/r/663203 (https://phabricator.wikimedia.org/T274347) (owner: 10David Caro) [14:30:42] (03PS2) 10David Caro: ci-common: Add a bypass for the ci-src-setup script [integration/config] - 10https://gerrit.wikimedia.org/r/663202 (https://phabricator.wikimedia.org/T274347) [14:33:48] (03PS2) 10David Caro: tox: Add a bypass for the ci-src-setup script [integration/config] - 10https://gerrit.wikimedia.org/r/663203 (https://phabricator.wikimedia.org/T274347) [14:43:56] (03CR) 10Volans: "A question and comment inline. I might miss some context in this repo." (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/663202 (https://phabricator.wikimedia.org/T274347) (owner: 10David Caro) [14:52:58] (03CR) 10David Caro: ci-common: Add a bypass for the ci-src-setup script (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/663202 (https://phabricator.wikimedia.org/T274347) (owner: 10David Caro) [14:54:06] (03PS1) 10David Caro: tox: Allow passing the injected parameters to tox itself [integration/config] - 10https://gerrit.wikimedia.org/r/663212 (https://phabricator.wikimedia.org/T274370) [14:58:21] 10Release-Engineering-Team (Deployment services), 10Patch-For-Review, 10SDAW-MediaSearch (MediaSearch-ReleaseCandidate), 10Structured-Data-Backlog (Current Work), 10Wikimedia-extension-review-queue: Split MediaSearch out into its own extension - https://phabricator.wikimedia.org/T265939 (10matthiasmullie... [15:07:59] hashar: confirming that you saw https://phabricator.wikimedia.org/T247653 [15:09:28] hashar: TLDR: 1) It seems scap deploy is broken for doc.wm.o, php code changes do not get affected until an SRE root manually reboots apache, maybe some kind of configuration issue in php.ini or opcache revalidation ttl detecting changed files, maybe scap needs to do reboot, or maybe php settings fixed, 2) would be nice to move to php 7.2+ instead of php7.0 since CI no longer lints for php 7.0, which caused outage yesterday [15:14:40] Krinkle: yeah I noticed a recent change adding Shellbox for the homepage was not showing up [15:15:11] I investigated a bit here and there but could not find the issue (paths looked fine, cache passes etc). But this morning the page was all fine [15:15:19] php7.0 yeah [15:15:35] the linter got removed by mistake as part of phasing out 7.0 [15:15:44] guess we can add components/php7.2 to the host [15:15:56] and migrate it to Buster, I should pair that with liw [15:16:17] but the last 3 weeks have been a magic black hole for most things :-\ [15:18:19] I dont even know the impact of moving to buster, like the new php version [15:19:38] hashar: see task, looks daniel has prepared everything ready for testing [15:19:49] yeah I got it [15:19:51] new php minor are back compat [15:19:53] but I have ZERO time to do it [15:19:58] literally [15:20:05] Worst thing we might get a warning [15:20:08] Okay [15:20:11] No problem [15:20:19] the idea we had with Tyler is for me to pair with liw and explain the doc.wm.o infra in a nutshell [15:20:43] and leveling up for some more Buster upgrade related stuff [15:20:48] but that hasn't happened yet :\ [15:20:57] Consider escalating yeah, or asking mgmr what else can be delayed since php70 and stretch are dying [15:21:08] guess the issue was that we had some patch that was not php7.0 compatible which did not get caught cause php70 linter got removed? [15:21:13] once fixed, the site is workinga gain? [15:21:18] Yes [15:21:41] It is not possible to supper php70 and 80 at the same time [15:23:07] s/possible/feasible for small projects like doc/ [15:23:33] 8.0? [15:25:04] hashar, yeah, we should pair, when we both have time and lack headaches [15:25:18] liw: yeah ;] [15:25:26] still have that nasty mw train blocker though [15:25:42] do we have other blockers than the featuredfeeds issue? [15:26:30] Majavah: so far no [15:26:50] .28 and .29 got briefly deployed and did not seem to surface too many errors [15:26:52] or they got solved [15:37:49] well it's likely not blocking the train now [15:38:15] I don't want to resolve it til we're out to all groups and have watched for a day and nothing exploded, but still [15:39:45] I probably just jinxed it... [15:42:14] apergos: yeah definitely wait before closing ;) [15:42:26] seems to happen mostly on wikipedias, so we will no for sure on thursday [15:42:28] I am off again [15:42:32] I am the slowest closer ever [15:42:32] driving kids around [15:42:44] 6 months a year later I'm lke "what is is still on the board, woops" [15:42:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10hashar) I can not tell why the homepage of doc.... [15:42:48] ok! [15:42:56] Krinkle: I gave a quick status update for the doc.wm.o migration to buster and spammed liw about it as well ;) [15:43:14] I am late, gotta rush [16:02:00] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) >>! In T247653#6819019, @hashar wrote: >... [16:23:32] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [16:24:27] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [16:26:12] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [16:50:06] 10Project-Admins, 10User-Ladsgroup: Archive #wb_vandalism? (Its Git repository is archived.) - https://phabricator.wikimedia.org/T274352 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup Done. [17:02:08] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO, 10Wikimedia-Site-requests, 10Readers-Web-Backlog (Kanbanana-FY-2020-21), and 2 others: [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10phuedx) >>! In T274210#6817031, @Jdlrobson wrote: >... [17:03:13] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO, 10Wikimedia-Site-requests, 10Readers-Web-Backlog (Kanbanana-FY-2020-21), and 2 others: [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10Jdforrester-WMF) I imagine that this is caused by t... [17:19:56] (03CR) 10Hashar: "Some assorted feedback. Thanks David!" (038 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/663202 (https://phabricator.wikimedia.org/T274347) (owner: 10David Caro) [17:29:18] hasharAway: docwmo is php 7.2 and 7.3 compat. Runs smoothly locally via composer start. [17:30:04] Do you want to give mutante the OK or are there other things to check? Or for liw to decide? [17:31:07] I'm not up to speed on doc.wikimedia.org stuff yet, so I should not decide [17:42:24] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1295.eqiad.wmnet'] ` an... [17:43:05] (03CR) 10David Caro: ci-common: Add a bypass for the ci-src-setup script (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/663202 (https://phabricator.wikimedia.org/T274347) (owner: 10David Caro) [17:51:30] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [17:58:45] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO, 10Wikimedia-Site-requests, 10Patch-For-Review, and 3 others: [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10Jdlrobson) @Jdforrester-WMF the above patch might do it, but I have no idea.... [18:00:32] Krinkle: hold for now :] we will do it eventually [18:00:53] just genuily need more than just me (and you) to maintain that piece of the infra [18:05:34] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1371.eqiad.wmnet'] ` Of... [18:06:53] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1379.eqiad.wmnet'] ` Of... [18:11:22] hashar: how do I debug a "stderr: fatal: remote error: access denied or repository not exported: /wikimedia/toolhub" failure in CI? https://integration.wikimedia.org/ci/job/wikimedia-toolhub-pipeline-test/625/console [18:11:47] it is failing to fetch from git://contint2001.wikimedia.org/wikimedia/toolhub [18:13:40] bd808: should be a one off error iirc [18:13:51] something something with git-daemon bailing out maybe due to some raec condition somewhere [18:14:18] and those repositories are not exposed to the internet [18:15:19] it has happened on https://gerrit.wikimedia.org/r/c/wikimedia/toolhub/+/663091 just now and https://gerrit.wikimedia.org/r/c/wikimedia/toolhub/+/663049 17 hours ago (same origin repo) [18:15:52] but https://gerrit.wikimedia.org/r/c/wikimedia/toolhub/+/663049 seems to have just passed, so yeah random race somewhere I guess [18:16:16] * bd808 does the 'recheck" magic [18:16:21] yeah pretty much [18:16:29] what I imagine is [18:16:37] the build tries to fetch [18:16:55] but at the exact sametime the zuul-merger running on contint2001 is doing a merge on that same repo [18:17:05] and maybe it is moving files with cp + rm [18:17:26] and if it delete some import .git file just as a client comes to fetch we get that error [18:17:47] *nod* that makes some sense. Both failures were during changes that would have touched the repo and then updated 2 patches [18:18:12] on contint2001: grep wikimedia/toolhub /var/log/zuul/merger-debug.log [18:18:23] that yields a bunch of activity at the time the fetch got attempd [18:18:57] oh https://phabricator.wikimedia.org/T187897 [18:19:23] bd808: so yeah recheck would "fix" it [18:20:09] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Scap, 10SRE: Re-imaged mw app servers can end up with missing l10n cache for old versions of MW needed for rollback - https://phabricator.wikimedia.org/T273334 (10Legoktm) This happened agai... [18:20:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Zuul, 10ci-test-error (WMF-deployed Build Failure): fatal: remote error: access denied or repository not exported: /mediawiki/extension... - https://phabricator.wikimedia.org/T187897 [18:24:08] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by legoktm on cumin1001.eq... [18:25:50] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Legoktm) [18:27:39] * DannyS712 is debugging T274312 on the beta cluster [18:36:05] !log deployment-prep deploy01: Run cd /srv/mediawiki-staging/php-master/extensions/PagedTiffHandler && git reset HEAD * && git checkout -- * to fix disappeared extension [18:36:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:36:16] !log deployment-prep: Run scap sync-world as jenkins-deploy [18:36:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:39:38] Project beta-scap-eqiad build #337429: 04FAILURE in 5 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/337429/ [18:40:49] Urbanecm ^ FYI a bunch of missing files/directory errors [18:48:58] Project beta-scap-eqiad build #337430: 15ABORTED in 4 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/337430/ [18:56:50] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T271344 (10mmodell) [18:57:26] new phabricator fields? [18:57:44] or when did the train status field appear? [18:58:01] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [18:58:40] Majavah think its been part of the form for a while, but only editable by some people (it doesn't show up for me when I click edit, and neither does the description text, so the missing space in "forward.This" has been bugging me for a while [19:00:20] DannyS712: I believe "forward.This" is coming from Phabricator code, not from the task description - at least I see an empty description field when editing that task [19:00:34] but now that you've mentioned it, it's going to bug me too until fixed :/ [19:01:57] Project beta-scap-eqiad build #337431: 04STILL FAILING in 7 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/337431/ [19:02:24] ^ for the record, I was not actually able to sync my debugging, and reverted my changes, so not me [19:03:29] you touched it last [19:04:41] 10Phabricator, 10User-DannyS712: Fix typo in description of train deployments tasks - https://phabricator.wikimedia.org/T274402 (10DannyS712) [19:04:47] no I never actually was able to touch it [19:04:53] also ^ for the missing space [19:06:15] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:06:39] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1294.eqiad.wmnet'] ` an... [19:08:57] Yippee, build fixed! [19:08:57] Project beta-scap-eqiad build #337432: 09FIXED in 4 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/337432/ [19:11:49] DannyS712: it has been years since I was active in deployment-prep, but even then trying to use the beta wikis to do ad-hoc debugging via scap deployed uncommitted temporary changes was a pretty delicate and likely to fail process. You probably need to invest more heavily in a local testing platform for your extension to work on cross-extension dependency issues. [19:12:25] if nothing else, the timer based jenkins jobs will stomp on you, or you on them, regularly [19:13:48] yeah, I finally managed to call scap sync-file between jenkins builds, hopefully it'll get me one or two debug log entries. I have it installed locally with central auth and the other extensions its is intended to interact with, but it works fine, so I thought the best way to do it would be to test where the issue actually is known to occur [19:14:26] DannyS712: jenkins will override it eventually [19:14:37] or some local change might cause the beta updater to eventually fail :] [19:14:50] 10Release-Engineering-Team (Logspam), 10MediaWiki-MergeHistory, 10Wikimedia-production-error: LinkBatch::addResultToCache: encountered invalid title ErrorException - https://phabricator.wikimedia.org/T274405 (10thcipriani) [19:15:06] I think a pattern is to use some feature flags [19:15:16] and turned them on for dpeloyment-prep only [19:15:29] but that doesn't fit for ongoing development of a change [19:17:14] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [19:17:36] I think I might have found it though - $wgEnableSidebarCache is true, and I need to vary the sidebar based on the page being viewed, so I should be using the onSidebarBeforeOutput instead of onSkinBuildSidebar hook, maybe [19:18:06] I just want to confirm first [19:19:55] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:19:59] DannyS712: no idea ;] But probably dont want the sidebar cache to vary too much. else I guess that defeat the purpose of that cache [19:20:05] then I know nothing about it so ... [19:20:20] you would want to hunt someone knowing about it I guess / ask around [19:20:27] (that was not that helpful sorry) [19:25:14] we can't really cache this since it changes the sidebar for different pages, but the code *should* be pretty performant [19:25:57] plus codesearch shows a bunch of other deployed extensions that use the hook already [19:33:31] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [19:53:00] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1321.eqiad.wmnet', 'mw13... [20:06:04] (03PS1) 10AndyRussG: WIP Fundraising images: Separate Payments and Civi setups [releng/dev-images] - 10https://gerrit.wikimedia.org/r/663294 (https://phabricator.wikimedia.org/T262975) [20:21:28] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T271344 (10mmodell) [20:21:33] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.29 deployment blockers - https://phabricator.wikimedia.org/T271343 (10mmodell) [20:24:39] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T271344 (10Ladsgroup) [20:24:42] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.29 deployment blockers - https://phabricator.wikimedia.org/T271343 (10Ladsgroup) [20:25:33] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T271344 (10mmodell) [20:26:56] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1370.eqiad.wmnet'] ` an... [20:27:45] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1378.eqiad.wmnet'] ` an... [20:36:05] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1293.eqiad.wmnet'] ` an... [20:45:03] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:46:03] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:52:50] 10Release-Engineering-Team (Local Dev), 10dev-images, 10docker-pkg, 10serviceops, 10User-brennen: docker-pkg: "certificate verify failed: unable to get local issuer certificate" for docker-registry.discovery.wmnet when publishing dev-images from contint2001 - https://phabricator.wikimedia.org/T274306 (10b... [20:59:27] !log Attempting one more update from dev-images docker-pkg on contint2001 for T274306 [20:59:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:59:34] T274306: docker-pkg: "certificate verify failed: unable to get local issuer certificate" for docker-registry.discovery.wmnet when publishing dev-images from contint2001 - https://phabricator.wikimedia.org/T274306 [21:00:04] Project mwcore-phpunit-coverage-master build #1209: 04FAILURE in 6 hr 0 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1209/ [21:00:36] 10Release-Engineering-Team (Local Dev), 10dev-images, 10docker-pkg, 10serviceops, 10User-brennen: docker-pkg: "certificate verify failed: unable to get local issuer certificate" for docker-registry.discovery.wmnet when publishing dev-images from contint2001 - https://phabricator.wikimedia.org/T274306 (10b... [21:07:28] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by legoktm on cumin1001.eq... [21:11:08] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10Patch-For-Review: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10dancy) >>! In T266055#6798797, @Krinkle wrote: > Okay, so we... [21:12:25] 10Phabricator, 10User-DannyS712: Fix typo in description of train deployments tasks - https://phabricator.wikimedia.org/T274402 (10Ammarpad) https://phabricator.wikimedia.org/source/phab-extensions/browse/wmf%252Fstable/src/customfields/ReleaseDetailsCustomField.php [21:58:28] 10Release-Engineering-Team (Logspam), 10MediaWiki-MergeHistory, 10Wikimedia-production-error: LinkBatch::addResultToCache: encountered invalid title ErrorException - https://phabricator.wikimedia.org/T274405 (10Ammarpad) [22:10:50] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1369.eqiad.wmnet'] ` an... [22:12:46] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1377.eqiad.wmnet'] ` an... [22:35:15] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1325.eqiad.wmnet', 'mw13... [22:43:21] bd808: did you ever see https://github.com/pyca/cryptography/blob/master/setup.py#L48 ? "CRYPTOGRAPHY_DONT_BUILD_RUST" ? [22:44:10] legoktm: I did, but I also saw something like "for this release only" in a changelog for that flag somewhere. [22:44:53] My current "fix" is installing the upstream wheel after getting a pip that supports the manylinux2014 format [22:45:17] 10Phabricator, 10User-DannyS712: Fix typo in description of train deployments tasks - https://phabricator.wikimedia.org/T274402 (10DannyS712) >>! In T274402#6820396, @Ammarpad wrote: > https://phabricator.wikimedia.org/source/phab-extensions/browse/wmf%252Fstable/src/customfields/ReleaseDetailsCustomField.php... [22:45:39] i did same for crypto problem in a different repo, had to update pip first then it would pull the appropriate manylinux wheel [22:46:35] ebernhardson: my special difficulty was doing that within Blubber and deploypipeline :) [22:47:09] legoktm: but per https://github.com/pyca/cryptography/issues/5771#issuecomment-775990406 we are going to have to figure out what y'all serviceops sres want to do going forward [22:47:33] * ebernhardson scared of blubber :P [22:48:02] its nice actually, but it forces you to be much more disciplined about how your containers work [22:48:15] it == Blubber [22:48:30] yea, but the only container i build today installs the entire hadoop ecosystem and a task scheduler. Blubber wants like 12 containers :P [22:49:04] I... but... that does not sound like a container so much as a vm in a tarball :) [22:49:19] i mean, we just install puppet, run a bunch of puppet. Thats dockerized right :P [22:49:54] its an integration environment anyways, noone is going to deploy hadoop this way :) [22:49:57] If you dig back in the mw-vagrant repo you can find a place where I did that horrible thing there before switching to lxc [22:50:32] with the added horrible of actually using Vagrant to apply the Puppet config within a Docker container! [22:50:38] lol [22:52:32] 10Release-Engineering-Team (Logspam), 10Readers-Web-Backlog, 10JavaScript, 10MediaWiki-Interface (Tables), 10Wikimedia-production-error: Some tables cannot be sorted (TypeError: $nextRows[i] is undefined in table sorting / Uncaught TypeError: Cannot read propert... - https://phabricator.wikimedia.org/T265503 [22:53:32] bd808: ack, I was trying to see exactly how much Rust is needed and stumbled upon that [22:55:02] install pip >=20 and then use the manylinux2014 wheel from upstream feels "right" to me but that brings the statically linked libssl baggage [22:55:02] !log Deploying zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/661796 [22:55:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:56:32] * bd808 is still confused that the cryptography upstream claims dropping support for multiple runtimes would not have been a semver breaking change [22:57:06] but they sidestep that in reality by not using semver! :) [23:17:52] (03PS2) 10Umherirrender: Add a new StaticClosureSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/662110 (https://phabricator.wikimedia.org/T274038) [23:17:54] (03PS1) 10Umherirrender: internal: Use static closures [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/663321 [23:18:32] (03CR) 10Umherirrender: "> Patch Set 1:" (033 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/662110 (https://phabricator.wikimedia.org/T274038) (owner: 10Umherirrender) [23:23:02] (03CR) 10jerkins-bot: [V: 04-1] internal: Use static closures [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/663321 (owner: 10Umherirrender) [23:23:21] (03PS2) 10Umherirrender: internal: Use static closures [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/663321 [23:29:19] (03CR) 10DannyS712: [C: 03+1] internal: Use static closures [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/663321 (owner: 10Umherirrender) [23:45:42] (03CR) 10Umherirrender: "It seems there is already a sniff, but SlevomatCodingStandard is not in use for the standard codesniffer (some extensions are using it)" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/662110 (https://phabricator.wikimedia.org/T274038) (owner: 10Umherirrender) [23:57:29] (03CR) 10Umherirrender: "Some closures using pre 5.4 (not sure about the version) binding like this" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/662110 (https://phabricator.wikimedia.org/T274038) (owner: 10Umherirrender) [23:58:09] (03PS3) 10Umherirrender: Add a new StaticClosureSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/662110 (https://phabricator.wikimedia.org/T274038) [23:58:23] (03CR) 10Umherirrender: "Rebased" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/662110 (https://phabricator.wikimedia.org/T274038) (owner: 10Umherirrender)