[00:00:48] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-Site-requests, 10Regression, 10Vector (Vector (Tracking)): [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10Jdlrobson) This requires a config change and the needed changes are de... [00:02:38] eileen: yeah, it doesn't really mean anything in the tooling as far as i know. [00:04:23] it's just the debian changelog format used for building deb packages, so not everything in it maps 1:1 to the docker use case. [00:04:44] (watch me turn out to be wrong about urgency specifically, but i can't think _what_ it'd control.) [00:09:08] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1387.eqiad.wmnet'] ` an... [00:11:47] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1386.eqiad.wmnet'] ` an... [00:17:21] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-Site-requests, 10Regression, 10Vector (Vector (Tracking)): [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10Reedy) >>! In T274210#6813759, @Jdlrobson wrote: > This requires a con... [00:26:42] brennen longma hey sorry was afk for a bit [00:27:10] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-Site-requests, 10Regression, 10Vector (Vector (Tracking)): [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10bd808) >>! In T274210#6813759, @Jdlrobson wrote: > This requires a con... [00:27:46] brennen longma here is the most recent readme and patch set to be used with the images we've proposed in dev-images: https://gerrit.wikimedia.org/g/wikimedia/fundraising/dev/+/a170097a8b7310a4860b3d439580e335af099457 [00:28:59] AndyRussG: thanks for the context. [00:32:41] brennen: thank u :) [00:45:13] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1302.eqiad.wmnet'] ` an... [00:46:08] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1301.eqiad.wmnet'] ` an... [01:02:17] (03CR) 10Jforrester: "https://doc.wikimedia.org/mediawiki-libs-Minify/master/ now exists. Time to land?" [integration/docroot] - 10https://gerrit.wikimedia.org/r/662785 (https://phabricator.wikimedia.org/T273247) (owner: 10Krinkle) [01:03:16] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-Site-requests, 10Regression, 10Vector (Vector (Tracking)): [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10Jdlrobson) Hey @bd808 Those logos only apply to old Vector and other s... [01:17:15] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-Site-requests, 10Regression, 10Vector (Vector (Tracking)): [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10bd808) >>! In T274210#6813829, @Jdlrobson wrote: > Hey @bd808 Those lo... [02:58:46] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10Release Pipeline, 10serviceops, 10Patch-For-Review: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10Legoktm) >>! In T273521#6812210, @Joe wrote: > So my proposal would boil down to havin... [04:09:45] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-Site-requests, 10Regression, 10Vector (Vector (Tracking)): [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10Jdlrobson) From what I can see here, the problem is that `wmf-config/I... [04:38:11] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-Site-requests, 10Regression, 10Vector (Vector (Tracking)): [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10Jdlrobson) [04:38:32] (03CR) 10Krinkle: [C: 04-1] "Packagist link is dead still" [integration/docroot] - 10https://gerrit.wikimedia.org/r/662785 (https://phabricator.wikimedia.org/T273247) (owner: 10Krinkle) [04:42:41] !log Submitted wikimedia/minify to Packagist. https://packagist.org/packages/wikimedia/minify [04:42:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:42:58] !log Submitted wikimedia/minify to Packagist. https://packagist.org/packages/wikimedia/minify ref T273247 [04:43:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:43:02] T273247: Publish wikimedia/minify as its own repo and package - https://phabricator.wikimedia.org/T273247 [04:44:15] (03CR) 10Krinkle: [C: 03+2] doc: Add wikimedia/minify [integration/docroot] - 10https://gerrit.wikimedia.org/r/662785 (https://phabricator.wikimedia.org/T273247) (owner: 10Krinkle) [04:44:16] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-Site-requests, 10Regression, 10Vector (Vector (Tracking)): [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10greg) Hey @Jdlrobson given the situation is one where the Beta Cluster... [04:45:20] (03Merged) 10jenkins-bot: doc: Add wikimedia/minify [integration/docroot] - 10https://gerrit.wikimedia.org/r/662785 (https://phabricator.wikimedia.org/T273247) (owner: 10Krinkle) [04:54:22] * Krinkle sees new docs at https://www.mediawiki.org/wiki/Continuous_integration/Documentation_generation#Updating_the_doc.wikimedia.org_site [04:54:34] Thanks hashar :) [05:01:21] (03PS1) 10Krinkle: build: Add /scap/log to gitignore [integration/docroot] - 10https://gerrit.wikimedia.org/r/662813 [05:08:20] hm.. it doesn't seem to have worked? [05:08:29] I confirm the code is on doc1001 [05:08:34] and bypassing the cdn cache [05:22:23] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Krinkle) I suspect this might have broken deplo... [05:22:41] James_F: legoktm: ^ in case either of you know something about this [05:22:48] * Krinkle is signing off for the day [05:23:38] I don't but I can look in a bit [05:36:42] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-Site-requests, 10Regression, 10Vector (Vector (Tracking)): [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10greg) p:05Triage→03High a:03Jdlrobson [06:59:54] (03PS1) 10Zoranzoki21: Archive the IfTemplates extension [integration/config] - 10https://gerrit.wikimedia.org/r/662824 (https://phabricator.wikimedia.org/T274046) [07:33:36] (03CR) 10Hashar: [C: 03+2] Archive the IfTemplates extension [integration/config] - 10https://gerrit.wikimedia.org/r/662824 (https://phabricator.wikimedia.org/T274046) (owner: 10Zoranzoki21) [07:34:58] (03Merged) 10jenkins-bot: Archive the IfTemplates extension [integration/config] - 10https://gerrit.wikimedia.org/r/662824 (https://phabricator.wikimedia.org/T274046) (owner: 10Zoranzoki21) [07:45:11] (03CR) 10Hashar: [C: 03+2] "Thanks!" [integration/docroot] - 10https://gerrit.wikimedia.org/r/662813 (owner: 10Krinkle) [07:46:19] (03Merged) 10jenkins-bot: build: Add /scap/log to gitignore [integration/docroot] - 10https://gerrit.wikimedia.org/r/662813 (owner: 10Krinkle) [08:13:58] (03PS1) 10Zoranzoki21: Archive the ListSignup extension [integration/config] - 10https://gerrit.wikimedia.org/r/662896 (https://phabricator.wikimedia.org/T274047) [08:18:34] Yippee, build fixed! [08:18:34] Project mwcore-phpunit-coverage-master build #1206: 09FIXED in 5 hr 18 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1206/ [08:20:47] (03CR) 10Hashar: [C: 03+2] Archive the ListSignup extension [integration/config] - 10https://gerrit.wikimedia.org/r/662896 (https://phabricator.wikimedia.org/T274047) (owner: 10Zoranzoki21) [08:22:04] (03Merged) 10jenkins-bot: Archive the ListSignup extension [integration/config] - 10https://gerrit.wikimedia.org/r/662896 (https://phabricator.wikimedia.org/T274047) (owner: 10Zoranzoki21) [08:58:10] 10Phabricator: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10Gilles) [09:06:40] 10Phabricator: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10Gilles) @epriestley seeing your old comment on a related issue: https://phabricator.wikimedia.org/T219978#5346100 We now have a demonstration that excessive traffic //can// melt d... [09:08:36] (03PS1) 10Zoranzoki21: Review access change [extensions/GoogleCustomWikiSearch] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/662682 [09:09:12] 10Phabricator, 10SRE, 10Traffic: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10MoritzMuehlenhoff) [09:12:19] 10Phabricator, 10SRE, 10Traffic: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10hashar) [09:30:35] 10Beta-Cluster-Infrastructure, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Beta cluster Swift backend instances are missing profile::swift::storage::rsync_limit_memory_percent (puppet fails) - https://phabricator.wikimedia.org/T274092 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Issue should b... [10:06:34] (03PS1) 10Hashar: dockerfiles: remove npm npm-stretch and ci-jessie images [integration/config] - 10https://gerrit.wikimedia.org/r/662912 (https://phabricator.wikimedia.org/T211784) [10:07:05] (03PS2) 10Hashar: dockerfiles: remove npm npm-stretch and ci-jessie images [integration/config] - 10https://gerrit.wikimedia.org/r/662912 (https://phabricator.wikimedia.org/T211784) [10:07:52] (03CR) 10Hashar: [C: 03+2] "Turned out to be straightforward. The npm image is no more used :] Bye bye NodeJS 6, npm 3 and Jessie!" [integration/config] - 10https://gerrit.wikimedia.org/r/662912 (https://phabricator.wikimedia.org/T211784) (owner: 10Hashar) [10:08:28] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Patch-For-Review: Drop jessie testing support - https://phabricator.wikimedia.org/T224908 (10hashar) [10:09:16] (03Merged) 10jenkins-bot: dockerfiles: remove npm npm-stretch and ci-jessie images [integration/config] - 10https://gerrit.wikimedia.org/r/662912 (https://phabricator.wikimedia.org/T211784) (owner: 10Hashar) [10:10:17] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Patch-For-Review: Drop jessie testing support - https://phabricator.wikimedia.org/T224908 (10hashar) 05Open→03Resolved a:03Jdforrester-WMF That was one of... [10:23:00] 10Beta-Cluster-Infrastructure, 10SRE, 10SRE-swift-storage: Beta cluster Swift backend instances are missing profile::swift::storage::rsync_limit_memory_percent (puppet fails) - https://phabricator.wikimedia.org/T274092 (10hashar) Puppet is all happy on both instances indeed. Thank you! [10:26:50] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.29 deployment blockers - https://phabricator.wikimedia.org/T271343 (10hashar) The patch for FeatureFeeds on T273242 got merged. I will promote 1.36.0-wmf.29 to group 0/group 1 thi... [10:28:13] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-Site-requests, 10Readers-Web-Backlog (Kanbanana-FY-2020-21), and 2 others: [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10ovasileva) [10:48:09] 10Phabricator, 10SRE, 10Traffic: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10Vgutierrez) p:05Triage→03Medium [11:02:38] 10Phabricator, 10SRE, 10Traffic: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10Majavah) Phabricator doesn't also seem to be caching user profile pictures at all, which are also stored in the `phabricator_file` database. [11:05:58] 10Jenkins, 10SRE-tools, 10ci-test-error: spicerack: tox creates py39-tests env even when python 3.9 isn't available - https://phabricator.wikimedia.org/T274232 (10Volans) p:05Medium→03High This is happening in CI too! ` 11:59:21 py39-unit run-test: commands[0] | py.test --strict-markers --cov-report=ter... [11:06:50] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10Release Pipeline, 10serviceops, 10Patch-For-Review: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10JMeybohm) >>! In T273521#6813953, @Legoktm wrote: > Is there a preference on whether t... [11:10:36] 10Jenkins, 10SRE-tools, 10ci-test-error: spicerack: tox creates py39-tests env even when python 3.9 isn't available - https://phabricator.wikimedia.org/T274232 (10Kormat) Bizarrely, even when python 3.9 _is_ available, it still doesn't get used: ` $ rm -rf .tox $ tox -e py39-flake8 --notest $ .tox/py39-tests... [11:39:31] 10Continuous-Integration-Infrastructure, 10SRE-tools, 10ci-test-error: spicerack: tox creates py39-tests env even when python 3.9 isn't available - https://phabricator.wikimedia.org/T274232 (10hashar) CI uses tox 3.10.0 but it should be able to recognize 3.9. Given spicerack tox.ini has: ` [tox] skip_missin... [11:58:44] (03CR) 10Lars Wirzenius: "recheck" [tools/scap] - 10https://gerrit.wikimedia.org/r/656951 (owner: 10Lars Wirzenius) [12:03:43] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release Pipeline (Blubber): Blubber forces installation of pip from PyPI, which no longer works with Python 2 - https://phabricator.wikimedia.org/T273793 (10LarsWirzenius) 05Open→03Resolved Confirmed that this fixes the problem Scap had in CI. [13:08:09] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Wikidata-Campsite, 10Patch-For-Review, and 2 others: Run browser tests as part of "npm test" for wikidata/query/gui in CI - https://phabricator.wikimedia.org/T222200 (10Addshore) [13:09:51] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Wikidata-Campsite, 10Patch-For-Review, and 2 others: Run browser tests as part of "npm test" for wikidata/query/gui in CI - https://phabricator.wikimedia.org/T222200 (10Addshore) [13:11:29] 10Project-Admins: Request for project MediaWiki-extensions-SemanticQueryInterface - https://phabricator.wikimedia.org/T138603 (10Aklapper) @Vedmaka: Hi, https://github.com/vedmaka/SemanticQueryInterface/issues also exists. Which place to use for bug reports and enhancement requests, in order not to duplicate? [13:40:18] hashar: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/662953 is incoming for .29, it should finish up just around the time you announced in your email for the rollout to groups 0/1, but if it's a few minutes late, can you wait for it? it's more logging [13:40:37] apergos: absolutely! :] [13:40:42] ty!! [13:40:54] apergos: and I should really get folks to make those CI jobs faster [13:41:10] that would be lovely [13:42:18] are there any that cna be run in parallel, to get any given patch through faster? [13:45:16] apergos: it is more about we rjust brute force run every single tests available [13:45:38] ugh [13:45:48] notably for the selenium tests, we blindly run all of them when we could find a way to only run a subset of them [13:46:06] like if one send a patch to EventBus, there is probably no need to run the tests to verify whether twocolconflict UI still works [13:46:25] or a change to Scribunto is not going to affect Echo alert notification system [13:46:27] something like that [13:50:02] (03PS4) 10Hashar: jjb: switch striker jobs to one based off buster [integration/config] - 10https://gerrit.wikimedia.org/r/651297 (owner: 10Jforrester) [13:54:25] (03CR) 10Hashar: "I have amended the image example-run.sh to run the image against both of the repositories. There are stacktraces here and there but that d" [integration/config] - 10https://gerrit.wikimedia.org/r/651297 (owner: 10Jforrester) [14:02:12] zuul claims there's about 4 minutes left, so maybe 10 minutes, for the gate-and-submit, and less than that for test-wmf. [14:05:51] hashar: merged! test-wmf is 'almost' done [14:05:58] yeah https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/79201/console [14:06:33] well if one looks at the full console output and head to the top of the doc https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/79201/consoleFull#console-section-0 [14:06:41] on the left side there is a summary of what is being run [14:06:45] Selenium extensions/TwoColConflict [14:06:48] Selenium extensions/Wikibase [14:06:49] etc [14:06:56] which well take a while :-] [14:07:13] lets follow up in -operations [14:15:36] 10Deployments, 10Release-Engineering-Team (Deployment services): deploy mediawiki/tools/releases on deployment server with scap - https://phabricator.wikimedia.org/T274255 (10hashar) [14:42:24] 10Phabricator, 10Project-Admins, 10Epic: Add links on all Phabricator/Maniphest project pages to the corresponding source code pages in Diffusion - https://phabricator.wikimedia.org/T144127 (10Aklapper) Coming back to this task, as I recognize a pattern of new technical contributors asking on tasks where the... [14:42:40] 10Phabricator, 10Project-Admins, 10Epic: Add links on all Phabricator/Maniphest project pages to the corresponding canonical code repository location - https://phabricator.wikimedia.org/T144127 (10Aklapper) [14:53:06] 10Continuous-Integration-Infrastructure, 10SRE-tools, 10ci-test-error: spicerack: tox creates py39-tests env even when python 3.9 isn't available - https://phabricator.wikimedia.org/T274232 (10Volans) a:05Volans→03None [15:09:16] (03PS1) 10Kormat: layout: Use tox-docker for wmfmariadbpy [integration/config] - 10https://gerrit.wikimedia.org/r/662992 [15:13:46] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.29 deployment blockers - https://phabricator.wikimedia.org/T271343 (10hashar) [15:45:45] (03CR) 10Giuseppe Lavagetto: [C: 03+1] "Rememeber to remove the image later, if we don't need it anymore" [integration/config] - 10https://gerrit.wikimedia.org/r/662992 (owner: 10Kormat) [16:00:38] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T271344 (10mmodell) p:05Medium→03High [16:02:19] o/ I'm just decoding part of a comment at https://phabricator.wikimedia.org/T272877#6813880which says "Images will be built per-extension anyway during the deployment process", is there a releng ticket related to that? [16:06:00] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Papaul) [16:06:37] addshore: nothing that I am aware of [16:31:41] (03Abandoned) 10Lars Wirzenius: test: make test failures under Python3 fail tox test run [tools/scap] - 10https://gerrit.wikimedia.org/r/656951 (owner: 10Lars Wirzenius) [16:47:51] (03PS6) 10Lars Wirzenius: have tox run scripts/check, require py3 to work, fix [tools/scap] - 10https://gerrit.wikimedia.org/r/661129 [17:18:09] 10Continuous-Integration-Config, 10Gerrit, 10Release-Engineering-Team-TODO, 10Developer Productivity: Jenkins comments no longer formatted on Gerrit 3 - https://phabricator.wikimedia.org/T256575 (10thcipriani) [17:19:35] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10serviceops: switch contint prod server back from contint2001 to contint1001 - https://phabricator.wikimedia.org/T256422 (10thcipriani) [17:19:48] (03CR) 10Jforrester: "Ah, neat." [integration/config] - 10https://gerrit.wikimedia.org/r/651297 (owner: 10Jforrester) [17:19:56] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10serviceops: switch contint prod server back from contint2001 to contint1001 - https://phabricator.wikimedia.org/T256422 (10thcipriani) p:05Triage→03Medium [17:21:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10Code-Health-Metrics: Move Dan's CI metrics cron script / HTML output from people.wikimedia.org to doc.wikimedia.org - https://phabricator.wikimedia.org/T255701 (10thcipriani) [17:21:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10Code-Health-Metrics: Move Dan's CI metrics cron script / HTML output from people.wikimedia.org to doc.wikimedia.org - https://phabricator.wikimedia.org/T255701 (10dduvall) a:05dduvall→03None >>! In T255701#6453608, @hashar wrote:... [17:22:10] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10Code-Health-Metrics: Move Dan's CI metrics cron script / HTML output from people.wikimedia.org to doc.wikimedia.org - https://phabricator.wikimedia.org/T255701 (10thcipriani) p:05Medium→03Low [17:26:33] 10Release-Engineering-Team, 10Release Pipeline: Support for deleting images in PipelineLib - https://phabricator.wikimedia.org/T255595 (10thcipriani) 05Open→03Declined I don't remember why I wanted this. [17:26:38] 10Release-Engineering-Team, 10Platform Engineering, 10Release Pipeline, 10SRE, and 5 others: Kask functional testing with Cassandra via the Deployment Pipeline - https://phabricator.wikimedia.org/T224041 (10thcipriani) [17:29:15] 10Release-Engineering-Team-TODO, 10Scap, 10SRE, 10Sustainability (Incident Followup), 10User-brennen: scap's logstash_checker.py is blissfully unaware of any logstash indexing latency - https://phabricator.wikimedia.org/T255197 (10thcipriani) [17:32:19] 10Deployments, 10Release-Engineering-Team-TODO, 10serviceops, 10Sustainability (Incident Followup), 10User-jijiki: Remove provisioning for 'mwscript', 'foreachwikiindblist' etc from deployment host - https://phabricator.wikimedia.org/T253822 (10thcipriani) [17:33:49] 10Release-Engineering-Team-TODO, 10observability, 10Developer Productivity: Add mwversion to php7-fatal-error.php logstash message - https://phabricator.wikimedia.org/T253781 (10thcipriani) [17:37:54] 10Release-Engineering-Team, 10Release Pipeline: Draft proposal for re-implementing pipelinelib for non-jenkins - https://phabricator.wikimedia.org/T246665 (10brennen) [17:38:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO: Move all Wikimedia CI (WMCS integration project) instances from stretch to buster - https://phabricator.wikimedia.org/T252071 (10thcipriani) [17:38:58] 10Release-Engineering-Team: Change deploy-promote to check previous groups have been promoted to - https://phabricator.wikimedia.org/T269242 (10brennen) [17:39:00] 10Release-Engineering-Team, 10Scap: Automate scap patch - https://phabricator.wikimedia.org/T245185 (10LarsWirzenius) We have "scap apply-patches" now, but {T269153} needs to worked on next for that. Adding it as a subtask. [17:39:09] 10Release-Engineering-Team, 10Scap: Automate scap patch - https://phabricator.wikimedia.org/T245185 (10LarsWirzenius) [17:39:11] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Scap, 10User-brennen: Applying security patches should be robust and also give some useful output - https://phabricator.wikimedia.org/T269153 (10LarsWirzenius) [17:43:44] 10Release-Engineering-Team-TODO: Change deploy-promote to check previous groups have been promoted to - https://phabricator.wikimedia.org/T269242 (10thcipriani) [17:45:49] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release Pipeline: Document pipelinelib - https://phabricator.wikimedia.org/T239423 (10thcipriani) [17:45:55] 10Release-Engineering-Team, 10Release Pipeline: Draft proposal for re-implementing pipelinelib for non-jenkins - https://phabricator.wikimedia.org/T246665 (10thcipriani) 05Open→03Declined we'll revisit this, probably not soon but some day. [17:46:45] (03CR) 10Hashar: "There are a few issues with the patches I have mentioned in the various files. But the biggest one is that all those repositories are test" (035 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/661829 (https://phabricator.wikimedia.org/T273822) (owner: 10Eileen) [17:47:21] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2264.codfw.wmnet'] ` an... [17:49:44] 10Release-Engineering-Team-TODO, 10MediaWiki-Release-Tools, 10User-brennen: branch.py: Convert deleted wmf branches to tags - https://phabricator.wikimedia.org/T250095 (10thcipriani) Maybe tag the tip of the branch rather than branch merge-base. [17:51:28] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1383.eqiad.wmnet'] ` an... [17:54:21] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team-TODO: Migrate CI Jenkins artifacts to a non wikimedia.org domain - https://phabricator.wikimedia.org/T248696 (10thcipriani) [17:54:56] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team-TODO: Migrate CI Jenkins artifacts to a non wikimedia.org domain - https://phabricator.wikimedia.org/T248696 (10thcipriani) p:05Triage→03Medium [17:59:53] 10Release-Engineering-Team-TODO, 10MediaWiki-Release-Tools: Change deploy-promote to check previous groups have been promoted to - https://phabricator.wikimedia.org/T269242 (10thcipriani) [18:01:52] (03PS1) 10David Caro: tox: upgrade all images to tox 3.21.4 [integration/config] - 10https://gerrit.wikimedia.org/r/663026 (https://phabricator.wikimedia.org/T274232) [18:02:43] (03CR) 10David Caro: "I'll need some help to test this propertly though, anyone willing to help?" [integration/config] - 10https://gerrit.wikimedia.org/r/663026 (https://phabricator.wikimedia.org/T274232) (owner: 10David Caro) [18:03:00] (03CR) 10Jforrester: "You'll need to update the changelogs so that these images would get built." [integration/config] - 10https://gerrit.wikimedia.org/r/663026 (https://phabricator.wikimedia.org/T274232) (owner: 10David Caro) [18:03:51] 10Continuous-Integration-Infrastructure, 10SRE-tools, 10Patch-For-Review, 10ci-test-error: spicerack: tox creates py39-tests env even when python 3.9 isn't available - https://phabricator.wikimedia.org/T274232 (10dcaro) I added a 'naive' patch, as it's the first time that I even see that repo xd, I'll need... [18:04:06] 10Continuous-Integration-Infrastructure, 10SRE-tools, 10Patch-For-Review, 10ci-test-error: spicerack: tox creates py39-tests env even when python 3.9 isn't available - https://phabricator.wikimedia.org/T274232 (10dcaro) a:03dcaro [18:04:35] (03CR) 10Lars Wirzenius: "Pushing a new patch set in a moment with the type fixed." (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/661129 (owner: 10Lars Wirzenius) [18:04:50] (03PS7) 10Lars Wirzenius: have tox run scripts/check, require py3 to work, fix [tools/scap] - 10https://gerrit.wikimedia.org/r/661129 [18:05:21] (03CR) 10Ahmon Dancy: [C: 03+1] have tox run scripts/check, require py3 to work, fix [tools/scap] - 10https://gerrit.wikimedia.org/r/661129 (owner: 10Lars Wirzenius) [18:06:12] (03CR) 10Ahmon Dancy: [C: 03+2] "Nice mods" [tools/scap] - 10https://gerrit.wikimedia.org/r/661129 (owner: 10Lars Wirzenius) [18:07:28] (03Merged) 10jenkins-bot: have tox run scripts/check, require py3 to work, fix [tools/scap] - 10https://gerrit.wikimedia.org/r/661129 (owner: 10Lars Wirzenius) [18:10:13] (03PS2) 10Jforrester: dockerfiles [tox] Upgrade all images to tox 3.21.4 [integration/config] - 10https://gerrit.wikimedia.org/r/663026 (https://phabricator.wikimedia.org/T274232) (owner: 10David Caro) [18:12:17] (03PS3) 10David Caro: tox: upgrade all images to tox 3.21.4 [integration/config] - 10https://gerrit.wikimedia.org/r/663026 (https://phabricator.wikimedia.org/T274232) [18:12:46] (03CR) 10Jforrester: "Ah, you over-wrote the cascade I made. :-(" [integration/config] - 10https://gerrit.wikimedia.org/r/663026 (https://phabricator.wikimedia.org/T274232) (owner: 10David Caro) [18:12:48] (03CR) 10David Caro: "> Patch Set 1:" [integration/config] - 10https://gerrit.wikimedia.org/r/663026 (https://phabricator.wikimedia.org/T274232) (owner: 10David Caro) [18:13:22] (03CR) 10David Caro: "> Patch Set 3:" [integration/config] - 10https://gerrit.wikimedia.org/r/663026 (https://phabricator.wikimedia.org/T274232) (owner: 10David Caro) [18:13:33] (03CR) 10jerkins-bot: [V: 04-1] tox: upgrade all images to tox 3.21.4 [integration/config] - 10https://gerrit.wikimedia.org/r/663026 (https://phabricator.wikimedia.org/T274232) (owner: 10David Caro) [18:21:15] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1300.eqiad.wmnet'] ` an... [18:23:03] (03CR) 10Jforrester: [C: 04-1] "> Patch Set 3:" [integration/config] - 10https://gerrit.wikimedia.org/r/663026 (https://phabricator.wikimedia.org/T274232) (owner: 10David Caro) [18:23:39] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2220.codfw.wmnet'] ` an... [18:25:52] (03PS4) 10David Caro: tox: upgrade all images to tox 3.21.4 [integration/config] - 10https://gerrit.wikimedia.org/r/663026 (https://phabricator.wikimedia.org/T274232) [18:26:49] (03CR) 10David Caro: "> Patch Set 3: Code-Review-1" [integration/config] - 10https://gerrit.wikimedia.org/r/663026 (https://phabricator.wikimedia.org/T274232) (owner: 10David Caro) [18:27:08] (03CR) 10jerkins-bot: [V: 04-1] tox: upgrade all images to tox 3.21.4 [integration/config] - 10https://gerrit.wikimedia.org/r/663026 (https://phabricator.wikimedia.org/T274232) (owner: 10David Caro) [18:31:09] (03PS5) 10David Caro: tox-buster: upgrade all images to tox 3.21.4 [integration/config] - 10https://gerrit.wikimedia.org/r/663026 (https://phabricator.wikimedia.org/T274232) [18:31:28] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T271344 (10RhinosF1) What is the plan for this week given the situation over the last 2 trains? [18:40:56] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO: deploy mediawiki/tools/releases on deployment server with scap - https://phabricator.wikimedia.org/T274255 (10thcipriani) [18:41:40] 10Deployments, 10Release-Engineering-Team: deploy mediawiki/tools/releases on deployment server with scap - https://phabricator.wikimedia.org/T274255 (10thcipriani) [18:41:56] 10Phabricator, 10SRE, 10Traffic: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10Gilles) a:0320after4 [18:46:51] 10Phabricator, 10SRE, 10Traffic: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10mmodell) Indeed I think it's caching static files correctly. [18:47:46] 10Phabricator, 10SRE, 10Traffic: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10epriestley) > Eventually Phabricator ran out of local sockets (30k limit): I'm not familiar with the particulars of your infrastructure, but if this was on th... [18:48:31] 10Phabricator, 10SRE, 10Traffic: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10Gilles) a:0520after4→03mmodell [18:48:35] (03PS1) 10David Caro: Bump versions to newer tox-buster and related images [integration/config] - 10https://gerrit.wikimedia.org/r/663029 (https://phabricator.wikimedia.org/T274232) [18:52:12] (03CR) 10David Caro: "I did not bump tox-labs-striker because it's already missing some other bumps, so I'm guessing that there's some dev work needed to make i" [integration/config] - 10https://gerrit.wikimedia.org/r/663029 (https://phabricator.wikimedia.org/T274232) (owner: 10David Caro) [18:52:19] 10Phabricator, 10SRE, 10Traffic: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10Majavah) Hmm. I'm looking at the Firefox developer console, and it looks like this: {F34098763} Note that styles and scripts are marked as "cached" instead of... [19:12:12] 10Phabricator, 10SRE, 10Traffic: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10epriestley) I suspect you're seeing that because when you reload a page by issuing a "Reload" command in your browser, most (all?) modern browsers interpret th... [19:26:49] (03CR) 10Jforrester: [C: 03+2] tox-buster: upgrade all images to tox 3.21.4 [integration/config] - 10https://gerrit.wikimedia.org/r/663026 (https://phabricator.wikimedia.org/T274232) (owner: 10David Caro) [19:28:02] (03Merged) 10jenkins-bot: tox-buster: upgrade all images to tox 3.21.4 [integration/config] - 10https://gerrit.wikimedia.org/r/663026 (https://phabricator.wikimedia.org/T274232) (owner: 10David Caro) [19:28:56] (03CR) 10Jforrester: "Psh, dupe of 662744 and 662745. ;-P" [integration/config] - 10https://gerrit.wikimedia.org/r/662912 (https://phabricator.wikimedia.org/T211784) (owner: 10Hashar) [19:29:50] (03Abandoned) 10Jforrester: dockerfiles: Drop npm, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/662744 (https://phabricator.wikimedia.org/T224908) (owner: 10Jforrester) [19:29:54] (03Abandoned) 10Jforrester: dockerfiles: Drop ci-jessie, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/662745 (https://phabricator.wikimedia.org/T224908) (owner: 10Jforrester) [19:31:24] !log Docker: Building and publishing tox-buster &c. with tox 3.21.4 T274232 [19:31:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:31:27] T274232: spicerack: tox creates py39-tests env even when python 3.9 isn't available - https://phabricator.wikimedia.org/T274232 [19:38:21] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO, 10Wikimedia-Site-requests, 10Readers-Web-Backlog (Kanbanana-FY-2020-21), and 2 others: [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10ovasileva) @greg - we’re looking over this today an... [19:39:19] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:40:30] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:41:01] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:41:54] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:46:59] (03PS2) 10Jforrester: jjb: Bump versions to newer tox-buster and related images [integration/config] - 10https://gerrit.wikimedia.org/r/663029 (https://phabricator.wikimedia.org/T274232) (owner: 10David Caro) [19:47:29] (03CR) 10Jforrester: "PS2: Fixed version for tox-docker (0.1.1, not 0.1.2); added job update list." [integration/config] - 10https://gerrit.wikimedia.org/r/663029 (https://phabricator.wikimedia.org/T274232) (owner: 10David Caro) [19:54:44] (03CR) 10Eileen: "Thanks @harshar - The main goal of switching to submodules was to be able to make one change that would allow us to work through the chang" [integration/config] - 10https://gerrit.wikimedia.org/r/661829 (https://phabricator.wikimedia.org/T273822) (owner: 10Eileen) [19:59:51] (03CR) 10Jforrester: [C: 03+2] "All updated jobs deployed; from testing, this seems fixed." [integration/config] - 10https://gerrit.wikimedia.org/r/663029 (https://phabricator.wikimedia.org/T274232) (owner: 10David Caro) [20:01:31] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO, 10Wikimedia-Site-requests, 10Readers-Web-Backlog (Kanbanana-FY-2020-21), and 2 others: [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10ovasileva) @Jdforrester-WMF - could we ask you for... [20:01:53] (03Merged) 10jenkins-bot: jjb: Bump versions to newer tox-buster and related images [integration/config] - 10https://gerrit.wikimedia.org/r/663029 (https://phabricator.wikimedia.org/T274232) (owner: 10David Caro) [20:02:21] 10Continuous-Integration-Infrastructure, 10SRE-tools, 10Patch-For-Review, 10ci-test-error: spicerack: tox creates py39-tests env even when python 3.9 isn't available - https://phabricator.wikimedia.org/T274232 (10Jdforrester-WMF) OK, this seems to work in CI. Before: > 00:05:20.152 py39-unit run-test: co... [20:06:00] (03PS2) 10Jforrester: layout: [operations/software/wmfmariadbpy] Use tox, not tox-mysqld [integration/config] - 10https://gerrit.wikimedia.org/r/662992 (owner: 10Kormat) [20:06:02] (03PS1) 10Jforrester: jjb: Drop wmfmariadbpy-tox-docker, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/663039 [20:07:38] (03CR) 10Jforrester: [C: 03+2] layout: [operations/software/wmfmariadbpy] Use tox, not tox-mysqld [integration/config] - 10https://gerrit.wikimedia.org/r/662992 (owner: 10Kormat) [20:09:43] (03Merged) 10jenkins-bot: layout: [operations/software/wmfmariadbpy] Use tox, not tox-mysqld [integration/config] - 10https://gerrit.wikimedia.org/r/662992 (owner: 10Kormat) [20:10:33] !log layout: [operations/software/wmfmariadbpy] Use tox, not tox-mysqld [20:10:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:16:03] (03CR) 10Jforrester: "> Patch Set 1: Code-Review+1" [integration/config] - 10https://gerrit.wikimedia.org/r/662992 (owner: 10Kormat) [20:16:39] kormat: Congratulations, works well. :-) [20:17:21] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/663039 (owner: 10Jforrester) [20:18:05] AndyRussG: i didn't get https://gerrit.wikimedia.org/g/wikimedia/fundraising/dev/+/a170097a8b7310a4860b3d439580e335af099457 running - looks like it's got a hardcoded username - but the relevant stuff on dev-images looks to build. gonna go ahead and merge/publish those. [20:18:30] wth.. so wmf.30 doesn't have submodules? [20:18:59] oh duh [20:19:00] brennen: ah huh oh there might be a mistke like that.... ahhh thanks!!! also just gonna check that [20:19:00] (03Merged) 10jenkins-bot: jjb: Drop wmfmariadbpy-tox-docker, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/663039 (owner: 10Jforrester) [20:19:03] * twentyafterfour didn't +2 the patch lol [20:19:13] twentyafterfour: :-D [20:19:19] That'd help. [20:19:30] twentyafterfour: i have had that moment at least twice. [20:19:44] brennen: yes that's a mistake heheh I can just substitute for the env var, one sec! [20:20:06] AndyRussG: want me to hold off on publishing images and do a check on that? [20:20:32] up to you - we can always iterate on the images as needed [20:20:39] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2263.codfw.wmnet'] ` an... [20:24:06] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1385.eqiad.wmnet'] ` an... [20:24:33] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1382.eqiad.wmnet'] ` an... [20:37:33] brennen: right of course! hmmm both feedback and +2 support are much appreciated, so whatever is you feel is best eh! thanks again... just fixed the username issue btw [20:37:55] brennen: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/dev/+/644884 [20:37:56] cool, i'll take another crack at it [20:38:12] will running setup.sh twice in a row be a bad idea? [20:38:23] brennen: nice thanks so much! I'm around for all afternoon and evening, so pls don't hesitate to reach out with questions or comments [20:38:31] no problem running it twice, it's meant for that [20:38:53] it'll ask about re-doing anything that could be destructive and more [20:39:36] right on, thanks. [20:42:07] AndyRussG: hmm - i also don't seem to have access to the private config repo, which i'm sure is for the best. [20:43:54] (or at any rate access to the linked task for it) [20:45:05] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:45:29] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:46:01] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:47:39] brennen: yeah the private repo is only fr-tech I'm afraid... it's just got credentials for payment providers [20:48:31] yeah - well, i'll go ahead and get these images published [20:50:02] (03CR) 10Brennen Bearnes: [V: 03+2 C: 03+2] "The" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/632173 (https://phabricator.wikimedia.org/T262976) (owner: 10AndyRussG) [20:51:01] (03CR) 10Brennen Bearnes: [V: 03+2 C: 03+2] "Builds cleanly locally. Difficult for me to test properly, but looks ok in general and we can iterate if something doesn't work." [releng/dev-images] - 10https://gerrit.wikimedia.org/r/632173 (https://phabricator.wikimedia.org/T262976) (owner: 10AndyRussG) [20:53:01] (03CR) 10Brennen Bearnes: [V: 03+2 C: 03+2] "LGTM. Publishing along with the earlier patch for creating fundraising-buster-php73-apache2-xdebug, per discussion in #wikimedia-releng y" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/635361 (https://phabricator.wikimedia.org/T262976) (owner: 10AndyRussG) [20:54:59] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1299.eqiad.wmnet'] ` an... [20:56:06] brennen: cool thanks so so much! [20:56:32] brennen: does that mean it'll be built and published automagically to the official registry? [20:57:37] there's a manual step at the moment [20:58:02] ...and i just realized that i neglected to check this rebased against master, so something is borked. investigating. [21:00:03] Project mwcore-phpunit-coverage-master build #1207: 04FAILURE in 6 hr 0 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1207/ [21:00:30] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:04:22] brennen: ahh oops... hmmm I can rebase [21:04:32] funny, I don't know what the conflict could be [21:04:47] i don't think there's a conflict per se, but a local test build failed. just running again to check. [21:06:15] realized i had an out-of-date local docker-pkg, so plausibly that's also a factor somehow, though i don't really know what... [21:06:32] well, it built fine this time, so i'm choosing to believe that was it. :) [21:08:12] !log Updating dev-images docker-pkg files on primary contint for [[gerrit:635361]] and [[gerrit:632173]] [21:08:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:29:16] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1384.eqiad.wmnet'] ` an... [21:29:22] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2260.codfw.wmnet'] ` an... [21:29:50] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1381.eqiad.wmnet'] ` an... [21:38:23] 10Release-Engineering-Team (Local Dev), 10SRE, 10dev-images, 10docker-pkg: docker-pkg: "certificate verify failed: unable to get local issuer certificate" for docker-registry.discovery.wmnet when publishing dev-images from contint2001 - https://phabricator.wikimedia.org/T274306 (10brennen) [21:39:31] 10Release-Engineering-Team (Local Dev), 10SRE, 10dev-images, 10docker-pkg, 10User-brennen: docker-pkg: "certificate verify failed: unable to get local issuer certificate" for docker-registry.discovery.wmnet when publishing dev-images from contint2001 - https://phabricator.wikimedia.org/T274306 (10brennen... [21:40:15] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:41:19] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:41:22] note for folks using docker-pkg on contint2001: currently broken for dev-images per T274306. unclear if it'll come up for other stuff. [21:41:23] T274306: docker-pkg: "certificate verify failed: unable to get local issuer certificate" for docker-registry.discovery.wmnet when publishing dev-images from contint2001 - https://phabricator.wikimedia.org/T274306 [21:43:32] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:54:36] 10Release-Engineering-Team, 10FR-Docker, 10Fundraising-Backlog, 10dev-images, and 2 others: dev-images +2 rights and Docker registry credentials for FR-Tech - https://phabricator.wikimedia.org/T274303 (10brennen) [22:09:54] !log attempting to run docker-pkg manually for dev-images on contint2001 [22:09:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:14:46] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1298.eqiad.wmnet'] ` an... [22:23:34] 10Release-Engineering-Team (Local Dev), 10SRE, 10dev-images, 10docker-pkg, 10User-brennen: docker-pkg: "certificate verify failed: unable to get local issuer certificate" for docker-registry.discovery.wmnet when publishing dev-images from contint2001 - https://phabricator.wikimedia.org/T274306 (10brennen... [22:24:08] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1373.eqiad.wmnet'] ` an... [22:25:33] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2259.codfw.wmnet'] ` an... [22:26:44] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:27:28] !log once more, with feeling: attempting docker-pkg run for dev-images again [22:27:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:27:55] legoktm: I'm going to blame you for someone putting Rust into my Python joy. ;) T274310 [22:27:56] T274310: Building wheels for cryptography package requires rust compiler - https://phabricator.wikimedia.org/T274310 [22:28:14] ooh [22:29:07] bd808: that's unfortunate. but let me know if I can help. we have the latest rust toolchain in the "releng/rust" docker image [22:30:02] not sure if you can do a multi-stage build for the rust part first and then copy it over to the python part, or if it has to be present for the Python part too [22:30:03] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:30:20] tried to scap sync-file on beta to add some quick debug logging, and got a failure: [22:30:20] 22:29:42 sync-file failed: [Errno 2] No such file or directory: '/srv/mediawiki-staging/php-master/extensions/.git/modules/WindowsAzureStorage' [22:30:45] legoktm: I'm going to start by trying to figure out how to hint to pip that it can just use the manylinux wheel [22:31:19] oh, https://gerrit.wikimedia.org/r/mediawiki/extensions/WindowsAzureStorage is "not found" ? [22:31:58] but https://gerrit.wikimedia.org/g/mediawiki/extensions/WindowsAzureStorage works fine, so the extension is still there... [22:32:41] tried again, got a different failure: 22:32:03 stderr: Fatal error: Uncaught Exception: Unable to open file /srv/mediawiki-staging/php-master/extensions/PagedTiffHandler/extension.json: filemtime(): stat failed for /srv/mediawiki-staging/php-master/extensions/PagedTiffHandler/extension.json in [22:32:41] /srv/mediawiki-staging/php-master/includes/registration/ExtensionRegistry.php:176 [22:32:42] any ideas? [22:35:27] What are you trying to sync? [22:35:29] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:36:23] I was trying to add a wfDebug line to figure out a bug with GlobalWatchlist [22:36:33] That doesn't answer the question [22:36:39] What file/directory are you trying to sync? [22:37:31] sorry. It was skin.php [22:37:31] dannys712@deployment-deploy01:/srv/mediawiki-staging$ scap sync-file php-master/includes/skins/Skin.php 'Debugging GlobalWatchlist' [22:37:46] I reverted my changes for now though [22:38:00] since I can reproduce the issue locally and its probably easier to debug there [22:40:32] hmm, actually I can't reproduce it locally... [22:43:56] Scap is taking a while [23:00:11] okay. I'll try my debugging again later [23:13:21] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1372.eqiad.wmnet'] ` an... [23:21:54] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1380.eqiad.wmnet'] ` Of... [23:24:26] beta cluster really isn't great for adhoc debugging like that [23:26:25] Anyone around to give this a look? should be pretty straightforward! https://gerrit.wikimedia.org/r/c/integration/config/+/661796 [23:28:54] (03CR) 10Reedy: [C: 03+2] CI jobs for image-suggestion-api service [integration/config] - 10https://gerrit.wikimedia.org/r/661796 (https://phabricator.wikimedia.org/T273225) (owner: 10Nikki Nikkhoui) [23:30:50] (03Merged) 10jenkins-bot: CI jobs for image-suggestion-api service [integration/config] - 10https://gerrit.wikimedia.org/r/661796 (https://phabricator.wikimedia.org/T273225) (owner: 10Nikki Nikkhoui) [23:31:19] Reedy: thanks! [23:39:33] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO, 10Wikimedia-Site-requests, 10Readers-Web-Backlog (Kanbanana-FY-2020-21), and 2 others: [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10Jdlrobson) a:05Jdlrobson→03None Just to set exp... [23:39:59] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1296.eqiad.wmnet'] ` an... [23:43:17] > beta cluster really isn't great for adhoc debugging like that [23:43:17] well, I can't reproduce with just the globalwatchlist extension enabled, so its likely due to either some config used on the WMF wikis or interactions with another extension - see T274312 [23:43:18] T274312: Special:Watchlist not showing a link to Special:GlobalWatchlist - https://phabricator.wikimedia.org/T274312 [23:43:27] marxarelli: are you about by chance? I'm working on T274310 and wondered if you have any strong opinions about the choice of `--only-binary cryptography` or `--prefer-binary` to address my `pip install` problem. [23:43:27] T274310: Building wheels for cryptography package requires rust compiler - https://phabricator.wikimedia.org/T274310 [23:43:42] legoktm: ^ your opinions are welcome as well [23:52:19] bd808: commented [23:52:51] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2250.codfw.wmnet'] ` an... [23:54:21] thanks legoktm. Your take makes sense and is probably something we can actually address with some updates to base images. I think I may have just found a quick hack to unbreak Toolhub. About to test it now locally. [23:56:12] :)) woot