[06:18:33] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO: Create WMF CI image for Node.js 14 - https://phabricator.wikimedia.org/T267888 (10Krinkle) [06:19:53] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO: Create WMF CI image for Node.js 14 - https://phabricator.wikimedia.org/T267888 (10Krinkle) [06:27:52] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO: Create WMF CI image for Node.js 14 - https://phabricator.wikimedia.org/T267888 (10Krinkle) [06:32:02] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services): Upgrade CI jobs for WMF-deployed projects from Node 10 to Node 14 LTS (tracking) - https://phabricator.wikimedia.org/T267890 (10Krinkle) [06:34:25] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services): Upgrade CI jobs for WMF-deployed projects from Node 10 to Node 14 LTS (tracking) - https://phabricator.wikimedia.org/T267890 (10Krinkle) [06:34:33] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services): Upgrade CI jobs for WMF-deployed projects from Node 10 to Node 14 LTS (tracking) - https://phabricator.wikimedia.org/T267890 (10Krinkle) [06:34:35] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO: Create WMF CI image for Node.js 14 - https://phabricator.wikimedia.org/T267888 (10Krinkle) [06:35:07] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO: Create WMF CI image for Node.js 14 - https://phabricator.wikimedia.org/T267888 (10Krinkle) [06:53:03] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services): Upgrade CI jobs for WMF-deployed projects from Node 10 to Node 14 LTS (tracking) - https://phabricator.wikimedia.org/T267890 (10Joe) [06:53:05] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO: Create WMF CI image for Node.js 14 - https://phabricator.wikimedia.org/T267888 (10Joe) [09:06:43] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services): Upgrade CI jobs for WMF-deployed projects from Node 10 to Node 14 LTS (tracking) - https://phabricator.wikimedia.org/T267890 (10hashar) [09:06:45] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO: Create WMF CI image for Node.js 14 - https://phabricator.wikimedia.org/T267888 (10hashar) [10:03:29] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Add prometheus exporter to Gerrit - https://phabricator.wikimedia.org/T184086 (10fgiunchedi) >>! In T184086#6616580, @fgiunchedi wrote: > The patch is live, unfortunately due to how our... [10:14:36] (03PS2) 10Jack Phoenix: Set up CI for AjaxShowEditors [integration/config] - 10https://gerrit.wikimedia.org/r/641038 [10:59:36] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Add prometheus exporter to Gerrit - https://phabricator.wikimedia.org/T184086 (10hashar) From a discussion with Filippo, the metrics are exposed on the service hostname. That makes it a... [11:00:17] (03CR) 10Hashar: [C: 03+2] Set up CI for AjaxShowEditors [integration/config] - 10https://gerrit.wikimedia.org/r/641038 (owner: 10Jack Phoenix) [11:01:25] (03Merged) 10jenkins-bot: Set up CI for AjaxShowEditors [integration/config] - 10https://gerrit.wikimedia.org/r/641038 (owner: 10Jack Phoenix) [11:02:29] (03CR) 10Hashar: "Deploy" [integration/config] - 10https://gerrit.wikimedia.org/r/641038 (owner: 10Jack Phoenix) [11:21:24] (03CR) 10Hashar: [C: 03+2] Readme updates [integration/composer] - 10https://gerrit.wikimedia.org/r/640573 (owner: 10Reedy) [11:21:42] (03Merged) 10jenkins-bot: Readme updates [integration/composer] - 10https://gerrit.wikimedia.org/r/640573 (owner: 10Reedy) [11:45:04] 10Continuous-Integration-Config, 10LibUp, 10Front-end-Standards-Group, 10MW-1.35-notes (1.35.0-wmf.30; 2020-04-28): Move eslint ignores out of Gruntfile.js or package.json and into .eslintignore for all repos - https://phabricator.wikimedia.org/T249858 (10Volker_E) Repeating from https://phabricator.wikime... [12:08:41] 10Project-Admins, 10Discovery-Search (Current work): Do not dump WikibaseCirrusSearch codebase tasks into some #Discovery-Search team project tag only - https://phabricator.wikimedia.org/T263725 (10Aklapper) I've added `#WikibaseCirrusSearch` as an alias for `#CirrusSearch` to https://phabricator.wikimedia.org... [12:15:06] 10Project-Admins, 10Epic: Replace all "tracking" tasks with tags/projects (if unbounded), relabel them to outcome tasks (if bounded), or kill them entirely (if pointless) - https://phabricator.wikimedia.org/T192655 (10Aklapper) [12:15:09] 10Project-Admins: Replace tracking bug T24710 by new project tag "IIS" - https://phabricator.wikimedia.org/T102505 (10Aklapper) 05Open→03Declined I won't create a tag for a topic with three (3) open tasks and zero tasks created since February 2016. Declining as it looks like nobody is tracking anything and a... [12:50:12] (03CR) 10Kosta Harlan: "Are these images published? I'm trying to test the core change that makes use of them." [releng/dev-images] - 10https://gerrit.wikimedia.org/r/622575 (https://phabricator.wikimedia.org/T245444) (owner: 10Brennen Bearnes) [12:58:53] 10Project-Admins, 10Discovery-Search (Current work): Do not dump WikibaseCirrusSearch codebase tasks into some #Discovery-Search team project tag only - https://phabricator.wikimedia.org/T263725 (10CBogen) >>! In T263725#6623717, @Aklapper wrote: > @CBogen: AFAIK anyone should be able to edit alias hashtags, i... [13:37:10] 10Beta-Cluster-Infrastructure: Puppet failures on many hosts - https://phabricator.wikimedia.org/T267006 (10jbond) [13:37:16] 10Beta-Cluster-Infrastructure, 10OTRS: profile::mail::mx fails Puppet on deployment-mx02 instance due to lack of otrs settings - https://phabricator.wikimedia.org/T267831 (10jbond) 05Open→03Resolved a:03jbond I have added the following to the puppet config in horizon, this equates to the config which is... [13:46:51] 10Beta-Cluster-Infrastructure: Puppet failures on many hosts - https://phabricator.wikimedia.org/T267006 (10jbond) Adding the following to [[ https://github.com/wikimedia/puppet/commit/adda6eb69bfb898cd8133884bef611e8d45a2fb2 | match production ]] has fixed deployment-ores01 ` lang=yanl profile::tlsproxy::env... [13:48:06] (03CR) 10Lars Wirzenius: [C: 03+2] Use SCAP_MW_LANG=en to speed up check script [tools/train-dev] - 10https://gerrit.wikimedia.org/r/640253 (owner: 10Ahmon Dancy) [13:48:14] (03Merged) 10jenkins-bot: Use SCAP_MW_LANG=en to speed up check script [tools/train-dev] - 10https://gerrit.wikimedia.org/r/640253 (owner: 10Ahmon Dancy) [13:53:39] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Puppet failures on many hosts - https://phabricator.wikimedia.org/T267006 (10jbond) [13:54:39] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Puppet failures on many hosts - https://phabricator.wikimedia.org/T267006 (10jbond) deployment-kafka-* servers where all working fine when tested [13:55:00] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Puppet failures on many hosts - https://phabricator.wikimedia.org/T267006 (10jbond) [14:00:29] jbond42: thank you so much for fixing all those beta cluster instances! :] [14:02:25] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Puppet failures on many hosts - https://phabricator.wikimedia.org/T267006 (10jbond) deployment-mdb01 [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/641167 | fixed ]] [14:06:11] 10Beta-Cluster-Infrastructure, 10OTRS: profile::mail::mx fails Puppet on deployment-mx02 instance due to lack of otrs settings - https://phabricator.wikimedia.org/T267831 (10hashar) `name=The ASCII art Barnstar ................. ........X........ ......XXXXX...... ..XXXXXXXXXXXXX.. .....XXXXXXX..... .....XX...... [14:24:45] (03CR) 10Lars Wirzenius: build scap.deb in Docker (034 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/634929 (owner: 10Lars Wirzenius) [14:30:57] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic, 10HTTPS: The certificate for upload.beta.wmflabs.org expired on November 13, 2020. - https://phabricator.wikimedia.org/T267858 (10Vgutierrez) >>! In T267858#6622251, @Krenair wrote: > @Vgutierrez FYI in case this could happen in prod too, I haven't be... [14:32:49] 10Beta-Cluster-Infrastructure: Puppet failures on many hosts - https://phabricator.wikimedia.org/T267006 (10jbond) deployment-wdqs01 should be fixed with https://gerrit.wikimedia.org/r/641171 however there is now a conflict wit a local commit which needs deleting: `c503964991 [LOCAL HACK] Fill placeholder for e... [14:33:03] 10Beta-Cluster-Infrastructure: Puppet failures on many hosts - https://phabricator.wikimedia.org/T267006 (10jbond) [14:34:59] (03PS4) 10Lars Wirzenius: build scap.deb in Docker [tools/scap] - 10https://gerrit.wikimedia.org/r/634929 [14:50:42] 10Beta-Cluster-Infrastructure: Puppet failures on many hosts - https://phabricator.wikimedia.org/T267006 (10jbond) to fix deployment-logstash03: * rename role::logstash::apifeatureusage to profile::logstash::apifeatureusage in horizon * rename role::logstash::collector to profile::logstash::collector in hori... [14:50:54] 10Beta-Cluster-Infrastructure: Puppet failures on many hosts - https://phabricator.wikimedia.org/T267006 (10jbond) [15:03:41] 10Beta-Cluster-Infrastructure: Puppet failures on many hosts - https://phabricator.wikimedia.org/T267006 (10jbond) >>! In T267006#6624104, @jbond wrote: > deployment-wdqs01 should be fixed with https://gerrit.wikimedia.org/r/641171 however there is now a conflict wit a local commit which needs deleting: > `c503... [15:03:54] 10Beta-Cluster-Infrastructure: Puppet failures on many hosts - https://phabricator.wikimedia.org/T267006 (10jbond) [15:09:03] 10Beta-Cluster-Infrastructure: Puppet failures on many hosts - https://phabricator.wikimedia.org/T267006 (10jbond) in relation to deployment-cache-upload06 puppet runs successfully but the following command fails on each execution @Vgutierrez may be able to quickly spot the issue ` lang=shell root@deployment-... [15:55:34] 10Beta-Cluster-Infrastructure: Puppet failures on many hosts - https://phabricator.wikimedia.org/T267006 (10dancy) [16:05:59] (03CR) 10Ahmon Dancy: [C: 04-1] "Works great! Just a typo remains." (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/634929 (owner: 10Lars Wirzenius) [16:14:01] 10Beta-Cluster-Infrastructure, 10DBA, 10Operations, 10serviceops, 10Patch-For-Review: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10ArielGlenn) Note that I ran a little dumps test on a non-latin1 wiki in deployment-prep (ruwiki to be precise) and the results look... [16:16:34] (03PS5) 10Lars Wirzenius: build scap.deb in Docker [tools/scap] - 10https://gerrit.wikimedia.org/r/634929 [16:16:48] (03CR) 10Lars Wirzenius: build scap.deb in Docker (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/634929 (owner: 10Lars Wirzenius) [16:17:29] (03CR) 10Ahmon Dancy: [C: 03+2] build scap.deb in Docker [tools/scap] - 10https://gerrit.wikimedia.org/r/634929 (owner: 10Lars Wirzenius) [16:19:54] (03Merged) 10jenkins-bot: build scap.deb in Docker [tools/scap] - 10https://gerrit.wikimedia.org/r/634929 (owner: 10Lars Wirzenius) [16:23:24] 10Beta-Cluster-Infrastructure: Puppet failures on many hosts - https://phabricator.wikimedia.org/T267006 (10Vgutierrez) it looks like deployment-cache-upload06 has been omitted in T267561: `vgutierrez@deployment-cache-upload06:/etc/varnish$ dpkg -l |grep varnish ii libvarnishapi1 5.1.3-1wm... [16:26:16] 10Beta-Cluster-Infrastructure: Puppet failures on many hosts - https://phabricator.wikimedia.org/T267006 (10hashar) @Vgutierrez definitely. T267561 only focused on the text06 instance and indeed we should have fixed that one as well. Maybe reopen that task to follow up? Some packages might be marked on hold (`... [16:29:22] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic, 10User-Ryasmeen: Beta needs to be upgraded to Varnish 6 - https://phabricator.wikimedia.org/T267561 (10Vgutierrez) 05Resolved→03Open Re-opening as mentioned in https://phabricator.wikimedia.org/T267006#6624466 deployment-cache-upload06 has been om... [16:36:22] 10Phabricator, 10Mail: Phab email account verification not received - https://phabricator.wikimedia.org/T267590 (10Jcross) HI @Aklapper Thanks so much for working on this. When you get to step #5 above - where you receive an email to the new address? I never get that email. [17:00:21] 10LibUp, 10Front-end-Standards-Group: LibUp should automatically re-write .eslintrc.json files into the new .eslint.config.js format when using an appropriate (forthcoming) version of eslint - https://phabricator.wikimedia.org/T267941 (10Jdforrester-WMF) [17:01:52] 10Beta-Cluster-Infrastructure: Puppet failures on many hosts - https://phabricator.wikimedia.org/T267006 (10Vgutierrez) puppet seems to be happy on deployment-cache-upload06 after upgrading to varnish 6 [17:03:35] !log Updating dev-images docker-pkg files on primary contint for T245444 [17:03:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:03:38] T245444: Create PHP 7.3 and PHP 7.4 variants of docker-registry.wikimedia.org/dev/stretch-php72-fpm-apache2-xdebug - https://phabricator.wikimedia.org/T245444 [17:04:57] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic, 10User-Ryasmeen: Beta needs to be upgraded to Varnish 6 - https://phabricator.wikimedia.org/T267561 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez manually ran `apt upgrade` and puppet afterwards.. everything seems ok on deployment-cache-upload06 [17:06:56] (03CR) 10Brennen Bearnes: "> Patch Set 10:" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/622575 (https://phabricator.wikimedia.org/T245444) (owner: 10Brennen Bearnes) [17:08:22] 10Phabricator, 10Mail: Phab email account verification not received - https://phabricator.wikimedia.org/T267590 (10Jcross) @Aklapper what magic did you work? I just got 5 verification emails :) [17:09:05] 10Phabricator, 10Mail: Phab email account verification not received - https://phabricator.wikimedia.org/T267590 (10Jcross) 05Open→03Resolved a:03Jcross [17:29:28] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic, 10User-Ryasmeen: Beta needs to be upgraded to Varnish 6 - https://phabricator.wikimedia.org/T267561 (10hashar) thank you! [18:10:09] 10Phabricator, 10Mail: Phab email account verification not received - https://phabricator.wikimedia.org/T267590 (10Aklapper) I have no idea as I don't have access to mail logs, but glad it worked finally for you :) [19:17:55] 10Beta-Cluster-Infrastructure: Beta cluster has reached its quota - https://phabricator.wikimedia.org/T257118 (10bd808) [19:17:57] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO, 10Code-Stewardship-Reviews: deployment-prep: Code stewardship request - https://phabricator.wikimedia.org/T215217 (10bd808) [19:19:49] 10Beta-Cluster-Infrastructure: Beta cluster has reached its quota - https://phabricator.wikimedia.org/T257118 (10thcipriani) [19:46:05] 10Project-Admins, 10MediaWiki-Vendor, 10User-DannyS712: Create component for `Assert` library - https://phabricator.wikimedia.org/T251076 (10DannyS712) [19:55:45] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic, 10HTTPS: The certificate for upload.beta.wmflabs.org expired on November 13, 2020. - https://phabricator.wikimedia.org/T267858 (10Krenair) 05Open→03Resolved [20:10:07] (03CR) 10Kosta Harlan: "> Patch Set 10:" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/622575 (https://phabricator.wikimedia.org/T245444) (owner: 10Brennen Bearnes) [20:12:05] (03CR) 10Jeena Huneidi: "> Patch Set 10:" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/622575 (https://phabricator.wikimedia.org/T245444) (owner: 10Brennen Bearnes) [20:15:59] 10phan, 10MediaWiki-Core-Testing: Future of tests/phan in MW Core - https://phabricator.wikimedia.org/T267859 (10EBernhardson) Plausibly the directory can be nuked. The post-process script certainly isn't needed anymore, that script implemented per-line issue suppression before phan had it built in. Similarly... [20:24:37] 10phan, 10MediaWiki-Core-Testing: Future of tests/phan in MW Core - https://phabricator.wikimedia.org/T267859 (10Legoktm) I mostly left it around because something (I don't remember what) was still relying on it. At this point it's probably best to just get rid of it and fix whatever fallout, if any, emerges. [20:27:34] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Math, 10Mathoid: Migrate mathoid to use PipelineLib - https://phabricator.wikimedia.org/T261809 (10jeena) [20:27:53] 10Project-Admins, 10MediaWiki-Vendor, 10User-DannyS712: Create component for `Assert` library - https://phabricator.wikimedia.org/T251076 (10Legoktm) Yes, having a proper project for this would be nice to match the other PHP libraries. I don't think there's a real maintainer, maybe @daniel as the primary aut... [20:28:40] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Operations, 10Release Pipeline, and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10jeena) [20:28:45] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Math, 10Mathoid: Migrate mathoid to use PipelineLib - https://phabricator.wikimedia.org/T261809 (10jeena) 05Open→03Resolved The title and description were a bit vague so I changed them to be specif... [20:32:34] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Release Pipeline, 10WVUI, 10Vue.js (Vue.js-Search): Add ability to use username-password credentials in pipelinelib pipelines - https://phabricator.wikimedia.org/T267702 (10jeena) @nnikkhoui Sorry, I... [20:51:19] 10Project-Admins, 10MediaWiki-Vendor, 10User-DannyS712: Create Phab project for wikimedia/assert library - https://phabricator.wikimedia.org/T251076 (10Krinkle) [21:54:16] 10Release-Engineering-Team (Pipeline): pipelinelib should automatically update image versions Cloud VPS nodes that use profile::docker::services - https://phabricator.wikimedia.org/T267975 (10Ottomata) [21:54:41] 10Release-Engineering-Team (Pipeline): pipelinelib should automatically update image versions Cloud VPS nodes that use profile::docker::services - https://phabricator.wikimedia.org/T267975 (10Ottomata) Please prioritize this accordingly, I may be the only one who actually regularly uses `profile::docker::services`. [23:31:22] 10Beta-Cluster-Infrastructure, 10DBA, 10Operations, 10serviceops, 10Patch-For-Review: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10jijiki) We upgraded to ICU 63: appservers, api, parsoid, jobrunners, mwmaint, and snapshot. What is left is deploy*. We are runnin... [23:32:26] 10Beta-Cluster-Infrastructure, 10DBA, 10Operations, 10serviceops, 10Patch-For-Review: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10jijiki)