[04:22:24] 10Diffusion, 10Release-Engineering-Team, 10Projects-Cleanup, 10Wikimedia-GitHub, 10User-MarcoAurelio: Delete/Archive Github and Diffusion mirrors of Gerrit repos related wikiba.se - https://phabricator.wikimedia.org/T239429 (10WMDE-leszek) Thanks a lot @MarcoAurelio and @hashar ! [05:17:09] 10Phabricator, 10DBA, 10Operations: replace phabricator db passwords with longer passwords - https://phabricator.wikimedia.org/T250361 (10mmodell) Works for me. We could also do {T146055} at the same time. [05:29:15] 10Phabricator, 10DBA, 10Operations: replace phabricator db passwords with longer passwords - https://phabricator.wikimedia.org/T250361 (10jcrespo) @mmodell could we schedule a specific date for this, so it is not forgotten? How much time do you need to prepare T146055? Work on our side is not too time consum... [05:36:57] 10Phabricator, 10Release-Engineering-Team (Development services), 10Developer Productivity: Make "Related Gerrit patches" easier to click and/or to visually associate its meta data - https://phabricator.wikimedia.org/T244601 (10mmodell) 05Open→03Resolved a:03mmodell @legoktm: good idea, I'll implement... [05:39:56] 10Phabricator, 10Wikimedia-Phabricator-Extensions, 10Developer Productivity: Link from "Related Gerrit patches" header directly to gerrit search results - https://phabricator.wikimedia.org/T253088 (10DannyS712) [05:50:26] 10Phabricator, 10EngProd-Virtual-Hackathon, 10Epic: Virtual Offsite: Phabricator hacking session - https://phabricator.wikimedia.org/T253090 (10mmodell) [05:50:41] 10Phabricator, 10EngProd-Virtual-Hackathon, 10Epic: Virtual Offsite: Phabricator hacking session - https://phabricator.wikimedia.org/T253090 (10mmodell) [05:52:16] 10Phabricator, 10EngProd-Virtual-Hackathon, 10Epic: Virtual Offsite: Phabricator hacking session - https://phabricator.wikimedia.org/T253090 (10mmodell) [05:52:19] 10Phabricator, 10Wikimedia-Phabricator-Extensions, 10Developer Productivity: Link from "Related Gerrit patches" header directly to gerrit search results - https://phabricator.wikimedia.org/T253088 (10mmodell) [05:52:21] 10Phabricator (Upstream), 10Upstream: Disable parent task metadata by default for new sub tasks - https://phabricator.wikimedia.org/T239378 (10mmodell) [05:52:23] 10Diffusion, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Developer Productivity: Reduce task notification noise/frequency of changes to associated open patchsets - https://phabricator.wikimedia.org/T143162 (10mmodell) [05:52:26] 10Deployments, 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Development services), and 2 others: Indicate deployed status (wmf branch) for Gerrit patches in Phabricator - https://phabricator.wikimedia.org/T88136 (10mmodell) [06:00:42] 10Phabricator, 10EngProd-Virtual-Hackathon, 10Epic: Virtual Offsite: Phabricator hacking session - https://phabricator.wikimedia.org/T253090 (10mmodell) p:05Triage→03Medium [06:20:33] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10ReleaseTaggerBot, 10Epic: [EPIC] Code Deploy Dashboard - https://phabricator.wikimedia.org/T280 (10Mynightmare1918) [07:09:51] 10Phabricator: Collapse "Related Gerrit Patches" when there are too many - https://phabricator.wikimedia.org/T237816 (10Ammarpad) [07:09:54] 10Phabricator, 10EngProd-Virtual-Hackathon, 10Epic: Virtual Offsite: Phabricator hacking session - https://phabricator.wikimedia.org/T253090 (10Ammarpad) [07:11:03] 10Phabricator: Restore old "merged" icon in Maniphest's Gerrit info block - https://phabricator.wikimedia.org/T252866 (10Ammarpad) [07:11:06] 10Phabricator, 10EngProd-Virtual-Hackathon, 10Epic: Virtual Offsite: Phabricator hacking session - https://phabricator.wikimedia.org/T253090 (10Ammarpad) [07:11:57] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10ReleaseTaggerBot, 10Epic: [EPIC] Code Deploy Dashboard - https://phabricator.wikimedia.org/T280 (10RhinosF1) [07:17:05] Whoever just disabled ^ I warned them on mediawiki.org first but thanks! [07:33:37] Reedy: can I steal you to do some code review soon? [07:38:55] Reedy: could you check https://phabricator.wikimedia.org/T253070's patches are ready and I will create backports this morning? [08:37:08] (03PS1) 10Filippo Giunchedi: operations-puppet: ship logstash and logstash-filter-verifier [integration/config] - 10https://gerrit.wikimedia.org/r/597227 (https://phabricator.wikimedia.org/T251869) [08:43:05] (03CR) 10Filippo Giunchedi: "This is the change I mentioned on the task, please let me know what you think!" [integration/config] - 10https://gerrit.wikimedia.org/r/597227 (https://phabricator.wikimedia.org/T251869) (owner: 10Filippo Giunchedi) [08:50:00] <_joe_> hi! I want to publish some go code from gerrit, and AIUI we need https://gerrit.googlesource.com/plugins/go-import/ in order to enable it [08:50:35] _joe_ that's already installed [08:50:38] <_joe_> and it's installed, but it doesn't seem to work with e.g. blubber [08:50:59] <_joe_> oh nevermind, it works [08:51:02] lol [08:51:15] <_joe_> I made s/blubber/blubberoid/ [08:51:17] <_joe_> sigh [08:51:27] _joe_ needs a cup o' joe (coffee) :) [08:52:03] <_joe_> hauskatze: nah this is a classic mistake, given blubberoid is the production service, and blubber the name of the repo [08:52:07] paladox: I don't think I've ever used go-import; I can't remember [08:52:23] heh [08:55:55] (03CR) 10Awight: "Great idea!" (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592415 (owner: 10Thiemo Kreuz (WMDE)) [09:37:15] 10Deployments, 10User-RhinosF1: Namespace & Sitename localisations deployment - 19 May 2020 - https://phabricator.wikimedia.org/T253070 (10RhinosF1) [09:40:15] (03CR) 10Thiemo Kreuz (WMDE): Full rewrite of the UnusedUseStatementSniff, 4x faster (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592415 (owner: 10Thiemo Kreuz (WMDE)) [09:42:17] 10Deployments, 10User-RhinosF1: Namespace & Sitename localisations deployment - 19 May 2020 - https://phabricator.wikimedia.org/T253070 (10RhinosF1) @MarcoAurelio has said a full scap & l10n update is needed. I checked wikiversions.json last night and see 4 impacted wikis to run namespaceDupes.php on. [09:58:22] 10Phabricator, 10EngProd-Virtual-Hackathon: Extract data for burn down charts - https://phabricator.wikimedia.org/T253109 (10dom_walden) [10:00:25] 10Deployments, 10User-RhinosF1: Namespace & Sitename localisations deployment - 19 May 2020 - https://phabricator.wikimedia.org/T253070 (10RhinosF1) [10:17:21] 10Phabricator, 10EngProd-Virtual-Hackathon: Extract data for burn down charts - https://phabricator.wikimedia.org/T253109 (10Aklapper) [Burndown charts already exist](https://phabricator.wikimedia.org/project/reports/1274/). [Random unmaintained external stuff](http://phlogiston.wmflabs.org/) parsing weird dat... [10:19:01] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Wikimedia-Logstash, 10observability, and 2 others: Regression test for Logstash filters - https://phabricator.wikimedia.org/T251869 (10Joe) I think we're at the point where it would be best if w... [10:28:06] PROBLEM - jenkins_service_running on releases2001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/java .*-jar /usr/share/jenkins/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [10:29:36] ... [10:29:38] ffs [10:34:38] May 19 10:33:54 releases2001 systemd[1]: Stopped Jenkins Continuous Integration Server. [10:34:41] May 19 10:33:54 releases2001 systemd[1]: Started Jenkins Continuous Integration Server. [10:43:53] 10Gerrit, 10Pywikibot: Recursive clone over ssh uses https for submodules - https://phabricator.wikimedia.org/T57654 (10Aklapper) 05Stalled→03Open Nothing stalled in this task as questoin has been answered, hence setting status to open [10:49:56] RECOVERY - jenkins_service_running on releases2001 is OK: PROCS OK: 1 process with regex args .*/bin/java .*-jar /usr/share/jenkins/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [10:50:56] ^ fixed with https://gerrit.wikimedia.org/r/c/operations/puppet/+/597255 [11:08:18] 10MediaWiki-Codesniffer, 10User-thiemowmde: Allow float|int documentation for strictly typed float parameters - https://phabricator.wikimedia.org/T251722 (10thiemowmde) 05Open→03Invalid I just realized we do not have a sniff that compares strict type hints with the types mentioned in `@param` tags. The onl... [11:26:10] addshore: I meant to also mention, there is a minimal amount of scripting contained in the base images in https://gerrit.wikimedia.org/r/plugins/gitiles/releng/dev-images/+/master (see e.g. entrypoint.sh in https://gerrit.wikimedia.org/r/plugins/gitiles/releng/dev-images/+/master/dockerfiles/stretch-php72-fpm-apache2/) [11:27:39] entrypoint.sh makes sense to keep, but probably once `mw` is packaged for distribution then the `install.sh` script in there should be removed. [11:51:56] mutante: good catch [12:07:29] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream: Duplicated 'should should' in Phabricator:arcanist-core english messages sent for translation - https://phabricator.wikimedia.org/T201138 (10Nikerabbit) My expectation is that Release Engineering updates the Phabricator message definitions in... [12:15:45] (03CR) 10Hashar: [C: 03+2] Add mediawiki/skins/WikimediaApiPortal to layout.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/596797 (https://phabricator.wikimedia.org/T247837) (owner: 10Reedy) [12:16:38] (03Merged) 10jenkins-bot: Add mediawiki/skins/WikimediaApiPortal to layout.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/596797 (https://phabricator.wikimedia.org/T247837) (owner: 10Reedy) [12:20:32] (03CR) 10Hashar: [C: 03+2] "> Deploying this is awkward, as it depends on a skin that isn't in gerrit so it's hard to add it as a dependancy..." [integration/config] - 10https://gerrit.wikimedia.org/r/596797 (https://phabricator.wikimedia.org/T247837) (owner: 10Reedy) [12:46:11] (03CR) 10Hashar: [C: 04-1] "I understand the intent but that sounds way too verbose to me. I have a few issues with that pattern:" [integration/quibble] - 10https://gerrit.wikimedia.org/r/546469 (owner: 10Awight) [13:04:41] (03CR) 10Hashar: [C: 04-1] "The retry module catch exceptions (by default instances of Exception which mean anything) and reinvoke the method." [integration/quibble] - 10https://gerrit.wikimedia.org/r/590393 (https://phabricator.wikimedia.org/T202893) (owner: 10Awight) [13:05:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services): Drop MediaWiki testing in stretch and instead test only in buster - https://phabricator.wikimedia.org/T252432 (10hashar) [13:11:30] (03CR) 10Reedy: "Well, yeah. That's another issue. I don't know how you really deal with that. It's seemingly decided a long time ago to deploy Chameleon, " [integration/config] - 10https://gerrit.wikimedia.org/r/596797 (https://phabricator.wikimedia.org/T247837) (owner: 10Reedy) [13:13:40] (03PS1) 10JMeybohm: helm-linter: New helm version (2.16.7-2) [integration/config] - 10https://gerrit.wikimedia.org/r/597278 (https://phabricator.wikimedia.org/T252428) [13:14:34] (03CR) 10jerkins-bot: [V: 04-1] helm-linter: New helm version (2.16.7-2) [integration/config] - 10https://gerrit.wikimedia.org/r/597278 (https://phabricator.wikimedia.org/T252428) (owner: 10JMeybohm) [13:15:34] (03PS2) 10JMeybohm: helm-linter: New helm version (2.16.7-2) [integration/config] - 10https://gerrit.wikimedia.org/r/597278 (https://phabricator.wikimedia.org/T252428) [13:18:47] 10Deployments, 10User-RhinosF1: Namespace & Sitename localisations deployment - 19 May 2020 - https://phabricator.wikimedia.org/T253070 (10Reedy) Why are we backporting namespace changes anyway? Can they just not ride the train? [13:19:35] 10Deployments, 10User-RhinosF1: Namespace & Sitename localisations deployment - 19 May 2020 - https://phabricator.wikimedia.org/T253070 (10RhinosF1) >>! In T253070#6148490, @Reedy wrote: > Why are we backporting namespace changes anyway? Can they just not ride the train? They require namespaceDupes.php to be... [13:22:28] 10phan, 10ci-test-error: seccheck-docker job failing on find delete step randomly/intermittently - https://phabricator.wikimedia.org/T196126 (10hashar) 05Open→03Declined Changed projects since we nowadays have #phan and #ci-test-error We lack the build context since the CI build got deleted a while ago.... [13:22:31] 10Deployments, 10User-RhinosF1: Namespace & Sitename localisations deployment - 19 May 2020 - https://phabricator.wikimedia.org/T253070 (10Reedy) >>! In T253070#6148491, @RhinosF1 wrote: >>>! In T253070#6148490, @Reedy wrote: >> Why are we backporting namespace changes anyway? Can they just not ride the train?... [13:23:00] 10Continuous-Integration-Infrastructure, 10Zuul: zuul should comment on gerrit when it fails to enqueue patches due to patch-chains - https://phabricator.wikimedia.org/T196910 (10hashar) [13:27:00] 10Continuous-Integration-Infrastructure, 10Jenkins: CI console output is messy for wmf-quibble-vendor-mysql-hhvm-docker - https://phabricator.wikimedia.org/T206867 (10hashar) 05Open→03Resolved a:03hashar We lack the console since it is garbage collected. Since this task has been opened, we have enhanced... [13:28:59] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Jenkins: Make the new Jenkins login page match the fancy gerrit login page - https://phabricator.wikimedia.org/T207762 (10hashar) 05Open→03Declined We lack time for this. I gu... [13:33:10] Reedy: simply, we decided there was less of chance of us needing to run it multiple times or changes due to train reverting etc if we just backported everything and ran the script in one session. [13:34:37] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Jenkins: Enable MFA on Jenkins - https://phabricator.wikimedia.org/T198814 (10hashar) I have read again the blog post which is merely a FAQ entry. There are plugins for SAML, Open... [13:34:55] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Jenkins: Enable MFA on Jenkins - https://phabricator.wikimedia.org/T198814 (10hashar) [13:39:43] 10Continuous-Integration-Config, 10LuaSandbox: Run LuaSandbox tests against PHP with ZTS - https://phabricator.wikimedia.org/T206591 (10hashar) [13:49:39] 10Deployments, 10User-RhinosF1: Namespace & Sitename localisations deployment - 19 May 2020 - https://phabricator.wikimedia.org/T253070 (10RhinosF1) > There's also no reason it can't be run after the train runs. I'd suspect the number of affected pages is going to be minimal It could be. We chose this method... [13:50:45] Reedy: I spoke with thcipriani first as well + the one for + sitename and meta namespace is a config patch [13:53:28] and? [13:56:15] Reedy: config part can’t ride with train [13:56:23] And? [13:56:30] We have SWAT windows for that [13:56:43] Reedy: they might as well all be done in one window together [13:57:20] It really makes minimal difference [13:57:56] my suggestion was that this go out with swat in case we hit the situation where we roll forward, run namespacedupes, theen have to roll back [13:58:15] RoanKattouw’s happy to do it and it gets it done. As long as we’re insane by the end, all’s good. [13:58:17] cleaner/less to think about doing swat [13:58:27] ^ [14:09:11] _joe_: blubber is also (was first) a stand-alone software, not powering a service ;) (iow: to me, blubberoid is secondary to blubber when I think of it) [14:09:19] Reedy: if it's common for the new namespaces to ride the train, and for the config to go out with SWAT where the swatter runs namespacedupes: that sounds fine to me. The part that rides the train is a little opaque to me. [14:09:46] my view of this has been: train conductors don't normally run namespacedupes :) [14:09:52] <_joe_> greg-g: it used to, but then blubberoid won [14:10:29] _joe_: g'morning :) [14:11:48] <_joe_> all hail the power of blubberoid [14:12:31] thcipriani: a lot of the time they are probably there on creation [14:12:38] Or the config patch at least [14:13:06] 10Deployments, 10User-RhinosF1: Namespace & Sitename localisations deployment - 19 May 2020 - https://phabricator.wikimedia.org/T253070 (10Reedy) >>! In T253070#6148573, @RhinosF1 wrote: > Overall, it’s slightly safer, ensures the script is ran straight away, ensures they remain translated once deployed even i... [14:14:45] Reedy: Are we still okay to go ahead with the deployment in your opinion? [14:15:04] Yes, if others have signed off on it [14:15:11] But it's really not needed [14:15:41] Reedy: thcipriani was happy and RoanKattouw agreed to deploy so should be fine. [14:16:30] Hauskatze originally suggested SWAT’ing them [14:29:59] (03CR) 10Awight: "Okay, I'm fine with not going this route. Mostly I'm just too sloppy to trust myself with a dozen parameters, u know I'm gonna swap them " [integration/quibble] - 10https://gerrit.wikimedia.org/r/546469 (owner: 10Awight) [14:30:12] (03Abandoned) 10Awight: PEP 3102: Require keyword arguments [integration/quibble] - 10https://gerrit.wikimedia.org/r/546469 (owner: 10Awight) [14:32:15] (03CR) 10Awight: "> The retry module catch exceptions (by default instances of" [integration/quibble] - 10https://gerrit.wikimedia.org/r/590393 (https://phabricator.wikimedia.org/T202893) (owner: 10Awight) [15:16:58] 10Release-Engineering-Team-TODO, 10Core Platform Team, 10CPT Initiatives (API Gateway), 10User-Ladsgroup, and 3 others: New Public Wiki for the API Portal - https://phabricator.wikimedia.org/T246945 (10apaskulin) [16:00:09] 10Beta-Cluster-Infrastructure: Setup monitoring for database servers in beta cluster - https://phabricator.wikimedia.org/T87093 (10Aklapper) 05Stalled→03Open The previous comments don't explain what/who exactly this task is stalled on (["If a report is waiting for further input (e.g. from its reporter or a t... [16:00:11] 10Beta-Cluster-Infrastructure, 10observability, 10Tracking-Neverending: Setup monitoring for Beta Cluster (tracking) - https://phabricator.wikimedia.org/T53497 (10Aklapper) [16:00:15] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Epic: Use Beta cluster as a true canary for code deployments (epic) - https://phabricator.wikimedia.org/T53494 (10Aklapper) [16:00:19] 10Beta-Cluster-Infrastructure, 10observability, 10Tracking-Neverending: Setup monitoring for Beta Cluster (tracking) - https://phabricator.wikimedia.org/T53497 (10Aklapper) 05Stalled→03Open The previous comments don't explain what/who exactly this task is stalled on (["If a report is waiting for further... [16:03:11] 10Phabricator, 10Puppet: Create puppet role for Phabricator hosted repo testing - https://phabricator.wikimedia.org/T104827 (10Aklapper) 05Stalled→03Declined Boldly declining as `phab-02` does not exist anymore and as it seems that nobody is interested in this anymore. If I misunderstood, please reopen. [16:13:34] 10Beta-Cluster-Infrastructure, 10observability, 10Tracking-Neverending: Setup monitoring for Beta Cluster (tracking) - https://phabricator.wikimedia.org/T53497 (10greg) 05Open→03Stalled Reflecting reality of team resourcing. [16:13:36] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Epic: Use Beta cluster as a true canary for code deployments (epic) - https://phabricator.wikimedia.org/T53494 (10greg) [16:13:41] 10Beta-Cluster-Infrastructure: Setup monitoring for database servers in beta cluster - https://phabricator.wikimedia.org/T87093 (10greg) 05Open→03Stalled Reflecting reality of team resourcing. [16:13:43] 10Beta-Cluster-Infrastructure, 10observability, 10Tracking-Neverending: Setup monitoring for Beta Cluster (tracking) - https://phabricator.wikimedia.org/T53497 (10greg) [16:20:26] kostajh: even I end up using a custom install script for my development, perhaps all those things can be included in the standard mediawiki install PHP [16:20:44] For my environment that is moving local settings out of the way doing an install the moving it back again [16:21:14] Basically only using installer.php to create database tables, add local settings somewhere else [16:21:44] Probably something as simple as --no-localsettings in install.php would work... [16:26:57] 10Continuous-Integration-Infrastructure: Rebuild PHP 7.3 containers for new PHP version - https://phabricator.wikimedia.org/T212815 (10hashar) 05Open→03Resolved We have rebuild the CI containers a few time since January 2019. releng/php73 currently has PHP 7.3.16 [16:29:50] (03CR) 10Jforrester: [C: 03+2] helm-linter: New helm version (2.16.7-2) [integration/config] - 10https://gerrit.wikimedia.org/r/597278 (https://phabricator.wikimedia.org/T252428) (owner: 10JMeybohm) [16:30:52] (03Merged) 10jenkins-bot: helm-linter: New helm version (2.16.7-2) [integration/config] - 10https://gerrit.wikimedia.org/r/597278 (https://phabricator.wikimedia.org/T252428) (owner: 10JMeybohm) [16:32:22] (03PS1) 10Jforrester: jjb: [helm-lint] Switch to new helm version (2.16.7-2) [integration/config] - 10https://gerrit.wikimedia.org/r/597307 (https://phabricator.wikimedia.org/T252428) [16:33:03] !log Docker: Publishing helm-linter:0.2.2 T252428 [16:33:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:33:05] T252428: Make helm upgrades atomic - https://phabricator.wikimedia.org/T252428 [16:35:57] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/597307 (https://phabricator.wikimedia.org/T252428) (owner: 10Jforrester) [16:36:53] (03Merged) 10jenkins-bot: jjb: [helm-lint] Switch to new helm version (2.16.7-2) [integration/config] - 10https://gerrit.wikimedia.org/r/597307 (https://phabricator.wikimedia.org/T252428) (owner: 10Jforrester) [16:56:33] 10Phabricator, 10Developer-Advocacy: Decrease issues created many years ago with no recent activity (aka stale tickets) - https://phabricator.wikimedia.org/T248034 (10Jdlrobson) > I think I am still wondering which actual underlying problem to solve by this proposal/solution, apart from folks working on stuff... [17:03:00] 10Phabricator, 10Release-Engineering-Team-TODO, 10User-brennen: Document a (Wikimedia-specific) Phabricator local install - https://phabricator.wikimedia.org/T253132 (10brennen) [17:05:02] 10Phabricator, 10Release-Engineering-Team-TODO, 10User-brennen: Document a (Wikimedia-specific) Phabricator local install - https://phabricator.wikimedia.org/T253132 (10brennen) I've got a page or two of notes from last time I did this we can probably clean up and put somewhere as a start. [17:35:49] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO: Stop using Differential for code review - https://phabricator.wikimedia.org/T191182 (10bd808) [18:12:45] 10Gerrit, 10Code-Review-Workgroup, 10Discovery, 10User-DannyS712: Cindy-the-browser-test-bot should use verified, not code review - https://phabricator.wikimedia.org/T249981 (10EBernhardson) 05Open→03Resolved a:03EBernhardson Looks to work as intended. [18:50:42] RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found) [18:50:42] PROBLEM - Free space - all mounts on deployment-logstash2 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid datapoints found) deployment-prep.deployment-logstash2.diskspace._var_lib_elasticsearch.byte_percentfree (No valid datapoints found)deployment-prep.deployment-logstash2.diskspace.root.byte_percentfree (<100.00%) [18:50:43] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 95918 bytes in 1.114 second response time [18:50:44] RECOVERY - Free space - all mounts on deployment-deploy02 is OK: OK: All targets OK [18:50:45] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 95629 bytes in 1.046 second response time [18:50:46] PROBLEM - Free space - all mounts on deployment-logstash2 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid datapoints found) deployment-prep.deployment-logstash2.diskspace._var_lib_elasticsearch.byte_percentfree (No valid datapoints found)deployment-prep.deployment-logstash2.diskspace.root.byte_percentfree (<100.00%) [18:50:47] PROBLEM - Parsoid on deployment-parsoid11 is CRITICAL: connect to address 172.16.1.115 and port 8000: Connection refused [18:50:48] PROBLEM - Parsoid on deployment-parsoid11 is CRITICAL: connect to address 172.16.1.115 and port 8000: Connection refused [18:50:50] RECOVERY - Free space - all mounts on integration-agent-docker-1006 is OK: OK: All targets OK [18:50:51] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 52122 bytes in 1.060 second response time [18:50:52] PROBLEM - Free space - all mounts on deployment-logstash2 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid datapoints found) deployment-prep.deployment-logstash2.diskspace._var_lib_elasticsearch.byte_percentfree (No valid datapoints found)deployment-prep.deployment-logstash2.diskspace.root.byte_percentfree (<100.00%) [18:59:05] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10EngProd-Virtual-Hackathon: Make deploy-promote more robust against failures - https://phabricator.wikimedia.org/T251750 (10thcipriani) [19:00:18] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10MediaWiki-Release-Tools, 10EngProd-Virtual-Hackathon: Automate weekly branch cut - https://phabricator.wikimedia.org/T196517 (10thcipriani) We have most of this complete, maybe we could polish thi... [19:09:49] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments: 1.35.0-wmf.32 deployment blockers - https://phabricator.wikimedia.org/T249964 (10Jdlrobson) [19:15:59] (03CR) 10Jforrester: "Good to go from the CI perspective; if SRE confirm they want this, I can roll it out." [integration/config] - 10https://gerrit.wikimedia.org/r/597227 (https://phabricator.wikimedia.org/T251869) (owner: 10Filippo Giunchedi) [19:56:07] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Core Platform Team, 10Quibble, and 2 others: Jenkins no longer fails build if mw-error.log or mw-dberror.log are non-empty - https://phabricator.wikimedia.org/T246358 (10Krinkle) [20:18:17] 10Release-Engineering-Team-TODO, 10Core Platform Team, 10MediaWiki-Debug-Logger, 10Readers-Web-Backlog (Tracking), 10Sustainability (Incident Prevention): Ensure flood of hard-deprecations are caught during (train) deployments - https://phabricator.wikimedia.org/T252923 (10Jdlrobson) [20:30:52] 10Release-Engineering-Team-TODO, 10Core Platform Team, 10MediaWiki-Debug-Logger, 10Readers-Web-Backlog (Tracking), 10Sustainability (Incident Prevention): Ensure flood of hard-deprecations are caught during (train) deployments - https://phabricator.wikimedia.org/T252923 (10BPirkle) p:05Triage→03Medium [20:52:35] 10Beta-Cluster-Infrastructure, 10Core Platform Team, 10Growth-Team, 10Notifications: echostore connection error in Beta Cluster - https://phabricator.wikimedia.org/T252898 (10BPirkle) p:05Triage→03Low @Eevans , do you have insight that will help us categorize this? [20:54:57] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:57:18] 10Beta-Cluster-Infrastructure, 10Core Platform Team, 10Growth-Team, 10Notifications: echostore connection error in Beta Cluster - https://phabricator.wikimedia.org/T252898 (10Eevans) echoseen in deployment-prep doesn't do HTTPS [20:59:46] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 95157 bytes in 1.097 second response time [21:00:27] 10Beta-Cluster-Infrastructure, 10Core Platform Team, 10Growth-Team, 10Notifications: echostore connection error in Beta Cluster - https://phabricator.wikimedia.org/T252898 (10BPirkle) 05Open→03Declined [21:24:57] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10Notifications, 10Core Platform Team Workboards (Clinic Duty Team): echostore connection error in Beta Cluster - https://phabricator.wikimedia.org/T252898 (10BPirkle) 05Declined→03Open a:03Clarakosi Correction: this is us after all. [23:58:31] 10Deployments, 10User-RhinosF1: Namespace & Sitename localisations deployment - 19 May 2020 - https://phabricator.wikimedia.org/T253070 (10RhinosF1) [23:59:44] 10Deployments, 10User-RhinosF1: Namespace & Sitename localisations deployment - 19 May 2020 - https://phabricator.wikimedia.org/T253070 (10RhinosF1) a:05RhinosF1→03Catrope