[00:25:32] <wikibugs>	 (03PS9) 10Jforrester: layout: Drop php70 and php71 from quibble jobs for master and REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/534229 (https://phabricator.wikimedia.org/T228342)
[00:25:49] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] layout: Drop php70 and php71 from quibble jobs for master and REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/534229 (https://phabricator.wikimedia.org/T228342) (owner: 10Jforrester)
[00:27:23] <wikibugs>	 (03Merged) 10jenkins-bot: layout: Drop php70 and php71 from quibble jobs for master and REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/534229 (https://phabricator.wikimedia.org/T228342) (owner: 10Jforrester)
[00:29:01] <James_F>	 !log Zuul: Dropping php70 and php71 from quibble jobs for master and REL1_34 T228342
[00:29:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[00:29:04] <stashbot>	 T228342: Define criteria for setting explicit PHP support target for MediaWiki - https://phabricator.wikimedia.org/T228342
[00:30:11] <wikibugs>	 (03PS1) 10Jforrester: layout: Actually drop HHVM jobs from *nohhvm templates [integration/config] - 10https://gerrit.wikimedia.org/r/537229
[00:30:32] <Krinkle>	 James_F: lol, well, that didn't take long
[00:31:00] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] layout: Actually drop HHVM jobs from *nohhvm templates [integration/config] - 10https://gerrit.wikimedia.org/r/537229 (owner: 10Jforrester)
[00:31:13] <James_F>	 Krinkle: Almost like I had the tabs open for a fortnight. ;-D
[00:31:23] <James_F>	 Krinkle: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/516829 if you're keen.
[00:31:25] <wikibugs>	 (03CR) 10Krinkle: layout: Drop php70 and php71 from quibble jobs for master and REL1_34 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/534229 (https://phabricator.wikimedia.org/T228342) (owner: 10Jforrester)
[00:31:51] <wikibugs>	 (03CR) 10Jforrester: layout: Drop php70 and php71 from quibble jobs for master and REL1_34 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/534229 (https://phabricator.wikimedia.org/T228342) (owner: 10Jforrester)
[00:31:53] <wikibugs>	 (03CR) 10Krinkle: layout: Drop php70 and php71 from quibble jobs for master and REL1_34 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/534229 (https://phabricator.wikimedia.org/T228342) (owner: 10Jforrester)
[00:33:59] <wikibugs>	 (03Merged) 10jenkins-bot: layout: Actually drop HHVM jobs from *nohhvm templates [integration/config] - 10https://gerrit.wikimedia.org/r/537229 (owner: 10Jforrester)
[00:34:14] <wikibugs>	 (03CR) 10Jforrester: [C: 04-2] [DNM] layout: Drop HHVM jobs from -wmf branches [integration/config] - 10https://gerrit.wikimedia.org/r/534520 (https://phabricator.wikimedia.org/T176370) (owner: 10Jforrester)
[00:37:51] <wikibugs>	 (03CR) 10Krinkle: "Yay :)" [integration/config] - 10https://gerrit.wikimedia.org/r/537229 (owner: 10Jforrester)
[01:39:22] <wikibugs>	 10Release-Engineering-Team-TODO (201909), 10MediaWiki-General, 10MW-1.34-release, 10PHP 7.1 support: Drop PHP 7.1 support from MediaWiki - https://phabricator.wikimedia.org/T216166 (10Jdforrester-WMF)
[01:39:24] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-General, 10MW-1.34-release, 10PHP 7.0 support: Drop PHP 7.0 support from MediaWiki - https://phabricator.wikimedia.org/T216165 (10Jdforrester-WMF) 05Stalled→03Resolved a:03Jdforrester-WMF Done....
[01:39:27] <wikibugs>	 10Release-Engineering-Team-TODO (201909), 10MediaWiki-General, 10MW-1.34-release, 10PHP 7.1 support: Drop PHP 7.1 support from MediaWiki - https://phabricator.wikimedia.org/T216166 (10Jdforrester-WMF) 05Stalled→03Resolved a:03Jdforrester-WMF Done. Live in master and will ship in MediaWiki 1.34.0+.
[02:05:10] <wmf-insecte>	 Project mediawiki-core-doxygen-docker build #9912: 04FAILURE in 1 min 6 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/9912/
[03:05:09] <wmf-insecte>	 Project mediawiki-core-doxygen-docker build #9913: 04STILL FAILING in 1 min 5 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/9913/
[04:02:28] <wikibugs>	 10Phabricator, 10Release-Engineering-Team-TODO (201909), 10Wikimedia-Logstash, 10observability, and 2 others: Deploy phatality into kibana - https://phabricator.wikimedia.org/T230752 (10mmodell) @fgiunchedi ^ this might be one way to avoid needing sudo.
[04:05:06] <wmf-insecte>	 Project mediawiki-core-doxygen-docker build #9914: 04STILL FAILING in 1 min 3 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/9914/
[05:05:07] <wmf-insecte>	 Project mediawiki-core-doxygen-docker build #9915: 04STILL FAILING in 1 min 3 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/9915/
[06:05:06] <wmf-insecte>	 Project mediawiki-core-doxygen-docker build #9916: 04STILL FAILING in 1 min 3 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/9916/
[07:05:10] <wmf-insecte>	 Project mediawiki-core-doxygen-docker build #9917: 04STILL FAILING in 1 min 6 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/9917/
[07:58:03] <leszek_wmde>	 top of the morning CI wizards! it looks that with https://gerrit.wikimedia.org/r/c/mediawiki/core/+/516829 landing on master, mwselenium-quibble-docker builds have turned red due to still running PHP 7.0
[07:58:36] <leszek_wmde>	 any plans to update containers soonish or should we go for reverting the mw change? All Wikibase changes are blocked currently
[07:58:53] <wikibugs>	 10Phabricator, 10Release-Engineering-Team-TODO (201909), 10Wikimedia-Logstash, 10observability, and 2 others: Deploy phatality into kibana - https://phabricator.wikimedia.org/T230752 (10fgiunchedi) Thanks @mmodell ! I think on balance I'm leaning towards sudo + `kibana-plugin install`, reason being is that...
[07:59:08] <shinken-wm>	 PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: connect to address 172.16.5.63 and port 8000: Connection refused
[07:59:17] <leszek_wmde>	 will create a phab ticket  if no one works on it already, but I figured I will check quickly here
[07:59:38] <leszek_wmde>	 oh hashar you might be a person I am looking for, but probably missed my essay I just wrote?
[08:00:12] <leszek_wmde>	 hashar: long story short: looks https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/537247 flipped https://integration.wikimedia.org/ci/job/mwselenium-quibble-docker/18136/consoleFull upside down
[08:00:23] <hashar>	 leszek_wmde: good morning :)
[08:00:36] <leszek_wmde>	 hashar: indeed good morning, sorry for my manners
[08:00:42] <hashar>	 leszek_wmde: and yeah I missed everything before my connection, I don't have a bouncer
[08:00:47] <hashar>	 though I can look at the irc log :]
[08:00:56] <leszek_wmde>	 nah,I will repeat
[08:01:05] <leszek_wmde>	 looks that with https://gerrit.wikimedia.org/r/c/mediawiki/core/+/516829 landing on master, mwselenium-quibble-docker builds have turned red due to still running PHP 7.0
[08:01:05] * hashar reads http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-releng/20190917.txt
[08:05:05] <wmf-insecte>	 Project mediawiki-core-doxygen-docker build #9918: 04STILL FAILING in 1 min 1 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/9918/
[08:08:02] <hashar>	 leszek_wmde: I will look at switching it to 7.2
[08:09:29] <hashar>	         - mwselenium-quibble-docker:
[08:09:29] <hashar>	             docker_image: docker-registry.wikimedia.org/releng/quibble-stretch-bundle:0.0.34-1
[08:09:30] <hashar>	 hmm
[08:09:32] <hashar>	 who knows :]
[08:11:02] <leszek_wmde>	 hashar: that's great, thanks
[08:12:41] <wikibugs>	 (03PS1) 10Hashar: docker: switch quibble-stretch-bundle to PHP 7.2 [integration/config] - 10https://gerrit.wikimedia.org/r/537327
[08:12:45] <hashar>	 leszek_wmde: :)
[08:12:46] <hashar>	 ^^
[08:12:53] <hashar>	 I have just changed the parent container
[08:13:01] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] docker: switch quibble-stretch-bundle to PHP 7.2 [integration/config] - 10https://gerrit.wikimedia.org/r/537327 (owner: 10Hashar)
[08:14:34] <wikibugs>	 (03Merged) 10jenkins-bot: docker: switch quibble-stretch-bundle to PHP 7.2 [integration/config] - 10https://gerrit.wikimedia.org/r/537327 (owner: 10Hashar)
[08:15:24] <wikibugs>	 (03PS1) 10Hashar: jjb: switch mw selenium jobs to php7.2 [integration/config] - 10https://gerrit.wikimedia.org/r/537329
[08:19:11] <hashar>	 !log Building docker container releng/quibble-stretch-bundle:0.0.34-2  (switch to php7.2)
[08:19:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[08:20:57] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Jobs updated" [integration/config] - 10https://gerrit.wikimedia.org/r/537329 (owner: 10Hashar)
[08:21:08] <hashar>	 leszek_wmde: I have updated the jobs!
[08:21:34] <leszek_wmde>	 hashar: merci beaucoup, will try rechecking one of the failed ones
[08:23:29] <wikibugs>	 10Continuous-Integration-Config, 10TechCom: On CI, stop testing MediaWiki with php7.0 ahead of dropping support - https://phabricator.wikimedia.org/T225628 (10hashar) The PHP 7.0 and 7.1 jobs have been removed from the pipelines for the master branch! T216165 T216166
[08:23:49] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: switch mw selenium jobs to php7.2 [integration/config] - 10https://gerrit.wikimedia.org/r/537329 (owner: 10Hashar)
[08:27:47] <wikibugs>	 (03Restored) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [selenium] - 10https://gerrit.wikimedia.org/r/330671 (owner: 10Hashar)
[08:27:51] <wikibugs>	 (03CR) 10Hashar: "recheck" [selenium] - 10https://gerrit.wikimedia.org/r/330671 (owner: 10Hashar)
[08:30:45] <hashar>	 leszek_wmde: that works on https://integration.wikimedia.org/ci/job/mediawiki-selenium-integration-docker/9/console
[08:30:50] <hashar>	 a job for mediawiki/selenium.git
[08:30:56] <hashar>	 which contains the ruby code for mediawik_selenium gem
[08:31:06] <hashar>	 and the jobs runs against mediawiki/core@master \o/
[08:31:09] <hashar>	 thank you for the report!
[08:31:25] <wikibugs>	 (03Abandoned) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [selenium] - 10https://gerrit.wikimedia.org/r/330671 (owner: 10Hashar)
[08:40:35] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Release, 10Train Deployments: 1.34.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T220748 (10zeljkofilipin)
[08:47:33] <wikibugs>	 10Release-Engineering-Team, 10Operations, 10observability: Export zuul metrics to Prometheus - https://phabricator.wikimedia.org/T233089 (10fgiunchedi)
[08:48:17] <leszek_wmde>	 hashar: thanks for the speedy fix! All looking in the Wikibase job as well
[08:54:11] <wikibugs>	 10Release-Engineering-Team, 10Operations, 10observability: Export zuul metrics to Prometheus - https://phabricator.wikimedia.org/T233089 (10fgiunchedi)
[08:58:45] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10CirrusSearch, and 2 others: selenium-daily-beta-CirrusSearch is broken - https://phabricator.wikimedia.org/T232495 (10zeljkofilipin) 05Open→03Resolved [[ https://inte...
[08:58:48] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Advanced-Search, 10CirrusSearch, and 8 others: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin)
[08:58:53] <wikibugs>	 10Release-Engineering-Team-TODO, 10MW-1.34-notes (1.34.0-wmf.23; 2019-09-17), 10Patch-For-Review: wdio-cucumber-framework fails on NodeJS due to fibers@2.x under nodejs10 - https://phabricator.wikimedia.org/T229033 (10zeljkofilipin)
[08:59:32] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Advanced-Search, 10CirrusSearch, and 8 others: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin)
[09:00:05] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Advanced-Search, 10CirrusSearch, and 8 others: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin) 05Open→03Resolv...
[09:05:20] <wmf-insecte>	 Project mediawiki-core-doxygen-docker build #9919: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/9919/
[09:11:33] <wikibugs>	 (03PS1) 10Hashar: docker: switch doxygen to use php7.2 [integration/config] - 10https://gerrit.wikimedia.org/r/537337
[09:14:28] <wikibugs>	 (03PS1) 10Hashar: jjb: upgrade doxygen jobs to PHP 7.2 [integration/config] - 10https://gerrit.wikimedia.org/r/537339
[09:14:36] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] docker: switch doxygen to use php7.2 [integration/config] - 10https://gerrit.wikimedia.org/r/537337 (owner: 10Hashar)
[09:15:01] <hashar>	 !log On doxygen jobs, change container to use php7.2 instead of php7.0
[09:15:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[09:16:21] <wikibugs>	 (03Merged) 10jenkins-bot: docker: switch doxygen to use php7.2 [integration/config] - 10https://gerrit.wikimedia.org/r/537337 (owner: 10Hashar)
[09:16:39] <hashar>	 !log Rebuilding releng/doxygen container  to use php7.2
[09:16:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[09:17:36] <hashar>	 [contint1001.wikimedia.org] out: 2019-09-17 09:16:58,142 [docker-pkg-build] INFO - /bin/sh: 1: cannot create /etc/apt/apt.conf.d/80_proxy: Permission denied
[09:17:37] <hashar>	 pfff
[09:19:28] <wikibugs>	 (03PS1) 10Hashar: docker: doxygen needs root for apt_install [integration/config] - 10https://gerrit.wikimedia.org/r/537341
[09:23:31] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "INFO:jenkins_jobs.builder:Number of jobs generated:  5" [integration/config] - 10https://gerrit.wikimedia.org/r/537339 (owner: 10Hashar)
[09:24:24] <hashar>	 !log Manually triggered https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/9920/ (switch to php7.2)
[09:24:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[09:26:26] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: upgrade doxygen jobs to PHP 7.2 [integration/config] - 10https://gerrit.wikimedia.org/r/537339 (owner: 10Hashar)
[09:30:23] <wmf-insecte>	 Yippee, build fixed!
[09:30:23] <wmf-insecte>	 Project mediawiki-core-doxygen-docker build #9920: 09FIXED in 6 min 25 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/9920/
[09:36:33] <hashar>	 hurrah
[09:49:42] <wikibugs>	 10Release-Engineering-Team-TODO, 10Packaging: gbp buildpackage with GIT_PBUILDER_AUTOCONF=no causes DIST to be ignored - https://phabricator.wikimedia.org/T233020 (10hashar)
[09:49:52] <wikibugs>	 10Release-Engineering-Team-TODO, 10Packaging, 10Upstream: gbp buildpackage with GIT_PBUILDER_AUTOCONF=no causes DIST to be ignored - https://phabricator.wikimedia.org/T233020 (10hashar)
[10:05:23] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Quibble, and 2 others: CI: Create a way to share a secret between MediaWiki and the testing framework. - https://phabricator.wikimedia.org/T233092 (10daniel)
[10:05:30] <wikibugs>	 (03PS4) 10Daniel Kinzler: Force value of $wgSecretKey. [integration/quibble] - 10https://gerrit.wikimedia.org/r/537118 (https://phabricator.wikimedia.org/T233092)
[10:33:01] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Quibble, and 3 others: CI: Create a way to share a secret between MediaWiki and the testing framework. - https://phabricator.wikimedia.org/T233092 (10mobrovac) What about using t...
[11:08:56] <wikibugs>	 10Project-Admins: Create a Project Hash Checking - https://phabricator.wikimedia.org/T233060 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #hash-checking has been created: https://phabricator.wikimedia.org/project/profile/4279/  I rephrased and shortened the project description a bit...
[11:11:46] <wikibugs>	 10Release-Engineering-Team-TODO (201909), 10Browser-Tests, 10User-zeljkofilipin: selenium-daily-beta-REPOSITORY Jenkins jobs broken since 2019-09-08 - https://phabricator.wikimedia.org/T232357 (10zeljkofilipin) [[ https://en.wikipedia.beta.wmflabs.org/wiki/Special:ListUsers?username=&group=&creationSort=1&de...
[11:12:08] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Browser-Tests, 10User-zeljkofilipin: selenium-daily-beta-REPOSITORY Jenkins jobs broken since 2019-09-08 - https://phabricator.wikimedia.org/T232357 (10zeljkofilipin)
[11:20:02] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth: [betalabs] Cannot create a new user account - https://phabricator.wikimedia.org/T232796 (10zeljkofilipin) p:05Triage→03High
[11:20:40] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 10User-zeljkofilipin: [betalabs] Cannot create a new user account - https://phabricator.wikimedia.org/T232796 (10zeljkofilipin)
[11:22:58] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Browser-Tests, 10User-zeljkofilipin: selenium-daily-beta-REPOSITORY Jenkins jobs broken since 2019-09-08 - https://phabricator.wikimedia.org/T232357 (10zeljkofilipin)
[11:23:00] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 10User-zeljkofilipin: [betalabs] Cannot create a new user account - https://phabricator.wikimedia.org/T232796 (10zeljkofilipin)
[11:23:30] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 10User-zeljkofilipin: [betalabs] Cannot create a new user account - https://phabricator.wikimedia.org/T232796 (10zeljkofilipin) This task is causing {T232357}.
[11:24:00] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 10User-zeljkofilipin: [betalabs] Cannot create a new user account - https://phabricator.wikimedia.org/T232796 (10zeljkofilipin) [[ https://en.wikipedia.beta.wmflabs.org/wiki/Special:ListUsers?username=&group=&creationSort=1&desc=1&wpsubmit=&w...
[11:24:20] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10MediaWiki-extensions-CentralAuth, 10User-zeljkofilipin: [betalabs] Cannot create a new user account - https://phabricator.wikimedia.org/T232796 (10zeljkofilipin)
[11:26:03] <hashar>	 !log Configured CI jenkins to use a webproxy for the update center due to mysterious "connection reset" when downloading plugins
[11:26:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[11:28:46] <mvolz>	 Can anyone help me interpret why the post-merge build failed here? "timed out waiting for the condition" https://integration.wikimedia.org/ci/job/service-pipeline-test-and-publish/612/console
[11:29:37] <wmf-insecte>	 Project beta-scap-eqiad build #267276: 04FAILURE in 1 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/267276/
[11:30:39] <shinken-wm>	 PROBLEM - Host deployment-prometheus02 is DOWN: CRITICAL - Host Unreachable (172.16.0.67)
[11:31:23] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:31:32] <shinken-wm>	 PROBLEM - Host deployment-aqs01 is DOWN: CRITICAL - Host Unreachable (172.16.1.190)
[11:31:59] <shinken-wm>	 PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:32:18] <shinken-wm>	 PROBLEM - Host deployment-acme-chief04 is DOWN: CRITICAL - Host Unreachable (172.16.6.213)
[11:32:39] <hashar>	 bah
[11:33:37] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2231 bytes in 9.206 second response time
[11:35:01] <shinken-wm>	 PROBLEM - Host deployment-aqs02 is DOWN: CRITICAL - Host Unreachable (172.16.1.5)
[11:35:16] <shinken-wm>	 PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2829 bytes in 6.059 second response time
[11:35:33] <shinken-wm>	 PROBLEM - Host deployment-db06 is DOWN: CRITICAL - Host Unreachable (172.16.4.147)
[11:36:38] <wmf-insecte>	 Project beta-scap-eqiad build #267277: 04STILL FAILING in 2 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/267277/
[11:46:07] <shinken-wm>	 RECOVERY - Host deployment-prometheus02 is UP: PING OK - Packet loss = 0%, RTA = 0.81 ms
[11:46:27] <wmf-insecte>	 Yippee, build fixed!
[11:46:27] <wmf-insecte>	 Project beta-scap-eqiad build #267278: 09FIXED in 1 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/267278/
[11:46:39] <shinken-wm>	 RECOVERY - Host deployment-aqs01 is UP: PING OK - Packet loss = 0%, RTA = 0.56 ms
[11:46:54] <shinken-wm>	 RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 37117 bytes in 6.245 second response time
[11:47:21] <shinken-wm>	 RECOVERY - Host deployment-acme-chief04 is UP: PING OK - Packet loss = 0%, RTA = 0.87 ms
[11:47:49] <shinken-wm>	 RECOVERY - Host deployment-db06 is UP: PING OK - Packet loss = 0%, RTA = 1.42 ms
[11:48:32] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Exception when accessing to a Phabricator mock: "Attempting to add more metadata after metadata has been locked." - https://phabricator.wikimedia.org/T218450 (10abian) 05Open→03Resolved a:03epriestley Thanks! :D
[11:48:38] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 48050 bytes in 6.789 second response time
[11:50:03] <shinken-wm>	 RECOVERY - Host deployment-aqs02 is UP: PING OK - Packet loss = 0%, RTA = 1.48 ms
[11:50:18] <shinken-wm>	 RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 48587 bytes in 6.572 second response time
[11:51:20] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 48070 bytes in 5.354 second response time
[11:57:20] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Quibble, and 3 others: CI: Create a way to share a secret between MediaWiki and the testing framework. - https://phabricator.wikimedia.org/T233092 (10daniel) >>! In T233092#54986...
[12:06:16] <wmf-insecte>	 Project mediawiki-core-doxygen-docker build #9923: 04FAILURE in 2 min 12 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/9923/
[12:10:11] <hashar>	 grblblblb
[12:11:20] <hashar>	 !log Remove Jenkins CI webproxy configuration , that breaks jobs using the git plugin!
[12:11:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[12:20:38] <wmf-insecte>	 Yippee, build fixed!
[12:20:38] <wmf-insecte>	 Project mediawiki-core-doxygen-docker build #9924: 09FIXED in 7 min 2 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/9924/
[12:23:08] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] [OAuth] Add phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/536748 (https://phabricator.wikimedia.org/T224761) (owner: 10Umherirrender)
[12:24:52] <wikibugs>	 (03Merged) 10jenkins-bot: [OAuth] Add phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/536748 (https://phabricator.wikimedia.org/T224761) (owner: 10Umherirrender)
[12:24:57] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10MediaWiki-extensions-CentralAuth, 10User-zeljkofilipin: [betalabs] Cannot create a new user account - https://phabricator.wikimedia.org/T232796 (10zeljkofilipin) a:03zeljkofilipin
[12:27:30] <wikibugs>	 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.34-notes, 10MW-1.34-release: Announce 1.34 "pencils down" and likely timeline to wikitech-l etc. - https://phabricator.wikimedia.org/T232026 (10CCicalese_WMF) 05Open→03Resolved a:03CCicalese_WMF https://lists.wikimedia.org/pipermail/wikitech-l/2019-...
[12:27:32] <wikibugs>	 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.34-notes, 10MW-1.34-release: Release MW 1.34 - https://phabricator.wikimedia.org/T232023 (10CCicalese_WMF)
[12:39:44] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10MediaWiki-extensions-CentralAuth, 10User-zeljkofilipin: [betalabs] Cannot create a new user account - https://phabricator.wikimedia.org/T232796 (10zeljkofilipin) I've checked https://logstash-beta.wmflabs.org. I have created a filte...
[12:41:23] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10MediaWiki-extensions-CentralAuth, 10User-zeljkofilipin: [betalabs] Cannot create a new user account - https://phabricator.wikimedia.org/T232796 (10zeljkofilipin) a:05zeljkofilipin→03None I'm blocked on this. I need help to figur...
[12:53:56] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/536748 (https://phabricator.wikimedia.org/T224761) (owner: 10Umherirrender)
[12:54:27] <hashar>	 zeljkof: logstash on beta is broken
[12:54:36] <hashar>	 zeljkof: there is nothing logged with _type:mediawiki
[12:54:45] <zeljkof>	 hashar: oh, nice :)
[12:55:06] <zeljkof>	 oh well, I do need help with figuring out what's wrong with beta
[12:55:07] <hashar>	 so that needs a task
[12:55:15] <hashar>	 add a few random projects tags
[12:55:35] <hashar>	 and hopefully what ever is broken in logstash ends up being logged in logstash?: D
[12:57:11] <zeljkof>	 recursion alert 
[13:11:23] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10puppet-compiler: Puppet catalog compiler - increasing max concurrent jobs - https://phabricator.wikimedia.org/T221969 (10hashar)
[13:48:01] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10puppet-compiler: Puppet catalog compiler - increasing max concurrent jobs - https://phabricator.wikimedia.org/T221969 (10hashar) 05Stalled→03Open To workaround the insanely slow puppet run from T228056, I commented out `base::resolvin...
[14:06:18] <wikibugs>	 10Release-Engineering-Team, 10Wikimedia-Site-requests: "empty" InitialiseSettings.php breaks a ton of links and references to that file - https://phabricator.wikimedia.org/T233069 (10Jdforrester-WMF)
[14:15:14] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-extensions-LoginNotify, 10MW-1.35-release: Bundle LoginNotify extension with MediaWiki - https://phabricator.wikimedia.org/T191746 (10Jdforrester-WMF)
[14:15:27] <wikibugs>	 10MediaWiki-Releasing, 10Growth-Team, 10Thanks, 10MW-1.35-release: Bundle Thanks extension with MediaWiki - https://phabricator.wikimedia.org/T191739 (10Jdforrester-WMF)
[14:15:36] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.34-release: Expand the set of bundled extensions and skins in MediaWiki 1.34 - https://phabricator.wikimedia.org/T209220 (10Jdforrester-WMF)
[14:15:38] <wikibugs>	 10MediaWiki-Releasing, 10Growth-Team, 10Notifications, 10MW-1.35-release: Bundle Echo extension with MediaWiki - https://phabricator.wikimedia.org/T191738 (10Jdforrester-WMF) 05Open→03Stalled Based on the determination above, and that clearly we are not going to get DB schemata written, tested, and mer...
[14:15:41] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-extensions-LoginNotify, 10MW-1.35-release: Bundle LoginNotify extension with MediaWiki - https://phabricator.wikimedia.org/T191746 (10Jdforrester-WMF)
[14:15:44] <wikibugs>	 10MediaWiki-Releasing, 10Growth-Team, 10Thanks, 10MW-1.35-release: Bundle Thanks extension with MediaWiki - https://phabricator.wikimedia.org/T191739 (10Jdforrester-WMF)
[14:18:39] <wikibugs>	 10MediaWiki-Releasing, 10Growth-Team, 10Thanks, 10MW-1.35-release: Bundle Thanks extension with MediaWiki - https://phabricator.wikimedia.org/T191739 (10CCicalese_WMF)
[14:18:43] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.35-release: Expand the set of bundled extensions and skins in MediaWiki 1.35 - https://phabricator.wikimedia.org/T232854 (10CCicalese_WMF)
[14:18:48] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.34-release: Expand the set of bundled extensions and skins in MediaWiki 1.34 - https://phabricator.wikimedia.org/T209220 (10CCicalese_WMF)
[14:19:39] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.34-release: Expand the set of bundled extensions and skins in MediaWiki 1.34 - https://phabricator.wikimedia.org/T209220 (10CCicalese_WMF)
[14:19:51] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.35-release: Expand the set of bundled extensions and skins in MediaWiki 1.35 - https://phabricator.wikimedia.org/T232854 (10CCicalese_WMF)
[14:22:41] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10cloud-services-team: Check bandwidth limitation on integration-castor03.integration.eqiad.wmflabs / cloudvirt1002 - https://phabricator.wikimedia.org/T232644 (10hashar) p...
[14:29:28] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10cloud-services-team: Check bandwidth limitation on integration-castor03.integration.eqiad.wmflabs / cloudvirt1002 - https://phabricator.wikimedia.org/T232644 (10hashar)
[14:30:43] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10cloud-services-team: Check bandwidth limitation on integration-castor03.integration.eqiad.wmflabs / cloudvirt1002 - https://phabricator.wikimedia.org/T232644 (10hashar) W...
[14:31:24] <hashar>	 !log Raise egress filter trafic shaping on integration-castor03 from 240Mbit to 400Mbit ( T232644 / T188375 )
[14:31:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:31:28] <stashbot>	 T232644: Check bandwidth limitation on integration-castor03.integration.eqiad.wmflabs / cloudvirt1002 - https://phabricator.wikimedia.org/T232644
[14:31:28] <stashbot>	 T188375: castor rsync's taking 3-5 minutes for mwgate-npm jobs - https://phabricator.wikimedia.org/T188375
[14:45:41] <hashar>	 so I flushed a good part of my email queue hurrah
[15:00:17] <wikibugs>	 10Continuous-Integration-Config, 10TechCom: On CI, stop testing MediaWiki with php7.0 ahead of dropping support - https://phabricator.wikimedia.org/T225628 (10Jdforrester-WMF) 05Open→03Invalid
[15:02:08] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "At least there is no more output buffering.  Note that the traffic happens on the loopback interface which (iirc) is a bit magic and is pr" [integration/quibble] - 10https://gerrit.wikimedia.org/r/534468 (https://phabricator.wikimedia.org/T219694) (owner: 10Ladsgroup)
[15:03:23] <wikibugs>	 (03Merged) 10jenkins-bot: Disable output buffering in running php [integration/quibble] - 10https://gerrit.wikimedia.org/r/534468 (https://phabricator.wikimedia.org/T219694) (owner: 10Ladsgroup)
[15:09:34] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 64.29% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[15:18:57] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10MediaWiki-extensions-CentralAuth, 10User-zeljkofilipin: [betalabs] Cannot create a new user account - https://phabricator.wikimedia.org/T232796 (10zeljkofilipin)
[15:18:59] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Release, 10Train Deployments: 1.34.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T220748 (10zeljkofilipin)
[15:32:07] <hashar>	 !log Tagging quibble 0.0.35 48f6b1d8db9eed8572f255fe86f920753149153c # T225730 T231862 T219694
[15:32:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:32:14] <stashbot>	 T225730: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730
[15:32:15] <stashbot>	 T231862: Selenium tests for Wikibase are being ran twice - https://phabricator.wikimedia.org/T231862
[15:32:15] <stashbot>	 T219694: Enable compression for MW web responses in Jenkins Jenkins (e.g. Quibble, Fresnel) - https://phabricator.wikimedia.org/T219694
[15:32:51] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10MediaWiki-extensions-CentralAuth, 10User-zeljkofilipin: [betalabs] Cannot create a new user account - https://phabricator.wikimedia.org/T232796 (10zeljkofilipin) p:05High→03Unbreak! In discussion with @Jdforrester-WMF we've deci...
[15:32:51] <James_F>	 Ooh, new quibble.
[15:35:02] <Lucas_WMDE>	 yaay
[15:35:11] <Lucas_WMDE>	 no more running Wikibase tests twice ^^
[15:38:03] <James_F>	 Lucas_WMDE: Maybe. We'll see if it indeed works. :-)
[15:38:10] <wikibugs>	 (03PS1) 10Hashar: docker: rebuild for quibble 0.0.35 [integration/config] - 10https://gerrit.wikimedia.org/r/537469 (https://phabricator.wikimedia.org/T219694)
[15:38:26] <hashar>	 I cant deploy it right now though
[15:38:32] <wikibugs>	 (03CR) 10Jforrester: [C: 03+1] docker: rebuild for quibble 0.0.35 [integration/config] - 10https://gerrit.wikimedia.org/r/537469 (https://phabricator.wikimedia.org/T219694) (owner: 10Hashar)
[15:38:39] <James_F>	 hashar: Want me to take it over?
[15:38:44] <hashar>	 hmm or maybe i can
[15:38:48] <hashar>	 the thing is
[15:38:56] <hashar>	 I am too afraid of blindly upgrading everything single jobs at once
[15:39:12] <hashar>	 so I think what I did last time was to craft several jjb changes
[15:39:24] <hashar>	 carefully update the jobs ie the non voting ones at first if there are any
[15:39:28] <hashar>	 and check what was going on
[15:39:40] <James_F>	 Sounds reasonable.
[15:39:42] <hashar>	 but maybe that can be blindly done instead and then deal with the aftermath
[15:39:56] <hashar>	 I will rebuild them
[15:39:59] <James_F>	 Well, certainly *non-voting as a first push isn't too hard to do. ;-)
[15:40:03] <hashar>	 !log Building docker images for Quibble 0.0.35
[15:40:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:40:20] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] docker: rebuild for quibble 0.0.35 [integration/config] - 10https://gerrit.wikimedia.org/r/537469 (https://phabricator.wikimedia.org/T219694) (owner: 10Hashar)
[15:40:28] <hashar>	 lets see
[15:40:38] <hashar>	 and those two weeks of train delayed all that madness and reviews
[15:40:55] <hashar>	 thanksfully I am not the only one to review / +1/+2 patchesanymore hyeahhhh
[15:40:55] <James_F>	 Yeah. :-(
[15:41:59] <wikibugs>	 (03Merged) 10jenkins-bot: docker: rebuild for quibble 0.0.35 [integration/config] - 10https://gerrit.wikimedia.org/r/537469 (https://phabricator.wikimedia.org/T219694) (owner: 10Hashar)
[15:43:28] <hashar>	 rebuilding
[15:46:01] <hashar>	 also fun thing
[15:46:09] <hashar>	 Debian does not upgrade chromium on Stretch :-\
[15:46:17] <hashar>	 so we are locked to 73
[15:46:35] <hashar>	 to grab 76 we would need buster
[15:46:38] <hashar>	 fun .. :-\
[15:47:05] <James_F>	 hashar: We could migrate to buster but maybe as a different change?
[15:47:19] <hashar>	 it is going to be a pain in the a** again :-\
[15:47:24] <hashar>	 but yeah
[15:47:37] <hashar>	 we would then need php7.2 ported to buster
[15:47:52] <hashar>	 and I don't think the mwapp servers are migrating anytime soon. I guess that depends on phasing out hhvm
[15:49:00] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.34-release: Expand the set of bundled extensions and skins in MediaWiki 1.34 - https://phabricator.wikimedia.org/T209220 (10Physikerwelt)
[15:51:40] <paladox>	 https://groups.google.com/forum/#!topic/repo-discuss/oDvOhpimkBQ ui survey if anyone wants to fill it out (For PolyGerrit)
[15:56:26] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[16:04:55] <hashar>	 !log Quibble 0.0.35 containers for CI have been build.
[16:04:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:04:58] <hashar>	 that is all for now
[16:04:59] <hashar>	 ;)
[16:05:47] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "I have build the containers and they are published." [integration/config] - 10https://gerrit.wikimedia.org/r/537469 (https://phabricator.wikimedia.org/T219694) (owner: 10Hashar)
[16:07:02] <wikibugs>	 (03CR) 10jenkins-bot: Disable output buffering in running php [integration/quibble] - 10https://gerrit.wikimedia.org/r/534468 (https://phabricator.wikimedia.org/T219694) (owner: 10Ladsgroup)
[16:38:57] <wikibugs>	 (03PS1) 10Jforrester: jjb: Upgrade all quibble jobs to 0.0.35 [integration/config] - 10https://gerrit.wikimedia.org/r/537485 (https://phabricator.wikimedia.org/T219694)
[16:39:36] <James_F>	 hashar: ^^ Yeah, it's a lot of jobs. :-) There are only 5 non-voting ones, so I'll push those first?
[16:44:02] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Patch-For-Review, 10User-zeljkofilipin: Cypress testing framework evaluation - https://phabricator.wikimedia.org/T230729 (10hashar) A note, from https://opensource.intuit.com/ I found cyphell to convert from wdio to cypress. https://github.com/intuit/cyph...
[16:46:53] <James_F>	 !log jjb: Experimentally updated mwext-php72-phan-seccheck-docker-non-voting
[16:46:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:47:32] <hashar>	 James_F: yeah the non voting first sounds good
[16:47:55] <hashar>	 I havent tested the containers for all the use case that got merged :-\
[16:48:16] <hashar>	 there are a few jobs that only affect a few repositories such as  * ooui-npm-special-node10-plus-composer-docker * parsoidsvc-parsertests-docker
[16:48:21] <James_F>	 hashar: Yeah, the magic of using `git add -f output/` to find out which jobs changed. ;-)
[16:48:34] <James_F>	 Oh, hmm, mwext-php72-phan-seccheck-docker-non-voting is entirely unused now.
[16:48:46] <hashar>	 * wikibase-client-docker * wikibase-repo-docker   as well, Wikibase should be quiet soon since it is the evening ini europe
[16:48:52] <James_F>	 As is mwskin-php72-phan-docker-non-voting
[16:49:29] <hashar>	 the non voting phan jobs are probably left over from the early days of adding phan
[16:49:30] <hashar>	 though
[16:49:47] <hashar>	 I gotta fill tasks / dig into it. the way the junit report is written is dated
[16:49:56] <James_F>	 Yes.
[16:50:02] <hashar>	 newer phan now has an option to write the junit xml directly to a file 
[16:50:04] <James_F>	 But no matches in zuul/
[16:50:08] <hashar>	 instead of us having to |tee foo.xml
[16:50:22] <hashar>	 and those ojbs could then have the junit plugin added to them for a nicer output
[16:50:26] <James_F>	 The only non-voting jobs we actually have live are mwcore-codehealth-master-non-voting and mwext-codehealth-master-non-voting
[16:50:36] <Amir1>	 let me know if tests break after the deployment
[16:51:08] <James_F>	 !log jjb: Experimentally updated mwext-codehealth-master-non-voting
[16:51:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:52:37] <James_F>	 Amir1: Thanks, will do.
[16:55:27] <James_F>	 OK, first build has run and all looks well.
[16:55:56] <James_F>	 https://integration.wikimedia.org/ci/job/mwext-codehealth-master-non-voting/1627/ reran as https://integration.wikimedia.org/ci/job/mwext-codehealth-master-non-voting/1629/ 
[16:56:23] <James_F>	 Seems to have run a bit faster even with the quibble image fetch at the start.
[16:56:57] <James_F>	 Same dependencies (different order, as expected).
[16:58:30] <James_F>	 OK, I'm going to declare that bit working and move on.
[16:58:52] <James_F>	 !log jjb: Updating mwcore-codehealth-patch experimentally
[16:58:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:16:44] <James_F>	 !log jjv: Updating mwcore-phpunit-coverage-master,mwcore-phpunit-coverage-patch
[17:16:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:26:21] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201909): logstash-beta.wmflabs.org does not receive any mediawiki events - https://phabricator.wikimedia.org/T233134 (10hashar)
[17:27:23] <James_F>	 Why does mwcore-phpunit-coverage-patch run `php -d zend_extension=xdebug.so tests/phpunit/phpunit.php --coverage-clover` twice? Surely once is enough? It almost doubles the execution time…
[17:27:59] <James_F>	 Oh, right, before/after. Never mind me. :-)
[17:29:13] <James_F>	 !log jjb: Updated mwext-codehealth-patch and mwext-codehealth-master-non-voting
[17:29:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:30:43] <James_F>	 !log jjv: Updated mwext-php72-phan-docker, mwext-php72-phan-seccheck-docker, mwskin-php72-phan-docker, mwskin-php72-phan-seccheck-docker
[17:30:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:33:44] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.35-release: Expand the set of bundled extensions and skins in MediaWiki 1.35 - https://phabricator.wikimedia.org/T232854 (10CCicalese_WMF)
[17:33:46] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic: Expand the set of bundled extensions and skins to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349 (10CCicalese_WMF)
[17:35:12] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.34-release: Expand the set of bundled extensions and skins in MediaWiki 1.34 - https://phabricator.wikimedia.org/T209220 (10CCicalese_WMF)
[17:35:29] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.35-release: Expand the set of bundled extensions and skins in MediaWiki 1.35 - https://phabricator.wikimedia.org/T232854 (10CCicalese_WMF)
[17:36:53] <James_F>	 !log jjb: Updating mediawiki-fresnel-patch-docker
[17:36:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:38:14] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10Wikimedia-Logstash, 10observability: logstash-beta.wmflabs.org does not receive any mediawiki events - https://phabricator.wikimedia.org/T233134 (10hashar) So the logstash input for udp tries to bind on 11514 but rsyslogd is already...
[17:39:48] <wikibugs>	 10MediaWiki-Releasing, 10AntiSpoof, 10MW-1.35-release: Bundle AntiSpoof extension with MediaWiki - https://phabricator.wikimedia.org/T191736 (10CCicalese_WMF) It looks like there is some work still necessary to getting this to be configured correctly by default for the web installer. I'm bumping it to 1.35.
[17:40:19] <hashar>	 and apparently logstash on beta has been broken for a few weeks (at least) :\  https://phabricator.wikimedia.org/T233134
[17:40:23] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.35-release: Expand the set of bundled extensions and skins in MediaWiki 1.35 - https://phabricator.wikimedia.org/T232854 (10CCicalese_WMF)
[17:40:25] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.34-release: Expand the set of bundled extensions and skins in MediaWiki 1.34 - https://phabricator.wikimedia.org/T209220 (10CCicalese_WMF)
[17:40:48] <James_F>	 !log jjb: Updated wikibase-client-docker, wikibase-repo-docker
[17:40:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:40:53] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.34-release: Expand the set of bundled extensions and skins in MediaWiki 1.34 - https://phabricator.wikimedia.org/T209220 (10CCicalese_WMF)
[17:41:05] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.35-release: Expand the set of bundled extensions and skins in MediaWiki 1.35 - https://phabricator.wikimedia.org/T232854 (10CCicalese_WMF)
[17:49:40] <James_F>	 !log jjb: Updating mwext-phpunit-coverage*
[17:49:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:51:00] <James_F>	 !log jjb: Updating parsoidsvc-parsertests-docker
[17:51:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:55:24] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki-jhuneidi is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 2046 bytes in 0.054 second response time
[17:56:11] <James_F>	 OK, time for the main event.
[17:56:25] <James_F>	 !log jjb: Upgrading mediawiki-quibble-vendor-mysql-php72-docker
[17:56:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:56:35] <hashar>	 James_F: :]
[17:59:27] <hauskatze>	 (things start to suddenly break royaly :P )
[17:59:37] <James_F>	 Well, let's hope not.
[17:59:43] <hauskatze>	 Amén.
[17:59:45] <James_F>	 We're 40 jjb jobs into this deploy. :-)
[18:00:09] <James_F>	 Entertainingly, I don't know if I broke the world yet, because CI is starved for executors and so the test job is waiting.
[18:00:12] * James_F sighs.
[18:03:37] <James_F>	 hashar: BTW, jenkins is moaning about version issues with Blue Ocean plugins; should I worry?
[18:13:36] <wikibugs>	 10Continuous-Integration-Config, 10Wikidata-Tainted-References-Sprint1: Make vue linting and testing run with CI - https://phabricator.wikimedia.org/T233107 (10Jdforrester-WMF)
[18:14:19] <James_F>	 !log jjb: Upgrading wmf-quibble-vendor-mysql-php72-docker
[18:14:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[18:28:40] <wikibugs>	 10Continuous-Integration-Infrastructure, 10MediaWiki-General, 10Performance-Team (Radar), 10Technical-Debt (Deprecation): Quibble should not rely on dynamically detecting the value of $wgServer - https://phabricator.wikimedia.org/T233140 (10daniel)
[18:29:10] <wikibugs>	 10Continuous-Integration-Infrastructure, 10MediaWiki-General, 10Performance-Team (Radar): Quibble should not rely on dynamically detecting the value of $wgServer - https://phabricator.wikimedia.org/T233140 (10daniel)
[18:31:08] <James_F>	 Woo-hoo. No duplicate Wikibase selenium jobs in https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-php72-docker/8692/console vs. https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-php72-docker/8663/consoleFull
[18:31:11] <James_F>	 Success!
[18:31:53] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/537485 (https://phabricator.wikimedia.org/T219694) (owner: 10Jforrester)
[18:32:20] <James_F>	 !log jjb: Updated the rest of the quibble jobs to 0.0.35.
[18:32:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[18:34:07] <James_F>	 Fingers crossed.
[18:37:14] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: Upgrade all quibble jobs to 0.0.35 [integration/config] - 10https://gerrit.wikimedia.org/r/537485 (https://phabricator.wikimedia.org/T219694) (owner: 10Jforrester)
[18:40:20] <wikibugs>	 10Continuous-Integration-Config, 10Quibble, 10ci-test-error: Selenium tests for Wikibase are being ran twice - https://phabricator.wikimedia.org/T231862 (10Jdforrester-WMF) 05Open→03Resolved Confirmed fixed – https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-php72-docker/8663/consoleFull...
[18:40:22] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201909): CI mediawiki/core run times have increased since July 26th - https://phabricator.wikimedia.org/T232626 (10Jdforrester-WMF)
[18:44:30] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quibble: Quibble jobs error (non-fatal) "ERROR:zuul.Repo:Unable to initialize repo for npm-test.git" - https://phabricator.wikimedia.org/T233143 (10Krinkle)
[18:46:48] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quibble: Quibble jobs error (non-fatal) "ERROR:zuul.Repo:Unable to initialize repo for npm-test.git" - https://phabricator.wikimedia.org/T233143 (10Jdforrester-WMF) (This task is from before the 0.0.35 quibble upgrade.)
[18:50:27] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quibble: Quibble jobs error (non-fatal) "ERROR:zuul.Repo:Unable to initialize repo for npm-test.git" - https://phabricator.wikimedia.org/T233143 (10Jdforrester-WMF) Is it possible that `quibble_args: '--skip selenium npm-test'` is being mis-inter...
[19:11:52] <hashar>	 James_F: yeah there is an issue with the jenkins plugins update :D
[19:11:54] <hashar>	 :\
[19:12:04] <hashar>	 anyway, restarting Jenkins should dimiss the warning
[19:12:12] <hashar>	 I should have filled a task I guess
[19:13:07] <hashar>	 I am off *wave*
[19:22:36] <James_F>	 Bye
[19:34:18] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quibble: Quibble jobs error (non-fatal) "ERROR:zuul.Repo:Unable to initialize repo for npm-test.git" - https://phabricator.wikimedia.org/T233143 (10Krinkle) Re-confirmed on 0.0.35 as well ([build](https://integration.wikimedia.org/ci/job/mediawik...
[19:34:58] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Patch-For-Review, 10User-zeljkofilipin: Cypress testing framework evaluation - https://phabricator.wikimedia.org/T230729 (10Jpita) Oh wow, that's amazing. This way we can actually compare times and error rate between frameworks and easily evaluate which i...
[19:35:17] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quibble: Quibble jobs error (non-fatal) "ERROR:zuul.Repo:Unable to initialize repo for npm-test.git" - https://phabricator.wikimedia.org/T233143 (10Jdforrester-WMF) integration/config is the only repo we have with the string `npm-test` in it, so...
[19:44:32] <wmf-insecte>	 Project beta-code-update-eqiad build #264073: 04FAILURE in 1 min 30 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/264073/
[19:54:31] <wmf-insecte>	 Yippee, build fixed!
[19:54:32] <wmf-insecte>	 Project beta-code-update-eqiad build #264074: 09FIXED in 1 min 31 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/264074/
[20:22:38] <wikibugs>	 (03CR) 10Brennen Bearnes: [C: 03+2] restbase: bump image to 0.0.1-2 for image rebuild [releng/dev-images] - 10https://gerrit.wikimedia.org/r/534516 (owner: 10Brennen Bearnes)
[20:37:43] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10MediaWiki-extensions-CentralAuth, 10User-zeljkofilipin: [betalabs] Cannot create a new user account - https://phabricator.wikimedia.org/T232796 (10hashar) I would recommend to push 1.34.0-wmf.23 to testwiki and attempt to reproduce...
[20:40:48] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T220746 (10mmodell) 05Open→03Resolved ^ log messages picked up the wrong task id becaus...
[20:46:12] <wikibugs>	 10Beta-Cluster-Infrastructure: Fatal error: Invalid operand type was used: cannot perform this operation with arrays in /srv/mediawiki/php-master/languages/Language.php on line 514 - https://phabricator.wikimedia.org/T233155 (10MarcoAurelio)
[20:46:38] <wikibugs>	 10Beta-Cluster-Infrastructure: Fatal error: Invalid operand type was used: cannot perform this operation with arrays in /srv/mediawiki/php-master/languages/Language.php on line 514 - https://phabricator.wikimedia.org/T233155 (10MarcoAurelio)
[20:48:00] <hauskatze>	 deployment-mediawiki-jhuneidi <-- what's that?
[20:49:25] <James_F>	 hauskatze: It's longma's deployment host.
[20:50:22] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Wikimedia-production-error: Fatal error: Invalid operand type was used: cannot perform this operation with arrays in /srv/mediawiki/php-master/languages/Language.php on line 514 - https://phabricator.wikimedia.org/T233155 (10MarcoAurelio) Re-adding #wikimedia-production-error....
[20:50:42] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Wikimedia-production-error: Fatal error: Invalid operand type was used: cannot perform this operation with arrays in /srv/mediawiki/php-master/languages/Language.php on line 514 - https://phabricator.wikimedia.org/T233155 (10MarcoAurelio)
[20:51:10] <hauskatze>	 James_F: aha, so it's flooding kibana with fatals :|
[20:52:06] <brennen>	 !log Updating dev-images docker-pkg files on contint1001 for new restbase image
[20:52:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[20:53:39] <James_F>	 I didn't even know it was hooked up.
[20:55:31] <longma>	 Should i do something?
[20:58:08] <James_F>	 I guess it shouldn't spam the real Beta Cluster kibana whilst it's not meant to be representative of the rest of the "staging" status of the Beta Cluster.
[20:59:06] <hauskatze>	 To clarify: is this also happening on the "real" mediawiki code?
[21:00:08] <James_F>	 Is what?
[21:00:44] <longma>	 How do I stop it from logging? should I just stop the instance for now?
[21:05:02] <hauskatze>	 James_F: the Language.php logspam I reported
[21:05:11] <hauskatze>	 anyway, got to go
[21:15:10] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10MediaWiki-extensions-CentralAuth, 10User-zeljkofilipin: [betalabs] Cannot create a new user account - https://phabricator.wikimedia.org/T232796 (10Tgr) This is a double-submit protection using a memcached lock. Memcached on beta ten...
[21:24:15] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10MediaWiki-extensions-CentralAuth, 10User-zeljkofilipin: [betalabs] Cannot create a new user account - https://phabricator.wikimedia.org/T232796 (10Tgr) ` Sep 17 21:09:02 deployment-deploy02:  PHP Warning:  PHP Startup: Unable to loa...
[21:26:09] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10MediaWiki-extensions-CentralAuth, 10User-zeljkofilipin: [betalabs] Cannot create a new user account - https://phabricator.wikimedia.org/T232796 (10Etonkovidova) I checked  `testwiki wmf.23` -  creating accounts seem to be ok; so it'...
[21:26:53] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10MediaWiki-extensions-CentralAuth, 10User-zeljkofilipin: [betalabs] Cannot create a new user account - https://phabricator.wikimedia.org/T232796 (10Jdforrester-WMF)
[21:26:56] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Release, 10Train Deployments: 1.34.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T220748 (10Jdforrester-WMF)
[21:27:06] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10MediaWiki-extensions-CentralAuth, 10User-zeljkofilipin: [betalabs] Cannot create a new user account - https://phabricator.wikimedia.org/T232796 (10Jdforrester-WMF) p:05Unbreak!→03High
[21:27:14] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10MediaWiki-extensions-CentralAuth, 10User-zeljkofilipin: [betalabs] Cannot create a new user account - https://phabricator.wikimedia.org/T232796 (10Jdforrester-WMF) Not a train-blocker, then.
[21:27:38] <James_F>	 twentyafterfour: Congrats, no train blockers any more. ;-)
[21:27:54] <twentyafterfour>	 nice, thanks James_F
[21:28:08] <James_F>	 Still a bad bug in Beta Cluster.
[21:28:36] <hauskatze>	 twentyafterfour: when you got some spare time I assigned a task to you today :-) re k18
[21:29:14] <twentyafterfour>	 hauskatze: ok I'll take care of it, thanks
[21:41:20] <Reedy>	 Is gerrit on a go slow?
[21:41:24] <Reedy>	 Or is it my end?
[21:41:45] <hauskatze>	 It's loading okay for me
[21:41:48] <hauskatze>	 I think
[21:41:55] * paladox checks
[21:42:20] <paladox>	 https://gerrit.wikimedia.org/r/monitoring looks fine to me
[21:42:38] <Reedy>	 push/pulls seem slow
[21:43:06] <paladox>	 Which repo?
[21:43:15] <Reedy>	 mw core, mw-config
[21:43:38] <paladox>	 I guess we could run gc on those repos
[21:43:47] <paladox>	 not sure if that'll make a difference though.
[21:44:04] <paladox>	 cc thcipriani ^
[21:44:31] <hauskatze>	 there's a button in gerrit for that
[21:44:39] <paladox>	 yup
[21:44:52] <paladox>	 You'll be waiting quite a while for mw-core to do :P
[21:44:53] <hauskatze>	 I misclicked it sometimes
[21:45:12] <paladox>	 it's probably better to use the cli rather then the ui to do gc for the larger repos.
[21:45:22] <hauskatze>	 Oh, for mw-core/config/puppet it'll probably time out
[21:45:23] <paladox>	 https://gerrit-review.googlesource.com/Documentation/cmd-gc.html
[21:48:35] <paladox>	 hauskatze i was about to ask does it ask for a confirmation. But i was the one that implemented that :P
[21:48:53] * paladox needs to add a confirmation prompt to that feature
[21:48:54] <thcipriani>	 Reedy: hrm, logs look normal, seems slower for you to upload than normal: https://phabricator.wikimedia.org/P9122
[21:48:56] <hauskatze>	 I can't remember, it was some time ago
[21:49:11] <Reedy>	 zomg, stalking ;)
[21:49:41] <thcipriani>	 :)
[21:49:54] <Reedy>	 It might just be my connection being hammered by various updates
[21:50:26] <paladox>	 How many updates do you have running that would cause that kind of degradation? :P
[21:50:48] <hauskatze>	 Just once suffices if it is big enough
[21:50:52] <Reedy>	 Steam :)
[21:50:55] <hauskatze>	 hah
[21:50:55] <paladox>	 lol
[21:50:56] <Reedy>	 That'll fill any connection
[21:50:57] <hauskatze>	 xD
[21:51:00] <Reedy>	 plus windows updates
[21:51:14] <paladox>	 Are you on 80mbps?
[21:51:31] <hauskatze>	 ... and today watch on Twitch @reedygames live streaming
[21:52:04] <Reedy>	 Yeah
[21:52:14] <Reedy>	 Didn't see much point paying for 160 down... yet
[21:52:24] <paladox>	 heh
[21:52:37] <paladox>	 Reedy would be awesome to have 160:P
[21:52:47] <Reedy>	 In theory, I can get 330
[21:52:54] <paladox>	 I think gigabit is comming to my town by the end of this year
[21:53:09] <hauskatze>	 I get 100 Mbps
[21:53:09] <paladox>	 Reedy bt ultrafast?
[21:53:15] <Reedy>	 BT FTTP, yes
[21:53:24] <paladox>	 lucky!
[21:53:38] <Reedy>	 One of the reasons I bought my house
[21:53:50] <paladox>	 :D
[21:56:32] <James_F>	 Reedy: FTTC with SDSL downlink, right?
[21:56:41] * paladox has fttc
[21:56:58] <Reedy>	 James_F: lol, no. BT don't offer that unless on ethernet/leased lines
[21:57:09] <James_F>	 Reedy: That's what my parents have.
[21:57:15] <Reedy>	 I've got FTTP
[21:57:21] <Reedy>	 like, proper fibre to my house. No VDSL involved
[21:57:34] <James_F>	 The fibre terminator is in the cabinet just outside the house.
[21:57:35] <Reedy>	 SDSL would normally affect the uplink
[21:57:39] <paladox>	 You doin't really need a VDSL anymore
[21:57:45] <paladox>	 rather it's built into the bt router
[21:57:48] <James_F>	 Total SDSL wire length is ~10 m.
[21:57:52] <Reedy>	 paladox: Eh?
[21:57:54] * James_F shrugs.
[21:57:58] <Reedy>	 VDSL is a technology
[21:58:03] <Reedy>	 like ADSL
[21:58:22] <paladox>	 Oh, righ.
[21:58:32] <paladox>	 thought you were talking about that white box they gave out
[21:59:06] <Reedy>	 That's a modem
[21:59:15] <Reedy>	 My FTTP terminates in my garage into the FTTP CPE
[21:59:37] <hauskatze>	 Oh look, my steam needs to update as well :|
[21:59:50] <Reedy>	 steam always needs an update
[21:59:59] <paladox>	 heh
[22:00:15] <hauskatze>	 and Minerva
[22:01:21] <paladox>	 Reedy sky had an outrage for over 24h a month and a bit ago. No one could use sky in the county i am in.
[22:01:41] <paladox>	 apparently they discovered a fault at the excahnge and decided to kill broadband access whilst they fixed it :P
[22:38:26] <wikibugs>	 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (201909), 10Developer Productivity, 10local-charts: Update local-charts repository to use restbase chart from deployment-charts repo - https://phabricator.wikimedia.org/T228915 (10jeena)
[22:38:45] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10Release Pipeline: Experiment with hosted kubernetes solutions for Beta - https://phabricator.wikimedia.org/T222820 (10jeena)
[22:40:31] <wikibugs>	 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (201909), 10Developer Productivity, 10local-charts: Move restbase chart from local-charts to deployment-charts repository - https://phabricator.wikimedia.org/T228910 (10jeena)
[22:47:21] <wikibugs>	 (03PS1) 10Jeena Huneidi: Use the restbase chart from releases.wikimedia.org/charts [releng/local-charts] - 10https://gerrit.wikimedia.org/r/537562 (https://phabricator.wikimedia.org/T228915)
[23:05:54] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10Wikimedia-Logstash, 10observability: logstash-beta.wmflabs.org does not receive any mediawiki events - https://phabricator.wikimedia.org/T233134 (10Krenair) The details of where we got to with this logstash03 instance are in the thr...
[23:07:17] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10Wikimedia-Logstash, 10observability: logstash-beta.wmflabs.org does not receive any mediawiki events - https://phabricator.wikimedia.org/T233134 (10Krenair) In fact according to openstack-browser, logstash-beta.wmflabs.org still poi...
[23:11:04] <wikibugs>	 10Phabricator (Upstream), 10Release-Engineering-Team-TODO (201909), 10Browser-Support-Google-Chrome, 10Upstream: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10Aklapper) Should be fixed in Chrome 77 (`M77 branch 3865`) according to https://bugs...