[00:03:23] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0]
[00:13:22] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[00:35:58] <Krenair>	 !log Fixed ferm status on deployment-ms* and deployment-memc* machines, was another case of removing AAAA things. Possibly fixed it on mediawiki ones too but there are other problems there
[01:03:24] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[01:30:13] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T220747 (10Krinkle)
[01:56:47] <wikibugs>	 (03PS1) 10Holger Knust: WIP: api-testing: Add job template and job for api-testing [integration/config] - 10https://gerrit.wikimedia.org/r/535747 (https://phabricator.wikimedia.org/T230340)
[01:57:41] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] WIP: api-testing: Add job template and job for api-testing [integration/config] - 10https://gerrit.wikimedia.org/r/535747 (https://phabricator.wikimedia.org/T230340) (owner: 10Holger Knust)
[02:08:10] <wikibugs>	 (03PS2) 10Holger Knust: WIP: api-testing: Add job template and job for api-testing [integration/config] - 10https://gerrit.wikimedia.org/r/535747 (https://phabricator.wikimedia.org/T230340)
[02:13:19] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] WIP: api-testing: Add job template and job for api-testing [integration/config] - 10https://gerrit.wikimedia.org/r/535747 (https://phabricator.wikimedia.org/T230340) (owner: 10Holger Knust)
[02:13:24] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[02:17:30] <wikibugs>	 (03PS3) 10Holger Knust: WIP: api-testing: Add job template and job for api-testing [integration/config] - 10https://gerrit.wikimedia.org/r/535747 (https://phabricator.wikimedia.org/T230340)
[02:22:05] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] WIP: api-testing: Add job template and job for api-testing [integration/config] - 10https://gerrit.wikimedia.org/r/535747 (https://phabricator.wikimedia.org/T230340) (owner: 10Holger Knust)
[03:51:22] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%)
[03:56:23] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK
[04:07:22] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%)
[04:13:24] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[04:27:23] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK
[04:58:22] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%)
[06:10:22] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%)
[06:13:22] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[06:26:54] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Parsoid-PHP, 10RESTBase, 10Core Platform Team Workboards (Purple): Make the parsoid server on the beta cluster a mediawiki app server - https://phabricator.wikimedia.org/T232538 (10Joe) hi @ssastry just a clarification: how would we load the parsoid code, if it can't be mer...
[06:40:19] <wmf-insecte>	 Yippee, build fixed!
[06:40:20] <wmf-insecte>	 Project mwcore-phpunit-coverage-master build #166: 09FIXED in 3 hr 40 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/166/
[06:50:15] <icinga-wm>	 PROBLEM - Gerrit Health Check on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus
[06:50:31] <icinga-wm>	 PROBLEM - Gerrit JSON on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Gerrit%23Monitoring
[07:00:23] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK
[07:00:32] <wmf-insecte>	 Project beta-code-update-eqiad build #263257: 04FAILURE in 7 min 31 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/263257/
[07:02:47] <icinga-wm>	 RECOVERY - Gerrit Health Check on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 865 bytes in 0.047 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus
[07:02:59] <icinga-wm>	 RECOVERY - Gerrit JSON on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 26353 bytes in 0.054 second response time https://wikitech.wikimedia.org/wiki/Gerrit%23Monitoring
[07:03:12] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10Marostegui) {F30298153}
[07:04:25] <wmf-insecte>	 Yippee, build fixed!
[07:04:25] <wmf-insecte>	 Project beta-code-update-eqiad build #263258: 09FIXED in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/263258/
[07:13:17] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10hashar) Gerrit got stuck again this morning. There was a SendEmail task stuck (from `gerrit show...
[07:16:51] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Scap, 10serviceops, and 3 others: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) 05Open→03Resolved This is now 99% done. We just need a confctl release...
[07:19:12] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10hashar)
[07:49:40] <wikibugs>	 10Continuous-Integration-Infrastructure: Need a mechanism for injecting a LocalSettings.php fragment before nodejs browser tests - https://phabricator.wikimedia.org/T223431 (10awight)
[07:55:16] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Jenkins, 10Patch-For-Review: integration-config-zuul-layout-validate-docker takes too long in Jenkins due to huge output - https://phabricator.wikimedia.org/T232287 (10hashar)...
[08:00:15] <wikibugs>	 10Continuous-Integration-Infrastructure: Need a mechanism for injecting a LocalSettings.php fragment before nodejs browser tests - https://phabricator.wikimedia.org/T223431 (10awight)
[08:13:23] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[08:13:26] <wikibugs>	 (03CR) 10Hashar: "The actual failure is the job is not created for Jenkins. The job template has to be applied to a jjb project in order to actually create " (035 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/535747 (https://phabricator.wikimedia.org/T230340) (owner: 10Holger Knust)
[08:16:06] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Sounds good enough to address the command line spammy output." [integration/config] - 10https://gerrit.wikimedia.org/r/535566 (https://phabricator.wikimedia.org/T232287) (owner: 10Hashar)
[08:17:35] <wikibugs>	 (03Merged) 10jenkins-bot: Filter out debug spam when validating zuul layout [integration/config] - 10https://gerrit.wikimedia.org/r/535566 (https://phabricator.wikimedia.org/T232287) (owner: 10Hashar)
[08:18:36] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Jenkins, 10Patch-For-Review: integration-config-zuul-layout-validate-docker takes too long in Jenkins due to huge output - https://phabricator.wikimedia.org/T232287 (10hashar)...
[08:21:16] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] jjb: Drop operations-mw-config-php70-composer-lint-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535714 (owner: 10Jforrester)
[08:21:31] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "I have deleted the job in Jenkins" [integration/config] - 10https://gerrit.wikimedia.org/r/535714 (owner: 10Jforrester)
[08:23:30] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: Drop operations-mw-config-php70-composer-lint-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535714 (owner: 10Jforrester)
[08:30:50] <wikibugs>	 (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/535715 (owner: 10Jforrester)
[08:30:59] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] jjb: Add operations-mw-config-hhvm-composer-lint-docker and operations-mw-config-php72-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535715 (owner: 10Jforrester)
[08:46:05] <wikibugs>	 (03CR) 10Daimona Eaytoy: "> I would replace it to have only one annotation." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/535241 (https://phabricator.wikimedia.org/T232256) (owner: 10Daimona Eaytoy)
[08:47:02] <wikibugs>	 (03PS3) 10Daimona Eaytoy: Also allow @phan-template as alias of @template [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/535241 (https://phabricator.wikimedia.org/T232256)
[08:50:27] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10elukey) Question asked on IRC: should a monitor/alarm be added for active thread count, coupled...
[08:53:19] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10Marostegui) >>! In T224448#5482259, @elukey wrote: > Question asked on IRC: should a monitor/ala...
[09:21:14] <fdans>	 ok folks who was it?
[09:21:26] <fdans>	 https://usercontent.irccloud-cdn.com/file/0iHpy4Sj/Screen%20Shot%202019-09-11%20at%2011.19.10%20AM.png
[09:21:34] <fdans>	 :D
[09:24:41] <wikibugs>	 10Phabricator, 10Project-Admins, 10Growth-Team (Current Sprint): Add GrowthExperiments to tasks tagged as GrowthExperiments-* - https://phabricator.wikimedia.org/T230831 (10MarcoAurelio) >>! In T230831#5453296, @Urbanecm wrote: >[...] It's a nonsense [...]  I presume #repository-admins where granted the abil...
[09:30:30] <wikibugs>	 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Cleanup, 10Maps, and 2 others: Delete (rather than archive) the maps/kartotherian and maps/tilerator repos - https://phabricator.wikimedia.org/T228483 (10MarcoAurelio) a:03MarcoAurelio Alright. Given that there's agreem...
[09:38:13] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 35.71% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[09:40:29] <wikibugs>	 10Continuous-Integration-Infrastructure: Need a mechanism for injecting a LocalSettings.php fragment before nodejs browser tests - https://phabricator.wikimedia.org/T223431 (10zeljkofilipin) See {T199939}.
[09:41:07] <effie>	 if a postbuild job is stuck 
[09:41:19] <effie>	 but jenkins as already merged 
[09:41:23] <effie>	 https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/535786/
[09:41:27] <effie>	 what should we do ?
[09:41:41] <effie>	 wait or go ahead and deploy
[09:55:35] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[09:59:25] <wikibugs>	 10Continuous-Integration-Infrastructure: Need a mechanism for injecting a LocalSettings.php fragment before nodejs browser tests - https://phabricator.wikimedia.org/T223431 (10awight) >>! In T223431#5482439, @zeljkofilipin wrote: > See {T199939}.  Even better!  Thanks
[09:59:34] <wikibugs>	 10Continuous-Integration-Infrastructure: Need a mechanism for injecting a LocalSettings.php fragment before nodejs browser tests - https://phabricator.wikimedia.org/T223431 (10awight)
[09:59:41] <wikibugs>	 10Continuous-Integration-Config, 10MediaViewer, 10MediaWiki-Core-Testing, 10MobileFrontend, and 7 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939 (10awight)
[10:00:18] <wikibugs>	 10Continuous-Integration-Config, 10MediaViewer, 10MediaWiki-Core-Testing, 10MobileFrontend, and 7 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939 (10awight)
[10:02:01] <wikibugs>	 10Continuous-Integration-Config, 10MediaViewer, 10MediaWiki-Core-Testing, 10MobileFrontend, and 7 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939 (10awight)
[10:13:23] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[10:19:51] <wikibugs>	 (03CR) 10DCausse: "Thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/535668 (owner: 10Kosta Harlan)
[10:20:15] <wikibugs>	 (03CR) 10DCausse: [C: 03+1] Revert "jjb: [mwext-codehealth*] Only clone ZUUL_PROJECT" [integration/config] - 10https://gerrit.wikimedia.org/r/535668 (owner: 10Kosta Harlan)
[10:28:34] <wikibugs>	 10Release-Engineering-Team (Code Health), 10Discovery-Search, 10MediaWiki-extensions-General, 10Code-Health: Migrate CirrusSearch MediaWikiIntegrationTestCase tests to on unit tests - https://phabricator.wikimedia.org/T232589 (10dcausse)
[10:30:05] <wikibugs>	 10Release-Engineering-Team (Code Health), 10Discovery-Search, 10MediaWiki-extensions-General, 10Code-Health: Migrate CirrusSearch MediaWikiIntegrationTestCase tests to MediaWikiUnitTestCase - https://phabricator.wikimedia.org/T232589 (10dcausse)
[11:27:25] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Patch-For-Review, 10User-zeljkofilipin: Cypress testing framework evaluation - https://phabricator.wikimedia.org/T230729 (10zeljkofilipin)
[11:39:53] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Update existing Selenium documentation - https://phabricator.wikimedia.org/T232598 (10zeljkofilipin)
[11:40:18] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Update existing Selenium documentation - https://phabricator.wikimedia.org/T232598 (10zeljkofilipin) p:05Triage→03High
[11:41:38] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Update existing Selenium documentation - https://phabricator.wikimedia.org/T232598 (10zeljkofilipin)
[11:45:11] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Core-Testing, 10Browser-Tests, 10User-zeljkofilipin: Usage instructions in tests/selenium/README.md are confusing - https://phabricator.wikimedia.org/T214708 (10zeljkofilipin)
[11:45:23] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK
[11:45:26] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Core-Testing, 10Browser-Tests, 10User-zeljkofilipin: Usage instructions in tests/selenium/README.md are confusing - https://phabricator.wikimedia.org/T214708 (10zeljkofilipin)
[11:45:38] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Update existing Selenium documentation - https://phabricator.wikimedia.org/T232598 (10zeljkofilipin)
[11:46:16] <wikibugs>	 10Phabricator: Cannot load Phabricator board on Chrome 77 - https://phabricator.wikimedia.org/T232600 (10Petar.petkovic)
[11:46:56] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Update files required by Selenium in core, extensions and skins - https://phabricator.wikimedia.org/T210726 (10zeljkofilipin)
[11:47:49] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Update Selenium/Node.js/Target Custom MediaWiki - https://phabricator.wikimedia.org/T210518 (10zeljkofilipin)
[11:48:46] <wikibugs>	 10Phabricator: Cannot load Phabricator board on Chrome 77 - https://phabricator.wikimedia.org/T232600 (10Petar.petkovic)
[11:48:52] <Lucas_WMDE>	 gerrit seems to be down
[11:49:02] <icinga-wm>	 PROBLEM - Gerrit Health Check on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus
[11:49:13] <Lucas_WMDE>	 ah, so it’s not just Amir1 and me ^^
[11:49:45] <Amir1>	 Yay, for once our office internet is not at fault 
[11:49:59] <icinga-wm>	 PROBLEM - Gerrit JSON on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Gerrit%23Monitoring
[11:50:30] <wikibugs>	 10Release-Engineering-Team-TODO (201909), 10Operations, 10Wikimedia-Mailing-lists: Please create private "testeng" team mailing list - https://phabricator.wikimedia.org/T232178 (10zeljkofilipin) 05Open→03Resolved Archives no longer public.
[11:52:15] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Update files required by Selenium in core, extensions and skins - https://phabricator.wikimedia.org/T210726 (10zeljkofilipin)
[11:52:38] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Core-Testing, 10Browser-Tests, 10User-zeljkofilipin: Usage instructions in tests/selenium/README.md are confusing - https://phabricator.wikimedia.org/T214708 (10zeljkofilipin) p:05Triage→03Normal
[11:52:48] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Update Selenium/Node.js/Target Custom MediaWiki - https://phabricator.wikimedia.org/T210518 (10zeljkofilipin) p:05High→03Normal
[11:54:53] <wikibugs>	 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Someday/maybe Selenium framework improvements - https://phabricator.wikimedia.org/T190995 (10zeljkofilipin)
[11:55:18] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Core-Testing, 10Documentation, and 2 others: Blog posts about new Selenium framework features - https://phabricator.wikimedia.org/T191982 (10zeljkofilipin)
[11:56:13] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Update existing Selenium documentation - https://phabricator.wikimedia.org/T232598 (10zeljkofilipin)
[11:56:15] <wikibugs>	 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Someday/maybe Selenium framework improvements - https://phabricator.wikimedia.org/T190995 (10zeljkofilipin)
[11:56:27] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Selenium tests should be easier to run - https://phabricator.wikimedia.org/T182691 (10zeljkofilipin)
[11:57:14] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Selenium tests should be easier to run - https://phabricator.wikimedia.org/T182691 (10zeljkofilipin) p:05Low→03Normal
[11:59:12] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10hashar) And that happened again https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMDk...
[12:01:41] <icinga-wm>	 RECOVERY - Gerrit Health Check on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 865 bytes in 0.134 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus
[12:02:10] <wmf-insecte>	 Project beta-code-update-eqiad build #263287: 04FAILURE in 9 min 9 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/263287/
[12:02:37] <icinga-wm>	 RECOVERY - Gerrit JSON on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 26003 bytes in 0.589 second response time https://wikitech.wikimedia.org/wiki/Gerrit%23Monitoring
[12:04:01] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit account cache has a faulty reentrant lock causing http/sendemail threads to stall completely - https://phabricator.wikimedia.org/T224448 (10hashar)
[12:04:32] <wmf-insecte>	 Yippee, build fixed!
[12:04:33] <wmf-insecte>	 Project beta-code-update-eqiad build #263288: 09FIXED in 1 min 32 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/263288/
[12:13:25] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[12:17:35] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:32:36] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 48060 bytes in 6.683 second response time
[12:45:12] <paladox>	 hmm
[12:45:16] <paladox>	 gerrit went down again?
[12:45:38] <paladox>	 https://gerrit.wikimedia.org/r/monitoring?part=graph&graph=activeThreads three times
[12:48:53] <apergos>	 gerrit is working for me right now
[12:49:16] <paladox>	 yup
[12:49:32] <paladox>	 It won't work if it reaches the max threads we have set :(
[12:53:09] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin)
[12:53:19] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin) p:05Triage→03Normal
[12:54:04] * paladox files a issue with guava.
[12:54:37] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin)
[12:54:48] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Advanced-Search, 10TCB-Team, 10User-zeljkofilipin: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin)
[12:55:00] <wikibugs>	 10Phabricator, 10Project-Admins, 10Growth-Team (Current Sprint): Add GrowthExperiments to tasks tagged as GrowthExperiments-* - https://phabricator.wikimedia.org/T230831 (10Aklapper) @mmodell: Following the list in T230831#5433325, I succeeded in silently removing #GrowthExperiments from 97 tasks, but next s...
[12:55:48] <wikibugs>	 10Phabricator, 10Project-Admins, 10Growth-Team (Current Sprint): Add GrowthExperiments to tasks tagged as GrowthExperiments-* - https://phabricator.wikimedia.org/T230831 (10Aklapper) a:05Aklapper→03mmodell
[12:57:21] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Advanced-Search, 10TCB-Team, 10User-zeljkofilipin: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin)
[12:57:40] <paladox>	 hashar thcipriani https://github.com/google/guava/issues/3602
[12:58:11] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit account cache has a faulty reentrant lock causing http/sendemail threads to stall completely - https://phabricator.wikimedia.org/T224448 (10Paladox) I've filed https://github.com/google/...
[12:58:39] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Advanced-Search, 10TCB-Team, 10User-zeljkofilipin: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin)
[12:58:52] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Advanced-Search, 10CirrusSearch, and 3 others: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin)
[13:01:36] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Advanced-Search, 10CirrusSearch, and 3 others: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10dcausse) I'm not sure we want to...
[13:02:07] <hashar>	 paladox: thanks :]
[13:02:30] <paladox>	 your welcome :), i know it lacks details. But it's to kickstart things.
[13:25:54] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10CirrusSearch, 10Discovery-Search: selenium-daily-beta-CirrusSearch is broken - https://phabricator.wikimedia.org/T232495 (10zeljkofilipin)
[13:25:56] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Advanced-Search, 10CirrusSearch, and 3 others: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin)
[13:26:23] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10CirrusSearch, and 2 others: selenium-daily-beta-CirrusSearch is broken - https://phabricator.wikimedia.org/T232495 (10zeljkofilipin)
[13:26:51] <paladox>	 hashar caffeine has been shown to work, so i've filed https://bugs.chromium.org/p/gerrit/issues/detail?id=11484
[13:28:07] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Advanced-Search, 10CirrusSearch, and 3 others: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin)
[13:30:04] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Advanced-Search, 10CirrusSearch, and 3 others: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin)
[13:31:43] <wikibugs>	 10Release-Engineering-Team-TODO (201909), 10Browser-Tests, 10User-zeljkofilipin: selenium-daily-beta-MediaWiki possibly broken by password changes - https://phabricator.wikimedia.org/T232357 (10zeljkofilipin)
[13:35:58] <wikibugs>	 10Release-Engineering-Team-TODO (201909), 10Browser-Tests, 10User-zeljkofilipin: selenium-daily-beta-REPOSITORY Jenkins jobs broken since 2019-09-08 - https://phabricator.wikimedia.org/T232357 (10zeljkofilipin)
[13:38:41] <wikibugs>	 10Release-Engineering-Team-TODO (201909), 10Browser-Tests, 10User-zeljkofilipin: selenium-daily-beta-REPOSITORY Jenkins jobs broken since 2019-09-08 - https://phabricator.wikimedia.org/T232357 (10zeljkofilipin)
[13:39:26] <wikibugs>	 10Release-Engineering-Team-TODO (201909), 10Browser-Tests, 10User-zeljkofilipin: selenium-daily-beta-REPOSITORY Jenkins jobs broken since 2019-09-08 - https://phabricator.wikimedia.org/T232357 (10zeljkofilipin)
[13:41:45] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Advanced-Search, 10CirrusSearch, and 3 others: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin)
[13:43:24] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10CirrusSearch, and 2 others: selenium-daily-beta-CirrusSearch is broken - https://phabricator.wikimedia.org/T232495 (10zeljkofilipin) p:05Triage→03Normal
[13:44:29] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Parsoid-PHP, 10RESTBase, 10Core Platform Team Workboards (Purple): Make the parsoid server on the beta cluster a mediawiki app server - https://phabricator.wikimedia.org/T232538 (10ssastry) >>! In T232538#5482002, @Joe wrote: > hi @ssastry just a clarification: how would we...
[13:44:55] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Advanced-Search, 10CirrusSearch, and 3 others: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin)
[13:45:51] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Advanced-Search, 10CirrusSearch, and 7 others: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin)
[13:47:31] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Advanced-Search, 10CirrusSearch, and 8 others: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin)
[13:48:46] <wikibugs>	 10Release-Engineering-Team-TODO (201909), 10Browser-Tests, 10User-zeljkofilipin: selenium-daily-beta-REPOSITORY Jenkins jobs broken since 2019-09-08 - https://phabricator.wikimedia.org/T232357 (10zeljkofilipin)
[13:50:03] <Lucas_WMDE>	 twentyafterfour: by the way, I assume your Phabricator upgrade (https://tools.wmflabs.org/sal/log/AWy2rXo8EHTBTPG-0XR_) is why boards now auto-update themselves? thank you <3
[13:53:12] <twentyafterfour>	 Lucas_WMDE: I can't take credit for it really, upstream did all the work ;) 
[13:53:35] <twentyafterfour>	 I just merge branches most of the time 
[13:55:48] <Lucas_WMDE>	 I can’t thank upstream as easily so you’ll just have to accept it ;)
[13:58:52] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Advanced-Search, 10CirrusSearch, and 8 others: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin)
[14:01:10] <wikibugs>	 10Project-Admins: Setup Phabricator tag for all Wikidata Documentation tasks - https://phabricator.wikimedia.org/T222711 (10Aklapper) 05Open→03Declined Thanks for the reply! I'm boldly declining this task for now, but please set the status of this report back to "Open" via the {nav name=Add Action... > Chang...
[14:02:12] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T220747 (10hashar)
[14:09:07] <wikibugs>	 10Phabricator: Cannot load Phabricator board on Chrome 77 - https://phabricator.wikimedia.org/T232600 (10RHo) Same issue occurring for me on #Growth-team and the project view [[https://phabricator.wikimedia.org/project/view/3450/ | Growth-Team (Current Sprint)]] using Chrome Beta Version 77.0.3865.65. Loads fine...
[14:09:27] <Nikerabbit>	 gate-and-submit seems stuck (or at least really slow)
[14:09:41] <Lucas_WMDE>	 just really slow, I think
[14:09:58] <Lucas_WMDE>	 Wikibase changes are slow in general (lots of tests and pulls in lots of dependent extensions with their tests)
[14:10:12] <Lucas_WMDE>	 and occasionally they fail, e. g. T232363
[14:10:15] <stashbot>	 T232363: Occasional Wikibase CI failures: Cannot read property 'set' of undefined - https://phabricator.wikimedia.org/T232363
[14:10:15] <Lucas_WMDE>	 :/
[14:10:37] <wikibugs>	 10Phabricator: Cannot load Phabricator board on Chrome 77 - https://phabricator.wikimedia.org/T232600 (10Paladox) Happens to me too (works on safari but breaks on chrome).
[14:10:44] <Nikerabbit>	 there are patches waiting 90 minutes just to get started :(
[14:10:52] <wikibugs>	 10Phabricator: Cannot load Phabricator board on Chrome 77 - https://phabricator.wikimedia.org/T232600 (10Paladox) p:05Triage→03High
[14:13:21] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[14:13:32] <wikibugs>	 10Release-Engineering-Team-TODO, 10Wikimedia-Site-requests, 10CPT Initiatives (PHP7 (TEC4)), 10Core Platform Team Workboards (Clinic Duty Team), and 2 others: Consider enabling wgExtensionInfoMTime in wmf-production - https://phabricator.wikimedia.org/T187154 (10eprodromou) p:05Triage→03Normal
[14:13:55] <wikibugs>	 10Release-Engineering-Team-TODO, 10Wikimedia-Site-requests, 10CPT Initiatives (PHP7 (TEC4)), 10Core Platform Team Workboards (Clinic Duty Team), and 2 others: Consider enabling wgExtensionInfoMTime in wmf-production - https://phabricator.wikimedia.org/T187154 (10WDoranWMF) @Anomie Could you review/evaluate...
[14:14:42] <wikibugs>	 10Phabricator: Phabricator boards do not load cards in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10Aklapper)
[14:14:51] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Phabricator boards do not load cards in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10Aklapper)
[14:18:27] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T220747 (10hashar)
[14:20:40] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10Aklapper)
[14:21:03] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10Aklapper) In the Developer Tools, could you take a look at the "network" panel please if there are any errors, and provide a screenshot of that? For more...
[14:21:43] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10mmodell) For me it locks up the tab so badly that I can't even get developer tools to work.
[14:23:07] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909): integration-agent-docker-1001.integration.eqiad.wmflabs is unrecheable - https://phabricator.wikimedia.org/T232619 (10hashar)
[14:24:59] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909): integration-agent-docker-1001.integration.eqiad.wmflabs is unrecheable - https://phabricator.wikimedia.org/T232619 (10hashar)
[14:25:37] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909): integration-agent-docker-1001.integration.eqiad.wmflabs is unrecheable - https://phabricator.wikimedia.org/T232619 (10Andrew) Oh, if this was created on Monday then it was pr...
[14:26:38] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909): integration-agent-docker-1001.integration.eqiad.wmflabs is unrecheable - https://phabricator.wikimedia.org/T232619 (10hashar)
[14:27:12] <hashar>	 !log  puppet cert clean integration-agent-docker-1001.integration.eqiad.wmflabs # T232619 | might unbreak puppet
[14:27:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:27:15] <stashbot>	 T232619:  integration-agent-docker-1001.integration.eqiad.wmflabs is unrecheable - https://phabricator.wikimedia.org/T232619
[14:29:23] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10Petar.petkovic) I don't see cards on #growth-team board, just like @RHo reported. >>! In T232600#5483706, @mmodell wrote: > For me it locks up the tab so...
[14:30:09] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909): integration-agent-docker-1001.integration.eqiad.wmflabs is unrecheable - https://phabricator.wikimedia.org/T232619 (10hashar) I think I have created an instance based on the...
[14:30:52] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909):  integration-agent-docker-1001.integration.eqiad.wmflabs is unreachable - https://phabricator.wikimedia.org/T232619 (10hashar) p:05Triage→03Normal
[14:32:28] <wikibugs>	 (03PS1) 10Zfilipin: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days [integration/config] - 10https://gerrit.wikimedia.org/r/535864 (https://phabricator.wikimedia.org/T232608)
[14:38:58] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10RHo) Same issues as @Petar.petkovic - unable to export a HAR file and can only get screenshots showing a bunch of requests pending: {F30299378} {F30299380}
[14:40:03] <wikibugs>	 (03PS4) 10Holger Knust: WIP: api-testing: Add job template and job for api-testing [integration/config] - 10https://gerrit.wikimedia.org/r/535747 (https://phabricator.wikimedia.org/T230340)
[14:42:14] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Advanced-Search, 10CirrusSearch, and 9 others: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin)
[14:45:08] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] WIP: api-testing: Add job template and job for api-testing [integration/config] - 10https://gerrit.wikimedia.org/r/535747 (https://phabricator.wikimedia.org/T230340) (owner: 10Holger Knust)
[14:45:54] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10mmodell) I managed to get script execution to break by setting a "script first statement" breakpoint in the sources debugger. So far I haven't been able...
[14:46:12] <wikibugs>	 (03CR) 10Zfilipin: [C: 03+2] Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days [integration/config] - 10https://gerrit.wikimedia.org/r/535864 (https://phabricator.wikimedia.org/T232608) (owner: 10Zfilipin)
[14:48:23] <wikibugs>	 (03Merged) 10jenkins-bot: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days [integration/config] - 10https://gerrit.wikimedia.org/r/535864 (https://phabricator.wikimedia.org/T232608) (owner: 10Zfilipin)
[14:59:54] <wikibugs>	 10Release-Engineering-Team-TODO, 10Wikimedia-Site-requests, 10CPT Initiatives (PHP7 (TEC4)), 10Core Platform Team Workboards (Clinic Duty Team), and 2 others: Consider enabling wgExtensionInfoMTime in wmf-production - https://phabricator.wikimedia.org/T187154 (10Anomie) `$wgExtensionInfoMTime` does not app...
[15:20:50] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Advanced-Search, 10CirrusSearch, and 8 others: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin) >>! In T232608#548...
[15:22:45] <wikibugs>	 (03CR) 10Hashar: [C: 04-1] "There is a little glitch in the jjb config for the packages_source variable, I think we should just use a string:" (034 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/535747 (https://phabricator.wikimedia.org/T230340) (owner: 10Holger Knust)
[15:25:52] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Parsoid-PHP, 10RESTBase, 10Core Platform Team Workboards (Purple): Make the parsoid server on the beta cluster a mediawiki app server - https://phabricator.wikimedia.org/T232538 (10ssastry)
[15:27:31] <wikibugs>	 (03PS2) 10Jforrester: jjb: Add operations-mw-config-hhvm-composer-lint-docker and operations-mw-config-php72-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535715
[15:27:42] <wikibugs>	 (03PS2) 10Jforrester: layout: [mediawiki-config] Add operations-mw-config-hhvm-composer-lint-docker and operations-mw-config-php72-composer-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535716
[15:28:17] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10epriestley) This loads okay for me in Chrome 77 on macOS 10.14:  {F30299504}  {F30299507}  Since the board is public, one possible test: does it work in...
[15:28:31] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909): integration-agent-docker-1001.integration.eqiad.wmflabs is unreachable - https://phabricator.wikimedia.org/T232619 (10thcipriani) >>! In T232619#5483743, @hashar wrote: > I t...
[15:29:16] <onimisionipe>	 thcipriani: hello..
[15:29:28] <onimisionipe>	 can you look at https://gerrit.wikimedia.org/r/c/mediawiki/services/kartotherian/+/534419
[15:29:35] <onimisionipe>	 pls
[15:30:10] <onimisionipe>	 Its docker build for kartotherian using config.yaml 
[15:31:58] <onimisionipe>	 Its based on what was implemented here: https://phabricator.wikimedia.org/T210267
[15:33:45] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10RHo) >>! In T232600#5483982, @epriestley wrote: > This loads okay for me in Chrome 77 on macOS 10.14: >  > {F30299504} >  > {F30299507} >  > Since the bo...
[15:37:08] <halfak>	 Hey folks.  I'm trying to rewrite the last couple of commits to a gerrit repo because some files ended up outside of LFS.  
[15:37:32] <halfak>	 I'm getting an error that I don't have "forge committer" right for rewriting the last couple of problematic commits. 
[15:38:13] <halfak>	 Can someone help me fix this issue?  
[15:38:16] <paladox>	 halfak you need "forge committer" added to your project acl :)
[15:38:34] <halfak>	 I think so.  This isn't the first time I've been doing git history cleaning. 
[15:38:47] <halfak>	 I've been given the rights temporarily before. 
[15:38:56] <halfak>	 Maybe it would make sense to give me them long term. 
[15:39:04] <paladox>	 you can :)
[15:39:10] <paladox>	 if your the project admin
[15:39:15] <paladox>	 you can grant it to your self.
[15:39:40] <paladox>	 project owner i mean
[15:41:08] <halfak>	 Oh..  Interesting. 
[15:41:13] <halfak>	 How would I find that setting?
[15:41:21] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10Petar.petkovic) >>! In T232600#5484021, @RHo wrote: > Spooky territory – incognito doesn't work for me either: > {F30299521} Incognito not working for me...
[15:41:39] <halfak>	 https://gerrit.wikimedia.org/r/#/admin/projects/research/ores/wheels
[15:42:38] <wikibugs>	 (03PS1) 10Paladox: Modify access rules [research/ores/wheels] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/535878
[15:42:41] <paladox>	 halfak ^
[15:43:20] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] jjb: Add operations-mw-config-hhvm-composer-lint-docker and operations-mw-config-php72-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535715 (owner: 10Jforrester)
[15:43:32] <wikibugs>	 (03CR) 10Halfak: [V: 03+2 C: 03+2] Modify access rules [research/ores/wheels] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/535878 (owner: 10Paladox)
[15:44:02] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] layout: [mediawiki-config] Add operations-mw-config-hhvm-composer-lint-docker and operations-mw-config-php72-composer-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535716 (owner: 10Jforrester)
[15:44:56] <paladox>	 halfak need to press the submit button
[15:48:47] <wikibugs>	 (03PS5) 10Holger Knust: WIP: api-testing: Add job template and job for api-testing [integration/config] - 10https://gerrit.wikimedia.org/r/535747 (https://phabricator.wikimedia.org/T230340)
[15:49:25] <wikibugs>	 (03PS1) 10Paladox: Grant force push to research-ores [research/ores/wheels] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/535880
[15:49:33] <wikibugs>	 (03Abandoned) 10Paladox: Modify access rules [research/ores/wheels] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/535721 (owner: 10Paladox)
[15:50:44] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10epriestley) Aha! I can reproduce on "Growth-Team", just not the original "Language-Team" project.  I'm not totally sure how to best debug this since all...
[15:50:55] <wikibugs>	 (03CR) 10Holger Knust: "Hopefully, PS5 is the charm :)" (039 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/535747 (https://phabricator.wikimedia.org/T230340) (owner: 10Holger Knust)
[16:02:49] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909): integration-agent-docker-1001.integration.eqiad.wmflabs is unreachable - https://phabricator.wikimedia.org/T232619 (10hashar) 05Open→03Resolved a:03hashar Thank you @th...
[16:06:24] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10mmodell) @epriestley: It may take several tries to get debugger: At least it did for me: I did several iterations of killing the hung task, setting vario...
[16:06:28] <shinken-wm>	 PROBLEM - Host integration-agent-docker-1001 is DOWN: CRITICAL - Host Unreachable (172.16.3.36)
[16:10:46] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10epriestley) Yeah, this is wildly difficult to debug with Chrome's tools. The profiler freezes and I can't get it to unfreeze, stopping execution freezes...
[16:13:25] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[16:14:19] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10CirrusSearch, and 2 others: selenium-daily-beta-CirrusSearch is broken - https://phabricator.wikimedia.org/T232495 (10zeljkofilipin) a:03zeljkofilipin
[16:16:04] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10mmodell) When I click "continue" from the beginning of WorkboardDropEffect.js it immediately freezes. Not sure if that's just the last js file or that's...
[16:16:56] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO: CI mediawiki/core run times have increased since July 26th - https://phabricator.wikimedia.org/T232626 (10thcipriani)
[16:24:32] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10epriestley) I think that's just "the freeze is after WorkboardDropEffect.js" is loaded. All the class definitions load first without actually executing a...
[16:31:52] <wikibugs>	 (03PS3) 10Jforrester: jjb: Add operations-mw-config-hhvm-composer-lint-docker and operations-mw-config-php72-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535715
[16:34:13] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Jenkins: integration-config-zuul-layout-validate-docker takes too long in Jenkins due to huge output - https://phabricator.wikimedia.org/T232287 (10Jdforrester-WMF) Is this Resol...
[16:36:46] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] jjb: Add operations-mw-config-hhvm-composer-lint-docker and operations-mw-config-php72-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535715 (owner: 10Jforrester)
[16:38:10] <wikibugs>	 10Phabricator: Phabricator issue on 2019-09-08; "Unable to establish a connection to any database host: Cannot assign requested address." - https://phabricator.wikimedia.org/T232276 (10Aklapper)
[16:38:19] <wikibugs>	 10Phabricator: Phabricator issue on 2019-09-08; "Unable to establish a connection to any database host: Cannot assign requested address." - https://phabricator.wikimedia.org/T232276 (10Aklapper) Thanks. Looking at the Grafana dashboard, it seems there was a peek around that time:  {F30299745}
[16:38:43] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10epriestley) Here's what I'm observing:    - The "project-boards" behavior executes and completes.   - The debugger visually advances to the next step. Th...
[16:41:27] <paladox>	 thcipriani so, spoke to someone from ericson (where hugo use to work). They forked gerrit and are running caffeine (hence why https://bugs.chromium.org/p/gerrit/issues/detail?id=7645 is resolved for them).
[16:44:09] <paladox>	 they are going to look at it for 2.15+
[16:44:17] <paladox>	 (since doing what they did for 2.14 broke on 2.15)
[16:46:44] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10epriestley) Here's a view of the board with exactly one task that freezes in Chrome for me:  https://phabricator.wikimedia.org/project/board/1114/?filter...
[16:51:13] <thcipriani>	 paladox: that'd be great! thanks for the guava task by the way!
[16:51:23] <paladox>	 your welcome :)
[16:56:21] <paladox>	 thcipriani they make a good case that this could be all gerrit's fault rather then guava. (that's a possibility)
[16:57:01] <thcipriani>	 from reading the original task that does seem like a potential fix
[16:58:17] <paladox>	 yeh
[16:58:28] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10epriestley) > There's nothing special about that task  Actually, I think I got lucky and there //is// something special about this task. This board does...
[17:03:57] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10epriestley) Broken in Chrome:  {F30299842}  Broken in Chrome:  {F30299844}  Broken in Chrome:  {F30299846}  Works great in Chrome:  {F30299848}  The only...
[17:10:17] <wikibugs>	 10Release-Engineering-Team-TODO (201909), 10User-greg, 10Wikimedia-extension-review-queue: Investigate and make improvements to the extension review process - https://phabricator.wikimedia.org/T195244 (10greg)
[17:11:07] <wikibugs>	 10Release-Engineering-Team-TODO, 10Product-Infrastructure-Team-Backlog, 10Wikimedia-Site-requests, 10Epic, and 2 others: Sunset Wikipedia Zero - https://phabricator.wikimedia.org/T187716 (10Jdforrester-WMF)
[17:12:08] <wikibugs>	 10Phabricator, 10Release-Engineering-Team-TODO (201909), 10Wikimedia-Logstash, 10observability, and 2 others: Deploy phatality into kibana - https://phabricator.wikimedia.org/T230752 (10mmodell)
[17:45:57] * Krinkle has +2'ed two patches for wmf branch & mw config 10 minutes ago, neither build has started yet.
[17:50:32] <bd808>	 Krinkle: :(( the thing at the top of gate-and-submit has been in queue for 3 hr 18 min
[17:51:56] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10epriestley) Here is a simplified document which freezes Chrome 77:  ` <html>   <head>     <style type="text/css">       .AAA {         width: 200px;...
[17:52:19] <Lucas_WMDE>	 there’s a change that’s over 1h50m old and hasn’t started building yet
[17:52:39] <bd808>	 hashar, thcipriani: how big are the integration-slave-docker-* instances? Wondering how many more we could grant you quota for to increase the pool size.
[17:52:49] <Krinkle>	 wmf extension gate job takes ~ 35min
[17:52:58] <Krinkle>	 Wikidata ~ 45-50min
[17:53:10] <Krinkle>	 the hour delay is presumably because one of them failed and caused all below it to retry
[17:53:13] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10puppet-compiler: Puppet catalog compiler - increasing max concurrent jobs - https://phabricator.wikimedia.org/T221969 (10thcipriani)
[17:53:32] <Krinkle>	 to reduce waste of this, I don't think it goes more than 5 ahead in the queue, so that other queues can process first
[17:53:35] <bd808>	 IMO (not a test engineer) we are running way too many of the same tests on different runtimes on each commit
[17:53:51] <Krinkle>	 https://phabricator.wikimedia.org/T225730
[17:53:58] <wikibugs>	 10Release-Engineering-Team-TODO, 10Operations, 10serviceops, 10Patch-For-Review, 10Wikimedia-Incident: docker-registry: some layers has been corrupted due to deleting other swift containers - https://phabricator.wikimedia.org/T228196 (10thcipriani)
[17:54:02] <Krinkle>	 Yep, we don't need wdio, qunit and npm-test on different PHP versions
[17:54:31] <Krinkle>	 although that's not where the bulk is spent. the bulk is phpunit and wdio for a single version which is the majority of those 30min/55min builds.
[17:54:53] <Krinkle>	 It used to be closer to 5min only a few weeks ago. Something has gotten a lot worse very quickly, and it's not the infra afaik. 
[17:55:24] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki-jhuneidi is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 hphp_invoke - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 353 bytes in 0.062 second response time
[17:55:36] <Krinkle>	 although more capacity would certainly help with getting non-mw-gate tests out the door quicker, e.g. pre-merge patch testing, perf, coverage, config patches etc.
[17:56:01] <Krinkle>	 bd808: That's pretty cool to hear. My impression was that "we" thought we were using as much from cloud as was feasible in the current setup
[17:56:13] <Krinkle>	 Even 1 or 2 more docker hosts would make a big impact I think.
[17:56:23] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<11.11%)
[17:56:29] <greg-g>	 see also: https://phabricator.wikimedia.org/T232626
[17:57:07] <bd808>	 we are not sitting on huge piles of unused capacity, but we can probably scrounge up some space
[17:57:50] <bd808>	 greg-g: ouch. that's a heck of a jump. Somebody did something "interesting" to cause that
[17:58:04] <greg-g>	 bd808: yeah :/
[17:58:15] <wikibugs>	 10Release-Engineering-Team-TODO (201909), 10Developer Productivity, 10dev-images, 10local-charts, 10Patch-For-Review: Move dev-images PHP image from php -S to Apache + php-fpm - https://phabricator.wikimedia.org/T222494 (10thcipriani)
[17:58:25] <greg-g>	 09:11:42 	<hashar>	so noticed by Tyler, the jenkins build time raised significantly around July 26th
[17:58:27] <greg-g>	 09:12:04 	<hashar>	which seems to match with Quibble running selenium tests directly from the extensions directory
[17:58:31] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, and 5 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Krinkle) >>! Task description > As of 11...
[17:58:37] <greg-g>	 but unconfirmed/investigated yet
[17:58:39] <Krinkle>	 greg-g: nice graph
[17:58:48] * bd808 punches selenium in the face
[17:59:10] <Krinkle>	 there is no <s>spoon</s>selenium.
[17:59:30] <bd808>	 again IMO browser regression tests should be periodic and not per-commit
[18:00:17] <Krinkle>	 bd808: https://phabricator.wikimedia.org/T225248
[18:00:22] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10cloud-services-team: Check bandwith limitation on integration-castor03.integration.eqiad.wmflabs / cloudvirt1002 - https://phabricator.wikimedia.org/T232644 (10hashar)
[18:01:01] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO: CI mediawiki/core run times have increased since July 26th - https://phabricator.wikimedia.org/T232626 (10Ladsgroup) I made a phabricator ticket at exact time: {T228739} @Jdforrester-WMF
[18:01:48] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO: CI mediawiki/core run times have increased since July 26th - https://phabricator.wikimedia.org/T232626 (10Ladsgroup) Another reason: {T231862} (Wikibase is part of gate, lots of selenium tests are being ran twice)
[18:02:48] <wikibugs>	 10Continuous-Integration-Config, 10Wikidata, 10ci-test-error: Jenkins runs Scribunto tests in Wikibase patches - https://phabricator.wikimedia.org/T228739 (10Ladsgroup)
[18:04:29] <Krinkle>	 bd808: for what it's worth, the Fresnel job uses the same CI worker nodes and wrapper scripts (quibble, docker, mw, mysql etc.) and same browser (chromium headless), but performs dozens of mw requests and full page loads in 2-3 seconds.
[18:04:36] <Krinkle>	 It's super fast.
[18:04:52] <Krinkle>	 There's something in our wdio test setup that's causing things to regularly halt for several seconds doing nothing.
[18:05:22] <Krinkle>	 It's reproducible locally as well. I might look into it later this week :)
[18:06:41] <Krinkle>	 but yeah, if we actually had a large number of valuable integration tests that genuinely need a long time to run, would probably make more sense to run as part of a larger pipeline somewhere after "merged in master" but before "sync to prod".
[18:08:56] <wikibugs>	 10Phabricator, 10Browser-Support-Google-Chrome: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10epriestley) I upstreamed this here:    - (Phabricator Upstream) https://secure.phabricator.com/T13413   - (Chrome Upstream) https://bugs.chromium.org/p/c...
[18:09:05] <James_F>	 Krinkle: If you save everyone 5 mins on every merge you'll be belovèd.
[18:11:56] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10cloud-services-team: Move integration-castor03.integration.eqiad.wmflabs to a newer cloudvirt machine - https://phabricator.wikimedia.org/T232646 (10hashar)
[18:13:23] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[18:14:44] <liw>	 James_F, you're implying we don't yet love Krinkle 
[18:18:28] <James_F>	 liw: Correct. Because CI is too slow. Clearly it's all his fault.
[18:19:07] <wikibugs>	 10Continuous-Integration-Infrastructure: castor rsync's taking 3-5 minutes for mwgate-npm jobs - https://phabricator.wikimedia.org/T188375 (10hashar) >>! In T188375#5378612, @tstarling wrote: > ... > If you add a varnishd on every integration slave and proxy via localhost, then some network traffic could be avoi...
[18:20:11] <wikibugs>	 10Continuous-Integration-Infrastructure: castor rsync's taking 3-5 minutes for mwgate-npm jobs - https://phabricator.wikimedia.org/T188375 (10hashar) I have filled T232644 about the network bandwidth limitation. Apparently WMCS rate limits instances as a safe guard measure (which is a good thing).  I have found...
[18:23:04] <hashar>	 bd808: the instances are bigram flavor (8 vCPU, 36G RAM),  though we dont need that much memory.  My intent is to instances with 24G RAM (mediumram)
[18:23:14] <hashar>	 which saves memory
[18:23:19] <hashar>	 and let us create new instances
[18:23:47] <hashar>	 though I am not going to add more CPU overall without consulting with cloud-admin (I dont want the cloudvirt to heat to much ;) )
[18:23:53] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Developer Productivity, 10MediaWiki-Core-Testing, 10Browser-Tests, 10Patch-For-Review: Consider moving browser based tests (Selenium and QUnit) to a non-voting pipeline - https://phabricator.wikimedia.org/T225248 (10Krinkle) >>! In T225248#5379085, @awight wrote...
[18:23:58] <hashar>	 and at the sametime migrate from Jessie to Stretch 
[18:24:02] <hashar>	 both are covered by anyway that is T226233
[18:24:03] <stashbot>	 T226233: Rebuild integration-slave-docker-* instances to use less RAM, new name and Stretch - https://phabricator.wikimedia.org/T226233
[18:24:10] <hashar>	 which got stall due to vacations / distrations etc
[18:25:52] <mutante>	 hashar: an option to do buster ?
[18:26:26] <hashar>	 mutante: hmm maybe. Not sure I am willing to deal with the havoc of a new distro though :-]  
[18:26:41] <hashar>	 mutante: more seriously, I think I filled the task before Buster got officially released
[18:26:56] <hashar>	 then I don't know whether we have the proper packages for buster, how puppet will behave etc
[18:27:36] <mutante>	 i am just saying it because i got reminded myself each time i want to do jessie-stretch in prod :)
[18:29:51] <hashar>	 :]]
[18:30:51] <hashar>	 also for the CI slow jobs
[18:30:55] <hashar>	 a good chunk of that is due to T188375
[18:30:56] <stashbot>	 T188375: castor rsync's taking 3-5 minutes for mwgate-npm jobs - https://phabricator.wikimedia.org/T188375
[18:31:04] <hashar>	 all jobs being retriggered simultaneously
[18:31:20] <hashar>	 and all hammering integration-castor03 to fetch/rsync the composer/npm cache
[18:31:37] <hashar>	 that castor system clearly is dated and does not scale :-\
[18:32:06] <hashar>	 but I don't htink I have the brain bandwidth to think about it :-/
[18:33:28] <hashar>	 also tyler found out an issue that emerged end of july, gotta fill a task about it and investigate
[18:33:30] <hashar>	 + 
[18:33:39] <hashar>	 quibble jobs doing way too many things serially
[18:33:49] <hashar>	 when some part can surely be moved to standalone jobs
[18:41:15] <hashar>	 I am off!
[18:41:51] <wikibugs>	 10Continuous-Integration-Infrastructure: castor rsync's taking 3-5 minutes for mwgate-npm jobs - https://phabricator.wikimedia.org/T188375 (10Krinkle)
[18:41:57] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, and 5 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Krinkle)
[18:42:47] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, and 5 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Krinkle)
[18:47:12] <wikibugs>	 10Phabricator (Upstream), 10Browser-Support-Google-Chrome, 10Upstream: Some Phabricator boards do not load cards anymore in Chrome 77 - https://phabricator.wikimedia.org/T232600 (10Aklapper)
[18:47:59] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10cloud-services-team: Check bandwidth limitation on integration-castor03.integration.eqiad.wmflabs / cloudvirt1002 - https://phabricator.wikimedia.org/T232644 (10Aklapper)
[18:49:00] <halfak>	 paladox, it looks like I still can't force-push to that repo but I don't get the "forge authorship" error anymore.  Instead it just says: 
[18:49:07] <halfak>	 " ! [remote rejected] master -> master (non-fast forward)   error: failed to push some refs to 'ssh://halfak@gerrit.wikimedia.org:29418/research/ores/wheels'"
[18:49:09] <halfak>	 Any suggestions?
[19:00:50] <paladox>	 halfak i have a patch for that
[19:00:59] <paladox>	 halfak https://gerrit.wikimedia.org/r/535880
[19:01:24] <wikibugs>	 (03CR) 10Halfak: [V: 03+2 C: 03+2] Grant force push to research-ores [research/ores/wheels] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/535880 (owner: 10Paladox)
[19:01:30] <halfak>	 Thanks paladox 
[19:01:37] <paladox>	 your welcome :)
[19:02:21] <halfak>	 :) 
[19:03:09] <halfak>	 paladox, I have another problem you might be able to help with.  My phab diffusion to gerrit mirror stopped working and I can't figure out why.  https://phabricator.wikimedia.org/source/ores won't push changes to https://gerrit.wikimedia.org/r/scoring/ores/ores
[19:03:22] <paladox>	 i think i know why
[19:03:32] <halfak>	 Looks like the last change up there was pushed on July 31st. 
[19:03:42] <halfak>	 \o/
[19:03:53] <paladox>	 K18 phabricator-gerrit
[19:03:55] <paladox>	 needs updating
[19:04:26] <halfak>	 Aha!  
[19:04:29] <halfak>	 Thanks
[19:05:02] <paladox>	 Unless it's using the lpap pass
[19:05:04] <paladox>	 *ldap
[19:05:52] <paladox>	 thcipriani twentyafterfour would we be able to have the pass updated for ^^ please? :)
[19:06:30] <twentyafterfour>	 paladox: hmm, I'll try 
[19:06:34] <paladox>	 thanks!
[19:07:22] <thcipriani>	 hrm, I don't see the pass in our secret password sharing repo...
[19:10:50] <paladox>	 :(
[19:19:11] <thcipriani>	 I'm sure someone has it or can reset it
[19:23:26] <halfak>	 Hmm.  This is a blocker for our deploy.  I could file a task if that's helpful. 
[19:24:19] <greg-g>	 halfak: sure, #repo-admins please
[19:24:24] <greg-g>	 sorry about this :(
[19:25:49] <halfak>	 https://phabricator.wikimedia.org/T232661
[19:25:59] <halfak>	 No worries greg-g.  Thanks for putting on your backlog :) 
[19:56:27] <wikibugs>	 (03PS4) 10Jforrester: jjb: Add operations-mw-config-hhvm-composer-lint-docker and operations-mw-config-php72-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535715
[19:58:35] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] jjb: Add operations-mw-config-hhvm-composer-lint-docker and operations-mw-config-php72-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535715 (owner: 10Jforrester)
[20:00:03] <wmf-insecte>	 Project mwcore-phpunit-coverage-master build #167: 04FAILURE in 5 hr 0 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/167/
[20:02:38] <wikibugs>	 (03PS5) 10Jforrester: jjb: Add operations-mw-config-hhvm-composer-lint-docker and operations-mw-config-php72-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535715
[20:05:55] <wikibugs>	 10Project-Admins: Create Project: Password-Reset-Update - https://phabricator.wikimedia.org/T232667 (10ifried)
[20:06:22] <wikibugs>	 10Project-Admins: Create Project: Password-Reset-Update - https://phabricator.wikimedia.org/T232667 (10ifried)
[20:06:53] <wikibugs>	 10Project-Admins: Create Project: Password-Reset-Update - https://phabricator.wikimedia.org/T232667 (10ifried)
[20:07:15] <wikibugs>	 10Project-Admins: Create Project: Password-Reset-Update - https://phabricator.wikimedia.org/T232667 (10ifried)
[20:11:07] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] jjb: Add operations-mw-config-hhvm-composer-lint-docker and operations-mw-config-php72-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535715 (owner: 10Jforrester)
[20:13:25] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[20:14:27] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: Add operations-mw-config-hhvm-composer-lint-docker and operations-mw-config-php72-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535715 (owner: 10Jforrester)
[20:14:34] <wikibugs>	 (03PS2) 10Jforrester: layout: [mediawiki-config] Drop operations-mw-config-hhvm-composer-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535717
[20:14:48] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] layout: [mediawiki-config] Drop operations-mw-config-hhvm-composer-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535717 (owner: 10Jforrester)
[20:16:09] <wikibugs>	 (03PS3) 10Jforrester: layout: [mediawiki-config] Add operations-mw-config-hhvm-composer-lint-docker and operations-mw-config-php72-composer-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535716
[21:03:24] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] layout: [mediawiki-config] Add operations-mw-config-hhvm-composer-lint-docker and operations-mw-config-php72-composer-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535716 (owner: 10Jforrester)
[21:03:55] <hauskatze>	 twentyafterfour: hello, you there?
[21:05:29] <wikibugs>	 (03Merged) 10jenkins-bot: layout: [mediawiki-config] Add operations-mw-config-hhvm-composer-lint-docker and operations-mw-config-php72-composer-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535716 (owner: 10Jforrester)
[21:05:56] <wikibugs>	 (03PS3) 10Jforrester: layout: [mediawiki-config] Drop operations-mw-config-hhvm-composer-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535717
[21:06:48] <James_F>	 !log Zuul: Reload to add operations-mw-config-php72-composer-test-docker and operations-mw-config-hhvm-composer-lint-docker
[21:06:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:08:03] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO: CI mediawiki/core run times have increased since July 26th - https://phabricator.wikimedia.org/T232626 (10hashar) I think that matches with the deployment of quibble 0.0.33/0.0.34. Notably b2f9e363046696ff1f547a9724a705bbd52dbde2 //Run node brow...
[21:08:40] <hauskatze>	 James_F: Hello. Got a minute?
[21:08:45] <James_F>	 hauskatze: Sure.
[21:08:59] <hauskatze>	 James_F: Can you see Gerrit logs by chance?
[21:09:11] <James_F>	 No, sorry.
[21:09:16] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO: CI mediawiki/core run times have increased since July 26th - https://phabricator.wikimedia.org/T232626 (10hashar)
[21:09:20] <James_F>	 What do you want to find out?
[21:09:20] <wikibugs>	 10Continuous-Integration-Config, 10Quibble, 10Patch-For-Review, 10ci-test-error: Selenium tests for Wikibase are being ran twice - https://phabricator.wikimedia.org/T231862 (10hashar)
[21:09:44] <wikibugs>	 10Continuous-Integration-Config, 10Quibble, 10Patch-For-Review, 10ci-test-error: Selenium tests for Wikibase are being ran twice - https://phabricator.wikimedia.org/T231862 (10hashar) p:05Triage→03High
[21:10:05] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO: CI mediawiki/core run times have increased since July 26th - https://phabricator.wikimedia.org/T232626 (10hashar) p:05Triage→03High
[21:10:20] <hauskatze>	 James_F: So... ORES repo (that uses GitHub -> Diffusion -> Gerrit scheme [expontaneous combustion]) isn't updating since July 31st. I was wondering if there's any error in the logs that might indicate what's going on.
[21:10:34] <paladox>	 there is
[21:10:44] <James_F>	 Hmm. Can we make it go GH->Gerrit directly to avoid that?
[21:10:45] <paladox>	 "WARN  [com.google.gerrit.httpd.ProjectBasicAuthFilter] Authentication failed for phabricator: account inactive or not provisioned in Gerrit"
[21:10:54] <James_F>	 Ha. Yeah, that's not ideal.
[21:11:00] <James_F>	 Why are we trying to mirror the wrong way?
[21:11:02] <hauskatze>	 I've switched to ssh paladox and still not updating
[21:11:03] <paladox>	 basically it's using the old http pass
[21:11:06] <hauskatze>	 phabricator@gerrit
[21:11:14] <hauskatze>	 aka k19
[21:11:20] <paladox>	 oh
[21:11:22] <hauskatze>	 paladox: where did you got that?
[21:11:30] <hauskatze>	 gerrit managers can see logs?
[21:11:33] <paladox>	 i see that on https://gerrit.wikimedia.org/r/monitoring
[21:11:39] <hauskatze>	 aah
[21:11:43] <hauskatze>	 javamelody
[21:12:52] <hauskatze>	 True: WARN  [com.google.gerrit.httpd.ProjectBasicAuthFilter] Authentication failed for phabricator: account inactive or not provisioned in Gerrit
[21:14:46] <paladox>	 hauskatze ssh will fail for phabricator too :)
[21:14:53] <paladox>	 since it's name does not match the shell
[21:14:56] <paladox>	 (i think)
[21:15:00] <paladox>	 same issue as twentyafterfour 
[21:15:26] <hauskatze>	 Username is Phabricator
[21:15:27] <wikibugs>	 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch/Buster - https://phabricator.wikimedia.org/T218729 (10Krenair)
[21:15:32] <hauskatze>	 with Uppercase P
[21:15:37] <hauskatze>	 does it matter?
[21:15:54] <hauskatze>	 ssh://Phabricator vs. ssh://phabricator ?
[21:16:11] <paladox>	 see https://phabricator.wikimedia.org/T225308
[21:16:12] <paladox>	 hauskatze ^
[21:16:18] <paladox>	 and yeh
[21:16:25] <paladox>	 the casing has to match when using git
[21:16:36] <paladox>	 login can work with both lowercase and uppercase
[21:17:56] <hauskatze>	 I can try using SSH again with P
[21:18:32] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] Deduplicate projects [integration/quibble] - 10https://gerrit.wikimedia.org/r/534134 (https://phabricator.wikimedia.org/T231862) (owner: 10Awight)
[21:18:48] <wikibugs>	 (03CR) 10Jforrester: [C: 03+1] Fix default logdir [integration/quibble] - 10https://gerrit.wikimedia.org/r/534219 (owner: 10Awight)
[21:18:55] <bd808>	 "<  James_F>	Why are we trying to mirror the wrong way?" -- TL;DR is that to deploy in prod data must source from gerrit and that project likes developing on github better
[21:19:23] <James_F>	 bd808: Yeah, sorry, "the wrong way" as in from Differential to gerrit.
[21:19:51] <bd808>	 ah. I think that answer has somethign to do with the pre-github setup
[21:20:18] <James_F>	 Mirroring a prod repo indirectly via Differential is just another thing to break that isn't critical path.
[21:20:20] <bd808>	 I think that it was diffusion->gerrit and then when they bailed on diffusion for github the chain was just extended
[21:20:52] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] Fix default logdir [integration/quibble] - 10https://gerrit.wikimedia.org/r/534219 (owner: 10Awight)
[21:21:02] <bd808>	 git-fat somehow played a role too, but my memory is fuzzy
[21:21:08] * James_F nods.
[21:21:12] <James_F>	 Fun times.
[21:21:50] <wikibugs>	 (03Merged) 10jenkins-bot: Fix default logdir [integration/quibble] - 10https://gerrit.wikimedia.org/r/534219 (owner: 10Awight)
[21:22:27] <wikibugs>	 (03CR) 10jenkins-bot: Fix default logdir [integration/quibble] - 10https://gerrit.wikimedia.org/r/534219 (owner: 10Awight)
[21:22:47] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] Reduce side-effects [integration/quibble] - 10https://gerrit.wikimedia.org/r/520240 (https://phabricator.wikimedia.org/T231862) (owner: 10Awight)
[21:23:38] <wikibugs>	 (03Merged) 10jenkins-bot: Reduce side-effects [integration/quibble] - 10https://gerrit.wikimedia.org/r/520240 (https://phabricator.wikimedia.org/T231862) (owner: 10Awight)
[21:23:40] <wikibugs>	 (03Merged) 10jenkins-bot: Deduplicate projects [integration/quibble] - 10https://gerrit.wikimedia.org/r/534134 (https://phabricator.wikimedia.org/T231862) (owner: 10Awight)
[21:24:15] <wikibugs>	 (03CR) 10jenkins-bot: Reduce side-effects [integration/quibble] - 10https://gerrit.wikimedia.org/r/520240 (https://phabricator.wikimedia.org/T231862) (owner: 10Awight)
[21:24:38] <wikibugs>	 (03CR) 10jenkins-bot: Deduplicate projects [integration/quibble] - 10https://gerrit.wikimedia.org/r/534134 (https://phabricator.wikimedia.org/T231862) (owner: 10Awight)
[21:25:15] <hauskatze>	 I've setup ssh://Phabricator@ with K19 to see if it will update
[21:25:24] <hauskatze>	 No errors so far, but no progress either
[21:26:28] <hauskatze>	 Nah, fails too, but no errors in monitoring paladox 
[21:26:52] <hauskatze>	 I think thcipriani can also update k18
[21:30:47] <paladox>	 ok
[21:31:19] <hauskatze>	 paladox: we need to update JavaMelody too apparently :)
[21:31:27] <hauskatze>	 still waiting for the upstream patch?
[21:31:28] <paladox>	 hauskatze oh?
[21:31:40] <paladox>	 i think we need to update the plugin :P
[21:31:41] <hauskatze>	 A new version of JavaMelody is available (1.79.0 instead of 1.74.0, see Release Notes)
[21:32:05] <hauskatze>	 Making a Task
[21:34:48] <wikibugs>	 10Gerrit, 10Release-Engineering-Team: Update JavaMelody to 1.79.0 - https://phabricator.wikimedia.org/T232678 (10MarcoAurelio)
[21:36:41] <hauskatze>	 It looks we have issues with "gerrit-newcomer-bot" too.
[21:37:44] <wikibugs>	 (03CR) 10Jforrester: "> Patch Set 3:" [integration/quibble] - 10https://gerrit.wikimedia.org/r/534468 (https://phabricator.wikimedia.org/T219694) (owner: 10Ladsgroup)
[21:38:33] <wikibugs>	 10Continuous-Integration-Config, 10Quibble, 10Patch-For-Review, 10ci-test-error: Selenium tests for Wikibase are being ran twice - https://phabricator.wikimedia.org/T231862 (10Jdforrester-WMF) Assuming the change to quibble works as expected, this should be fixed by minting the next version of quibble and...
[21:46:16] <wikibugs>	 10Release-Engineering-Team-TODO, 10Product-Infrastructure-Team-Backlog, 10Wikimedia-Site-requests, 10Epic, and 2 others: Sunset Wikipedia Zero - https://phabricator.wikimedia.org/T187716 (10MaxSem) Update on `zero.wikipedia.org`: in August, the subdomain had received 5398 requests. What do we think is a re...
[21:47:10] <wikibugs>	 10Gerrit, 10Release-Engineering-Team: Update JavaMelody to 1.79.0 - https://phabricator.wikimedia.org/T232678 (10MarcoAurelio) We however use `v2.14-82-g0636afe`. Do we have an update from Upstream Gerrit @Paladox ?
[21:50:59] <hauskatze>	 paladox: task filed ^^
[21:52:12] <wikibugs>	 10Gerrit, 10Release-Engineering-Team: Update JavaMelody to 1.79.0 - https://phabricator.wikimedia.org/T232678 (10Paladox) The 2.14 is from the tag, the rest i'm not sure.  But anyways, yeh there is updates we need to apply from the stable-2.15 branch.
[21:56:56] <wikibugs>	 10Project-Admins, 10Password-Reset-Update: Create Project: Password-Reset-Update - https://phabricator.wikimedia.org/T232667 (10Samwilson) 05Open→03Resolved a:03Samwilson Created: https://phabricator.wikimedia.org/project/view/4275/
[21:58:20] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: (Service Check Timed Out)
[22:03:19] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 48016 bytes in 8.257 second response time
[22:04:39] <wikibugs>	 10Gerrit, 10Operations: setup/install gerrit1001 - https://phabricator.wikimedia.org/T231046 (10Paladox)
[22:13:21] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[22:14:56] <wikibugs>	 10Release-Engineering-Team-TODO (201909), 10ORES, 10Repository-Admins, 10Scoring-platform-team (Current): Update k18 password in phab diffusion - https://phabricator.wikimedia.org/T232661 (10thcipriani) p:05Triage→03Normal a:03mmodell >>! In T232661#5485794, @MarcoAurelio wrote: > I've tried to force...
[22:31:37] <Krinkle>	 !log Disable "publishing" for mediawiki/skins/Vector in Phabricator Diffusion mirror - T143162
[22:31:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[22:31:40] <stashbot>	 T143162: Reduce task notification noise/frequency of changes to associated open patchsets - https://phabricator.wikimedia.org/T143162
[23:02:40] <wikibugs>	 10Project-Admins, 10Password-Reset-Update: Create Project: Password-Reset-Update - https://phabricator.wikimedia.org/T232667 (10Aklapper) Thanks. I've rephrased the project description slightly ("we"; "For this project...")  as the first sentence is shown in the Phab project search.
[23:05:22] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10Release Pipeline: Gerrit/Argo CI proof of concept - https://phabricator.wikimedia.org/T229246 (10dduvall)
[23:07:25] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10Release Pipeline: Gerrit/Argo CI proof of concept - https://phabricator.wikimedia.org/T229246 (10dduvall) @thcipriani, @hashar, @LarsWirzenius, @zeljkofilipin, @brennen see the updated description for my summary.
[23:07:49] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10Release Pipeline: Gerrit/Argo CI proof of concept - https://phabricator.wikimedia.org/T229246 (10dduvall)
[23:12:35] <mutante>	 hi, could somebody confirm for me 2 things  a) releases-jenkins.wikimedia.org looks normal for them after login   b) it is expected that i get "Dzahn is missing the Overall/Read permission" 
[23:18:55] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, 10serviceops: Gerrit Hardware Upgrade - https://phabricator.wikimedia.org/T222391 (10Dzahn) p:05Low→03Normal
[23:19:04] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, 10serviceops: Gerrit Hardware Upgrade - https://phabricator.wikimedia.org/T222391 (10Dzahn) 05Stalled→03Open
[23:19:09] <wikibugs>	 10Gerrit, 10Release-Engineering-Team, 10VPS-project-codesearch, 10Patch-For-Review: Gerrit thread use GC thrashing - https://phabricator.wikimedia.org/T221026 (10Dzahn)
[23:20:09] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, 10serviceops: Gerrit Hardware Upgrade - https://phabricator.wikimedia.org/T222391 (10Paladox)
[23:20:12] <wikibugs>	 10Gerrit, 10Operations: setup/install gerrit1001 - https://phabricator.wikimedia.org/T231046 (10Paladox)
[23:42:02] <wikibugs>	 10Gerrit, 10Operations: setup/install gerrit1001 - https://phabricator.wikimedia.org/T231046 (10Dzahn) The additional steps needed before this can be switched to active are tracked in the parent task T222391.
[23:47:44] <wikibugs>	 10Phabricator: Not able to edit Phabricator task T218184: ERR_SPDY_PROTOCOL_ERROR - https://phabricator.wikimedia.org/T232478 (10Krinkle)
[23:47:54] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[23:54:12] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1