[02:11:01] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [04:11:01] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:54:02] 10Project-Admins: New Phabricator project for Wikicontrib - https://phabricator.wikimedia.org/T231268 (10Rammanojpotla) [05:54:39] 10Project-Admins: New Phabricator project for Wikicontrib - https://phabricator.wikimedia.org/T231268 (10Rammanojpotla) [06:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:27:19] PROBLEM - Puppet errors on deployment-mediawiki-09 is CRITICAL: CRITICAL: 4.49% of data above the critical threshold [3.0] [07:02:46] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:15:33] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T220745 (10Simetrical) [07:21:51] 10Release-Engineering-Team, 10Wikidata, 10User-Ladsgroup, 10ci-test-error: Wikibase and WikibaseLexeme tests now take more than 1 hour - https://phabricator.wikimedia.org/T231198 (10hashar) 05Open→03Resolved a:03Ladsgroup [07:22:50] 10Release-Engineering-Team, 10Wikidata, 10User-Ladsgroup, 10ci-test-error: Wikibase and WikibaseLexeme tests now take more than 1 hour - https://phabricator.wikimedia.org/T231198 (10hashar) [07:22:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Zuul, 10Patch-For-Review: CI performance issues - https://phabricator.wikimedia.org/T231200 (10hashar) [07:45:06] (03CR) 10Hashar: [V: 03+2 C: 03+2] "Any reason this has not been merged yet?" [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/528772 (https://phabricator.wikimedia.org/T230015) (owner: 10Ladsgroup) [07:45:33] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201908), 10WMDE-Analytics-Engineering, and 2 others: Make "analytics/wmde/toolkit-analyzer-build" use git lfs - https://phabricator.wikimedia.org/T230015 (10hashar) [07:45:51] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201908), 10WMDE-Analytics-Engineering, and 2 others: Make "analytics/wmde/toolkit-analyzer-build" use git lfs - https://phabricator.wikimedia.org/T230015 (10hashar) p:05Triage→03Normal [07:45:57] (03CR) 10Hashar: [C: 03+1] Make "analytics/wmde/toolkit-analyzer-build" use git lfs [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/528772 (https://phabricator.wikimedia.org/T230015) (owner: 10Ladsgroup) [07:48:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201908): CI performance issues - https://phabricator.wikimedia.org/T231200 (10hashar) The root cause was a faulty patch merged in mediawiki/core on Friday. It roughly doubled the time... [07:49:06] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201908), 10Wikidata, 10User-Ladsgroup, 10ci-test-error: Wikibase and WikibaseLexeme tests now take more than 1 hour - https://phabricator.wikimedia.org/T231198 (10hashar) [08:11:03] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [08:22:19] RECOVERY - Puppet errors on deployment-mediawiki-09 is OK: OK: Less than 1.00% above the threshold [2.0] [08:24:36] 10Release-Engineering-Team-TODO (201908), 10Lexicographical data, 10MediaWiki-Core-Testing, 10Wikidata, 10ci-test-error: WikibaseLexeme test broken by refactor of MediaWiki's Language class - https://phabricator.wikimedia.org/T231103 (10Tarrow) > Tests that are not run by CI Ah, that's the bit I was mis... [08:44:26] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T220745 (10zeljkofilipin) [09:12:43] does anyone know why keyholder on deployment servers is not armed since a couple weeks? [09:12:55] it is an alert but also it's acked [09:13:18] guess it's not blocking deployment if that is the case [09:32:52] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T220745 (10zeljkofilipin) [09:50:20] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [10:00:11] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T220745 (10pmiazga) [10:11:01] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [10:14:47] 10Continuous-Integration-Infrastructure: castor rsync's taking 3-5 minutes for mwgate-npm jobs - https://phabricator.wikimedia.org/T188375 (10hashar) With releng/castor:0.2.3 which no more use `--compress`, a build of quibble-vendor-mysql-php72-docker took 7 minutes to fetch the cache: ` 00:00:00.763 Defined: CA... [10:17:08] 10Continuous-Integration-Config, 10MediaWiki-extensions-TwnMainPage, 10Language-Team (Language-2019-July-September): Stop running HHVM tests for TwnMainPage - https://phabricator.wikimedia.org/T231289 (10Nikerabbit) p:05Triage→03Normal [10:17:23] 10Continuous-Integration-Config, 10MediaWiki-extensions-TwnMainPage, 10Language-Team (Language-2019-July-September): Stop running HHVM tests for TwnMainPage - https://phabricator.wikimedia.org/T231289 (10Nikerabbit) a:05abi_→03Nikerabbit [10:33:35] 10Phabricator (2019-08-22): Cannot delete pages in the Mediawiki. Fatal error: Uncaught RuntimeException: Transaction callbacks are still pending: WikiPage::archiveRevisions, WikiPage::archiveRevisions, WikiPage::doDeleteArticleBatched, ManualLogEntry::insert, WikiPage::... - https://phabricator.wikimedia.org/T231290 [10:34:44] 10Continuous-Integration-Infrastructure: castor rsync's taking 3-5 minutes for mwgate-npm jobs - https://phabricator.wikimedia.org/T188375 (10hashar) When the system is mostly idle, that specific cache is fetched as fast as in 11 seconds. When busy, I looked at the transfer rate using `bwm-ng` and it shows a net... [10:37:00] 10Continuous-Integration-Infrastructure: castor rsync's taking 3-5 minutes for mwgate-npm jobs - https://phabricator.wikimedia.org/T188375 (10hashar) And last thing, the instance hosting the cache is `integration-castor03.integration.eqiad.wmflabs` (172.16.5.161). It is running on the server cloudvirt1002. The... [11:36:12] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [11:39:46] (03CR) 10D3r1ck01: "what's the update on this?" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/492634 (https://phabricator.wikimedia.org/T216971) (owner: 10Mainframe98) [12:04:53] 10Continuous-Integration-Config, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201908), 10User-zeljkofilipin: Jenkins jobs not running after pushing to gerrit for Jpita user - https://phabricator.wikimedia.org/T231003 (10zeljkofilipin) 05Open→03Resolved The problem... [12:06:01] 10Continuous-Integration-Config, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201908), 10User-zeljkofilipin: Jenkins jobs not running after pushing to gerrit for Jpita user - https://phabricator.wikimedia.org/T231003 (10zeljkofilipin) I've prepared patch for [[ https:... [12:11:05] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [12:20:28] 10Continuous-Integration-Config, 10MediaWiki-extensions-TwnMainPage, 10Language-Team (Language-2019-July-September): Stop running HHVM tests for TwnMainPage - https://phabricator.wikimedia.org/T231289 (10Nikerabbit) [12:28:40] 10Continuous-Integration-Config, 10Operations: add ci test for admin module indentation - https://phabricator.wikimedia.org/T190766 (10hashar) 05Open→03Resolved I introduced the same faulty indentation and `modules/admin/data/data_test.py` does fail since it can not parse the yaml: ` ParserError: while par... [12:30:01] 10Continuous-Integration-Config, 10MediaWiki-Debug-Logger, 10Patch-For-Review: Log suppressed errors with level=DEBUG - https://phabricator.wikimedia.org/T193472 (10hashar) Might be the same issue as T228838 [12:31:01] 10Release-Engineering-Team, 10MediaWiki-Debug-Logger, 10Core Platform Team Workboards (Clinic Duty Team), 10Wikimedia-production-error: Unreferenced log channels are not logged at all causing "logged" errors to be missed - https://phabricator.wikimedia.org/T228838 (10hashar) Probably related: {T193472} [12:35:09] 10Release-Engineering-Team-TODO (201908), 10ContentTranslation, 10User-zeljkofilipin: error while trying to run wdio tests - https://phabricator.wikimedia.org/T231305 (10zeljkofilipin) [12:37:26] 10Continuous-Integration-Config, 10MediaWiki-extensions-TranslationNotifications, 10phan-taint-check-plugin, 10MW-1.34-notes (1.34.0-wmf.3; 2019-04-30): phan-taint-check-plugin should ignore /extensions/ folder - https://phabricator.wikimedia.org/T201794 (10hashar) 05Open→03Resolved a:03sbassett Look... [12:37:30] 10Continuous-Integration-Config, 10Wikimedia-General-or-Unknown, 10phan-taint-check-plugin, 10Patch-For-Review: Enable phan-taint-check-plugin on all Wikimedia-deployed repositories where it is currently passing - https://phabricator.wikimedia.org/T201219 (10hashar) [12:37:37] 10Release-Engineering-Team-TODO (201908), 10ContentTranslation, 10User-zeljkofilipin: error while trying to run wdio tests - https://phabricator.wikimedia.org/T231305 (10zeljkofilipin) The problem seems to be in wdio 5. I didn't test it yet. We're still on wdio 4. Try using sample files from this task: T210726. [12:38:26] 10Continuous-Integration-Config, 10Composer: parallel-lint reports lint errors on line -1 - https://phabricator.wikimedia.org/T202581 (10hashar) 05Open→03Declined Decline for now since I have never been able to find a way to reproduce it. [12:40:37] 10Continuous-Integration-Config, 10MediaWiki-Releasing: php55lint is not running for REL1_30 mediawiki/core patches - https://phabricator.wikimedia.org/T203054 (10hashar) 05Open→03Declined PHP 5.5 and REL1_30 are gone now. [12:41:42] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10phan-taint-check-plugin: Configure CI to run phan-taint-check-plugin for MediaWiki core - https://phabricator.wikimedia.org/T203630 (10hashar) 05Open→03Resolved It seems that has been solved. [12:44:37] 10Continuous-Integration-Config, 10Composer: parallel-lint reports lint errors on line -1 - https://phabricator.wikimedia.org/T202581 (10Daimona) There's an upstream bug report by @Krinkle: https://github.com/JakubOnderka/PHP-Parallel-Lint/issues/136. I tried locally and I also cannot reproduce, but it keeps... [12:45:01] 10Continuous-Integration-Config, 10Operations, 10Patch-For-Review: rspec-puppet fails with Could not find the daemon directory (tested [/etc/sv,/var/lib/service]) - https://phabricator.wikimedia.org/T203645 (10hashar) 05Open→03Resolved a:03hashar When someone encounters the issue, the module spec_helpe... [12:46:08] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10phan-taint-check-plugin: Configure CI to run phan-taint-check-plugin for MediaWiki core - https://phabricator.wikimedia.org/T203630 (10Daimona) >>! In T203630#5441293, @hashar wrote: > It seems that has been solved. Yes, or at least, CI has been c... [12:47:39] 10Continuous-Integration-Infrastructure, 10Jenkins: CI console output is messy for wmf-quibble-vendor-mysql-hhvm-docker - https://phabricator.wikimedia.org/T206867 (10hashar) That is due to the configuration for the [[ https://wiki.jenkins.io/display/JENKINS/Collapsing+Console+Sections+Plugin | Collapsing Cons... [12:48:16] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10phan-taint-check-plugin: Enable seccheck for MW core - https://phabricator.wikimedia.org/T231311 (10Daimona) [12:48:29] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10phan-taint-check-plugin: Enable seccheck for MW core - https://phabricator.wikimedia.org/T231311 (10Daimona) [12:59:17] 10Beta-Cluster-Infrastructure, 10User-LokalProfil: Beta cluster: Campaign editor rights request - https://phabricator.wikimedia.org/T230768 (10JeanFred) 05Open→03Resolved a:03JeanFred Done. [13:00:23] 10Project-Admins: New Phabricator project for Wikicontrib - https://phabricator.wikimedia.org/T231268 (10Aklapper) 05Open→03Resolved a:03Aklapper (Personally I'm not a huge fan of having differing names for stuff... `contrabandapp` in the URL vs `WikiContribs` in the UI; for example I cannot find "WikiCont... [13:12:18] 10Project-Admins: Request project for parliamentdiagram - https://phabricator.wikimedia.org/T230460 (10Luke081515) a:03Slashme Hello @Slashme, please answer the question above to continue here. [13:14:20] 10Project-Admins: Create a tag on Phabricator for fawiki elections - https://phabricator.wikimedia.org/T230615 (10Luke081515) 05Open→03Declined Looks solved for me. Choosing declined, because there was no new tag created. [13:37:28] 10Phabricator, 10Developer-Advocacy (Jul-Sep 2019): Cover in monthly Phabricator statistics email to wikitech-l@ where to find list of users who created the most tasks - https://phabricator.wikimedia.org/T231320 (10Aklapper) p:05Triage→03Low [13:44:37] 10Continuous-Integration-Config, 10MediaWiki-extensions-TwnMainPage, 10Language-Team (Language-2019-July-September): Stop running HHVM tests for TwnMainPage - https://phabricator.wikimedia.org/T231289 (10Nikerabbit) It seems `extension-quibble` should be replaced with either `extension-quibble-composer-nohhv... [13:45:41] zeljkof: so lets triage the blockers :) [13:45:42] https://phabricator.wikimedia.org/T231071#5433964 [13:45:52] the issue is php7 serialize() something rather large [13:45:59] the result is stored in the cache [13:46:17] then if anoyher process runs on hhvm and unserialize() it explodes because that exceed hhvm default serialize limit [13:46:18] so [13:46:21] hashar: maybe -operations is better? [13:46:23] that is an issue related to the php7.2 migration [13:46:24] but here is fine [13:46:29] and really that is not a blocker to this train [13:46:45] I am editing it :) [13:46:50] hashar: also, feel free to comment directly on the tasks, and resolve/remove them as you see fit [13:52:21] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T220745 (10hashar) [13:54:24] eek [13:54:30] neighbor rininging the bell :D [13:54:33] bbl [13:56:53] zeljkof: re :) [13:58:42] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T220745 (10hashar) [13:58:47] two less [14:01:36] zeljkof: so yeah looking at the two others, I think you should promote group0 anyway [14:02:34] hashar: will do, thanks [14:03:45] syncing testwiki, will push wmf.20 to group 0 as soon as it's done [14:04:29] 10Project-Admins: New Phabricator project for Wikicontrib - https://phabricator.wikimedia.org/T231268 (10Rammanojpotla) Thanks for the tag @Aklapper The tool is presently hosted with the name "Contrabandapp". It will be migrated to "WikiContrib" soon :) [14:08:41] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T220745 (10hashar) [14:10:46] zeljkof: last is https://phabricator.wikimedia.org/T231029 """DefaultPreferencesFactory.php: Global default '' is invalid for field incubatortestwiki-code""" [14:10:50] which I have no idea what it is :\ [14:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [14:14:54] 10Release-Engineering-Team, 10MediaWiki-Debug-Logger, 10Core Platform Team Workboards (Clinic Duty Team), 10Wikimedia-production-error: Consider enabling all MW log channels by default for WMF - https://phabricator.wikimedia.org/T228838 (10Krinkle) [14:27:05] 10Release-Engineering-Team-TODO (201908), 10ContentTranslation, 10User-zeljkofilipin: error while trying to run wdio tests - https://phabricator.wikimedia.org/T231305 (10Jpita) after removing wdio 5 and following the instructions on T210726 , now I'm getting this error when trying to run: `➜ ContentTranslat... [14:28:43] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201908), 10Documentation: Document how to deploy a new Quibble version to CI - https://phabricator.wikimedia.org/T231251 (10Krinkle) Keeping it in the repo is best I think. Yeah. A... [15:49:44] 10Project-Admins: New Phabricator project for Wikicontrib - https://phabricator.wikimedia.org/T231268 (10Rammanojpotla) @Aklapper I have created a new tool with name "wikicontrib" almost 12 hours ago. But I am not sure why it is not in the list of tools here (https://tools.wmflabs.org/admin/tools). I can find it... [15:51:30] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T220745 (10zeljkofilipin) [15:55:43] 10Project-Admins: Estonian version of "needs-volunteer" tag - https://phabricator.wikimedia.org/T209354 (10tramm) [16:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [16:15:19] 12:14:25 rsync: failed to set times on "/cache/.": Operation not permitted (1) [16:15:24] first time I've seen this from CI [16:15:26] https://integration.wikimedia.org/ci/job/conftool-tox-docker/353/console [16:17:15] I think it's mostly something you can ignore [16:17:24] well, it failed the job [16:17:30] but looks like a 'recheck' is working fine [16:17:44] h sorry that was the wrong error [16:17:51] 12:14:34 Unable to find image 'docker-registry.wikimedia.org/releng/tox-conftool:0.4.0' locally [16:17:57] 12:14:35 docker: Error response from daemon: received unexpected HTTP status: 502 connect failed. [16:22:40] 10Phabricator, 10I18n, 10User-Nikerabbit: Please add Brazilian Portuguese (pt-br) language to Phabricator - https://phabricator.wikimedia.org/T215697 (10mmodell) @Eduaddad it's updated 2 to 3 times each month. [16:24:28] 10Phabricator, 10I18n, 10User-Nikerabbit: Please add Brazilian Portuguese (pt-br) language to Phabricator - https://phabricator.wikimedia.org/T215697 (10mmodell) >>! In T215697#5435645, @Eduaddad wrote: > https://translatewiki.net/wiki/Phabricator:phabricator-maniphest-9d0e05d653ce25e7/pt-br or https://trans... [16:28:16] 10Phabricator, 10I18n, 10User-Nikerabbit: Please add Brazilian Portuguese (pt-br) language to Phabricator - https://phabricator.wikimedia.org/T215697 (10Eduaddad) I realize that some of my translations were not placed Thank you so much for verifying the implementation [16:31:03] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, and 5 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Simetrical) I think the best returns here... [16:34:28] 10Project-Admins: Estonian version of "needs-volunteer" tag - https://phabricator.wikimedia.org/T209354 (10tramm) 05Declined→03Open Just come back at this... First of all, most of our tasks are not coding tasks. What we are doing now is that we are putting tags on tasks that are explained well enough to be c... [16:43:11] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T220745 (10zeljkofilipin) [16:43:39] 10Project-Admins: Estonian version of "needs-volunteer" tag - https://phabricator.wikimedia.org/T209354 (10tramm) By the way, I find suggestions of `#good-first-task` and `#mentoring` (or maybe `#mentoring-available` or `#mentoring-needed`) in T194657 really meaningful. Feel free to change the title of this issu... [16:44:03] (03CR) 1020after4: [C: 03+2] local-charts: CLI for managing minikube, helm, etc [releng/local-charts] - 10https://gerrit.wikimedia.org/r/525563 (https://phabricator.wikimedia.org/T224939) (owner: 1020after4) [16:44:59] (03CR) 1020after4: [V: 03+2 C: 03+2] local-charts: CLI for managing minikube, helm, etc [releng/local-charts] - 10https://gerrit.wikimedia.org/r/525563 (https://phabricator.wikimedia.org/T224939) (owner: 1020after4) [16:55:00] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T220745 (10zeljkofilipin) [17:31:45] 10Phabricator, 10Developer-Advocacy (Jul-Sep 2019): Cover in monthly Phabricator statistics email to wikitech-l@ where to find list of users who created the most tasks - https://phabricator.wikimedia.org/T231320 (10Aklapper) 05Open→03Resolved https://lists.wikimedia.org/pipermail/wikitech-l/2019-August/092... [17:37:52] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 53.33% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [17:53:45] (03PS2) 10Thcipriani: Move puppet jobs to dedicated small node [integration/config] - 10https://gerrit.wikimedia.org/r/532437 (https://phabricator.wikimedia.org/T231200) [18:03:29] (03PS3) 10Jforrester: jjb: Drop mwext-npm-doc-publish [integration/config] - 10https://gerrit.wikimedia.org/r/532420 [18:03:45] (03CR) 10Jforrester: [C: 04-1] "MF is not ready yet (this job breaks)." [integration/config] - 10https://gerrit.wikimedia.org/r/532417 (https://phabricator.wikimedia.org/T230841) (owner: 10Jforrester) [18:11:05] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [18:12:36] (03CR) 10Bstorm: "> Patch Set 1: Code-Review-1" [integration/config] - 10https://gerrit.wikimedia.org/r/530606 (https://phabricator.wikimedia.org/T228499) (owner: 10Bstorm) [18:13:37] (03CR) 10Bstorm: "Sorry, may be fixed after another patch or so." [integration/config] - 10https://gerrit.wikimedia.org/r/530606 (https://phabricator.wikimedia.org/T228499) (owner: 10Bstorm) [18:14:46] (03CR) 10Bstorm: "> Patch Set 1:" [integration/config] - 10https://gerrit.wikimedia.org/r/530606 (https://phabricator.wikimedia.org/T228499) (owner: 10Bstorm) [18:16:12] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Core Platform Team: Audit all existing code to ensure that any extension currently or previously adding blobs to ExternalStore has been registering a reference in the text table (and fix up if wrong) - https://phabricator.wikimedia.org/T106388 (10... [18:24:50] PROBLEM - Puppet staleness on integration-agent-puppet-docker-1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [43200.0] [18:35:04] James_F: I possibly identified the issue with storybook on https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MobileFrontend/+/532736/ ?are we just backed up again? [18:36:46] 10Continuous-Integration-Config, 10MediaWiki-extensions-TwnMainPage, 10Language-Team (Language-2019-July-September): Stop running HHVM tests for TwnMainPage - https://phabricator.wikimedia.org/T231289 (10hashar) For historical reasons CI default to use `mediawiki/vendor.git` to ship the composer dependencies... [18:37:39] Jdlrobson: Hmm, no, not backed up. Let me have a poke. [18:40:06] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [18:41:37] Jdlrobson: Just looks like jenkins has too many patches in test at once, sorry. [18:41:48] All the executors are running. [18:44:04] But waiting over an hour for an experimental job is extreme. [18:44:06] * James_F sighs. [18:46:50] 10Continuous-Integration-Config, 10MediaWiki-extensions-TwnMainPage, 10Language-Team (Language-2019-July-September): Stop running HHVM tests for TwnMainPage - https://phabricator.wikimedia.org/T231289 (10Nikerabbit) We're still on PHP 7.1. TwnMainPage doesn't have much dependencies, so `extension-quibble-com... [18:49:21] 10Continuous-Integration-Config, 10Composer: parallel-lint reports lint errors on line -1 - https://phabricator.wikimedia.org/T202581 (10hashar) 05Declined→03Open Reopening since that still happening and we have a recent reproduction case. git fetch "https://gerrit.wikimedia.org/r/mediawiki/extensions/Ab... [18:52:28] 10Continuous-Integration-Config, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201908), 10User-zeljkofilipin: Jenkins jobs not running after pushing to gerrit for Jpita user - https://phabricator.wikimedia.org/T231003 (10hashar) `@wikimedia.org` emails are whitelisted,... [18:57:42] James_F: so I think the issue with storybook stuff is that it's trying to access files inside core that don't exist [18:58:04] one way I can work around that is use a script to curl the files it needs into a temporary directory [18:58:05] Jdlrobson: Oh, yes, documentation runs are isolated to the repo. [18:58:10] Eww. [18:58:14] Can you just… not? [18:58:16] https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MobileFrontend/+/532736/3/dev-scripts/storybook.sh [18:58:22] it needs access to the styles :/ [18:58:46] Create a local copy of them in the repo? [18:58:50] ideally those modules in core would be on npm and I could just pull them that way [18:59:03] I don't think that model is ever going to fly. [18:59:06] a local copy would work but we'd need to keep it in sync [18:59:20] which also feels icky to me [18:59:23] Have a job in your unit tests that checks they're the same and fails if they aren't. [18:59:26] which is why i preferred the curl model [18:59:28] 5 mins' work. [18:59:37] External curl is not great. [18:59:50] James_F: is there a way to access a local version of core? [19:00:07] Yes, when you're running unit/integration tests we provide it. [19:00:18] But not for the "lightweight" tasks like documentation runs. [19:00:33] (Ultimately we should split the unit ones to not have access to MW either, but that's for the future.) [19:02:44] so inside `npm run doc` command what would be the best way to get copies of those files? The documentation is UI documentation so ideally I'd love it to reflect what's in core (and the actual UI) rather than copy and paste it and give false impressions of what that should look like. [19:03:16] (the curl for the time being will at least help me confirm that's why we're seeing the issue) [19:04:33] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T220745 (10Jdforrester-WMF) [19:04:43] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T220745 (10Jdforrester-WMF) [19:06:03] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T220745 (10Jdforrester-WMF) [19:07:50] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Request Sauce Labs access for niedzielski - https://phabricator.wikimedia.org/T206358 (10Niedzielski) 05Resolved→03Open @zeljkofilipin, I'm getting "Your trial period has ended. Upgrade now..." messages again :[ Can you take a look? [19:20:48] Hi! I'm getting this weird error whenever I try to pull down an image from docker-registry.wikimedia.org: `Error response from daemon: received unexpected HTTP status: 502 connect failed` [19:21:22] Any ideas? thcipriani [19:39:24] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [20:05:42] (03PS1) 10Nikerabbit: TwnMainPage: switch to extension-quibble-composer-nohhvm [integration/config] - 10https://gerrit.wikimedia.org/r/532779 (https://phabricator.wikimedia.org/T231289) [20:11:09] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [20:21:55] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T220745 (10MaxSem) [20:22:12] (03CR) 10Jforrester: [C: 03+2] TwnMainPage: switch to extension-quibble-composer-nohhvm [integration/config] - 10https://gerrit.wikimedia.org/r/532779 (https://phabricator.wikimedia.org/T231289) (owner: 10Nikerabbit) [20:23:00] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [20:24:48] (03Merged) 10jenkins-bot: TwnMainPage: switch to extension-quibble-composer-nohhvm [integration/config] - 10https://gerrit.wikimedia.org/r/532779 (https://phabricator.wikimedia.org/T231289) (owner: 10Nikerabbit) [20:25:35] !log Zuul: TwnMainPage: switch to extension-quibble-composer-nohhvm T231289 [20:25:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:25:40] T231289: Stop running HHVM tests for TwnMainPage - https://phabricator.wikimedia.org/T231289 [20:33:23] clarakosi: maybe ask in #wikimedia-operations? [20:34:39] clarakosi: did you get it figured out? What command are you trying? [20:36:20] (03CR) 10Thcipriani: [C: 03+2] Move puppet jobs to dedicated small node [integration/config] - 10https://gerrit.wikimedia.org/r/532437 (https://phabricator.wikimedia.org/T231200) (owner: 10Thcipriani) [20:38:00] thcipriani: no haven't figured it out yet. i'm running `docker pull docker-registry.wikimedia.org/wikimedia/mediawiki-services-kask:v1.0.3` [20:39:39] (03Merged) 10jenkins-bot: Move puppet jobs to dedicated small node [integration/config] - 10https://gerrit.wikimedia.org/r/532437 (https://phabricator.wikimedia.org/T231200) (owner: 10Thcipriani) [20:41:06] clarakosi: weird, works for me [20:41:49] clarakosi: that's strange, I can definitely pull that image, I don't have any access to the registry box itself (afaik) serviceops may be able to help. Can you netcat to port 443? nc -vz docker-registry.wikimedia.org -w 1 443 ? Or pull other images? [20:42:18] kostajh: yeah urandom tried it too and it worked for him [20:42:39] what docker are you using? Docker machine, docker for Mac, etc? [20:42:46] Docker for mac [20:43:02] And I can't get other images from docker-registry.wikimedia [20:43:12] But I can from docker hub [20:45:57] well that's odd. [20:46:53] what about: curl https://docker-registry.wikimedia.org/v2/wikimedia/mediawiki-services-kask/tags/list [20:46:57] does that get you anything? [20:50:15] thcipriani: curl works. Gets a json blob [20:51:25] well there you go, no browser for you clarakosi [20:51:56] Reedy: 😂 [20:58:31] hrm, could it be your docker version somehow? what version are you running? I'm kind of grasping at straws... [20:59:42] version 19.03.1 [21:00:19] hrm, that's the same version I've got [21:02:20] Are you also on mac? [21:04:57] I am not, I'm on linux [21:05:39] ah ok :/ [21:06:06] (03CR) 10Bstorm: "I am now confident in the tests in master having merged the various patches needed." [integration/config] - 10https://gerrit.wikimedia.org/r/530606 (https://phabricator.wikimedia.org/T228499) (owner: 10Bstorm) [21:06:20] well, the first step of pulling is pulling the manifest, so: curl https://docker-registry.wikimedia.org/v2/wikimedia/mediawiki-services-kask/manifests/v1.0.3 [21:08:38] the next step would be downloading layers so: curl https://docker-registry.wikimedia.org/v2/wikimedia/mediawiki-services-kask/blobs/sha256:eca4fd889718e5c1856e8b52d84b981e7ec54770f60827995ee15915923b267f -o /dev/null [21:08:49] if those both succeed then I'm at a loss :\ [21:09:26] thcipriani i've finally managed to work on supporting our theme under Polymer 2 https://gerrit-review.googlesource.com/c/gerrit/+/234734/17 :) [21:10:01] paladox: nice :) [21:10:25] thcipriani https://gerrit-review.googlesource.com/c/gerrit/+/234734/17#message-efd5136ffb6d942bb0aa3ed1c09528bdf7ef7265 is how our theme will look [21:11:01] heh, I was expecting a screenshot :) [21:11:07] I don't know why [21:11:10] oh [21:11:19] i have it running locally :P [21:11:31] looks nice and compact [21:11:36] thcipriani: second one works. first one gave an html error page [21:11:49] thcipriani https://phabricator.wikimedia.org/F30132547 [21:12:21] clarakosi: ah, that's interesting, that must be where it's failing: getting the manifest from the registry, that's something! [21:12:47] thcipriani: progress! [21:13:21] clarakosi: indeed. Lets see if there's someone with access to that machine that can look at a log file or give us other pointers from here [21:14:04] I also know how we can do this under the dark theme too :P [21:14:16] so 2 themes, one for the app and one for the dark mode [21:23:01] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (201908), 10Developer Productivity, 10local-charts: Move parsoid chart from local-charts to deployment-charts repository - https://phabricator.wikimedia.org/T228909 (10greg) 05Open→03Resolved [21:23:02] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (201908), 10Developer Productivity, 10local-charts, 10Patch-For-Review: Move local-charts helm charts to a chart repository - https://phabricator.wikimedia.org/T224935 (10greg) [21:23:19] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (201908), 10Developer Productivity, 10local-charts: Update local-charts repository to use parsoid chart from deployment-charts repo - https://phabricator.wikimedia.org/T228914 (10greg) 05Open→03Resolved [21:23:21] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (201908), 10Developer Productivity, 10local-charts, 10Patch-For-Review: Move local-charts helm charts to a chart repository - https://phabricator.wikimedia.org/T224935 (10greg) [21:34:54] clarakosi: I think it's the new roll-out of ATS to replace Varnish that has bitten you. [21:38:44] thcipriani 2.15.16 will be released tomorrow too! [21:38:45] https://gerrit-review.googlesource.com/c/gerrit/+/234875 [21:41:28] James_F: ahh ok. And thanks for tagging traffic to the ticket! [21:42:05] clarakosi: Hope it's easy to fix. [21:42:27] 🤞 [22:11:04] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [22:18:46] 10Continuous-Integration-Config, 10VisualEditor: No CI for wmf branches of VE-core - https://phabricator.wikimedia.org/T231394 (10Jdforrester-WMF) p:05Triage→03Low [22:18:54] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO, 10VisualEditor: No CI for wmf branches of VE-core - https://phabricator.wikimedia.org/T231394 (10Jdforrester-WMF) [22:20:21] Project beta-update-databases-eqiad build #36300: 04FAILURE in 20 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/36300/ [23:21:29] Yippee, build fixed! [23:21:30] Project beta-update-databases-eqiad build #36301: 09FIXED in 1 min 28 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/36301/ [23:34:26] (03PS1) 10Thcipriani: php-fpm: sudo_check_call takes a string [tools/scap] - 10https://gerrit.wikimedia.org/r/532828 (https://phabricator.wikimedia.org/T224857) [23:34:28] (03PS1) 10Thcipriani: php-fpm: restart as mwdeploy as root [tools/scap] - 10https://gerrit.wikimedia.org/r/532829 [23:37:04] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Beta cluster - requesting bureaucrat rights - https://phabricator.wikimedia.org/T228352 (10DannyS712) @Jdforrester-WMF @TheDJ @Nemo_bis do you think you could take a look at this? [23:40:54] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Beta cluster - requesting bureaucrat rights - https://phabricator.wikimedia.org/T228352 (10Peachey88) I think from memory @greg normally deals with these. [23:44:16] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Beta cluster - requesting bureaucrat rights - https://phabricator.wikimedia.org/T228352 (10greg) I can't do it right now, but I'm fine with this. [23:50:34] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Beta cluster - requesting bureaucrat rights - https://phabricator.wikimedia.org/T228352 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF Done. [23:53:17] (03PS1) 10Jrbranaa: added Popups and MobileFrontend extensions to codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/532832