[00:01:05] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [00:12:11] Project beta-scap-eqiad build #265566: 04FAILURE in 0.85 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/265566/ [00:14:25] Project beta-scap-eqiad build #265567: 04STILL FAILING in 0.73 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/265567/ [00:24:25] Project beta-scap-eqiad build #265568: 04STILL FAILING in 0.73 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/265568/ [00:27:30] Project beta-scap-eqiad build #265569: 04STILL FAILING in 0.7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/265569/ [00:28:18] James_F: Looks like you upset beta [00:28:23] 01:27:30 RuntimeError: Scap failed!: Call to mwscript eval.php stderr: Warning: rename(/tmp/conf2-enwikiWewz9U,/tmp/mw-cache-master//tmp/mw-cache-master/conf2-enwiki): No such file or directory in /srv/mediawiki-staging/multiversion/MWConfigCacheGenerator.php on line 120 [00:28:23] 01:27:30 00:27:30 scap failed: RuntimeError Scap failed!: Call to mwscript eval.php stderr: Warning: rename(/tmp/conf2-enwikiWewz9U,/tmp/mw-cache-master//tmp/mw-cache-master/conf2-enwiki): No such file or directory in /srv/mediawiki-staging/multiversion/MWConfigCacheGenerator.php on line 120 (duration: 00m 00s) [00:29:42] Oh, fun. [00:29:48] Let me finish prod first. [00:30:15] But /tmp shouldn't be cleaned that often? [00:34:26] Project beta-scap-eqiad build #265570: 04STILL FAILING in 0.74 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/265570/ [00:35:26] Yeah yeah. [00:44:30] Project beta-scap-eqiad build #265571: 04STILL FAILING in 3.8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/265571/ [00:45:54] * James_F grumbles at people re-factoring code and re-using variables for different purposes. :-) [00:51:57] Yippee, build fixed! [00:51:58] Project beta-scap-eqiad build #265572: 09FIXED in 2 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/265572/ [00:55:16] Reedy: There you go. [00:55:32] (Whoops.) [01:01:01] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [01:16:04] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [01:16:45] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<20.00%) [02:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [02:15:34] 10Phabricator: Rename to MJL - https://phabricator.wikimedia.org/T232062 (10MattLongCT) [02:15:56] 10Phabricator: Rename Phabricator account to MJL - https://phabricator.wikimedia.org/T232062 (10MattLongCT) [03:13:22] 10Continuous-Integration-Infrastructure, 10DataValues, 10Wikidata: RuntimeException breaking CI builds - https://phabricator.wikimedia.org/T232063 (10Mholloway) [03:13:44] 10Continuous-Integration-Infrastructure, 10DataValues, 10Wikidata: RuntimeException breaking CI builds - https://phabricator.wikimedia.org/T232063 (10Mholloway) p:05Triage→03High [03:17:10] 10Continuous-Integration-Infrastructure, 10DataValues, 10Wikidata: RuntimeException breaking CI builds - https://phabricator.wikimedia.org/T232063 (10Mholloway) [04:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:04:05] paladox: https://gerrit.wikimedia.org/r/c/operations/puppet/+/511614 [06:11:01] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:40:38] 10Release-Engineering-Team-TODO (201909), 10Wikidata, 10Wikidata-Termbox, 10serviceops, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10hashar) [06:41:46] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:07:25] 10Continuous-Integration-Infrastructure, 10DataValues, 10Wikidata: RuntimeException breaking CI builds - https://phabricator.wikimedia.org/T232063 (10hashar) I have looked at the builds for https://integration.wikimedia.org/ci/job/wikibase-repo-docker/ , downloaded the console log for the last successful one... [07:09:54] 10Continuous-Integration-Infrastructure, 10DataValues, 10Wikidata: RuntimeException breaking CI builds - https://phabricator.wikimedia.org/T232063 (10hashar) Or the lib has to be updated in mediawiki/vendor.git ? [07:28:50] 10Release-Engineering-Team-TODO (201909), 10Wikidata, 10Wikidata-Termbox, 10serviceops, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10Tarrow) I think this is probably the same as T229313. We sus... [07:40:01] 10Continuous-Integration-Infrastructure, 10DataValues, 10Wikidata, 10Patch-For-Review: RuntimeException breaking CI builds - https://phabricator.wikimedia.org/T232063 (10hashar) Pinning data-values/data-values < 2.2.0 in Wikibase does fix the composer install when having mediawiki/vendor.git. So I guess w... [07:41:00] 10Release-Engineering-Team-TODO (201909), 10Wikidata, 10Wikidata-Termbox, 10serviceops, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10Joe) >>! In T232035#5467309, @Tarrow wrote: > I think this i... [07:43:46] 10Continuous-Integration-Infrastructure, 10DataValues, 10Wikidata, 10Patch-For-Review: RuntimeException breaking CI builds - https://phabricator.wikimedia.org/T232063 (10hashar) Fun thing, I have the same issue when attempting to update the lib in mediawiki/vendor: ` lang=diff diff --git a/composer.json b/... [07:58:38] 10Release-Engineering-Team-TODO (201909), 10Wikidata, 10Wikidata-Termbox, 10serviceops, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10hashar) Some debugging thingies, reaching out to https://ww... [08:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [08:12:47] 10Release-Engineering-Team-TODO (201909), 10Wikidata, 10Wikidata-Termbox, 10serviceops, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10Tarrow) I tried the same from codfw's deployment host and sa... [08:26:05] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Scap, 10serviceops, and 3 others: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) I did some tests, and we still have one problem with `scap pull`: - It is... [08:26:42] 10Release-Engineering-Team-TODO (201909), 10Wikidata, 10Wikidata-Termbox, 10serviceops, and 4 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10hashar) Promoted wikidatawiki again and I ran the service ch... [08:52:17] 10Release-Engineering-Team-TODO (201909), 10Wikidata, 10Wikidata-Termbox, 10serviceops, and 4 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10hashar) Looking at `icinga_contact.raw: "irc"`, `Termbox` ov... [08:55:37] 10Release-Engineering-Team-TODO (201909), 10Wikidata, 10Wikidata-Termbox, 10serviceops, and 4 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10Joe) So the real issue was: - termbox **correctly** uses the... [08:58:17] 10Release-Engineering-Team-TODO (201909), 10Wikidata, 10Wikidata-Termbox, 10serviceops, and 4 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10hashar) Rolled back to 1.34.0-wmf.20. The latency is similar... [08:58:21] 10Continuous-Integration-Infrastructure, 10DataValues, 10Wikidata, 10Patch-For-Review, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞): RuntimeException breaking CI builds - https://phabricator.wikimedia.org/T232063 (10alaa_wmde) [08:58:38] 10Continuous-Integration-Infrastructure, 10DataValues, 10Wikidata, 10Patch-For-Review, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞): RuntimeException breaking CI builds - https://phabricator.wikimedia.org/T232063 (10alaa_wmde) a:03alaa_wmde [09:05:46] 10Release-Engineering-Team-TODO (201909), 10Wikidata, 10Wikidata-Termbox, 10serviceops, and 4 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10hashar) 05Open→03Resolved a:03hashar Fixed with the he... [09:05:48] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T220746 (10hashar) [09:09:48] be back this afternoon [09:09:51] train looks fine on group1 [09:09:52] :) [09:09:54] * hashar waves [09:39:37] 10Continuous-Integration-Infrastructure, 10DataValues, 10Wikidata, 10Patch-For-Review, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞): RuntimeException breaking CI builds - https://phabricator.wikimedia.org/T232063 (10alaa_wmde) The issue is caused by the fact that DataValues started excluding test... [09:52:26] 10Continuous-Integration-Infrastructure, 10DataValues, 10Wikidata, 10Patch-For-Review, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞): RuntimeException breaking CI builds - https://phabricator.wikimedia.org/T232063 (10Pablo-WMDE) 2.2.1 release is out: https://github.com/DataValues/DataValues/releases [10:01:45] 10Phabricator: Rename Phabricator account to MJL - https://phabricator.wikimedia.org/T232062 (10Aklapper) 05Open→03Resolved a:03Aklapper Sure, done in https://phabricator.wikimedia.org/people/manage/19017/ - cheers :) [10:11:01] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [10:54:51] 10Release-Engineering-Team-TODO (201909), 10AbuseFilter, 10Patch-For-Review, 10User-zeljkofilipin: The first Selenium test for AbuseFilter - https://phabricator.wikimedia.org/T214478 (10zeljkofilipin) [11:03:21] 10Project-Admins, 10translatewiki.net: Convert T41480 ("Issues affecting translatewiki.net") to a project tag - https://phabricator.wikimedia.org/T231991 (10Aklapper) Hmm I see. Sounds like two separate tags to me then. Something like `#translatewiki.net-Infrastructure` (or such; currently `#translatewiki.net`... [12:11:04] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [12:48:26] 10Project-Admins, 10translatewiki.net: Convert T41480 ("Issues affecting translatewiki.net") to a project tag - https://phabricator.wikimedia.org/T231991 (10Nikerabbit) Yeah two tags. And naming is hard. [12:57:31] 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Request Sauce Labs access for niedzielski - https://phabricator.wikimedia.org/T206358 (10zeljkofilipin) @Niedzielski do you have the same problem as @Etonkovidova? Or do you get the error message all the time? [12:57:35] 10Project-Admins, 10translatewiki.net: Convert T41480 ("Issues affecting translatewiki.net") to a project tag - https://phabricator.wikimedia.org/T231991 (10abi_) I also prefer the approach of having multiple tags rather than a single one. The proposed columns under the new //external// project tag also sound... [12:58:32] 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Request Sauce Labs access for niedzielski - https://phabricator.wikimedia.org/T206358 (10zeljkofilipin) a:05zeljkofilipin→03None @Niedzielski seems to be out of the office until November. [12:59:03] 10Release-Engineering-Team-TODO (201909): Request Sauce Labs access for niedzielski - https://phabricator.wikimedia.org/T206358 (10zeljkofilipin) [13:44:07] hashar: Is there an extension I can look at that shows how to get unit tests running on Jenkins? It looks like mw.o is out of date. [13:44:25] hexmode: hi [13:44:35] hey [13:44:37] hexmode: not sure what is the new standard but kostajh might now [13:44:43] some stuff has been refactored recently [13:45:06] right.... ok, I guess I'll ask kostajh ... [13:45:17] hexmode: which extension are you working on? [13:45:22] theoretically , mediawiki/extensions/BoilerPlate might be up to date [13:45:27] PageForms [13:45:40] ok, I'll look at BoilerPlate [13:45:43] hexmode: if I understand right, you are talking about integration/config and getting the extension added to the right layout, yeah? [13:45:52] kostajh: also WhoIsWatching [13:46:06] Or are you talking about writing unit (not integration) tests? [13:46:38] kostajh: yes, but allso getting jenkins to run the test suite to block merges ... put a -1 on commits with failing tests [13:47:19] I want to do some refactoring on PageForms, but thought it would be good to have tests, first :) [13:47:39] * kostajh looks [13:47:50] hexmode: also, maybe we should chat in #wikimedia-dev [13:48:14] kostajh: ok, I just saw hashar was here... switching [13:48:45] I didn’t even know that channel, what’s the difference to #wikimedia-tech? [13:48:47] I think mediawiki core just look at tests/phpunit and adds that automatically when using tests/phpunit.php --testsuite extensions [13:48:56] then there is some refactor made by kostajh to also run true unit tests [13:49:00] (lots of wikibugs spam, judging by the logs ^^) [13:49:25] Lucas_WMDE: #wikimedia-dev was a general chat for WMF developers to talk about development [13:49:46] Lucas_WMDE: but that -dev chan eventually went crippled by robots spamming it constantly so folks migrated to other new channels [13:49:53] afaik it is mostly spam bot now :] [13:50:14] #wikimedia-tech is the original channel for the community and volunteers devs to talk together about the infrastructure [13:50:18] or development [13:50:28] but its usage has shifted to I believe a support channel for the community [13:50:42] ok thanks [13:50:45] (all of that is my outdated knowledge about our IRC channels, I am not in either of them) [13:51:01] hashar gerrit-review are going to test zuul as their ci :) [13:51:16] (and i think they are planning to support the new checkers plugin) [13:51:19] hashar: BoilerPlate doesn't have tests fwiw [13:51:46] paladox: ahh nice :] And I guess Zuul could well have some integration tests against various versions of Gerrit. But last time I checked the Gerrit calls are all mocked [13:52:22] hashar 2 members of the opendev infra team went to the hackathon at gerrit last week: http://lists.openstack.org/pipermail/openstack-infra/2019-September/006475.html [13:54:38] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 53.85% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [13:55:07] hashar https://bugs.chromium.org/p/gerrit/issues/detail?id=11418 is the task :) [14:11:03] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [14:15:07] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 4 others: install.php --with-extensions silently ignores extensions whose dependencies are not satisfied - https://phabricator.wikimedia.org/T225512 (10hashar) **TLDR**... [14:15:38] paladox: I am very happy to see dynamic between those groups :] [14:15:43] oh [14:15:46] two members *cough* [14:15:55] yeh [14:15:59] Monty Taylor and James Blair [14:16:27] James Blair literally wrote most of the CI stack (JJB, Zuul etc) [14:16:36] and Monty did a good share of work on that front as well [14:16:38] (among others) [14:16:44] so yeah, that is a lot [14:17:07] yeh [14:17:52] The Gerrit maintainers are interested in using Zuul to run Gerrit's [14:17:52] upstream CI system. Monty and I plan on helping to implement that. [14:18:01] yup [14:18:06] fun [14:18:12] i voted for zuul :) [14:19:57] https://bugs.chromium.org/p/gerrit/issues/list?q=label:Retrospective [14:20:11] https://groups.google.com/forum/#!topic/repo-discuss/CqFvLzs4Leg [14:23:09] I really gotta spend sometime on migrating the zuul deployment to scap [14:23:10] :-\ [14:28:47] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO: MW core doxygen / jsduck build doesn't run for tags, just branches - https://phabricator.wikimedia.org/T232055 (10hashar) [14:30:02] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO: MW core doxygen / jsduck build doesn't run for tags, just branches - https://phabricator.wikimedia.org/T232055 (10hashar) eek, note the job is explicitly made to NOT process tags. Either because I could not figure out how to trigger on tag chang... [14:32:13] hexmode: Good idea, tests would be a nice subtask for https://phabricator.wikimedia.org/T97105 [14:32:52] :) [14:32:57] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [14:33:19] awight: kostajh already has a patch [14:33:47] :-) that was fast [14:35:19] tests pass and look good to me. +2ing [14:57:49] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 53.33% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [14:59:37] 10Release-Engineering-Team-TODO (201909): Evaluate GitLab PoC CI system - https://phabricator.wikimedia.org/T229257 (10LarsWirzenius) [15:00:17] 10Release-Engineering-Team-TODO (201909): Update architecture document draft for future CI - https://phabricator.wikimedia.org/T229258 (10LarsWirzenius) [15:03:54] hashar seems to have the ability to always quit in the 5-min leading up to me needing to ask him something. [15:04:04] no matter what time of day [15:09:43] * awight And I miss you, when you're gone... :music: [16:07:29] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [16:11:01] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [16:33:24] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [16:47:07] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [16:50:20] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 4 others: install.php --with-extensions silently ignores extensions whose dependencies are not satisfied - https://phabricator.wikimedia.org/T225512 (10Anomie) @WDoranW... [17:13:09] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10Zuul, 10Documentation: Document Zuul problems caused by force merge - https://phabricator.wikimedia.org/T225955 (10hashar) [17:13:21] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909): Rebuild integration-slave-docker-* instances to use less RAM, new name and Stretch - https://phabricator.wikimedia.org/T226233 (10hashar) [17:26:43] "docker-in-docker containers inherit MTU=1500 setting from docker0 bridge" "Kubernetes clusters with the flannel CNI, and integrate with GitLab AutoDevOps to deploy some of our smaller support applications" https://github.com/coreos/flannel/issues/1171 <-- some Internet Archive issue with Kubernetes for those interested [17:47:59] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 5 others: install.php --with-extensions silently ignores extensions whose dependencies are not satisfied - https://phabricator.wikimedia.org/T225512 (10hashar) >>! In T... [17:51:33] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 5 others: install.php --with-extensions silently ignores extensions whose dependencies are not satisfied - https://phabricator.wikimedia.org/T225512 (10Anomie) Oh, we s... [17:55:23] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-jhuneidi is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 hphp_invoke - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 353 bytes in 0.040 second response time [17:56:47] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 5 others: install.php --with-extensions silently ignores extensions whose dependencies are not satisfied - https://phabricator.wikimedia.org/T225512 (10hashar) Yes sorr... [18:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [18:31:24] (03PS1) 10Holger Knust: layout: Remove unneeded test from cassandra-table-properties [integration/config] - 10https://gerrit.wikimedia.org/r/534655 [18:52:59] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10CPT Initiatives (API Integration Tests), 10Core Platform Team Workboards (Purple): Set up CI for mediawiki/tools/api-testing - https://phabricator.wikimedia.org/T230340 (10holger.knust) a:03holger.knust [19:06:51] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 42.86% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [19:08:55] (03CR) 10Jforrester: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/534655 (owner: 10Holger Knust) [19:30:47] (03CR) 10Jforrester: [C: 03+2] layout: Remove unneeded test from cassandra-table-properties [integration/config] - 10https://gerrit.wikimedia.org/r/534655 (owner: 10Holger Knust) [19:35:15] (03Merged) 10jenkins-bot: layout: Remove unneeded test from cassandra-table-properties [integration/config] - 10https://gerrit.wikimedia.org/r/534655 (owner: 10Holger Knust) [19:36:10] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [19:36:31] !log Zuul: Drop debian-glue-non-voting from cassandra-table-properties [19:36:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:02:55] (03PS1) 10Alex Paskulin: doc: Update RESTBase link [integration/docroot] - 10https://gerrit.wikimedia.org/r/534678 [20:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [20:17:41] (03CR) 10Zoranzoki21: "https://phabricator.wikimedia.org/T227766" [integration/docroot] - 10https://gerrit.wikimedia.org/r/534678 (owner: 10Alex Paskulin) [20:25:59] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 5 others: install.php --with-extensions silently ignores extensions whose dependencies are not satisfied - https://phabricator.wikimedia.org/T225512 (10Clarakosi) a:05... [20:26:58] 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO (201908), 10Code-Stewardship-Reviews, 10Graphoid, and 3 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10dr0ptp4kt) We're mulling this over still. [20:29:03] (03CR) 10Alex Paskulin: "> Patch Set 1:" [integration/docroot] - 10https://gerrit.wikimedia.org/r/534678 (owner: 10Alex Paskulin) [20:29:33] (03Abandoned) 10Alex Paskulin: doc: Update RESTBase link [integration/docroot] - 10https://gerrit.wikimedia.org/r/534678 (owner: 10Alex Paskulin) [21:13:29] 10Continuous-Integration-Config, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201909), 10Scap, and 4 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) Tagging scap as this implies changes t... [21:25:41] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.34-release: Enhance installer to support dependencies between skins and extensions - https://phabricator.wikimedia.org/T196670 (10Jdforrester-WMF) Hey there, should this be moved to 1.35? The cut is a couple of weeks away. If it needs to go out in 1.34, is... [21:30:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO: Bump CI php72 images given we've moved production to 7.2.22 - https://phabricator.wikimedia.org/T232165 (10Jdforrester-WMF) [21:46:49] 10Release-Engineering-Team-TODO, 10Wikimedia-Site-requests, 10MW-1.34-notes (1.34.0-wmf.22; 2019-09-10), 10Performance-Team (Radar): Consider enabling wgExtensionInfoMTime in wmf-production - https://phabricator.wikimedia.org/T187154 (10Jdforrester-WMF) [22:11:03] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [22:41:53] awesome to see i have been thanked for my contribution to PolyGerrit in https://bugs.chromium.org/p/gerrit/issues/detail?id=11415 ! [23:27:16] kudos paladox :) [23:27:46] :) [23:27:47] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<30.00%)