[00:00:16] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4113662 (10Legoktm) At this point maybe we should file separate tickets for each ext... [00:00:47] though https://gerrit.git.wmflabs.org/r/c/3/ is fast for me [00:04:10] 10Beta-Cluster-Infrastructure, 10Discovery, 10Wikimedia-Portals, 10Discovery-Portal-Sprint, and 2 others: Wikimedia.org portal broken in Beta Cluster (Domain unavailable) - https://phabricator.wikimedia.org/T173887#4113664 (10EddieGP) >>! In T173887#4113631, @gerritbot wrote: > Change 424361 **merged** by... [00:27:17] 10Release-Engineering-Team (Watching / External), 10Operations, 10puppet-compiler: Integrate the puppet compiler in the puppet CI pipeline - https://phabricator.wikimedia.org/T166066#4113673 (10EddieGP) [00:36:13] 10Continuous-Integration-Infrastructure: cumin not working in integration cloud project: "Default backend 'openstack' is not registered" - https://phabricator.wikimedia.org/T191680#4113678 (10Legoktm) [00:36:49] Project mwext-phpunit-coverage-publish build #3073: 04FAILURE in 3.6 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3073/ [00:36:52] Project mwext-phpunit-coverage-publish build #3074: 04STILL FAILING in 2.2 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3074/ [00:36:54] Project mwext-phpunit-coverage-publish build #3075: 04STILL FAILING in 1.6 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3075/ [00:38:20] Yippee, build fixed! [00:38:20] Project mwext-phpunit-coverage-publish build #3076: 09FIXED in 1 min 24 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3076/ [00:42:59] 10Continuous-Integration-Config, 10AbuseFilter, 10Upstream: stylelint is just outputting dots and number of errors, making it impossible to fix - https://phabricator.wikimedia.org/T190072#4061790 (10Umherirrender) Seems not to help on GettingStarted: https://integration.wikimedia.org/ci/job/mwgate-npm-node-6... [01:11:26] 10Phabricator, 10Wikimedia-Planet: missing html links for Phabricator Blog feeds - https://phabricator.wikimedia.org/T191683#4113733 (10Dzahn) [03:37:14] Project mediawiki-core-code-coverage-php7 build #191: 04STILL FAILING in 37 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/191/ [04:27:37] Project mediawiki-core-code-coverage build #3430: 04STILL FAILING in 1 hr 27 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3430/ [05:55:36] 10Phabricator (Upstream), 10Wikimedia-Planet, 10Upstream: missing html links for Phabricator Blog feeds - https://phabricator.wikimedia.org/T191683#4113816 (10Peachey88) [06:35:45] (03PS1) 10Legoktm: Add common replacements for invalid license tag sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/424737 [06:36:50] (03PS2) 10Legoktm: Add common autofix replacements for invalid license tag sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/424737 [06:42:09] (03CR) 10jerkins-bot: [V: 04-1] Add common autofix replacements for invalid license tag sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/424737 (owner: 10Legoktm) [06:44:13] 10Continuous-Integration-Infrastructure, 10MediaWiki-Codesniffer, 10Test-Coverage: Post-merge build failed for mediawiki/tools/codesniffer - https://phabricator.wikimedia.org/T191637#4113832 (10Legoktm) ``` 6edf7d939f499137e40aab1aca2739ce5ce4d805 is the first bad commit commit 6edf7d939f499137e40aab1aca2739... [07:13:32] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:26:15] 10Continuous-Integration-Infrastructure, 10MediaWiki-Codesniffer, 10Test-Coverage: Post-merge build failed for mediawiki/tools/codesniffer - https://phabricator.wikimedia.org/T191637#4113849 (10Legoktm) `CodeCoverage::initializeData()` `include_once` every sniff file to "...initialize the data before we star... [07:27:09] (03PS3) 10Legoktm: Add common autofix replacements for invalid license tag sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/424737 [07:47:11] 10Continuous-Integration-Infrastructure: cumin not working in integration cloud project: "Default backend 'openstack' is not registered" - https://phabricator.wikimedia.org/T191680#4113860 (10hashar) [07:47:13] 10Continuous-Integration-Infrastructure, 10Operations-Software-Development, 10Patch-For-Review: cumin 3.0.1-1 is broken on labs master - https://phabricator.wikimedia.org/T188112#4113858 (10hashar) [07:50:34] 10Continuous-Integration-Infrastructure, 10Operations-Software-Development, 10Patch-For-Review: cumin 3.0.1-1 is broken on labs master - https://phabricator.wikimedia.org/T188112#4113861 (10hashar) **Left over note from March 13th** > Caught InvalidQueryError exception: Default backend 'openstack' is not re... [08:47:57] (03CR) 10Umherirrender: Add common autofix replacements for invalid license tag sniff (035 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/424737 (owner: 10Legoktm) [12:38:41] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:54:28] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:36:32] PROBLEM - Puppet errors on deployment-eventlog05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:43:53] when you'll quit failing beta-cluster puppet? :) [13:56:40] ou know shinken will regularly post here if the alert does not go away, right? So on the plus side, there's no new failures ;-) [13:57:58] Hauskatze: when it becomes sentiale and fixes itself [13:58:00] spamming people until fixed? sounds good to me [13:59:03] The result of that is people stop caring about the errors. Still sounds good? ;-) [13:59:20] not really [13:59:37] Elder wand to fix puppet please [14:00:24] One could probably stop that by acknowledging the alerts which are known to be okay to ignore for now in shinken, but to do that I'd need more access to shinken than guest/guest provides. [14:01:42] Yep. Who can create an account for us? [14:02:41] E.g. about 4/14 alerts are hosts failing because they still run trusty. Fixing that needs replacing the instance. Obvs people don't always have time for that. [14:03:22] That'd probably be any of the maintainers of the shinken project: https://tools.wmflabs.org/openstack-browser/project/shinken [14:13:15] (03CR) 10Zfilipin: [C: 04-1] "Argh. Not good. If wdio fails, the job still passes. :|" [integration/config] - 10https://gerrit.wikimedia.org/r/424592 (https://phabricator.wikimedia.org/T179190) (owner: 10Zfilipin) [14:38:12] (03CR) 10Zfilipin: [C: 04-1] "The problem should be fixed by https://gerrit.wikimedia.org/r/#/c/424764/" [integration/config] - 10https://gerrit.wikimedia.org/r/424592 (https://phabricator.wikimedia.org/T179190) (owner: 10Zfilipin) [14:39:45] (03CR) 10Zfilipin: Run `npm run selenium` instead of `grunt webdriver:test` [integration/config] - 10https://gerrit.wikimedia.org/r/424592 (https://phabricator.wikimedia.org/T179190) (owner: 10Zfilipin) [14:40:14] (03PS4) 10Zfilipin: Run `npm run selenium` instead of `grunt webdriver:test` [integration/config] - 10https://gerrit.wikimedia.org/r/424592 (https://phabricator.wikimedia.org/T179190) [15:03:31] (03CR) 10Zfilipin: "Tested with a commit that has broken Selenium tests https://gerrit.wikimedia.org/r/#/c/424763/4" [integration/config] - 10https://gerrit.wikimedia.org/r/424592 (https://phabricator.wikimedia.org/T179190) (owner: 10Zfilipin) [15:16:58] 10Beta-Cluster-Infrastructure, 10Puppet: deployment-secureredirexperiment puppet error - https://phabricator.wikimedia.org/T191663#4114240 (10MarcoAurelio) [15:27:45] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:33:42] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:36:29] eddiegp: so the "labs-puppetmaster/Labs Puppetmaster HTTPS is UNKNOWN since 4M 3w 4d 20h 32m 40s" is because the server tries to fetch the https status and finds none, right? [15:36:53] looks like we have to point him to the right place to check? [15:37:02] Project mediawiki-core-code-coverage-php7 build #192: 04STILL FAILING in 37 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/192/ [15:43:39] eddie@eddie-thinkpad:~$ curl -I http://labs-puppetmaster.wikimedia.org:8140/ [15:43:41] HTTP/1.1 400 Bad Request [15:43:48] eddie@eddie-thinkpad:~$ curl -I https://labs-puppetmaster.wikimedia.org:8140/ [15:43:50] curl: (60) SSL certificate problem: self signed certificate in certificate chain [15:45:10] Hauskatze: It does check the right place. The server replies with the right status code (the one expected by the check) over http, so I suspect it'd do so over https as well if an TLS session could be established. [15:46:42] UNKNOWN basically means "something failed, so the check couldn't even be performed correctly". My guess it that this "something" is the TLS session failing over a self signed cert [15:48:09] so some lets encrypt thing? [15:51:12] Rather not, self-signed cert doesn't sound like something that would magically pop up when letsencrypt fails. [15:52:15] Tbh I have no clue what this check or the port it's connecting to is even for. That's why I linked in WMCS on the ticket. [16:25:14] Project mediawiki-core-code-coverage build #3431: 04STILL FAILING in 1 hr 25 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3431/ [18:39:22] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:58:41] (03PS4) 10Legoktm: Add common autofix replacements for invalid license tag sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/424737 [21:16:43] Project mwext-phpunit-coverage-publish build #3101: 04FAILURE in 2 min 8 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3101/ [21:18:01] Yippee, build fixed! [21:18:02] Project mwext-phpunit-coverage-publish build #3102: 09FIXED in 1 min 18 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3102/