[01:17:56] (03CR) 10MaxSem: Prohibit nested functions (038 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/420231 (https://phabricator.wikimedia.org/T183756) (owner: 10MaxSem) [01:18:16] (03PS3) 10MaxSem: Prohibit nested functions [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/420231 (https://phabricator.wikimedia.org/T183756) [01:19:07] (03PS4) 10MaxSem: Prohibit nested functions [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/420231 (https://phabricator.wikimedia.org/T183756) [01:24:24] (03CR) 10jerkins-bot: [V: 04-1] Prohibit nested functions [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/420231 (https://phabricator.wikimedia.org/T183756) (owner: 10MaxSem) [01:57:53] (03PS4) 10MaxSem: Add checks for invalid annotations [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/420159 (https://phabricator.wikimedia.org/T182057) [01:59:51] (03PS5) 10MaxSem: Add checks for invalid annotations [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/420159 (https://phabricator.wikimedia.org/T182057) [02:03:33] (03CR) 10MaxSem: "*Scratches head*" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/420231 (https://phabricator.wikimedia.org/T183756) (owner: 10MaxSem) [02:24:14] PROBLEM - Free space - all mounts on deployment-mediawiki04 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<11.11%) [02:34:13] RECOVERY - Free space - all mounts on deployment-mediawiki04 is OK: OK: All targets OK [02:47:25] PROBLEM - Free space - all mounts on deployment-mediawiki05 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki05.diskspace.root.byte_percentfree (<11.11%) [02:52:24] RECOVERY - Free space - all mounts on deployment-mediawiki05 is OK: OK: All targets OK [03:05:21] PROBLEM - Puppet staleness on deployment-eventlog05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [07:04:33] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<30.00%) [07:19:33] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:07:43] RECOVERY - Puppet errors on deployment-zookeeper02 is OK: OK: Less than 1.00% above the threshold [0.0] [08:42:15] (03PS1) 10Hashar: Move Webdriver run command to quibble.test [integration/quibble] - 10https://gerrit.wikimedia.org/r/423138 [08:42:17] (03PS1) 10Hashar: Move Zuul clone helper to quibble.zuul [integration/quibble] - 10https://gerrit.wikimedia.org/r/423139 [09:40:47] hashar: could you please check what's going on at https://gerrit.wikimedia.org/r/#/c/423010/ ? -- jenkins failure makes no sense there. [09:43:22] also http://shinken.wmflabs.org/problems displays a flood of puppet downs at deployment-* [09:43:52] Hauskatze: yeah bunch of deployment-prep instances must have puppet failures [09:44:03] labs/tools/stewardbots I am looking at it [09:44:19] hashar: running sudo puppet agent -tv to debug [09:45:21] Hauskatze: did you fill a task for stewardbot ? [09:45:35] hashar: yep, on continuous-integration-config [09:45:59] T191077 [09:45:59] T191077: zuul merge cloner might be broken - https://phabricator.wikimedia.org/T191077 [09:46:02] hashar: ^^ [09:47:00] 10Continuous-Integration-Config, 10Jenkins, 10Zuul: zuul merge cloner might be broken - https://phabricator.wikimedia.org/T191077#4093827 (10hashar) [09:47:06] GitCommandError: 'git fetch --tags -v origin' returned with exit code 128 [09:47:07] stderr: 'fatal: internal server error [09:47:07] remote: internal server error [09:47:07] fatal: protocol error: bad pack header' [09:47:38] hmm [09:48:40] 10Continuous-Integration-Config, 10Jenkins, 10Zuul: zuul merge cloner might be broken - https://phabricator.wikimedia.org/T191077#4092537 (10hashar) And on #gerrit server side: ``` [2018-03-30 09:39:36,971] [SSH git-upload-pack /labs/tools/stewardbots (jenkins-bot)] ERROR com.google.gerrit.sshd.BaseCommand :... [09:50:09] 10Beta-Cluster-Infrastructure, 10Puppet: deployment-etcd-01 puppet errors - https://phabricator.wikimedia.org/T191107#4093831 (10MarcoAurelio) p:05Triage>03Normal [09:55:01] 10Continuous-Integration-Config, 10Jenkins, 10Zuul: zuul merge cloner might be broken - https://phabricator.wikimedia.org/T191077#4093854 (10hashar) On the Zuul merger: ``` zuul@contint1001:/srv/zuul/git/labs/tools/stewardbots $ git remote -v origin ssh://jenkins-bot@gerrit.wikimedia.org:29418/labs/tools/ste... [09:56:06] !log Nuking /srv/zuul/git/labs/tools/stewardbots on zuul-merger hosts (contint1001 and contint2001). Fetch fails with org.eclipse.jgit.transport.UploadPackInternalServerErrorException | TT191077 [09:56:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:56:09] !log Nuking /srv/zuul/git/labs/tools/stewardbots on zuul-merger hosts (contint1001 and contint2001). Fetch fails with org.eclipse.jgit.transport.UploadPackInternalServerErrorException | T191077 [09:56:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:56:12] T191077: zuul merge cloner might be broken - https://phabricator.wikimedia.org/T191077 [09:57:52] Hauskatze: fixed! [09:57:57] 10Continuous-Integration-Config, 10Jenkins, 10Zuul: zuul merge cloner might be broken - https://phabricator.wikimedia.org/T191077#4093863 (10hashar) 05Open>03Resolved a:03hashar I have no idea what might have been going on really :( After nuking the local git repositories on the Zuul-merger and doing... [09:58:14] Hauskatze: CI has some local git repositories to merge your patch against the tip of the branch, the end result is what is being used by Jenkins to run tests [09:58:26] the local git repo used for the merge operation must have been corrupted somehow [09:58:32] or maybe it has object that gerrit doesn't know about [09:58:36] or whatever strange madness [09:58:41] anyway, deleting the local repo fixed it [09:58:47] thanks hashar ! [09:58:53] was a weird error [09:59:19] +2 and merging [09:59:28] yeah definitely an infra one [10:00:05] (Merged) jenkins-bot: Reinstate "build: Updating mediawiki/mediawiki-codesniffer to 17.0.0" [labs/tools/stewardbots] - https://gerrit.wikimedia.org/r/423010 (owner: MarcoAurelio) [10:00:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Zuul: zuul merge cloner might be broken - https://phabricator.wikimedia.org/T191077#4093866 (10hashar) [10:01:11] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Zuul: zuul merge cloner might be broken - https://phabricator.wikimedia.org/T191077#4093868 (10MarcoAurelio) Thank you! [10:03:39] 10Beta-Cluster-Infrastructure, 10Puppet: deployment-eventlog05 puppet errors - https://phabricator.wikimedia.org/T191109#4093870 (10MarcoAurelio) [10:04:49] RECOVERY - Puppet staleness on deployment-eventlog05 is OK: OK: Less than 1.00% above the threshold [3600.0] [10:05:29] 10Beta-Cluster-Infrastructure, 10Puppet: deployment-eventlog05 puppet errors - https://phabricator.wikimedia.org/T191109#4093881 (10MarcoAurelio) ``` maurelio@deployment-eventlog05:~$ sudo puppet agent -tv Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info... [10:09:48] for eventlog05 I fixed some, but remains a critical error there [10:09:56] puppet is a headache isn't it? [10:13:40] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet: Puppet broken on deployment-mira - https://phabricator.wikimedia.org/T191110#4093928 (10MarcoAurelio) [10:33:31] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: Update page object pattern in Selenium tests - https://phabricator.wikimedia.org/T185094#4094179 (10zeljkofilipin) Thanks a lot @Krinkle, things are more clear to me now. I like our current ES6 syntax. I... [10:45:36] 10Phabricator, 10Discourse, 10Developer-Relations (Jan-Mar-2018): Enable Wikimedia Phabricator login in discourse-mediawiki.wmflabs.org - https://phabricator.wikimedia.org/T184987#4094236 (10yana_agun) @Tgr I don't think I do have any access to Wikimedia org related to this project. Anyway, I added you as... [10:54:57] (03CR) 10Zoranzoki21: [C: 04-1] Add mediawiki/extension/SecureAuth in zuul (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/422960 (owner: 10Jayprakash12345) [10:58:37] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin, 10Wikimedia-log-errors (Jenkins Failure): Warning: Task "stylelint:src" failed due to postcss-less@1.1.4 - https://phabricator.wikimedia.org/T190269#4094239 (10ze... [11:08:00] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin, 10Wikimedia-log-errors (Jenkins Failure): Warning: Task "stylelint:src" failed due to postcss-less@1.1.4 - https://phabricator.wikimedia.org/T190269#4094252 (10ze... [11:38:35] (03PS1) 10Hashar: Drop version from setup.py [integration/quibble] - 10https://gerrit.wikimedia.org/r/423147 [11:38:37] (03PS1) 10Hashar: Embed zuul-cloner [integration/quibble] - 10https://gerrit.wikimedia.org/r/423148 [11:38:38] !log deployment-prep reindexing with forceSearchIndex all beta wikis (T189694) [11:38:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:38:41] T189694: forceSearchIndex on testwiki, mediawikiwiki, labswiki, labtestwiki, and svwiki. And everything on Beta Cluster - https://phabricator.wikimedia.org/T189694 [11:39:09] (03CR) 10jerkins-bot: [V: 04-1] Embed zuul-cloner [integration/quibble] - 10https://gerrit.wikimedia.org/r/423148 (owner: 10Hashar) [11:46:50] (03PS2) 10Hashar: Move Zuul clone helper to quibble.zuul [integration/quibble] - 10https://gerrit.wikimedia.org/r/423139 [11:46:52] (03PS2) 10Hashar: Drop version from setup.py [integration/quibble] - 10https://gerrit.wikimedia.org/r/423147 [11:46:54] (03PS2) 10Hashar: Embed zuul-cloner [integration/quibble] - 10https://gerrit.wikimedia.org/r/423148 [11:47:20] (03CR) 10jerkins-bot: [V: 04-1] Embed zuul-cloner [integration/quibble] - 10https://gerrit.wikimedia.org/r/423148 (owner: 10Hashar) [11:51:57] (03PS3) 10Hashar: Embed zuul-cloner [integration/quibble] - 10https://gerrit.wikimedia.org/r/423148 [11:54:13] (03CR) 10Hashar: [C: 032] Move Webdriver run command to quibble.test [integration/quibble] - 10https://gerrit.wikimedia.org/r/423138 (owner: 10Hashar) [11:54:24] (03CR) 10Hashar: [C: 032] Move Zuul clone helper to quibble.zuul [integration/quibble] - 10https://gerrit.wikimedia.org/r/423139 (owner: 10Hashar) [11:54:27] (03CR) 10Hashar: [C: 032] Drop version from setup.py [integration/quibble] - 10https://gerrit.wikimedia.org/r/423147 (owner: 10Hashar) [11:54:40] (03Merged) 10jenkins-bot: Move Webdriver run command to quibble.test [integration/quibble] - 10https://gerrit.wikimedia.org/r/423138 (owner: 10Hashar) [11:54:48] (03Merged) 10jenkins-bot: Move Zuul clone helper to quibble.zuul [integration/quibble] - 10https://gerrit.wikimedia.org/r/423139 (owner: 10Hashar) [11:54:50] (03Merged) 10jenkins-bot: Drop version from setup.py [integration/quibble] - 10https://gerrit.wikimedia.org/r/423147 (owner: 10Hashar) [12:11:51] PROBLEM - Free space - all mounts on deployment-ores01 is CRITICAL: CRITICAL: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (No valid datapoints found)deployment-prep.deployment-ores01.diskspace.root.byte_percentfree (<100.00%) [12:15:21] (03PS4) 10Hashar: Embed zuul-cloner [integration/quibble] - 10https://gerrit.wikimedia.org/r/423148 [12:16:01] (03CR) 10Hashar: [C: 032] "Forgot zuul/lib/__init__.py zuul/merger/__init__.py" [integration/quibble] - 10https://gerrit.wikimedia.org/r/423148 (owner: 10Hashar) [12:16:30] (03Merged) 10jenkins-bot: Embed zuul-cloner [integration/quibble] - 10https://gerrit.wikimedia.org/r/423148 (owner: 10Hashar) [12:22:35] 10Beta-Cluster-Infrastructure: Create mediawiki::maintenance server (aka terbium) in deployment-prep - https://phabricator.wikimedia.org/T187826#4094358 (10MarcoAurelio) [12:24:52] (03PS1) 10Hashar: Add color to logging [integration/quibble] - 10https://gerrit.wikimedia.org/r/423152 [12:25:40] (03CR) 10Hashar: [C: 032] Add color to logging [integration/quibble] - 10https://gerrit.wikimedia.org/r/423152 (owner: 10Hashar) [12:26:06] (03Merged) 10jenkins-bot: Add color to logging [integration/quibble] - 10https://gerrit.wikimedia.org/r/423152 (owner: 10Hashar) [12:28:26] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet: Puppet broken on deployment-mira - https://phabricator.wikimedia.org/T191110#4094366 (10MarcoAurelio) ``` maurelio@deployment-mira:/etc/puppet$ cd modules -bash: cd: modules: No such file or directory ``` It makes sense therefore that puppet can't find t... [12:38:41] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:44:39] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#4094387 (10zeljkofilipin) Created testing job: [[ https://integration.wikimedia.org/ci/view/Selenium/job/mediawiki-core-qunit-selenium-jessi... [12:54:27] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:36:33] PROBLEM - Puppet errors on deployment-eventlog05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:46:44] (03PS3) 10Jayprakash12345: Add mediawiki/extensions/SecureAuth in zuul [integration/config] - 10https://gerrit.wikimedia.org/r/422960 [14:29:33] (03CR) 10Hashar: [C: 032] Add mediawiki/extensions/SecureAuth in zuul [integration/config] - 10https://gerrit.wikimedia.org/r/422960 (owner: 10Jayprakash12345) [14:30:44] (03Merged) 10jenkins-bot: Add mediawiki/extensions/SecureAuth in zuul [integration/config] - 10https://gerrit.wikimedia.org/r/422960 (owner: 10Jayprakash12345) [15:06:41] (03PS1) 10Hashar: Run extensions/skins 'composer test' [integration/quibble] - 10https://gerrit.wikimedia.org/r/423162 [15:27:45] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:33:43] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:36:02] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet: Puppet broken on deployment-mira - https://phabricator.wikimedia.org/T191110#4094555 (10Dzahn) @MarcoAurelio This looks like it's about data missing in Hiera. In production we have: hieradata/role/common/deployment_server.yaml:profile::kubernetes::depl... [15:52:07] (03PS2) 10Hashar: Run extensions/skins 'composer test' [integration/quibble] - 10https://gerrit.wikimedia.org/r/423162 [15:52:31] (03CR) 10jerkins-bot: [V: 04-1] Run extensions/skins 'composer test' [integration/quibble] - 10https://gerrit.wikimedia.org/r/423162 (owner: 10Hashar) [15:55:29] (03PS3) 10Hashar: Run extensions/skins 'composer test' [integration/quibble] - 10https://gerrit.wikimedia.org/r/423162 [15:55:31] (03PS1) 10Hashar: log: default: INFO, quibble: DEBUG [integration/quibble] - 10https://gerrit.wikimedia.org/r/423164 [15:55:55] (03CR) 10jerkins-bot: [V: 04-1] Run extensions/skins 'composer test' [integration/quibble] - 10https://gerrit.wikimedia.org/r/423162 (owner: 10Hashar) [16:10:35] (03PS4) 10Hashar: Run extensions/skins 'composer test' [integration/quibble] - 10https://gerrit.wikimedia.org/r/423162 [16:12:25] (03CR) 10Hashar: [C: 032] log: default: INFO, quibble: DEBUG [integration/quibble] - 10https://gerrit.wikimedia.org/r/423164 (owner: 10Hashar) [16:12:29] (03CR) 10Hashar: [C: 032] Run extensions/skins 'composer test' [integration/quibble] - 10https://gerrit.wikimedia.org/r/423162 (owner: 10Hashar) [16:12:51] (03Merged) 10jenkins-bot: log: default: INFO, quibble: DEBUG [integration/quibble] - 10https://gerrit.wikimedia.org/r/423164 (owner: 10Hashar) [16:12:55] (03Merged) 10jenkins-bot: Run extensions/skins 'composer test' [integration/quibble] - 10https://gerrit.wikimedia.org/r/423162 (owner: 10Hashar) [17:15:35] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<50.00%) [17:19:51] @seen Hauskatze [18:39:23] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:08:58] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:41:00] no_justification https://gerrit-review.googlesource.com/c/gerrit/+/169330 [19:41:04] yay first step [19:48:58] RECOVERY - Puppet errors on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:05:06] no_justification guess what! [20:05:36] you can test test what master will look like against gerrit.wikimedia.org using there script polygerrit-ui/run-server.sh they use for gerrit-review [20:08:40] well you carn't view the actual changes due to the rest api change in 2.15 [20:08:44] but the admin page works [20:16:27] no_justification this is what mediawiki/core looks like under admin [20:16:28] https://phabricator.wikimedia.org/F16483213 [20:17:56] oh i found dashboard does not use base url [20:17:57] sigh [20:18:00] * paladox fixes it :) [20:24:50] oh i found bugs now heh [20:25:00] the trunication of projects looks ugly [20:25:26] see https://phabricator.wikimedia.org/F16483256 [20:28:14] 10Beta-Cluster-Infrastructure, 10RelEng-Archive-FY201718-Q1, 10media-storage, 10Patch-For-Review: deployment-ms-be03.deployment-prep and deployment-ms-be04.deployment-prep have high load / system CPU - https://phabricator.wikimedia.org/T160990#4094803 (10hashar) [20:28:16] 10Beta-Cluster-Infrastructure, 10Operations, 10media-storage, 10Patch-For-Review: nscd does not cache localhost causing high CPU usage when localhost is often resolved - https://phabricator.wikimedia.org/T171745#4094801 (10hashar) 05Open>03declined No time to look into it, so lets archive this task. [20:31:13] (03Abandoned) 10Hashar: HUGE WIP [integration/quibble] - 10https://gerrit.wikimedia.org/r/354997 (owner: 10Hashar) [20:39:01] PROBLEM - Puppet errors on deployment-mediawiki07 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:14:12] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Puppet: Puppet broken on deployment-mira - https://phabricator.wikimedia.org/T191110#4094818 (10MarcoAurelio) Puppet still failing: ``` maurelio@deployment-tin:~$ sudo puppet agent -tv Info: Using configured environment 'future' Info: Retrieving p... [21:56:02] (03PS2) 10Hashar: Experimental Quibble job [integration/config] - 10https://gerrit.wikimedia.org/r/423026 [22:00:00] PROBLEM - Free space - all mounts on integration-slave-docker-1004 is CRITICAL: CRITICAL: integration.integration-slave-docker-1004.diskspace.root.byte_percentfree (<40.00%) [22:10:50] PROBLEM - Puppet staleness on deployment-eventlog05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [43200.0] [22:11:56] 10Phabricator: Integrate login with SUL - https://phabricator.wikimedia.org/T179124#4094867 (10Aklapper) 05Open>03declined >>! In T179124#3714009, @Kerry_Raymond wrote: > At the very least, when you get the error message about the user name, tell the user that what they have to do is replace all the illegal... [22:14:56] RECOVERY - Free space - all mounts on integration-slave-docker-1004 is OK: OK: All targets OK [23:01:45] woo i finally figured out how to fix groups [23:01:47] https://gerrit-review.googlesource.com/c/gerrit/+/168955 [23:02:12] that's a medium size change [23:02:19] but looks so much better [23:02:35] well only that it links properly on the audit page