[01:18:40] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<11.11%) [03:00:15] Project mediawiki-core-code-coverage-php7 build #166: 04STILL FAILING in 15 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/166/ [03:00:17] Project mediawiki-core-code-coverage build #3405: 04STILL FAILING in 16 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3405/ [03:05:20] PROBLEM - Puppet staleness on deployment-eventlog05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [03:35:17] !log Marked integration-slave-docker-1003 offline because / is full. Needs cleanup. (cc hashar, thcipriani) [03:35:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:40:30] I will let you know when I see hashar and I will deliver that message to them [03:40:30] @notify hashar /var/lib/docker is 16G on integration-slave-docker-1003 . Needs cleanup and probably moving to /srv [04:22:53] (03PS19) 10Zoranzoki21: Add few extensions in zuul/layout.yaml to Jenkins can run builds and remove mediawiki/extensions/Collection/OfflineContentGenerator/node_modules [integration/config] - 10https://gerrit.wikimedia.org/r/406524 (https://phabricator.wikimedia.org/T183674) [04:24:03] (03CR) 10jerkins-bot: [V: 04-1] Add few extensions in zuul/layout.yaml to Jenkins can run builds and remove mediawiki/extensions/Collection/OfflineContentGenerator/node_modules [integration/config] - 10https://gerrit.wikimedia.org/r/406524 (https://phabricator.wikimedia.org/T183674) (owner: 10Zoranzoki21) [04:36:29] (03CR) 10Zoranzoki21: "I am small unsure what I should to do, so Jenkins can give +2 vote." [integration/config] - 10https://gerrit.wikimedia.org/r/406524 (https://phabricator.wikimedia.org/T183674) (owner: 10Zoranzoki21) [05:44:11] 10Phabricator (Upstream), 10Upstream: Can't preview new titles and/or changes to task titles on the Phabricator - https://phabricator.wikimedia.org/T190610#4079680 (10gh87) I've never written a [[https://en.wikipedia.org/wiki/Use_case|use case]] before, honestly. However, here goes my first use case based on a... [07:03:40] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:43:58] (03CR) 10Hashar: [C: 032] "good enough for now" [integration/quibble] - 10https://gerrit.wikimedia.org/r/421548 (owner: 10Hashar) [08:44:23] (03Merged) 10jenkins-bot: Add webhost and qunit support [integration/quibble] - 10https://gerrit.wikimedia.org/r/421548 (owner: 10Hashar) [09:18:23] (03PS2) 10Hashar: Allow skipping zuul and npm/composer [integration/quibble] - 10https://gerrit.wikimedia.org/r/421600 [09:20:58] (03CR) 10Hashar: [C: 032] Allow skipping zuul and npm/composer [integration/quibble] - 10https://gerrit.wikimedia.org/r/421600 (owner: 10Hashar) [09:21:22] (03Merged) 10jenkins-bot: Allow skipping zuul and npm/composer [integration/quibble] - 10https://gerrit.wikimedia.org/r/421600 (owner: 10Hashar) [09:29:15] (03PS1) 10Thiemo Kreuz (WMDE): Optimize DirUsage sniff for performance [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/421845 [09:32:16] PROBLEM - Puppet staleness on deployment-maps01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [10:12:18] (03PS4) 10Hashar: Port mw-phpunit.sh to python [integration/quibble] - 10https://gerrit.wikimedia.org/r/421592 [10:12:20] (03PS1) 10Hashar: Add chromedriver and webdriver:test support [integration/quibble] - 10https://gerrit.wikimedia.org/r/421852 [11:02:58] (03PS1) 10KartikMistry: Add apertium-separable package [integration/config] - 10https://gerrit.wikimedia.org/r/421861 (https://phabricator.wikimedia.org/T189075) [11:04:48] hashar: ^ [11:06:52] kart_: easy :] [11:06:59] (03CR) 10Hashar: [C: 032] Add apertium-separable package [integration/config] - 10https://gerrit.wikimedia.org/r/421861 (https://phabricator.wikimedia.org/T189075) (owner: 10KartikMistry) [11:08:12] (03Merged) 10jenkins-bot: Add apertium-separable package [integration/config] - 10https://gerrit.wikimedia.org/r/421861 (https://phabricator.wikimedia.org/T189075) (owner: 10KartikMistry) [11:08:44] kart_: done! [11:09:25] (03PS5) 10Hashar: Port mw-phpunit.sh to python [integration/quibble] - 10https://gerrit.wikimedia.org/r/421592 [11:12:02] (03CR) 10Hashar: [C: 032] "And they pass with mysql !" [integration/quibble] - 10https://gerrit.wikimedia.org/r/421592 (owner: 10Hashar) [11:12:28] (03Merged) 10jenkins-bot: Port mw-phpunit.sh to python [integration/quibble] - 10https://gerrit.wikimedia.org/r/421592 (owner: 10Hashar) [11:16:52] hashar: thanks! [11:22:50] (03PS1) 10Hashar: Split PHPUnit Database tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/421866 (https://phabricator.wikimedia.org/T190554) [11:24:42] 10Project-Admins, 10Africa-Wikimedia-Developers: Project work board request for WikiFundi - https://phabricator.wikimedia.org/T186754#4080161 (10Anthere) Hi Aklapper. Sorry I was mostly off. WikiIndaba took some time of me, followed by French strikes which got me stuck away from home. I need to refocus and wil... [11:25:22] (03PS2) 10Hashar: Add chromedriver and webdriver:test support [integration/quibble] - 10https://gerrit.wikimedia.org/r/421852 [11:32:24] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<100.00%) [11:45:54] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Lexicographical data, 10Wikidata, and 3 others: CirrusSearch smoke selenium tests cause failures of mediawiki-core-qunit-selenium-jessie job for extensions - https://phabricator.wikimedia.org/T190039#4080225 (10zeljkofilipin) m... [11:46:28] 10Continuous-Integration-Infrastructure, 10Front-end-Standards-Group, 10MediaWiki-extensions-General, 10Services (designing): Decide whether we want the package-lock.json to commit or ignore - https://phabricator.wikimedia.org/T179229#4080228 (10Jhernandez) > Exactly. Using locked deps on each level quickl... [11:56:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Lexicographical data, 10Wikidata, and 3 others: CirrusSearch smoke selenium tests cause failures of mediawiki-core-qunit-selenium-jessie job for extensions - https://phabricator.wikimedia.org/T190039#4080254 (10zeljkofilipin) F... [11:58:08] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin: Replace nodemw with mwbot - https://phabricator.wikimedia.org/T181284#4080265 (10zeljkofilipin) [11:58:10] 10Release-Engineering-Team (Kanban), 10Advanced-Search, 10TCB-Team, 10Patch-For-Review, 10User-zeljkofilipin: Cannot find module nodemw - https://phabricator.wikimedia.org/T190307#4080263 (10zeljkofilipin) 05Open>03Invalid Nothing to do in this task, this will be fixed with {T190426}. [11:58:58] 10Release-Engineering-Team, 10Epic, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review, 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#4080271 (10zeljkofilipin) [11:59:00] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin: Replace nodemw with mwbot - https://phabricator.wikimedia.org/T181284#3785137 (10zeljkofilipin) 05Open>03Resolved This is resolved. F... [12:05:51] RECOVERY - Free space - all mounts on deployment-ores01 is OK: OK: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (No valid datapoints found) deployment-prep.deployment-ores01.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) [12:09:19] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Document differences between Ruby and Node.js Selenium frameworks - https://phabricator.wikimedia.org/T182692#4080289 (10zeljkofilipin) p:05High>03Normal [12:09:30] RECOVERY - Free space - all mounts on integration-slave-docker-1003 is OK: OK: integration.integration-slave-docker-1003.diskspace._srv.byte_percentfree (More than half of the datapoints are undefined) integration.integration-slave-docker-1003.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) [12:11:51] PROBLEM - Free space - all mounts on deployment-ores01 is CRITICAL: CRITICAL: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (No valid datapoints found)deployment-prep.deployment-ores01.diskspace.root.byte_percentfree (<100.00%) [12:15:30] PROBLEM - Free space - all mounts on integration-slave-docker-1003 is CRITICAL: CRITICAL: integration.integration-slave-docker-1003.diskspace.root.byte_percentfree (<100.00%) [12:37:49] (03PS3) 10Hashar: Add chromedriver and webdriver:test support [integration/quibble] - 10https://gerrit.wikimedia.org/r/421852 [12:38:39] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:40:38] (03PS1) 10Hashar: Common class for backend servers [integration/quibble] - 10https://gerrit.wikimedia.org/r/421877 [12:41:32] (03PS4) 10Hashar: Add chromedriver and webdriver:test support [integration/quibble] - 10https://gerrit.wikimedia.org/r/421852 [12:45:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Zuul: Zuul no longer starts gate-and-submit when parent is in gate-and-submit pipeline - https://phabricator.wikimedia.org/T186949#4080375 (10Lucas_Werkmeister_WMDE) This seems to be working now, e. g. on https://gerrit.wikimedia... [12:47:41] 10Gerrit, 10Operations: git pull fails for MW core filas with "fatal: protocol error: bad pack header" - https://phabricator.wikimedia.org/T190676#4080384 (10daniel) [12:48:40] 10Gerrit, 10Operations: git pull fails for MW core filas with "fatal: protocol error: bad pack header" - https://phabricator.wikimedia.org/T190676#4080395 (10daniel) p:05Triage>03High Bumping to high. May even be UBN. I'm trying a fresh clone now. [12:50:44] 10Gerrit, 10Operations: git pull fails for MW core filas with "fatal: protocol error: bad pack header" - https://phabricator.wikimedia.org/T190676#4080408 (10daniel) Ftr: [14:49] worked for me [14:49] https://www.irccloud.com/pastebin/U3V1obs5/ [12:53:55] 10Gerrit, 10Operations: git pull fails for MW core filas with "fatal: protocol error: bad pack header" - https://phabricator.wikimedia.org/T190676#4080422 (10daniel) A fresh clone seems to fix the problem. So no UBN. [12:54:25] 10Gerrit, 10Operations: git pull fails for MW core filas with "fatal: protocol error: bad pack header" - https://phabricator.wikimedia.org/T190676#4080423 (10daniel) [12:54:28] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:58:34] mutante: ^ [12:58:37] 10Gerrit, 10Operations: git pull fails for MW core filas with "fatal: protocol error: bad pack header" - https://phabricator.wikimedia.org/T190676#4080384 (10Paladox) Hi, please run "git remote prune origin" [12:59:47] 10Gerrit, 10Operations: git pull fails for MW core filas with "fatal: protocol error: bad pack header" - https://phabricator.wikimedia.org/T190676#4080478 (10Paladox) 05Open>03Resolved As long as you run "git remote prune origin" that should resolve the problem, please reopen if it dosen't. [13:00:53] The last Puppet run was at Thu Mar 22 12:18:13 UTC 2018 (5802 minutes ago). <-- I'll try to re-run it [13:01:44] 10Gerrit, 10Operations: git pull fails for MW core filas with "fatal: protocol error: bad pack header" - https://phabricator.wikimedia.org/T190676#4080384 (10hashar) The reason is you have local branches pointing to remote branches that no more exist. Specially, the wmf/ branches are deleted after some weeks a... [13:03:48] 10Gerrit, 10Operations: git pull fails for MW core filas with "fatal: protocol error: bad pack header" - https://phabricator.wikimedia.org/T190676#4080495 (10daniel) Ah, thank you! Is this somehow a new thing? I wonder why I'm running into this for the first time now. [13:05:03] 10Gerrit, 10Operations: git pull fails for MW core filas with "fatal: protocol error: bad pack header" - https://phabricator.wikimedia.org/T190676#4080497 (10daniel) I feel like filing an upstream bug. "fatal: protocol error: bad pack header" is NOT a good way to say "the branch you are tracking no longer exis... [13:05:45] 10Gerrit, 10Operations: git pull fails for MW core filas with "fatal: protocol error: bad pack header" - https://phabricator.wikimedia.org/T190676#4080498 (10Paladox) >>! In T190676#4080495, @daniel wrote: > Ah, thank you! > Is this somehow a new thing? I wonder why I'm running into this for the first time no... [13:06:57] 10Scap, 10Language-Team, 10MediaWiki-extensions-LocalisationUpdate, 10User-Nikerabbit: Alert when l10update fails - https://phabricator.wikimedia.org/T171925#4080500 (10Arrbee) [13:07:12] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-ms-be03 - https://phabricator.wikimedia.org/T190683#4080501 (10MarcoAurelio) [13:08:43] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-ms-be03 - https://phabricator.wikimedia.org/T190683#4080501 (10Paladox) You should run puppet to find out what the error is please? [13:09:02] paladox: I can, let me know what [13:09:13] Hauskatze sudo puppet agent -tv [13:09:18] should tell you the error :) [13:09:33] I did that already and started creating weird things on home/maurelio/.puppet [13:09:45] I guess I should find the right folder [13:10:05] I'm in the middle of swat, will you still be around later? [13:10:16] I may be around later yes [13:10:33] though i wont know how to fix this, if it's swift errors :) [13:10:37] I guess I can take a look now though [13:12:24] RECOVERY - Free space - all mounts on integration-slave-jessie-1001 is OK: OK: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found) [13:15:08] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-ms-be03 - https://phabricator.wikimedia.org/T190683#4080550 (10MarcoAurelio) I ran puppet agent -tv but I guess I did that on the wrong folder as it started to create stuff on ~/maurelio/.puppet -- I shall find the right place to run this. [13:15:26] (03CR) 10Hashar: [C: 032] Common class for backend servers [integration/quibble] - 10https://gerrit.wikimedia.org/r/421877 (owner: 10Hashar) [13:15:39] (03CR) 10Hashar: [C: 032] "It is magic :]" [integration/quibble] - 10https://gerrit.wikimedia.org/r/421852 (owner: 10Hashar) [13:15:52] (03Merged) 10jenkins-bot: Common class for backend servers [integration/quibble] - 10https://gerrit.wikimedia.org/r/421877 (owner: 10Hashar) [13:16:00] Hauskatze did you run it with sudo? [13:16:04] (03Merged) 10jenkins-bot: Add chromedriver and webdriver:test support [integration/quibble] - 10https://gerrit.wikimedia.org/r/421852 (owner: 10Hashar) [13:16:09] paladox: would that make a difference? [13:16:12] yeh [13:16:16] nope, just "puppet agent -tv" [13:16:24] I'll try with 'sudo' [13:16:39] you have to run it with sudo if you are running it as non root ie with a user account [13:17:24] would that break anything? [13:17:30] I think -t is for testing? [13:17:59] well puppet agent -tv runs puppet so it applys [13:18:27] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<40.00%) [13:19:55] _joe_: got a minute for a puppet question? [13:21:26] <_joe_> Hauskatze: ask, I'll answer as soon as I can :) [13:21:59] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-ms-be03 - https://phabricator.wikimedia.org/T190683#4080558 (10MarcoAurelio) ``` maurelio@deployment-ms-be03:~$ sudo puppet agent -tv Info: Using configured environment 'future' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loadi... [13:22:26] _joe_ / paladox --> https://phabricator.wikimedia.org/T190683#4080558 [13:24:34] ok [13:24:43] unable to init lv-a for swift at [13:24:55] i guess it must be trying to create something [13:25:09] though i am not sure how to fix that (as i am unsure how to use swift) [13:25:14] but thanks for running puppet. [13:31:51] no_justification https://gerrit-review.googlesource.com/c/plugins/simple-submit-rules/+/168490 :) [13:33:02] paladox: no problems, hopefully _joe_ will know. I could check the other puppet errors I saw on other servers and put some more info in the tickets. [13:33:10] yep [13:36:34] PROBLEM - Puppet errors on deployment-eventlog05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:38:42] 10Beta-Cluster-Infrastructure, 10Puppet: PROBLEM - Puppet errors on deployment-mediawiki07 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] - https://phabricator.wikimedia.org/T190632#4080601 (10MarcoAurelio) ``` maurelio@deployment-mediawiki07:~$ sudo puppet agent -tv Info: Using conf... [13:39:59] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet errors on deployment-mediawiki07 - https://phabricator.wikimedia.org/T190632#4080603 (10MarcoAurelio) [13:40:43] ^^ apparently this is "Dependency Package[mcrouter] has failures: true" paladox [13:41:00] E: Unable to locate package mcrouter [13:41:21] is this a stretch instance or a jessie one? [13:41:33] https://github.com/wikimedia/puppet/search?utf8=✓&q=mcrouter&type= [13:41:40] i doin't see it in the puppet repo ^^ [13:41:58] Creating directory '/home/maurelio'. [13:41:58] Linux deployment-mediawiki07 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-0 1-04) x86_64 [13:41:58] Debian GNU/Linux 9.2 (stretch) [13:41:58] deployment-mediawiki07 is mediawiki::appserve [13:42:07] r [13:42:23] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: Update page object pattern in Selenium tests - https://phabricator.wikimedia.org/T185094#4080606 (10zeljkofilipin) a:05zeljkofilipin>03None [13:42:34] so 'stetch' I'd say [13:43:09] ok, though mcrouter is not in the puppet repo [13:43:24] RECOVERY - Free space - all mounts on integration-slave-jessie-1001 is OK: OK: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found) [13:43:31] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Retrospective for T139740 Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T188740#4080608 (10zeljkofilipin) [13:43:34] so some how either the change was enver removed from the puppet master in deployment-prep or there's a change cherry picked [13:43:46] https://gerrit.wikimedia.org/r/c/392221/ [13:43:52] yep change was cherry picked ^^ [13:44:37] https://apt.wikimedia.org/wikimedia/pool/main/m/mcrouter/ [13:44:50] i guess that needs to be copied to stretch-wikimedia too [13:45:42] 10Beta-Cluster-Infrastructure: Puppet broken on deployment-logstash2.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T188921#4080610 (10MarcoAurelio) I feel this is fixed: ``` maurelio@deployment-logstash2:~$ sudo puppet agent -tv Info: Using configured environment 'future' Info: Retrieving pl... [13:47:03] paladox: I feel https://phabricator.wikimedia.org/T188921#4080610 is fixed though. [13:47:15] yep [13:47:22] if puppet runs, it is fixed :) [13:47:45] 10Beta-Cluster-Infrastructure, 10Puppet, 10Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#4080616 (10MarcoAurelio) [13:47:47] 10Beta-Cluster-Infrastructure, 10User-MarcoAurelio: Puppet broken on deployment-logstash2.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T188921#4080613 (10MarcoAurelio) 05Open>03Resolved a:03MarcoAurelio Looks like it's resolved by now. [13:48:48] http://shinken.wmflabs.org/host/deployment-eventlog05 <-- sigh [13:49:25] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<20.00%) [13:53:01] deployment-eventlog05 is a EventLogging analytics processes (eventlogging::analytics) [13:53:01] The last Puppet run was at Fri Mar 23 15:03:42 UTC 2018 (4248 minutes ago). Puppet is disabled. reason not specified [13:55:26] !log restarting CI Jenkins . Upgrades Mail plugin from 1.20 to 1.21 | T190393 [13:55:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:59:25] RECOVERY - Free space - all mounts on integration-slave-jessie-1001 is OK: OK: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found) [14:08:15] Yippee, build fixed! [14:08:15] Project beta-scap-eqiad build #201108: 09FIXED in 4 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/201108/ [14:19:40] 10Continuous-Integration-Infrastructure, 10Release Pipeline: Update integration docker agents to stretch - https://phabricator.wikimedia.org/T190584#4080686 (10akosiaris) [14:19:42] 10Continuous-Integration-Infrastructure, 10Release Pipeline: Package docker-ce for stretch - https://phabricator.wikimedia.org/T190585#4080683 (10akosiaris) 05Open>03Resolved a:03akosiaris I 've just added `docker-ce` under the `thirdparty/ci` component for stretch as well. Resolving, feel free to reopen [14:21:47] (03PS1) 10Hashar: Properly register all python packages [integration/quibble] - 10https://gerrit.wikimedia.org/r/421897 [14:23:42] (03PS1) 10Hashar: Pass os.environ to subprocess commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/421898 [14:24:57] (03PS2) 10Hashar: Pass os.environ to subprocess commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/421898 [14:25:20] (03CR) 10jerkins-bot: [V: 04-1] Pass os.environ to subprocess commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/421898 (owner: 10Hashar) [14:25:26] (03PS1) 10Hashar: (WIP) Docker container with Stretch [integration/quibble] - 10https://gerrit.wikimedia.org/r/421900 [14:25:49] (03CR) 10jerkins-bot: [V: 04-1] (WIP) Docker container with Stretch [integration/quibble] - 10https://gerrit.wikimedia.org/r/421900 (owner: 10Hashar) [14:25:51] (03PS3) 10Hashar: Pass os.environ to subprocess commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/421898 [14:26:02] (03CR) 10Hashar: [C: 032] Properly register all python packages [integration/quibble] - 10https://gerrit.wikimedia.org/r/421897 (owner: 10Hashar) [14:26:25] (03Merged) 10jenkins-bot: Properly register all python packages [integration/quibble] - 10https://gerrit.wikimedia.org/r/421897 (owner: 10Hashar) [14:26:36] (03CR) 10Hashar: [C: 032] Pass os.environ to subprocess commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/421898 (owner: 10Hashar) [14:26:58] (03Merged) 10jenkins-bot: Pass os.environ to subprocess commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/421898 (owner: 10Hashar) [14:27:35] 10Continuous-Integration-Infrastructure, 10Release Pipeline: Update integration docker agents to stretch - https://phabricator.wikimedia.org/T190584#4080732 (10akosiaris) [14:27:38] 10Continuous-Integration-Infrastructure, 10Release Pipeline: Package minikube for stretch - https://phabricator.wikimedia.org/T190586#4080729 (10akosiaris) 05Open>03Resolved a:03akosiaris I 've just added minikube to `thirdparty/ci` component for stretch-wikimedia. I 'll resolve this, feel free to reopen [14:29:05] 10Release-Engineering-Team (Kanban), 10Wikimedia-Hackathon-2018, 10User-zeljkofilipin: Pair on writing Selenium tests in JavaScript/Node.js - https://phabricator.wikimedia.org/T190687#4080739 (10zeljkofilipin) [14:29:41] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Retrospective for T139740 Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T188740#4080743 (10zeljkofilipin) [14:30:18] 10Release-Engineering-Team, 10Epic, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review, 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#4080751 (10zeljkofilipin) [14:30:21] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Retrospective for T139740 Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T188740#4017888 (10zeljkofilipin) 05Open>03Resolved {J88} [14:30:35] 10Scap, 10Patch-For-Review, 10Scoring-platform-team (Current): [blocked] Support git-lfs - https://phabricator.wikimedia.org/T180627#4080752 (10Halfak) [14:35:17] 10Scap, 10Patch-For-Review, 10Scoring-platform-team (Current): [Blocked] Support git-lfs - https://phabricator.wikimedia.org/T180627#4080777 (10Halfak) [14:36:04] 10Scap, 10Scoring-platform-team, 10Patch-For-Review: [Blocked] Support git-lfs - https://phabricator.wikimedia.org/T180627#3764014 (10Halfak) [14:38:03] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Retrospective for T139740 Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T188740#4080786 (10zeljkofilipin) Sent link to the blog post to participants of the meeting. [14:40:25] 10Release-Engineering-Team, 10Epic, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review, 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#4080801 (10zeljkofilipin) [14:40:30] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin: Replace nodemw with mwbot - https://phabricator.wikimedia.org/T181284#4080799 (10zeljkofilipin) 05Resolved>03Open I have forgot I was... [15:00:10] Project mediawiki-core-code-coverage-php7 build #167: 04STILL FAILING in 10 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/167/ [15:00:11] Project mediawiki-core-code-coverage build #3406: 04STILL FAILING in 10 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3406/ [15:10:29] 10Release-Engineering-Team (Watching / External), 10Operations, 10Release Pipeline: Update Debian package of Blubber - https://phabricator.wikimedia.org/T190551#4080909 (10akosiaris) 05Open>03Resolved a:03akosiaris Package built and uploaded to `stretch-wikimedia` and `jessie-wikimedia`. Resolving this... [15:27:43] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:27:55] 10Gerrit, 10Operations: git pull fails for MW core with "fatal: protocol error: bad pack header" when local branches point to remote branches that no more exist - https://phabricator.wikimedia.org/T190676#4080932 (10Aklapper) [15:33:42] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:41:28] 10Gerrit, 10Operations: git pull fails for MW core with "fatal: protocol error: bad pack header" when local branches point to remote branches that no more exist - https://phabricator.wikimedia.org/T190676#4080949 (10demon) This also isn't a Gerrit bug. Can easily replicate with Github or any other git server.... [15:42:56] Project mwext-phpunit-coverage-publish build #2586: 04FAILURE in 3.6 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2586/ [15:43:59] 10Gerrit, 10Operations: git pull fails for MW core with "fatal: protocol error: bad pack header" when local branches point to remote branches that no more exist - https://phabricator.wikimedia.org/T190676#4080953 (10demon) >>! In T190676#4080498, @Paladox wrote: > You could also do "git config remote.origin.pr... [15:44:22] Yippee, build fixed! [15:44:23] Project mwext-phpunit-coverage-publish build #2587: 09FIXED in 1 min 23 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2587/ [15:46:41] (03PS1) 10Hashar: Revert "Pass os.environ to subprocess commands" [integration/quibble] - 10https://gerrit.wikimedia.org/r/421922 [15:46:48] (03CR) 10Hashar: [C: 032] Revert "Pass os.environ to subprocess commands" [integration/quibble] - 10https://gerrit.wikimedia.org/r/421922 (owner: 10Hashar) [15:47:15] (03Merged) 10jenkins-bot: Revert "Pass os.environ to subprocess commands" [integration/quibble] - 10https://gerrit.wikimedia.org/r/421922 (owner: 10Hashar) [15:48:20] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-ms-be03 - https://phabricator.wikimedia.org/T190683#4080501 (10Joe) @MarcoAurelio if you look at the code: ``` define swift::init_device($partition_nr='1') { if ($title !~ /^\/dev\/([hvs]d[a-z]+|md[0-9]+)$/) {... [16:05:24] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<10.00%) [16:10:24] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Phlogiston: Phlogiston reports don't have new data since mid-February - https://phabricator.wikimedia.org/T188149#4081072 (10mmodell) I can run a phabricator database migration to convert everything to the new format. That could help with processing time... [16:17:18] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Phlogiston: Phlogiston reports don't have new data since mid-February - https://phabricator.wikimedia.org/T188149#4081127 (10JAufrecht) I found the missing indexes and put them in and tested over the weekend; processing time is back to what it was before... [16:25:23] RECOVERY - Free space - all mounts on integration-slave-jessie-1001 is OK: OK: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found) [16:31:27] 10Release-Engineering-Team, 10Page-Previews, 10Readers-Web-Backlog, 10Epic, 10MobileFrontend (MobileFrontend.js): [EPIC] Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980#4081189 (10Jhernandez) FTR here's that email thread I sent to discuss last year: https:... [16:32:39] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Verify functionality of the 'production' image in the context of an isolated k8s deployment - https://phabricator.wikimedia.org/T183165#4081198 (10dduvall) [16:33:44] 10Release-Engineering-Team (Kanban), 10Release Pipeline: install helm on integration agents - https://phabricator.wikimedia.org/T188934#4081215 (10dduvall) a:03dduvall [16:33:58] 10Release-Engineering-Team (Kanban), 10Release Pipeline: install helm on integration agents - https://phabricator.wikimedia.org/T188934#4024513 (10dduvall) p:05Triage>03Normal [16:34:04] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin, 10Wikimedia-log-errors (Jenkins Failure): Warning: Task "stylelint:src" failed due to postcss-less@1.1.4 - https://phabricator.wikimedia.org/T190269#4081220 (10ze... [16:34:57] 10Release-Engineering-Team (Kanban), 10Release Pipeline: ensure minikube running on integration agents - https://phabricator.wikimedia.org/T188936#4081224 (10dduvall) p:05Triage>03High a:03dduvall [16:35:47] 10Release-Engineering-Team (Kanban), 10Release Pipeline: install kubectl on integration agents - https://phabricator.wikimedia.org/T188933#4081228 (10dduvall) p:05Triage>03High a:03dduvall [16:37:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikimedia-Incident, 10Zuul: CI is running against parent patches, not the patches themselves for chained patches - https://phabricator.wikimedia.org/T187567#3979150 (10hashar) 05Open>03Resolved [16:37:10] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Zuul: Zuul no longer starts gate-and-submit when parent is in gate-and-submit pipeline - https://phabricator.wikimedia.org/T186949#3960209 (10hashar) 05Open>03Resolved [16:37:15] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Exception while launching job: TypeError: 'int' object has no attribute '__getitem__' - https://phabricator.wikimedia.org/T186381#4081238 (10hashar) 05Open>03Resolved [16:37:51] 10Release-Engineering-Team (Someday), 10scap2, 10Patch-For-Review: Eliminate symlinks in mediawiki-config (as much as possible) - https://phabricator.wikimedia.org/T126306#4081252 (10greg) [16:38:07] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Packaging, 10Zuul: Upload new zuul and jenkins-debian-glue packages to apt.wikimedia.org - https://phabricator.wikimedia.org/T186786#4081254 (10hashar) [16:38:09] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin, 10Wikimedia-log-errors (Jenkins Failure): Warning: Task "stylelint:src" failed due to postcss-less@1.1.4 - https://phabricator.wikimedia.org/T190269#4068266 (10ze... [16:38:42] 10Gerrit, 10Release-Engineering-Team (Watching / External), 10Cleanup, 10Repository-Admins, 10Scoring-platform-team: Investigate and clean up deprecated ORES repos - https://phabricator.wikimedia.org/T179610#4081256 (10greg) [16:38:52] 10Release-Engineering-Team (Kanban), 10Release Pipeline: install helm on integration agents - https://phabricator.wikimedia.org/T188934#4081258 (10dduvall) We should be sure to add a necessary workaround to integration puppet for {{T190589}}. [16:45:54] 10Release-Engineering-Team, 10Epic, 10Tracking, 10User-zeljkofilipin: Selenium framework improvements - https://phabricator.wikimedia.org/T182986#3840359 (10zeljkofilipin) a:03zeljkofilipin [16:47:52] 10Release-Engineering-Team, 10Epic, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review, 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#4081307 (10zeljkofilipin) [16:47:56] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Selenium tests in CI for extensions - https://phabricator.wikimedia.org/T164721#4081306 (10zeljkofilipin) [16:49:14] 10Release-Engineering-Team, 10Epic, 10Tracking, 10User-zeljkofilipin: Selenium framework improvements - https://phabricator.wikimedia.org/T182986#4081313 (10zeljkofilipin) [16:49:19] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Selenium tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3345285 (10zeljkofilipin) [16:50:32] (03PS3) 10Umherirrender: Simplify UnusedGlobalVariables sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/421332 (owner: 10Thiemo Kreuz (WMDE)) [16:50:36] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160#4081319 (10zeljkofilipin) [16:50:37] (03CR) 10Umherirrender: [C: 032] Simplify UnusedGlobalVariables sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/421332 (owner: 10Thiemo Kreuz (WMDE)) [16:50:39] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Patches in Gerrit deleting Selenium+Ruby tests for repositories that still have them - https://phabricator.wikimedia.org/T183162#4081318 (10zeljkofilipin) [16:51:25] 10Release-Engineering-Team, 10Epic, 10Tracking, 10User-zeljkofilipin: Selenium framework improvements - https://phabricator.wikimedia.org/T182986#3840359 (10zeljkofilipin) [16:51:29] 10Release-Engineering-Team, 10Epic, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review, 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#4081331 (10zeljkofilipin) [16:51:38] (03Merged) 10jenkins-bot: Simplify UnusedGlobalVariables sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/421332 (owner: 10Thiemo Kreuz (WMDE)) [16:51:53] 10Release-Engineering-Team, 10Epic, 10Tracking, 10User-zeljkofilipin: Selenium framework improvements - https://phabricator.wikimedia.org/T182986#4081335 (10zeljkofilipin) [16:51:58] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Should selenium-EXTENSION-jessie run for all repositores with Selenium tests? - https://phabricator.wikimedia.org/T188742#4081334 (10zeljkofilipin) [16:52:00] 10Release-Engineering-Team, 10Epic, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review, 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3822892 (10zeljkofilipin) [16:52:31] (03CR) 10jenkins-bot: Simplify UnusedGlobalVariables sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/421332 (owner: 10Thiemo Kreuz (WMDE)) [16:55:21] (03CR) 10Umherirrender: [C: 031] "I would not expect a big difference in the order, but looks okay to me" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/421845 (owner: 10Thiemo Kreuz (WMDE)) [17:02:08] 10Release-Engineering-Team (Kanban), 10Wikimedia-Apache-configuration, 10Patch-For-Review: Cleanup remaining WikipediaMobileFirefoxOS references - https://phabricator.wikimedia.org/T187850#4081406 (10jcrespo) > I don't think we use mediawiki-config for anything We use them to run check_private_data.py, whic... [17:23:41] o/ What is the gerrit project to add some volunteer Gerrit users to trigger CI builds? [17:31:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10Ops-Access-Requests, 10Patch-For-Review: contint-admins sudo for service jenkins - https://phabricator.wikimedia.org/T190277#4081546 (10RobH) This was approved in today's SRE meeting. I'll rebase a... [17:31:52] bearND: integration/config [17:32:25] legoktm: thanks! [17:32:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10Ops-Access-Requests, 10Patch-For-Review: contint-admins sudo for service jenkins - https://phabricator.wikimedia.org/T190277#4081552 (10RobH) 05Open>03Resolved a:03RobH This is now live on the... [17:35:16] I hate how resolving a task also assigns it to oneself. [17:35:27] is that a customizable option per use? [17:35:28] user [17:37:53] robh, when resolving you can set the assignee to override the default being yourself [17:37:57] IIRC [17:38:19] well, when its not assigned, and i resolve [17:38:23] it makes me assignee [17:38:46] Krenair: im not sure what you mean with what you stated [17:38:52] its likely im just not parsing it right ;] [17:39:09] so what you do is set the status to resolved, just above your comment, right? [17:39:39] correct [17:39:43] robh: If you as well as changing the status you change the assignee (and set the box blank), it'll warn that you're not changing anything but it'll also not change you to be the assignee. [17:39:47] You can take multiple actions like the status change in one go [17:39:58] For example, in addition to changing status, you can set assignee [17:40:06] James_F: yeah, i figured that may be what i had to do but i was tired of the dialoge confirm [17:40:07] heh [17:40:09] this should override the default of self-assigning [17:40:28] Krenair: yeah i just dislike that i have to specify no asignee when i never asked it to make me the assignee [17:40:48] its assuming i should be the assigned task person since i changed status, which isn't correct for many of our workflows, heh. [17:40:52] (03PS1) 10Umherirrender: Improve performance of PrefixedGlobalFunctionsSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/421954 [17:40:53] * Krenair shrug [17:40:59] I can see why they did it [17:41:04] I can also see why you might not want to do it [17:41:07] but seems the only way around is to set no assignee in the same action and confirm the option [17:41:23] i didnt see a way to easily set it to behave the way i want for just my user [17:41:26] but figured id ask ;] [17:41:32] it's probably correct in the most obvious case [17:41:42] maybe not if you're just closing out old stuff done ages ago [17:41:49] but thanks both Krenair and James_F for suggestions its appreciated! =] [17:41:56] yeah, its mostly when i close something as 'clinic duty' [17:42:05] well [17:42:06] * James_F nods. [17:42:07] i dont want it reopened in a week with me as the assignee, since im not on clinic duty then [17:42:14] hashar: it looks like mediawiki-phpunit-coverage-patch might be stuck? [17:42:27] indeed though i can see why it would be a default behavior [17:42:34] notice im not asking for an upstream bug ;D [17:42:38] robh, hmm... sometimes I've seen people incorrectly reopen [17:42:55] for example when they want to do something extra [17:43:20] yeah, that happens regularly as well. then we have to ask them to open a new task if its not a correction or clarification of the old, etc... [17:43:36] in other cases, I guess reopening is in case the task wasn't completed properly? or maybe they just don't like the way you closed it? [17:43:53] (03PS1) 10Ejegg: Get rid of php55 tests in DonationInterface [integration/config] - 10https://gerrit.wikimedia.org/r/421955 [18:00:05] hi rel-engers! I'm trying to get rid of the php5.5 tests for DonationInterface ^^^ [18:00:12] does that look like the way to do it? [18:00:40] I'm trying to test locally, but jenkins-jobs is crashing with NoneType object is not iterable [18:01:15] basically, we just want to remove the mwgate-composer-php55-docker test that's breaking on https://gerrit.wikimedia.org/r/421950 [18:01:43] since we upped the minimum php version of one of our dependencies to 5.6 [18:27:22] legoktm: my bet is nodepool pool is full and since coverage patches are low priority they will be delayed until there is some room available [18:27:42] iirc the gear implementation first run all high precedence job [18:27:48] if there is still some room, the normal one [18:27:54] and finally the low precedences ones [18:28:09] so essentially when the pool is full, the low precedences jobs never have a chance to run [18:29:02] so.. comparing tin and deploy1001.. after letting puppet do it's thing.. we get a /srv of 31G on deploy1001. but on tin we have 43G in /srv . i wonder if i should rsync it all now [18:29:15] mutante or hashar: can you help get fr-tech unstuck? https://gerrit.wikimedia.org/r/421955 should (i think) stop running php5.5 tests for DOnationInterface [18:29:19] and what the diff really is.. [18:29:30] tht doesn't get automatically pulled / isnt puppetized? [18:29:37] just 1 line deleted [18:29:38] or just logs.. will look [18:30:15] ejegg: i dont have +2 in that repo [18:30:21] oops, sorry [18:30:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10Ops-Access-Requests, 10Patch-For-Review: contint-admins sudo for service jenkins - https://phabricator.wikimedia.org/T190277#4081805 (10hashar) Looks good. Thank you everyone :] [18:30:58] hashar: ^ can you help fr-tech [18:31:47] ejegg: mutante : looking at the diff ( https://integration.wikimedia.org/ci/job/integration-zuul-layoutdiff/15075/console ) :D [18:32:05] ooh, right, I forgot we got a diff there [18:32:11] was trying to generate it locally [18:32:17] thanks for the reminder [18:32:24] so that drops wgate-composer-php55-docker and wgate-composer-hhvm-docker [18:32:29] but iirc you are running on php 5.6 [18:32:36] yeah [18:32:55] (03CR) 10Hashar: [C: 032] "That drops wgate-composer-php55-docker and wgate-composer-hhvm-docker, the cluster is on PHP 5.6. ;D" [integration/config] - 10https://gerrit.wikimedia.org/r/421955 (owner: 10Ejegg) [18:32:58] easy! [18:33:03] thanks! [18:34:10] (03Merged) 10jenkins-bot: Get rid of php55 tests in DonationInterface [integration/config] - 10https://gerrit.wikimedia.org/r/421955 (owner: 10Ejegg) [18:34:51] $ git pull --all [18:34:51] Fetching gerrit [18:34:51] fatal: internal server error [18:36:33] ah, so the part that doesn't get synced between deployment servers is mostly mediawiki-staging but also mediawiki is missing some things [18:38:04] MaxSem: you need to run git remote prune origin [18:38:06] Please [18:38:54] why? [18:39:22] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:16] ejegg: and I have deployed the change [18:41:25] MaxSem: https://phabricator.wikimedia.org/T190676#4080492 [18:41:45] thanks much hashar [18:42:34] MaxSem: to fix that internal server error [18:42:38] When you were pulling [18:44:11] it helped [18:44:14] thanks paladox [18:49:54] 10Continuous-Integration-Infrastructure (shipyard), 10MediaWiki-Parser, 10Quibble, 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review: Parsertest fails: Namespace takes precedence over interwiki link (T53680) - https://phabricator.wikimedia.org/T190554#4081860 (10hashar)... [18:51:17] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin, 10Wikimedia-log-errors (Jenkins Failure): Warning: Task "stylelint:src" failed due to postcss-less@1.1.4 - https://phabricator.wikimedia.org/T190269#4081869 (10ha... [18:53:38] bad pack header? [18:54:13] I'm getting that on pull now, for the fundraising/REL1_27 branch [18:54:36] of core [18:55:41] ejegg: git remote prune origin (or gerrit if that's your remote name) [18:55:47] can't even get fetch [18:55:56] k, trying prune [18:56:04] * greg-g restrains from cc'ing chad :) [18:56:41] (03CR) 10BearND: [C: 031] "@Hashar Any thoughts about this? Can this be merged soon? The dependency got merged a couple of weeks ago." [integration/config] - 10https://gerrit.wikimedia.org/r/415886 (https://phabricator.wikimedia.org/T177896) (owner: 10Mholloway) [19:14:02] 10Phabricator, 10Release-Engineering-Team (Kanban), 10User-greg: Please create a phame blog for surveys and other type of aggregated analysis - https://phabricator.wikimedia.org/T190598#4081939 (10greg) From Maria via email: Name: The surveyor Subtitle: Surveys and other aggregated analysis that focuses on... [19:25:55] (03PS1) 10Umherirrender: Allow globals to start with numbers [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/421978 [19:53:04] PROBLEM - Puppet staleness on integration-slave-docker-1003 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [43200.0] [19:55:57] (03CR) 10Legoktm: [C: 04-1] Add mobileapps-diff-test periodic job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/415886 (https://phabricator.wikimedia.org/T177896) (owner: 10Mholloway) [20:12:36] (03PS1) 10Hashar: Port mw-run-update-script.sh to python [integration/quibble] - 10https://gerrit.wikimedia.org/r/421990 [20:15:13] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T183965#4082233 (10Krinkle) [20:17:04] (03CR) 10Hashar: [C: 032] Port mw-run-update-script.sh to python [integration/quibble] - 10https://gerrit.wikimedia.org/r/421990 (owner: 10Hashar) [20:17:30] (03Merged) 10jenkins-bot: Port mw-run-update-script.sh to python [integration/quibble] - 10https://gerrit.wikimedia.org/r/421990 (owner: 10Hashar) [20:21:59] paladox: https://archiva.wikimedia.org/#basicsearch/2.14.7-9 [20:22:18] no_justification thank you :) [20:24:08] Let's get that out now [20:24:21] :) [20:25:41] (03CR) 10Legoktm: [C: 032] Allow globals to start with numbers [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/421978 (owner: 10Umherirrender) [20:26:39] (03Merged) 10jenkins-bot: Allow globals to start with numbers [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/421978 (owner: 10Umherirrender) [20:26:43] (03CR) 10Legoktm: [C: 032] Optimize DirUsage sniff for performance [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/421845 (owner: 10Thiemo Kreuz (WMDE)) [20:27:01] (03CR) 10Legoktm: [C: 032] Use File::getMethodProperties to get visibility [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/421615 (owner: 10Umherirrender) [20:27:26] (03Merged) 10jenkins-bot: Optimize DirUsage sniff for performance [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/421845 (owner: 10Thiemo Kreuz (WMDE)) [20:27:31] (03CR) 10jenkins-bot: Allow globals to start with numbers [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/421978 (owner: 10Umherirrender) [20:27:46] (03Merged) 10jenkins-bot: Use File::getMethodProperties to get visibility [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/421615 (owner: 10Umherirrender) [20:28:00] (03CR) 10jenkins-bot: Optimize DirUsage sniff for performance [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/421845 (owner: 10Thiemo Kreuz (WMDE)) [20:28:27] (03CR) 10jenkins-bot: Use File::getMethodProperties to get visibility [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/421615 (owner: 10Umherirrender) [20:32:07] 10Gerrit, 10Analytics-Tech-community-metrics, 10Upstream: Gerrit patchset 99101 cannot be accessed: "500 Internal server error" - https://phabricator.wikimedia.org/T161206#4082284 (10Paladox) 05Open>03Resolved https://gerrit.wikimedia.org/r/#/c/99101/ is now working :) [20:33:26] PROBLEM - Puppet errors on deployment-eventlogging04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:39:01] PROBLEM - Puppet errors on deployment-mediawiki07 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:39:14] 10Phabricator, 10Release-Engineering-Team (Kanban), 10User-greg: Please create a phame blog for surveys and other type of aggregated analysis - https://phabricator.wikimedia.org/T190598#4082293 (10greg) 05Open>03Resolved Created: https://phabricator.wikimedia.org/phame/live/12/ I couldn't use the given... [20:42:52] (03PS1) 10Hashar: Drop hardcoded zuul-cloner path and use color/verbose [integration/quibble] - 10https://gerrit.wikimedia.org/r/422057 [20:43:29] (03CR) 10Hashar: [C: 032] Drop hardcoded zuul-cloner path and use color/verbose [integration/quibble] - 10https://gerrit.wikimedia.org/r/422057 (owner: 10Hashar) [20:43:32] !log deployed to BC: [mobileapps/deploy@e223f51]: Update mobileapps to 534f95d [20:43:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:43:54] (03Merged) 10jenkins-bot: Drop hardcoded zuul-cloner path and use color/verbose [integration/quibble] - 10https://gerrit.wikimedia.org/r/422057 (owner: 10Hashar) [21:13:24] RECOVERY - Puppet errors on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [21:29:38] 10Release-Engineering-Team (Watching / External), 10Analytics-Kanban, 10Operations, 10Patch-For-Review, and 2 others: Deprecation of mw.errors.* metrics - https://phabricator.wikimedia.org/T188749#4082451 (10Nuria) 05Open>03Resolved [21:40:05] 10Phabricator, 10Release-Engineering-Team (Kanban), 10User-greg: Please create a phame blog for surveys and other type of aggregated analysis - https://phabricator.wikimedia.org/T190598#4082476 (10mcruzWMF) Thank you, Greg! I really appreciate all the help, and how fast you solved this =) {meme, src="cat-st... [22:09:10] 10Release-Engineering-Team, 10Page-Previews, 10Readers-Web-Backlog, 10Epic, 10MobileFrontend (MobileFrontend.js): [EPIC] Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980#4082728 (10Niedzielski) We discussed this in passing a little today and mentioned that... [22:12:41] paladox: Does TopMenu not work in polygerrit 2.14.x? [22:12:48] no_justification nope [22:12:50] not currently [22:12:56] Fixed in 2.15? [22:12:59] nope [22:13:02] not yet [22:13:14] they are still redesgning the whole thing [22:13:24] Oh, dangit. So that was a waste of an hour :( [22:13:30] oh why? [22:13:33] it's in gwtui [22:13:43] Yeah I guess [22:13:47] though yes, let me see if i can find a way to get upstream to support it [22:15:33] no_justification hmm documentation tab (should support extra configuations) [22:16:21] no_justification but actually with the redesgned admin page [22:16:54] it would look different [22:21:26] no_justification ah, documentation links are hard coded it seems [22:21:34] https://github.com/GerritCodeReview/gerrit/blob/master/polygerrit-ui/app/elements/core/gr-main-header/gr-main-header.js#L38 [22:31:17] i guess it needs to be hardcoded [22:31:29] if looking at https://github.com/GerritCodeReview/gerrit/blob/61b19d28912dc13d3f1838ed5b57b06b612042ed/gerrit-gwtui/src/main/java/com/google/gerrit/client/Gerrit.java#L819 is correctly [22:36:31] Can't you extend it here? https://github.com/GerritCodeReview/gerrit/blob/61b19d28912dc13d3f1838ed5b57b06b612042ed/gerrit-gwtui/src/main/java/com/google/gerrit/client/Gerrit.java#L907 [22:37:39] 10Beta-Cluster-Infrastructure: Request for access to the beta cluster - https://phabricator.wikimedia.org/T190755#4082855 (10chelsyx) [22:38:16] no_justification: Hm.. I also ran into the issue of 'git remote update' not working (fatal: internal server error, fatal: protocol error: bad pack header). I heard you say that 'git remote prune origin' fixes it. And indeed it is. I know about prune and run it maybe once a week or so, but removing branches like this is not new. Yet, it only started to break this time. Did anythign change as far as you know? this is the first time I've seen [22:38:16] this error. [22:38:41] I ran GC after the branch was pruned [22:38:45] But I do that from time to time anyway [22:39:15] Sure, but why would that cause an internal error. [22:39:43] I've got for weeks without pruning locally and then I run it once and it catches up without issue, and removes all the local branch references at once, in between , git updates used to work fine. [22:39:54] Is this a bug in JGit? [22:40:01] That it can't handle the update? [22:40:39] I don't think it's a jgit bug, I think Gerrit isn't handling it right [22:40:49] Here, I'll put the stacktrace [22:45:43] https://phabricator.wikimedia.org/P6899 [22:56:06] Krinkle: I did http://wisercoder.com/fixing-the-protocol-error-bad-pack-header-error-in-git/ to fix it earlier. [23:01:35] James_F: 'git remote prune origin' also works. [23:01:58] * James_F nods. [23:02:46] Which is essentially the part of 'git remote update' that Git doesn't do by default (which is to not only update and add branches, but also remove old ones that no longer exist remotely). Note that 'git remote prune origin' only affects the internal branches locally under your own copy of refs/remotes/*, it does not affect "real" branches you have. So even if you have a local 1.20.0-wmf.1 branch with an important local commit, that'll never [23:02:46] get wiped or anything. [23:07:09] I wish git did this by default [23:07:38] As I said on the task, I have it enables globally [23:37:20] no_justification: hmm I guess we could [23:37:34] But I would need to look into that more at some point [23:37:55] As we need to make sure it correctly adds it to the right tab and add additional tabs [23:39:21] I suppose that menu extender plugin only supports the old way too [23:40:06] Yeh [23:40:19] no_justification: polygerrit only supports JavaScript plugins [23:40:27] HTML + JavaScript [23:40:31] Yeah [23:40:34] But you can build that in Java [23:40:41] Like codemirror-editor [23:40:58] We could add it in Javascript easy enough [23:41:03] Yeh [23:41:03] We do anyway [23:41:40] Speaking of, can the login page stuff be there or does it have to be injected by css? [23:42:05] Has to still be injected by css [23:42:09] Figured [23:42:20] Apparently implementing PG there would require a lot of work [23:42:34] As there is currently not rest api where you can use to login [23:42:46] I’ve created tasks for this some where a while ago [23:42:48] Well also there's a security concern of running Javascript on login pages [23:42:56] Yeh ^^ [23:42:57] We disable user js on login page [23:43:30] Presumably a plugin is secure but if you enable remote plugin administration it's a potential escalation [23:43:38] Yeh [23:44:02] Polygerrit is securer [23:44:10] They have some kind of xss plugin [23:44:15] To prevent attacks [23:44:49] I remember this as I reported it broke building it in the first place :) [23:44:55] They had to fix that [23:46:30] no_justification: https://github.com/GerritCodeReview/gerrit/commit/5ee6971a61a04b0ec0b7a28df4c191b7e1c99a77 [23:47:58] https://github.com/Polymer/polymer-resin [23:48:50] no_justification: so upstream should try to migrate to pg as fast as they can [23:48:57] That will improve security [23:51:22] They may decide to do that in 3.1 as 3.0 will be a big change with the removal of reviewdb