[00:13:07] 10Beta-Cluster-Infrastructure, 10Graphoid, 6Services: Graphoid deploy on beta cluster is failing / salt problems in Beta - https://phabricator.wikimedia.org/T125067#1980371 (10thcipriani) 5Open>3Resolved a:3thcipriani Got it working through some combination of: 1. `sudo salt '*' saltutil.refresh_pilla... [00:14:29] thcipriani: thnx! \o/ [00:15:08] mobrovac: :D [00:15:12] thcipriani: your comment sums up salt/trebuchet pretty good - "I threw a bunch of commands at it and at some point it started working" :D [00:15:31] true story. [00:15:36] haha [00:16:15] have you tried every salt command you can think of? no? That's probably your problem right there. [00:16:40] :( [00:18:16] thcipriani: salt even manages to resist to the famous cure in IT "have you tried turning it off and on again?" ;) [00:18:25] 10Deployment-Systems, 3Scap3: sync-masters slow on mira - https://phabricator.wikimedia.org/T125108#1980381 (10bd808) The first thing I would do to debug (if I had the root powers to do it) would be to run this from tin: ``` $ sudo /usr/bin/rsync \ --verbose \ --debug=CONNECT2 \ --archive --delete-... [00:19:47] 10Deployment-Systems, 3Scap3: sync-masters slow on mira - https://phabricator.wikimedia.org/T125108#1980382 (10bd808) Possibly related: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1300367 (rsync 3.1.0 and 3.0.9 incompatibility) [00:49:11] twentyafterfour: Oops, I hadn't realized that you submitted a patch for the EINTR issue [00:49:47] I wouldn't have submitted mine if I had seen it in time; I did not mean to imply there was anything wrong with yours -- I just didn't see it. [00:53:14] 3Scap3: scap should handle EINTR gracefully - https://phabricator.wikimedia.org/T125136#1980516 (10ori) >>! In T125136#1980008, @mmodell wrote: > ok nevermind D111 @mmodell, my bad -- I did not notice that you had submitted a patch too. I would have not worked on this if I had known. No slight intended. [01:20:12] ori: I didn't think you were dissing my patch ;) [01:20:26] your solution is cleaner and more thorough. [01:26:18] 3Scap3: scap should handle EINTR gracefully - https://phabricator.wikimedia.org/T125136#1980594 (10mmodell) @ori: No worries, I figured you hadn't seen it, I am not offended. :) Anyway your patch is much more thorough and what I did took all of 5 minutes, so it's no loss really. [01:35:56] Yippee, build fixed! [01:35:57] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #516: 09FIXED in 18 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/516/ [02:56:04] Project beta-scap-eqiad build #87989: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/87989/ [05:09:28] 10Continuous-Integration-Config, 10MediaWiki-extensions-Other, 10Wikidata: [LifeWeb] CI test: The module 'wikibase.RepoApi' required by 'ext.LifeWeb.libLW' must exist - https://phabricator.wikimedia.org/T104085#1980835 (10JanZerebecki) [05:16:51] 10Continuous-Integration-Config, 10Gerrit, 10MediaWiki-extensions-Other, 5Patch-For-Review: [LightweightRDFa] The module 'ext.wikiEditor' required by 'ext.LightweightRDFa.button' must exist - https://phabricator.wikimedia.org/T93727#1980848 (10JanZerebecki) https://www.mediawiki.org/wiki/Gerrit/Inactive_pr... [07:48:18] legoktm: I need to add an extension in the mediawiki namespace to packagist, can you give my account access https://packagist.org/users/JanZerebecki/ or is it nece? [07:48:26] ssary to use the mediawiki account? [07:52:32] -> https://phabricator.wikimedia.org/T123970#1980978 [07:55:35] bd808: ^^? [08:33:59] Yippee, build fixed! [08:34:00] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #859: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/859/ [10:10:33] (03PS1) 10Phedenskog: Use bash to collect return for different WebPageTest scripts [integration/config] - 10https://gerrit.wikimedia.org/r/267226 [10:11:30] (03Abandoned) 10Phedenskog: Reimpl for catching errors in WebPageTest runs [integration/config] - 10https://gerrit.wikimedia.org/r/259783 (https://phabricator.wikimedia.org/T120365) (owner: 10Phedenskog) [10:12:47] woah [10:12:59] come on beta-scap-eqiad! [10:54:25] Yippee, build fixed! [10:54:26] Project beta-scap-eqiad build #88035: 09FIXED in 29 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/88035/ [11:33:57] woo! [11:58:24] <_joe_> !log upgraded hhvm to 3.6 wm8 in deployment-prep [11:58:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:00:03] Yippee, build fixed! [12:00:04] Project browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #373: 09FIXED in 3 min 3 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/373/ [12:46:11] Project browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce build #776: 04FAILURE in 10 sec: https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce/776/ [12:55:08] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #737: 04FAILURE in 1 min 7 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/737/ [15:32:41] https://phabricator.wikimedia.org/D110 trivialllllll [15:56:38] 7Blocked-on-RelEng, 6Phabricator: iridium:/var/log/phd/daemons.log is growing too much (took 20% of filesystem space) - https://phabricator.wikimedia.org/T124651#1981705 (10demon) We should have it in place already. From `/etc/logrotate.d/phd`: ``` /var/log/phd { daily compress missingok notife... [16:04:42] thcipriani: thx :) [16:05:25] ostriches: meant to accept sooner, just fell down the list :( [16:05:54] wanted to actually run the code before accept, in case my eyes did deceive me. [16:06:21] hehe fair enough [16:09:32] thcipriani: I was gonna land ori's EINTR patch (D112) unless you had any objections. [16:10:41] ostriches: no objections, gave it a look, seemed sane. If you're good with it, that's good enough for me. [16:11:14] (well, I guess, if you and twentyafterfour are good with it...etc.) [16:11:44] I tested it. It works for the general non-failing case fine. [16:12:23] Bleh merge conflict [16:12:30] Prolly my pep8 change [16:12:55] * thcipriani shakes fist at pep8 [16:15:28] Errrr [16:15:30] Hm [16:17:58] lol, no wonder it conflicted. ori's parent was back in October :p [16:21:37] 3Scap3: scap should handle EINTR gracefully - https://phabricator.wikimedia.org/T125136#1981751 (10demon) 5Open>3Resolved a:3demon [16:22:27] I think D102 and D101 should probably be the last 2 we look at, then call it "done" [16:22:37] 102 especially [16:23:19] I'd like to get 104 in there, too. Something's wonky in my testing of that patch. [16:24:01] but, yeah, it's definitely not a requirement. Just a nice-to-have. Dunno what's wrong with that patch intuitively though. [16:25:00] * thcipriani reviews some patches [16:30:59] Hi could someone help me with composer. When you have an extension that requires another extension but through composer. When you go to install it the path looks like extensions/Example/extensions/Example2/ I would like to get it so it looks like extensions/Example/ and extensions/Example2/ i am aware there is an extra plugin in composer i am not sure how to do this please could i have help. [16:44:43] !log deployed scap updates on beta [16:44:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:24:51] thcipriani: Question, should setup.py use python2 explicitly as well, or is it fine with env? [17:24:57] I know we're keeping tests/* as env [17:26:14] ostriches: might be fine keeping with env. Wouldn't be using setup.py if you're using the package. [17:27:00] True. [17:27:02] mmk [17:28:17] Meh, I don't wanna commandeer first. [17:30:25] Heh I can't self-accept the thing now that I took it [17:30:48] There should be a "give it back I'm done commandeering" button lol [17:34:10] ostriches: I can accept. Super quick nitpick: tests/scap/php-data/good_notphp.py is not a file that actually runs anything, it's just there for testing, but it's no longer valid python2 code... [17:35:23] well, it was valid python3, now the interpreters changed to python2. I think the whole point of that file is to be not a php file, so it probably doesn't matter at all. [17:37:22] lol yeah I didn't know [17:38:39] It works for me on 2.7... [17:42:11] huh, so it does. That's weird, I thought you had to do: from __future__ import print_function for python2 to use print as a function [17:43:06] * ostriches shrugs [17:43:13] confusion: https://docs.python.org/2/library/functions.html#print [17:44:21] Yay python! [18:00:30] marxarelli: https://phabricator.wikimedia.org/D114 :) [18:19:35] iirc the future print function is for things like print("foo", file=sys.stderr) [18:19:42] but basic print works in py2 [18:30:39] ugh ...so I guess I missed most of that meeting [18:35:35] 10Deployment-Systems, 3Scap3: Merge `git_deploy_user` and `ssh_user` - https://phabricator.wikimedia.org/T124460#1982242 (10dduvall) 5Open>3Resolved [18:35:57] twentyafterfour: We're still in the meeting. Peer reviewed ^ [18:42:33] Phab timed out? [18:42:58] !log updated scap on beta [18:43:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:46:29] No, just my wifi being dumb [19:04:50] ostriches: can (or who do i ask) add the wikidata group to https://gerrit.wikimedia.org/r/#/admin/groups/1138,members ? [19:05:06] some reason it doesn't let me do that myself [19:07:22] aude: ostriches just left for the day and I don't have permission either [19:07:36] greg-g: ok :/ [19:07:38] not urgent [19:08:08] probably anyone in https://gerrit.wikimedia.org/r/#/admin/groups/1,members can [19:08:31] twentyafterfour: around? :) [19:12:40] aude: done. [19:12:45] thcipriani: thanks [19:13:10] now someone other than myself can review patches there [19:44:05] aude yo [19:44:32] twentyafterfour: thcipriani already helped me (needed someoen who is a gerrit admin) [19:44:44] ok sorry I was lost in google docs for a minute [19:44:49] it's ok [19:48:07] :Q [19:48:13] ah [21:54:54] 7Blocked-on-RelEng, 6Phabricator: iridium:/var/log/phd/daemons.log is growing too much (took 20% of filesystem space) - https://phabricator.wikimedia.org/T124651#1982852 (10mmodell) so it was growing fast enough to fill the disk even on a short rotation schedule? That's no good. NOTE: The root partition on ir... [22:27:46] 7Blocked-on-RelEng, 6Phabricator: iridium:/var/log/phd/daemons.log is growing too much (took 20% of filesystem space) - https://phabricator.wikimedia.org/T124651#1982968 (10jcrespo) Yes, I got it on the 6% alert. It doesn't have the flexibility of LVM, which was the first thing I checked to solve this. [22:32:03] 6Release-Engineering-Team, 10MediaWiki-extensions-Cards: Cards stuck on version 0.3.0 on beta labs - https://phabricator.wikimedia.org/T125182#1982987 (10Jdlrobson) [22:32:16] twentyafterfour: any ideas? ^ [22:33:26] thcipriani: weren't you testing updated scap in beta cluster recenlty? [22:33:54] greg-g: I did deploy scap to beta earlier [22:33:56] today [22:34:06] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 10MediaWiki-extensions-Cards: Cards stuck on version 0.3.0 on Beta Cluster - https://phabricator.wikimedia.org/T125182#1982991 (10greg) [22:34:21] thcipriani: ^ [22:34:29] the gerrit magic is controlled by the gitmodules config. I don't actually know how beta cluster handles tracking the submodules [22:34:56] I'm guessing this has to do with special:version stuff. I think there were other instances of this earlier this week. [22:35:00] with the production deployments the extension versions are controlled by make-wmf-branch [22:35:02] * thcipriani looks [22:35:14] special:version stuff? [22:35:50] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 10MediaWiki-extensions-Cards: Cards stuck on version 0.3.0 on Beta Cluster - https://phabricator.wikimedia.org/T125182#1982994 (10mmodell) The black magic in gerrit is controlled by the git module configuration. If the submodule has "branch=" set to some b... [22:36:02] oh hi [22:36:07] hai [22:36:08] special:version wasn't showing the latest version for something else earlier in the week [22:36:15] I just came to ask if anyone knows how mediawiki/extensions.git is updated [22:36:31] related to bug that greg-g is flagging [22:36:32] bd808: I just replied in the task [22:36:46] it's based on the .gitmodules having branch= setting [22:36:58] IIRC it's updated by gerrit and ^ [22:37:04] and it's not 100% reliable, gerrit only does it most of the time [22:37:27] does not handle sub-sub-modules and sometimes it seems to just forget [22:37:28] jdlrobson: is that one of the extensions that recently changed from a dev branch to master? [22:37:32] yes [22:37:47] so the gitmodules in extensions.git needs to be updated, most likely? [22:38:29] I didn't realize the beta cluster used extensions.git to track the extensions... [22:38:42] yeah. that's what gets cloned [22:38:49] the whole ball of everything [22:39:40] mmmm the ball of everything. [22:42:11] 7Blocked-on-RelEng, 6Phabricator: iridium:/var/log/phd/daemons.log is growing too much (took 20% of filesystem space) - https://phabricator.wikimedia.org/T124651#1983001 (10mmodell) @jcrespo: Couldn't we just make /var into a symlink to /srv/var or would that cause problems that I'm not aware of? Actually I c... [22:43:27] 7Blocked-on-RelEng, 6Phabricator: iridium:/var/log/phd/daemons.log is growing too much (took 20% of filesystem space) - https://phabricator.wikimedia.org/T124651#1983002 (10mmodell) I _could_ just move phd.log off /var, if that isn't some kind of violation of #ops policies / preferences. [22:45:13] all of the extensions submodules have "branch = ." [22:45:17] I don't know what that means [22:47:09] I asked hashar about this wednesday. https://review.openstack.org/Documentation/user-submodules.html#_defining_the_submodule_branch [22:58:44] will this impact what gets deployed next week? [22:58:59] or is it BC specific? [22:59:58] BC specific [23:00:00] what gets deployed is controlled by make-wmf-branch/config.json for now (though that is soon to be replaced) [23:00:16] but, I'm not a fan of deploying things that havne't really been on BC yet.. so hopefully this is fixed soon :) [23:00:17] that's in the release.git repo [23:03:42] so the extensions.git .gitmodules setting seems to imply that it would be tracking master in the cards submodule [23:04:07] which is the desired behavior, right? (not sure if I followed the discussion correctly) [23:08:00] okay that's good to know :) [23:08:04] so how can i fix this on the short term? [23:10:11] which branch is it supposed to be tracking? [23:10:45] extensions should be tracking master for Cards (looking at .gitmodules) which is correct, but it doesn't seem like it's getting updated (since it's not currently on the HEAD of any upstream branch). [23:11:01] also it's interesting that this is its last bump: https://github.com/wikimedia/mediawiki-extensions/commit/2b89a0e4c3392aebb867202cafeb73b820814bea [23:12:21] so I guess in the short term we update to the current head of cards master, then hope gerrit wakes up, track it via the ticket. Does that sound right?