[00:24:52] packagist is extremely slow for me right now, is there any alternative? [00:25:01] I tried the vendor repo already but it doesn't have dev dependencies [00:50:33] TimStarling: Override the repository? [00:50:34] https://getcomposer.org/doc/05-repositories.md [00:50:41] "Loading a package from a VCS repository" [00:50:52] But I guess you'd need to do it for all the dependancies, which sounds like a pita [00:52:22] Is it slow from everywhere? Or would it be quicker to run it on your vps and download the whole folder gzip? [00:58:56] 06Release-Engineering-Team, 06Project-Admins, 10ReleaseTaggerBot: Move all WMF-deploy-* projects to be milestones of ?? the "MW-1.XX-release-notes" projects? - https://phabricator.wikimedia.org/T152999#3090058 (10mmodell) 05Open>03Resolved a:03mmodell ok this is done. [01:04:10] solution was to go do something else for half an hour, it got there in the end [01:06:29] the whole set of package names specified in composer.json are only known to packagist.org [01:07:13] maybe you could mirror it via their API [01:09:22] a lot of them usually map 1:1 with a github repo [01:09:24] But not always [01:10:27] 65 [01:10:40] line starting with slash was eaten [01:10:57] that was ./composer.phar show | wc -l [01:11:29] this would have to be automated, I'm not going to figure out the git URL for 65 packages manually, even once [01:27:38] 06Release-Engineering-Team, 06Project-Admins, 10ReleaseTaggerBot: Move all WMF-deploy-* projects to be milestones of ?? the "MW-1.XX-release-notes" projects? - https://phabricator.wikimedia.org/T152999#3090089 (10Jdforrester-WMF) Thank you! [03:41:21] jenkins/zuul/nodepool seems to be messed up. Jobs in queue on jenkins for 10 hours. Zuul logging to gerrit patches that it is starting gate and submit but then not showing on the UI [03:59:04] Project selenium-MultimediaViewer » firefox,mediawiki,Linux,BrowserTests build #325: 04FAILURE in 2 min 59 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=mediawiki,PLATFORM=Linux,label=BrowserTests/325/ [06:51:33] Yippee, build fixed! [06:51:33] Project selenium-Wikibase » chrome,beta,Linux,BrowserTests build #295: 09FIXED in 2 hr 11 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/295/ [07:03:09] PROBLEM - Puppet run on deployment-imagescaler01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [07:45:46] 10Gerrit, 06Release-Engineering-Team, 06Labs, 06Operations, 07LDAP: Remove user gerrit2 from ldap - https://phabricator.wikimedia.org/T160122#3090315 (10MoritzMuehlenhoff) @demon Can you confirm that the user is no longer needed? [07:52:20] 10Browser-Tests-Infrastructure, 07Easy: Remove lines from Gemfile that are used by RVM - https://phabricator.wikimedia.org/T1331#23427 (10Harjotsingh) Hi I'm interested to work on this, Do I remove RVM on github and make a pull request or do I remove it and push to gerrit for review ? [08:47:15] The times mentioned in postmerge part looks worrying: https://integration.wikimedia.org/zuul/ [09:03:13] 10Browser-Tests-Infrastructure, 07Easy: Remove lines from Gemfile that are used by RVM - https://phabricator.wikimedia.org/T1331#3090517 (10hashar) @Harjotsingh Gerrit :-} The repositories on Github are usually just mirrors. [09:03:49] 10Browser-Tests-Infrastructure, 07Easy: Remove lines from Gemfile that are used by RVM - https://phabricator.wikimedia.org/T1331#3090518 (10hashar) Also you can usually reach both of us in the freenode IRC channel `#wikimedia-releng` though I am not connected this morning. [10:03:11] hi, I need to deploy a mw-config change for elastic5 (affects only deployment-prep), https://gerrit.wikimedia.org/r/#/c/342040/ [10:05:14] 10Browser-Tests-Infrastructure, 07Easy, 13Patch-For-Review: Remove lines from Gemfile that are used by RVM - https://phabricator.wikimedia.org/T1331#3090618 (10Harjotsingh) I don't know if it is the right way, but I added different commits for different extensions. Wanted to ask if it is right before adding... [10:05:23] 10Browser-Tests-Infrastructure, 07Easy, 13Patch-For-Review: Remove lines from Gemfile that are used by RVM - https://phabricator.wikimedia.org/T1331#3090619 (10Harjotsingh) a:03Harjotsingh [10:05:49] are there any objections? this patch is to fix search on deployment-prep [10:13:09] 10Browser-Tests-Infrastructure, 07Easy, 13Patch-For-Review: Remove lines from Gemfile that are used by RVM - https://phabricator.wikimedia.org/T1331#3090627 (10hashar) Excellent @Harjotsingh . I got the change for PoolCounter merged. Looks like you can do the others. The one for Wikidata should be done in W... [10:13:18] dcausse: bonjour [10:13:27] salut [10:13:46] je me suis connecté juste avant ta question me manque l'historique [10:13:50] que se passe t'il? [10:14:23] hashar: rien de grave, j'ai besoin d'activer une petite config pour réparer la recherche sur deployment-prep [10:14:39] GOGO !!! :] [10:14:48] je peux poser un +1 si ca peut te rassurer [10:15:23] si tu veux mais le patch est vraiement tout simple : https://gerrit.wikimedia.org/r/#/c/342040 :) [10:15:43] pour les patches n'affectant que les fichiers -labs.php [10:15:56] on est quasi 101% assuré que ca ne touche que beta [10:16:13] donc généralement pas besoin de permission spécifique [10:16:24] si ce n'est de faire un rebase sur les serveurs de prod [10:16:25] oui, mais je dois quand même lancer scap en prod pour éviter quelques alertes? [10:16:33] ou just rebase? [10:16:35] pour éviter que quelqu'un se demande pourquoi le patch n'a pas été poussé sur la prod [10:16:41] je ne fait que rebase sur tin oui [10:16:49] ok ca marche [10:16:51] certains font un scap sync-file en plus [10:17:04] mais vu que ca n'est pas utilisé par la prod, ce n'est pas vraiment nécessaire [10:17:23] ok, je fais çà, merci! [10:17:45] est ce que vous avez tenté l'upgrade d'elastic search sur beta? [10:18:08] oui c'est en place, il manque just cette config qu'on a oublié :/ [10:19:23] magique! [10:19:25] hashar: je rebase tin [10:19:35] vaut mieux l'oublier sur beta qu'en prod :] [10:19:41] oui :) [10:20:40] et si je comprend bien codfw aura ES5 [10:20:51] et un switch de config permet de passer l'un à l'autre non? [10:21:17] oui on teste la version mediawiki [10:21:32] on a fait comme ca pour l'upgrade elastic2 [10:21:50] pas mal [10:21:57] comme rien n'est compatible entre les deux versions utiliser le train nous aide pas mal [10:29:13] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T158996#3090641 (10mmodell) 05Open>03Resolved [11:03:33] should I manualy deploy mw-config changes from deployment-tin.deployment-prep.eqiad.wmflabs? I thought that such changes were deployed automatically in the beta cluster [11:08:15] ah I see some queued changes for beta-mediawiki-config-update-eqiad in zuul, that should be it, will wait [11:26:42] hmm, something must be blocked, some changes are pending for more than 11 hours [12:04:02] Which instance in Beta labs has wikishared db? I need to update index for contenttranslation DB (T159800) [12:04:03] T159800: Update DB index for T146450 - https://phabricator.wikimedia.org/T159800 [12:17:49] kart_: Well, there's only 2 DB hosts in beta at all... :P [12:18:01] deployment-db03 and deployment-db04 [12:18:27] Reedy: :) [12:18:33] https://noc.wikimedia.org/conf/highlight.php?file=db-labs.php [12:18:41] Thanks. Just saw the list. [12:19:36] Reedy: it doesn't have mwscript command? [12:19:44] db03 [12:19:49] Very unlikely [12:19:55] Same as the way they don't in production :P [12:19:59] try deployment-tin [12:20:02] 10Continuous-Integration-Config, 06Operations: Enforce reference to Phabricator task for all commits to modules/admin/data/data.yaml - https://phabricator.wikimedia.org/T142827#3090778 (10hashar) [12:22:48] Reedy: yes. I got these errors in deployment-tin now: http://pastebin.com/mFxBiH8k [12:23:18] did your update work though? [12:23:37] Reedy: it seems --wikidb is not needed. [12:23:56] I can get mysql prompt now [12:28:24] Reedy: I can use !log from #wikimedia-operations for Beta changes, right? [12:28:46] You can just !log in here and it goes to the releng sal :) [12:29:39] !log Beta: T159800: Update DB index for T146450 [12:29:42] cool [12:29:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:29:45] T159800: Update DB index for T146450 - https://phabricator.wikimedia.org/T159800 [12:29:46] T146450: Unable to translate an article for which another user has a deleted draft - https://phabricator.wikimedia.org/T146450 [12:29:58] Thanks! [13:00:05] 10Browser-Tests-Infrastructure, 07Easy, 05MW-1.29-release (WMF-deploy-2017-03-14_(1.29.0-wmf.16)), 13Patch-For-Review: Remove lines from Gemfile that are used by RVM - https://phabricator.wikimedia.org/T1331#3090832 (10Harjotsingh) @hashar Some file show in https://github.com/search?q=org%3Awikimedia+%22ru... [13:27:12] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure: Zuul postmerge blocked on beta-mediawiki-config-update-eqiad - https://phabricator.wikimedia.org/T160168#3090888 (10dcausse) [13:30:02] hashar: ^^^ [13:30:45] dcausse: Zppix ahhhh deadlock of some sort [13:30:49] gotta restart jenkins [13:31:03] let me know when/if you do [13:39:16] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure: Zuul postmerge blocked on beta-mediawiki-config-update-eqiad - https://phabricator.wikimedia.org/T160168#3090888 (10hashar) Jenkins had some deadlock issue overloading it + another issue with the deployment-tin Jenkins slave. I have rest... [13:47:23] hashar: thanks! all good, search if ok on beta now [13:48:03] dcausse: scap still running though [13:48:09] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure: Zuul postmerge blocked on beta-mediawiki-config-update-eqiad - https://phabricator.wikimedia.org/T160168#3090936 (10dcausse) 05Open>03Resolved a:03hashar Thanks! [13:48:10] ah ok :) [13:48:11] but yeah maybe the config file got deployed in a previous scap run [13:48:14] root cause is [13:48:17] blame java [13:48:20] hehe :) [13:48:24] well really blame some kind of deadlock [13:48:36] and I have absolutely no idea how to debug deadlock in java. So I end up restarting it [13:49:08] PROBLEM - Puppet run on buildlog is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:49:11] yes... we could take a stackdump but without knowing jenkins internals that will be hard to read [13:50:05] yup [13:50:11] though one can ready the source and try to guess [13:50:17] unzip the jar? ;P [13:50:21] :) [13:50:24] then I can't figure out which resource is holding the lock [13:50:48] I need access to the secret lair of wikimedia javaist [13:50:56] and trick some of them to look at jenkins [13:51:11] hashar: is it the java process owned by jenkins on deployment-tin? [13:51:20] jenkins-deploy [13:51:25] ok [13:51:26] for historical / bad reason [13:51:30] but the lock is on the master [13:51:37] ah [13:51:41] hashar: perhaps having some sort of ichinga or shinken alert when/if it appears jenkins is in a deadlock to maybe see if theres a pattern related to uptime and/or time of day, etc [13:51:46] basically the idea is the master ssh to deployment-tin as jenkins-deploy [13:51:55] then there are a bunch of sudo rules to let it run command as mwdeploy [13:52:11] Zppix: it is more complicated than that unfortunately [13:52:43] dcausse: but eventually I will get a standalone Jenkins server on an instance in beta [13:53:02] hashar: i mean zuul could report to ichinga or something that if a job(s) is queued for x time then alert -releng [13:53:41] ahh [13:53:44] yeah maybe that is doable [13:53:57] hashar: ok, feel free to ping me when it happens again I'll try to investigate, I'm very pessimistic on my chances to find the cause but I can try [13:53:59] it could be a starting point [13:55:20] or do a failsafe to where if beta job isnt ran in x time then remove it from zuul's queue and restart jenkins instance on deployment.tin (or whatever beta runs on for jenkins) then requeue it hashar [14:04:40] 10Browser-Tests-Infrastructure, 07Easy, 05MW-1.29-release (WMF-deploy-2017-03-14_(1.29.0-wmf.16)), 13Patch-For-Review: Remove lines from Gemfile that are used by RVM - https://phabricator.wikimedia.org/T1331#3090955 (10zeljkofilipin) @Harjotsingh only remove the mentioned two lines for Gemfiles. Cleaning u... [14:08:00] PROBLEM - zuul_merger_service_running on contint2001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-merger [14:08:24] and theres the complaint [14:11:52] 06Release-Engineering-Team, 10Phabricator: Running bin/search init fails with Call to undefined method PhabricatorSearchCluster::getDisplayName() - https://phabricator.wikimedia.org/T160085#3090961 (10mmodell) 05Open>03Resolved [14:13:00] RECOVERY - zuul_merger_service_running on contint2001 is OK: PROCS OK: 1 process with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-merger [14:13:59] PROBLEM - git_daemon_running on contint2001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/lib/git-core/git-daemon --user [14:17:07] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:17:09] PROBLEM - git_daemon_running on contint1001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/lib/git-core/git-daemon --user [14:19:00] PROBLEM - zuul_merger_service_running on contint2001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-merger [14:20:00] RECOVERY - zuul_merger_service_running on contint2001 is OK: PROCS OK: 1 process with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-merger [14:30:30] ACKNOWLEDGEMENT - git_daemon_running on contint1001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/lib/git-core/git-daemon --user amusso I broke it. Fix is https://gerrit.wikimedia.org/r/#/c/342212/ [14:31:08] ACKNOWLEDGEMENT - git_daemon_running on contint2001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/lib/git-core/git-daemon --user amusso https://gerrit.wikimedia.org/r/#/c/342212/ [14:33:43] Yippee, build fixed! [14:33:44] Project selenium-WikiLove » firefox,beta,Linux,BrowserTests build #327: 09FIXED in 1 min 43 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/327/ [14:49:13] RECOVERY - git_daemon_running on contint1001 is OK: PROCS OK: 1 process with regex args ^/usr/lib/git-core/git-daemon [14:51:35] Hey zeljkof! [14:51:45] hi Tobi_WMDE_SW! [14:51:48] Thanks for your effort in updating to selenium 3 yay! [14:52:06] I've poked a bit at the wikibase tests lately [14:52:56] zeljkof: one problem there was, that we have several spots with something like e.g. expect(on(ItemPage).snak_value_input_field.when_visible).to be_visible [14:53:44] we have this to wait at least for a small timeout before checking if a certain element is there [14:54:11] but it seems that in one of the later page_object versions, when_visible is working a bit different.. [14:55:34] changing when_visible to when_present fixed most of the problems there [14:56:12] zeljkof: but there is also https://github.com/cheezy/page-object/issues/417 which worries me a bit and which goes in a similar direction [14:56:12] ok [14:56:49] I have to go in a few minutes [14:57:00] I think I will need to spend a bit more time investigating the wikibase tests when I find time [14:57:03] nP [14:57:06] is there a problem? besides the reported issue? [14:57:50] zeljkof: no particular one. :) I'll have a look again next week when I find time for it and see how far I get [14:58:15] at least the number of failures went a bit down now [14:58:17] :) [14:58:21] Tobi_WMDE_SW: let me know if you need help with something [14:58:30] * zeljkof has to go [14:58:30] zeljkof: will do! [14:58:37] have a nice weekend! [15:00:33] (03PS1) 10Harjotsingh: Remove RVM lines from MediaWiki API from Ruby [ruby/api] - 10https://gerrit.wikimedia.org/r/342216 (https://phabricator.wikimedia.org/T1331) [15:03:02] RECOVERY - git_daemon_running on contint2001 is OK: PROCS OK: 1 process with regex args ^/usr/lib/git-core/git-daemon [15:22:06] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [15:28:03] PROBLEM - git_daemon_running on contint2001 is CRITICAL: PROCS CRITICAL: 2 processes with regex args ^/usr/lib/git-core/git-daemon [15:29:02] RECOVERY - git_daemon_running on contint2001 is OK: PROCS OK: 1 process with regex args ^/usr/lib/git-core/git-daemon [15:36:46] 2 processes grrr [15:58:42] 06Release-Engineering-Team, 10Phabricator: Phabricator daemons do not stop - https://phabricator.wikimedia.org/T160167#3091230 (10Paladox) [16:26:21] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review: zuul-merger git-daemon process is not started properly by systemd ? - https://phabricator.wikimedia.org/T157785#3091254 (10hashar) 05Open>03Resolved a:03hashar Solved. We no more use `git-daemon-sysvinit` and now r... [16:28:08] 06Release-Engineering-Team, 10Phabricator: Phabricator daemons do not stop - https://phabricator.wikimedia.org/T160167#3091258 (10mmodell) I think we need to change the systemd unit file to Type=forking as in https://secure.phabricator.com/T4181#204679 [16:29:13] 06Release-Engineering-Team, 10Phabricator: Phabricator daemons do not stop - https://phabricator.wikimedia.org/T160167#3091259 (10Paladox) @demon it is already type=forking also i tested locally without any systemd files as a mac does not support systemd. SO i ran manually bin/phd [16:29:44] Woops wrong user ^^ thats meant to be twentyafterfour [16:31:33] twentyafterfour: paladox usually you don't need to use forking [16:31:42] oh [16:31:48] I have no clue how that "daemon" works,but most probably you can just run the underlying command [16:31:55] in foreground mode [16:31:58] it uses php pcntl [16:32:08] example: /bin/bash -c 'sleep 55000' [16:32:18] oh [16:32:19] systemd takes care of finding the PID of the process, runs it in the background [16:32:33] and a systemctl stop would send SIGTERM to the PID systemd is tracking [16:32:52] Yep, i did try running the command in bash like bin/phd including on my mac [16:33:08] well that depends what bin/phd is doing [16:33:12] it seems to take 10+ mins for it to stop [16:33:13] probably it is forking behind the scene [16:33:17] oh [16:33:19] maybe you can make it to run in foreground ? [16:33:33] Oh not sure [16:34:22] hashar: usually, maybe not, but phd is not like the usual daemon [16:34:48] phd implements it's own supervisor functionality [16:34:58] it's not a single daemon it's a daemon monitoring service [16:35:20] written in PHP ? :D [16:35:51] yes, phabricator reinvents every wheel in php [16:36:30] they really put the PH in everything [16:36:48] okkkk [16:36:56] oh [16:36:59] i have a log [16:37:09] twentyafterfour my log is pretty huge all from today [16:37:13] do you have the output of "systemctl status phd" somewhere ? [16:37:13] full of daemon errors [16:37:23] [10-Mar-2017 16:35:35 UTC] [2017-03-10 16:35:35] ERROR 2: fprintf(): 3 is not a valid stream resource at [/Users/patrickmulhall/sites/libphutil/src/console/PhutilConsoleServer.php:155] [16:37:26] you should see multiple process. And definitely try type=forking [16:37:42] paladox: on which instance are you running it? [16:37:45] systemd will notice the ExecStart command has forked and track the fork PID [16:37:50] Its from my mac [16:37:53] oh [16:37:56] i can look on instance [16:38:00] wait systemd on a mac? [16:38:04] no [16:38:11] on Windows! [16:38:17] lol [16:38:19] theres no systemd on mac [16:38:20] lol [16:39:43] launchctl + systemd + bootcamp + parallels + windows 10 + ubuntu + .... [16:40:15] paladox: I don't think phd is designed to work on mac os [16:40:24] yeh it is [16:40:29] as php pcntl is on mac [16:40:37] Also i found the same log on the test instances [16:40:44] [10-Mar-2017 16:39:42 UTC] [2017-03-10 16:39:42] ERROR 2: fprintf(): 3 is not a valid stream resource at [/srv/deployment/phabricator/deployment/libphutil/src/console/PhutilConsoleServer.php:155] [16:41:13] https://phabricator.wikimedia.org/P5049 [16:42:19] twentyafterfour ^^ [16:43:26] Hmm i found this [16:43:26] https://secure.phabricator.com/T11486 [16:45:05] try it on a labs instance ? [16:45:29] That is on labs [16:45:42] ahhhh [16:47:47] it is trying to write to an invalid / closed file descriptor # 3 [16:47:55] who knows how that is defined in php [16:48:18] Oh not sure [16:48:27] though the logs are full of those errors from march 10 [16:50:31] oh wow [16:50:45] twentyafterfour the log has gown by 99% in the last 20-30mins [16:51:22] scrolling down to 16:39:40 it shows i am at 1& [16:51:24] 1% [16:52:31] 06Release-Engineering-Team, 10Phabricator: Phabricator daemons do not stop - https://phabricator.wikimedia.org/T160167#3091291 (10Paladox) Investigating it further and have found when you try to stop phd daemons it grows the logs by as much as 99% in 20-30mins. See https://phabricator.wikimedia.org/P5049 [16:52:56] (03CR) 10Hashar: [C: 032] Remove RVM lines from MediaWiki API from Ruby [ruby/api] - 10https://gerrit.wikimedia.org/r/342216 (https://phabricator.wikimedia.org/T1331) (owner: 10Harjotsingh) [16:53:35] (03Merged) 10jenkins-bot: Remove RVM lines from MediaWiki API from Ruby [ruby/api] - 10https://gerrit.wikimedia.org/r/342216 (https://phabricator.wikimedia.org/T1331) (owner: 10Harjotsingh) [16:53:57] (03CR) 10jenkins-bot: Remove RVM lines from MediaWiki API from Ruby [ruby/api] - 10https://gerrit.wikimedia.org/r/342216 (https://phabricator.wikimedia.org/T1331) (owner: 10Harjotsingh) [17:05:01] 10Gerrit, 06Release-Engineering-Team, 06Labs, 06Operations, 07LDAP: Remove user gerrit2 from ldap - https://phabricator.wikimedia.org/T160122#3091314 (10demon) >>! In T160122#3090315, @MoritzMuehlenhoff wrote: > @demon Can you confirm that the user is no longer needed? For a little history, it **used**... [17:09:25] twentyafterfour there was recent updates to the daemon i belive [17:09:46] https://github.com/wikimedia/phabricator-libphutil/commit/be546154255ccfcd5db10eabdabcadf1d8ea73b7 [17:09:48] ah [17:10:03] twentyafterfour it is showing the third param as $context = null; [17:12:21] i guess it has allways been null [17:16:11] I am out have a good week-end [17:20:07] Shoulden't there be break points here [17:20:07] https://github.com/wikimedia/phabricator-libphutil/blob/a5fa31459c1ffa6157dd1d6783073050fca25bc7/src/console/PhutilConsoleServer.php#L13 [17:29:21] paladox: You don't need break if you're doing return [17:29:28] Oh [17:29:30] thanks [17:29:45] I belive it's the new daemon update they did [17:30:58] 10Gerrit, 06Release-Engineering-Team, 06Labs, 06Operations, 07LDAP: Remove user gerrit2 from ldap - https://phabricator.wikimedia.org/T160122#3091387 (10MoritzMuehlenhoff) Ok, I'll make the change on Tuesday when you're around (since Monday is a holiday for US staff) [17:31:04] 06Release-Engineering-Team, 10Phabricator: Phabricator daemons do not stop - https://phabricator.wikimedia.org/T160167#3091388 (10Dzahn) If this result happened on a system not even running systemd, AND you tested the systemd unit file on labs before we merged this.. ist it even relevant? [17:32:55] 06Release-Engineering-Team, 10Phabricator: Phabricator daemons do not stop - https://phabricator.wikimedia.org/T160167#3091390 (10Paladox) I tested it but i didn't check the daemon status on pabricator's ui. I used the systemd status to tell me if it stopped it or started it. [17:38:12] 06Release-Engineering-Team, 10Phabricator: Phabricator daemons do not stop - https://phabricator.wikimedia.org/T160167#3091419 (10Paladox) I upgraded phabricator test instance yesturday to match prod's update [17:40:54] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 05MW-1.29-release (WMF-deploy-2017-02-28_(1.29.0-wmf.14)), 05MW-1.29-release-notes, and 3 others: Update Ruby tests to Selenium 3 - https://phabricator.wikimedia.org/T158074#3091424 (10Jdlrobson) [17:42:15] 06Release-Engineering-Team, 10Phabricator: Phabricator daemons do not stop - https://phabricator.wikimedia.org/T160167#3091439 (10Paladox) [17:42:44] 10Gerrit, 06Release-Engineering-Team, 06Labs, 06Operations, 07LDAP: Remove user gerrit2 from ldap - https://phabricator.wikimedia.org/T160122#3091441 (10demon) p:05High>03Normal Sounds good [17:44:36] 06Release-Engineering-Team, 10Phabricator: Phabricator daemons do not stop - https://phabricator.wikimedia.org/T160167#3091448 (10Paladox) [17:45:59] 06Release-Engineering-Team, 10Phabricator: Phabricator daemons do not stop - https://phabricator.wikimedia.org/T160167#3090875 (10Paladox) I have an idea on what's causing that error in the logs, this https://github.com/wikimedia/phabricator-libphutil/commit/be546154255ccfcd5db10eabdabcadf1d8ea73b7 [18:01:22] 06Release-Engineering-Team, 10Phabricator: Phabricator daemons do not stop - https://phabricator.wikimedia.org/T160167#3091499 (10Paladox) I see this in the log now [10-Mar-2017 17:58:14 UTC] [2017-03-10 17:58:14] ERROR 2: posix_isatty(): 2 is not a valid stream resource at [/srv/deployment/phabricator/deploy... [18:18:26] 06Release-Engineering-Team, 10Phabricator: Phabricator daemons do not stop - https://phabricator.wikimedia.org/T160167#3091578 (10Dzahn) >>! In T160167#3091390, @Paladox wrote: > I tested it but i didn't check the daemon status on pabricator's ui. I used the systemd status to tell me if it stopped it or starte... [18:19:37] 06Release-Engineering-Team, 10Phabricator: Phabricator daemons do not stop - https://phabricator.wikimedia.org/T160167#3091580 (10Paladox) Yes it seems so, but it seems systemd only checks that the pid is deleted on stop which it is. [19:42:59] marxarelli|afk: how did the docker registry stuff go? [20:03:14] 06Release-Engineering-Team, 10Phabricator: Phabricator daemons do not stop - https://phabricator.wikimedia.org/T160167#3092002 (10Paladox) It's this https://github.com/wikimedia/phabricator-libphutil/commit/2e26268fb8540589afd6458f92dfb28a24bef71d commit that caused the problem as checking out the one previous... [20:05:20] (03PS1) 10Mholloway: Hygiene: Remove Jenkins plugin-driven install-android-sdk builder [integration/config] - 10https://gerrit.wikimedia.org/r/342263 [20:20:58] (03PS1) 10Mholloway: Hygiene: Get and display lint reports from /app/build/reports [integration/config] - 10https://gerrit.wikimedia.org/r/342265 [20:41:47] Yippee, build fixed! [20:41:48] Project selenium-Echo » chrome,beta,Linux,BrowserTests build #329: 09FIXED in 47 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/329/ [20:41:54] Yippee, build fixed! [20:41:54] Project selenium-Echo » firefox,beta,Linux,BrowserTests build #329: 09FIXED in 53 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/329/ [20:44:56] I found the problem commit [20:45:00] it's this one [20:45:01] https://secure.phabricator.com/rPHU0625e4d28b16cba8b0154e49dc31845559b4e704 [20:45:05] twentyafterfour ^^ [20:45:11] hashar i found the problem [20:45:11] https://secure.phabricator.com/rPHU0625e4d28b16cba8b0154e49dc31845559b4e704 [20:53:15] paladox: how is that causing problems? [20:53:26] It is keeping at least one daemon [20:53:39] tested it by revert to the prevous commit before that one [20:53:50] and saw no daemons running after running bin/phd stop [20:54:07] after applying that patch i saw at least one daemon running after running bin/phd stop [20:55:46] 10Beta-Cluster-Infrastructure, 06Labs, 13Patch-For-Review: deployment-upload won't start, upload.beta.wmflabs.org down - https://phabricator.wikimedia.org/T131322#3092127 (10Andrew) [20:57:49] twentyafterfour ^^ [21:09:06] twentyafterfour hibernate should not happen if you stop the daemons [21:13:05] paladox: `service phd stop` works on phabricator.phabricator.eqiad.wmflabs [21:13:10] Yes [21:13:15] but go to the daemon [21:13:18] status page [21:13:36] https://phab-01.wmflabs.org/daemon/ [21:13:39] Shows as running [21:13:55] 983 phabricator 2638 PhabricatorTaskmasterDaemon Running Fri, Mar 10, 9:12 PM [21:13:55] 982 phabricator 2638 PhabricatorTriggerDaemon Waiting Fri, Mar 10, 9:12 PM [21:13:55] 981 phabricator 2638 PhabricatorRepositoryPullLocalDaemon Running Fri, Mar 10, 9:12 PM [21:13:55] 979 phabricator 26636 PhabricatorTriggerDaemon Running Fri, Mar 10, 6:24 PM [21:14:54] twentyafterfour ^^ [21:15:25] now it shows [21:15:26] 982 phabricator 2638 PhabricatorTriggerDaemon Running Fri, Mar 10, 9:12 PM [21:15:26] 979 phabricator 26636 PhabricatorTriggerDaemon Running Fri, Mar 10, 6:24 PM [21:15:59] that isn't really a problem though [21:16:20] Oh [21:16:35] but if you do bin/phd stop then bin/phd start [21:16:41] it will start alot of daemons [21:16:54] as it would not have stopped [21:17:03] !pastebin | paladox [21:17:12] ok [21:17:25] ty wm-bot [21:18:54] ah [21:18:59] twentyafterfour found it [21:19:00] paladox: those daemons are ghosts in the UI [21:19:00] https://secure.phabricator.com/rP6f50729a91710f21ec8e323b27c8624303043410 [21:19:04] yep [21:19:11] $pool_size = pht('(Pool: %s)', idx($daemon, 'pool', 1)); [21:19:21] Didnt need a paste bin as it's a one liner [21:20:54] Seems to have alot of ghosts in the ui [21:20:55] after stopping them [21:22:19] paladox: don't they go away after a while? [21:22:34] Yes after a while [21:22:48] I don't see it as a problem then really... [21:22:54] oh ok [21:23:11] it's always had problems with ghosts occasionally, never worried me any [21:24:54] Oh [21:24:59] twentyafterfour should we remove search.elastic.version config's from the local.json? [21:40:04] yeah [21:42:15] :) [21:42:16] ok [21:42:17] done [21:42:28] twentyafterfour https://gerrit.wikimedia.org/r/342275 [21:52:14] Does vagrant have a replacement for mwscript ? [21:52:27] i need to execute some maintenance scripts on a vagrant instance and mwscript is not available [21:52:29] jdlrobson: it has mwscript [21:52:44] are you actually inside the vm? [21:52:53] bd808: yeh i sshed in and i am getting command not found [21:52:59] hmmm... [21:53:06] even when i sudo su mwvagrant [21:53:25] you should be the 'vagrant' user inside the vm by default [21:53:34] and mwscript should be in its $PATH [21:53:49] existing but not in PATH? [21:54:10] jdlrobson: it should be at /usr/local/bin/mwscript [21:54:13] maybe "locate mwscript" if locate is installed [21:54:30] i see an mwvagrant there but not mwscript [21:54:43] it's not anywhere to be found [21:54:57] anyway i can install it? [21:55:13] you aren't inside the mv-vagrant managed vm then I bet. You ssh'ed to a Labs instance? [21:55:36] if so then you still need to cd /srv/mediawiki-vagrant and vagrant ssh [21:56:32] on a Labs instance the mw-vagrant role sets up an LXC container with the Vagrant managed stuff inside it [22:03:25] bd808: yup labs instance [22:03:50] ahhh vagrant inception [22:04:19] sort of, yes :) LXC in the OpenStack VM [22:04:35] bd808: so i've sshed inside the labs instnce [22:04:40] but i'm in an empty dir [22:04:52] cd /srv/mediawiki-vagrant [22:04:58] vagrant ssh [22:04:58] mediawiki-vagrant is not at /srv/ [22:05:08] hmmm.. [22:05:11] really old instance? [22:05:14] when i've sshed in where can i find mediawiki [22:05:22] (vagrant sshed) [22:05:28] ah. ok [22:05:38] mwscript should be in your $PATH now [22:05:43] yup got that [22:05:44] i see /srv/pages/wiki [22:05:55] cool beans [22:06:10] vagrant@mediawiki-vagrant:~$ cd /srv/ [22:06:15] centralauthtestimages/ frwiktionaryimages/ images/ mobileimages/ redis/ frimages/ heimages/ loginimages/ pages/ zhwikivoyageimages/ [22:06:26] mediawiki will be at /vagrant/mediawiki [22:06:36] ahhh [22:06:38] there we go [22:06:43] just like it would be in a vm on your laptop [22:06:47] thanks bd808 for being my vagrant spirit guide [22:06:56] np [22:10:20] hashar twentyafterfour i guess you doint know the macs have the apt-get commands. Found a package that provides it [22:10:28] so it is like debian [22:11:50] paladox: we all use homebrew [22:12:01] Oh yep i use homebrew [22:12:02] https://brew.sh [22:12:13] I am trying to figure out how to publish gerrit [22:12:14] there [22:12:20] per lucas feedback [22:13:37] look at the jenkins formula [22:13:47] Oh [22:13:49] the one for Gerrit should be pretty similar since boths are java apps [22:14:12] they most probably have an IRC channel here [22:14:23] Im trying to publish it from https://gerrit.googlesource.com/gerrit-installer/+/refs/heads/master/macOS/ [22:14:34] Which i maintain :) [22:14:47] oh with full compilation hehe [22:15:02] Yeh [22:15:10] I actually have someone that used it [22:15:23] so you end up maintaining an installer of Gerrit ? \O/ [22:16:09]