[00:21:11] 06Labs, 10Labs-Infrastructure: Install xml2 on labs - https://phabricator.wikimedia.org/T134146#2255706 (10scfc) Do you need `xml2` in your Labs project (you can install it there yourself) or in the #Tool-Labs project? [00:24:54] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Diffusion, 15User-bd808: Create application to manage Diffusion repositories for a Tool Labs project - https://phabricator.wikimedia.org/T133252#2258340 (10mmodell) [01:26:19] 10Quarry: Quarry should preserve the protocol on redirects - https://phabricator.wikimedia.org/T92600#2258455 (10Huji) 05Open>03declined Somehow in the last two weeks this is working correctly. Not sure if Yuvi_Panda has applied a patch that fixed this, or if its dependencies are updated. At any rate, I will... [01:41:34] !log deployment-prep ran package updates on deployment-parsoid06 so that exim4 would start so puppet will run [01:41:34] Please !log in #wikimedia-releng for beta cluster SAL [01:41:47] !log test test [01:41:48] Did you mean tools.test instead of test? [01:42:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, Master [01:42:30] test is not a valid project. [01:43:21] oh, there you are [01:50:36] 10Quarry: Quarry should preserve the protocol on redirects - https://phabricator.wikimedia.org/T92600#1115720 (10Legoktm) Probably was fixed by {T107627}? [04:25:31] Hi, I got a ToolLabs account yesterday and am trying to clone pywikibot and mw-core into it using git+gerrit [04:26:02] When running `git clone...`, the command gets stuck "Cloning into 'mw-core'..." for nearly an hour [04:26:57] After which *sometimes* it begins the cloning at ~10-20 KBps and twice not it just hung up at ~40 and 70% making me redo (and again wait for an hour before cloning starts ...) [04:28:39] I;m wondering if I'm doing something wrong ? [04:28:56] Or is this expected behaviour ? [04:43:43] AbdealiJK: the best would be if you can just copy/paste the last couple lines into a phabricator ticket. people are not awake at the same time.. and it will get more replies [04:43:57] Sure, will do that [04:43:58] Thanks :) [04:44:24] yw, i would say a little slowness for mw-core is expected but not that extreme :) [04:45:11] on first clone it's a bit [04:45:49] Hmm, this happens for pywikibot-core too [04:46:27] I'll create the phabricator and see if this is really expected :) (A little irritating since I've been trying to clone since the last 2-3 hrs now and it keeps failing) [04:46:55] AbdealiJK: it's a special time too [04:47:00] there is currently a migration going on [04:47:08] of the software hosting the git repos [04:47:15] Oh. I see, I didn't know that [04:47:17] so a ticket is best, yea [04:47:38] it was gitblit [04:47:44] and it should become phabricator itself [04:47:49] afaik [04:47:56] like .. these days [04:49:12] mutante, The phabricator diffusion tool ? [04:49:21] [13nagf] 15yuvipanda 04force-pushed 06docker from 14ca31076 to 14ab73221: 02https://github.com/wikimedia/nagf/commits/docker [04:49:22] 13nagf/06docker 14dca64fd 15YuviPanda: Support local dev with Docker... [04:49:22] 13nagf/06docker 142b3338d 15YuviPanda: Move pid file to /tmp to avoid permission issues [04:49:22] 13nagf/06docker 14ca0236e 15YuviPanda: Do not install recommended packages... [04:49:44] AbdealiJK: yea [04:49:59] well, "a phabricator solution" is what i really know [04:50:14] mutante, so gerrit is not going to be used anymore ? [04:50:28] that's the plan of the releng team, yes [04:50:35] at some point [04:50:37] I see, cool [04:50:48] this was about git.wm.org being gitblit for now [04:50:55] wikimedia/nagf#39 (docker - ab73221: YuviPanda) The build passed. - https://travis-ci.org/wikimedia/nagf/builds/127418667 [04:51:01] so that's 2 separate things [04:51:08] code review tool and the tool hosting git.wm.org [04:51:14] depending what you clone from [04:51:18] you could try the other one [04:51:50] Right, got it. hmm [04:54:57] 06Labs, 10Tool-Labs: ToolLabs git clone is really slow - https://phabricator.wikimedia.org/T134232#2258764 (10AbdealiJK) [04:55:00] also, you could try to use ssh and https protocols [04:55:02] and compare [04:55:38] I've been using https, because I didn't want to setup ssh keys and so on (Wasn't going to push into them) [04:56:32] *nod* yea [06:42:03] (03CR) 10Lokal Profil: "Is this crashing because old composer still gets run?" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286467 (owner: 10Jean-Frédéric) [06:43:40] 06Labs, 10Tool-Labs, 10pywikibot-core: ToolLabs git clone is really slow - https://phabricator.wikimedia.org/T134232#2258834 (10jayvdb) Just noting that https://www.mediawiki.org/wiki/Manual:Pywikibot/Installation/Labs does suggest doing a clone. Over at https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/De... [06:52:38] (03PS5) 10Lokal Profil: [NOT FULLY TESTED] Better support for sister projects [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) [06:53:09] (03CR) 10Lokal Profil: "This need to be rebased and rechecked once https://gerrit.wikimedia.org/r/#/c/286467/ is resolved." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) (owner: 10Lokal Profil) [06:53:32] (03CR) 10jenkins-bot: [V: 04-1] [NOT FULLY TESTED] Better support for sister projects [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) (owner: 10Lokal Profil) [06:54:01] PROBLEM - Puppet run on tools-exec-1219 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:26:41] (03CR) 10Hashar: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286467 (owner: 10Jean-Frédéric) [07:27:41] (03CR) 10Hashar: "The HHVM job is gone T134207 :-}" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286467 (owner: 10Jean-Frédéric) [07:28:26] (03CR) 10Hashar: [C: 031] Downgrade requirement of phpunit from 5.3.x to 4.8.x [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286467 (owner: 10Jean-Frédéric) [07:33:40] 06Labs, 10Tool-Labs, 10pywikibot-core: ToolLabs git clone is really slow - https://phabricator.wikimedia.org/T134232#2258875 (10AbdealiJK) The above was done using `https://`. I was not planning on pushing from the Tools Lab and was only planning on pulling when I make a change on my local system. Over at I... [07:34:01] RECOVERY - Puppet run on tools-exec-1219 is OK: OK: Less than 1.00% above the threshold [0.0] [07:44:29] 06Labs, 10Tool-Labs, 10pywikibot-core: ToolLabs git clone is really slow - https://phabricator.wikimedia.org/T134232#2258879 (10AbdealiJK) I finally got mediawiki-core to clone successfully. I times it using the `time` command and here is the data: ``` $ time git clone --recursive https://gerrit.wikimedia.or... [08:38:05] 06Labs, 06Developer-Relations, 06WMF-Legal: Provide an easy way for Tool Labs tools to expose their source code - https://phabricator.wikimedia.org/T102081#2258985 (10Qgil) [08:39:00] 06Labs, 06Developer-Relations, 06WMF-Legal: Make sure tools can be taken over after they are abandoned - https://phabricator.wikimedia.org/T102066#2258989 (10Qgil) [08:42:39] 10Tool-Labs-tools-Other, 13Patch-For-Review: tools.persondata 504's (sockets disabled, connection limit reached) - https://phabricator.wikimedia.org/T133697#2258995 (10valhallasw) 05Open>03Resolved a:03valhallasw This seems to have resolved the issue! [09:02:16] (03CR) 10Lokal Profil: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) (owner: 10Lokal Profil) [09:15:36] (03CR) 10Hashar: "You would need PHPUnit 4.8.x, if you rebase this change on top of https://gerrit.wikimedia.org/r/#/c/286467/ that should do it :-}" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) (owner: 10Lokal Profil) [09:40:12] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:44:09] ^ false alarm [09:44:12] * YuviPanda goes back to food [09:50:09] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 824120 bytes in 5.236 second response time [10:09:28] 06Labs, 06Developer-Relations, 06WMF-Legal: Provide an easy way for Tool Labs tools to expose their source code - https://phabricator.wikimedia.org/T102081#2259396 (10Qgil) [10:09:33] 06Labs, 06Developer-Relations, 06WMF-Legal: Make sure tools can be taken over after they are abandoned - https://phabricator.wikimedia.org/T102066#2259397 (10Qgil) [10:10:30] (03CR) 10Jean-Frédéric: [C: 032] Downgrade requirement of phpunit from 5.3.x to 4.8.x [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286467 (owner: 10Jean-Frédéric) [10:11:18] (03Merged) 10jenkins-bot: Downgrade requirement of phpunit from 5.3.x to 4.8.x [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286467 (owner: 10Jean-Frédéric) [10:12:37] (03CR) 10Jean-Frédéric: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) (owner: 10Lokal Profil) [10:51:12] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:56:10] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 824020 bytes in 7.451 second response time [11:22:50] (03CR) 10Jean-Frédéric: "This is now failing because of expectException()..." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) (owner: 10Lokal Profil) [11:39:12] (03PS1) 10Lokal Profil: Drop use of preg_replace() with the /e modifier [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286629 (https://phabricator.wikimedia.org/T134236) [11:39:42] (03PS6) 10Lokal Profil: [NOT FULLY TESTED] Better support for sister projects [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) [11:40:50] (03CR) 10jenkins-bot: [V: 04-1] [NOT FULLY TESTED] Better support for sister projects [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) (owner: 10Lokal Profil) [11:41:47] 06Labs, 10Tool-Labs, 13Patch-For-Review, 03Scap3 (Scap3-Adoption-Phase1): Setup a proper deployment strategy for Kubernetes - https://phabricator.wikimedia.org/T129311#2259646 (10yuvipanda) p:05High>03Normal It's been a month, and this actually continues to work out ok for us... [11:42:18] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Write a k8s admission controller to enforce that all containers running come from our private repository - https://phabricator.wikimedia.org/T133515#2259648 (10yuvipanda) I've eliminated all running dockerhub based containers now \o/ This was just moving a few straggle... [11:45:51] 06Labs, 10Tool-Labs, 10pywikibot-core: ToolLabs git clone is really slow - https://phabricator.wikimedia.org/T134232#2258764 (10Luke081515) I make that experience too: at a normal medium labs instance, git clone is about two or three times faster than at the toollabs-bastion. [12:37:46] (03CR) 10Lokal Profil: "So expectException is not supported in PhpUnit 4.8?... Any ideas suggestions for how to get around this? Guess I can possibly rewrite the " [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) (owner: 10Lokal Profil) [12:56:24] 06Labs, 10Labs-Sprint-115, 10Tool-Labs, 10labs-sprint-116, and 3 others: Attribute cache issue with NFS on Trusty - https://phabricator.wikimedia.org/T106170#2259860 (10JeanFred) [13:06:39] YuviPanda: I notice last few days we have gotten a lot of the labs shinken alerts for tools home page [13:06:41] any idea why? [13:06:59] (03CR) 10Jean-Frédéric: "> So expectException is not supported in PhpUnit 4.8?..." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) (owner: 10Lokal Profil) [13:07:17] gah didn't realize I was guest [13:08:10] chasemp: so my theory is that it's more sensitive than the icinga one, and fails due to either DNS or NFS spikes that are gone by the time I look at them [13:08:14] (03CR) 10Jean-Frédéric: [C: 032] Drop use of preg_replace() with the /e modifier [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286629 (https://phabricator.wikimedia.org/T134236) (owner: 10Lokal Profil) [13:08:15] it's a poor one [13:08:23] the other theory is that it's just shinken flaking out [13:08:28] but the load on that box is fairly low [13:08:28] ah [13:08:46] I'm not sure yeah, i can say I haven't caught it at all where shinken says down and icinga says up [13:08:48] as down I mean [13:08:49] there was a ticket to get rid of this check from icinga [13:08:54] right. I did earlier today. [13:08:58] interesting [13:09:00] so it was a 5min interval but it always is [13:09:07] because of shinken checking every 5min I guess [13:09:15] (03Merged) 10jenkins-bot: Drop use of preg_replace() with the /e modifier [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286629 (https://phabricator.wikimedia.org/T134236) (owner: 10Lokal Profil) [13:09:17] and I checkd NFS, was ok (there was the codfw backup running) [13:09:18] icinga checks every 1? [13:09:20] or so I thought [13:09:46] so NFS was under load but otherwise ok [13:09:59] yeah, and not unreasonable load either - a manual load of homepage worked [13:10:13] I honestly don't think that's the cause here (nfs) [13:10:25] but I don't have a good ide either [13:11:05] yeah, I don't think it's either [13:11:25] ok good to know I will try to catch it too [13:11:43] ok! [13:12:05] I killed all the things from our k8s that were from dockerhub. will try to kill gcr.io tonight [13:17:03] * YuviPanda starts writing self review stuff [13:27:02] tools web page is typically mostly impacted by server load in the webgrid host it's running on [13:32:15] tools.taxonbot: login {result WrongPass} - what's going on? [13:33:34] on tools-bastion-02 only [13:34:52] works for me: [13:34:53] luke081515@tools-bastion-02:~$ become luke081515bot-beta [13:34:53] tools.luke081515bot-beta@tools-bastion-02:~$ [13:36:57] no, login to dewiki by token [13:37:25] you mean the connection from bastion-02 to other wikis? [13:37:37] I can replicate *that* too [13:37:37] valhallasw`cloud - do you know anything about [13:37:59] doctaxon: I can't magically guess what's happening [13:38:28] Is the password correct...? [13:38:31] i cannot login to dewiki by token [13:38:35] yes it is [13:38:41] 'by token'?? [13:38:48] only tools-bastion-02 [13:39:02] -03 is okay [13:39:56] valhallasw`cloud: He means the normal API login with a login token [13:40:55] doctaxon: from the information you're providing my answer is 'I have no clue'. [13:41:06] I'm using the same logindata at -03 and -02, but when I'm trying to start a login at a wiki at -02, I get a 'wrong pass' eeror [13:41:09] *error [13:41:21] and I don't have time to debug the issue at the moment. [13:41:26] state the error line, Luke [13:42:07] doctaxon: My error line would not help you, because that's only the part at the script, where the scripts stop, if the bot is not logged in [13:42:20] it started about 13:30 UTC [13:45:14] PHP Fatal error: Uncaught exception 'Exception' with message 'login failed with message WrongPass' [13:45:47] at tools-bastion-02 only - any other labs admin here [13:47:34] doctaxon: to the best of our knowledge, nothing has changed on tools-bastion-02 specifically. In terms of configuration, all trusty tools bastions are the same. [13:48:11] but it works on 03 ?! [13:59:26] 06Labs, 10Tool-Labs: No API login with login token possible at tools-bastion-02 possible only - https://phabricator.wikimedia.org/T134262#2260068 (10doctaxon) [13:59:57] 06Labs, 10Tool-Labs: No API login with login token possible at tools-bastion-02 only - https://phabricator.wikimedia.org/T134262#2260086 (10doctaxon) [14:02:25] 06Labs, 10DBA, 13Patch-For-Review: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2260092 (10Andrew) I propose that we do the final switchover for this from 14:00 to 15:00 UTC on Thursday, 2015-05-05. [14:04:10] 06Labs, 10Tool-Labs: No API login with login token possible at tools-bastion-02 only - https://phabricator.wikimedia.org/T134262#2260097 (10doctaxon) errors reported by tools.taxonbot and tools.luke081515bot-beta [14:07:20] 06Labs, 10Tool-Labs: No API login with login token possible at tools-bastion-02 only - https://phabricator.wikimedia.org/T134262#2260117 (10Luke081515) I can reprocude it, that exception at the description above was from my script, it is: PHP Fatal error: Uncaught exception 'Exception' with message 'login f... [14:17:20] 06Labs, 10Tool-Labs: No API login with login token possible at tools-bastion-02 only - https://phabricator.wikimedia.org/T134262#2260131 (10valhallasw) @Anomie, do you have any clue what could cause this? The response from the API is not very verbose: 2016-05-03 14:02:40 api.py, 1980 in... [14:52:40] heyaaaa andrewbogott, do you know if/how I can log into grafana.wmflabs.org? [14:53:13] and, if i am sending stuff to statsd at labmon1001.eqiad.wmnet, will it show up there? [14:55:31] ottomata: I don't know — at the moment I can't log in either. [14:56:00] hm ok! [14:56:00] I don't use it [14:56:05] wonder who does... [14:56:09] maybe releng folks? [14:57:14] 06Labs, 10Tool-Labs: No API login with login token possible at tools-bastion-02 only - https://phabricator.wikimedia.org/T134262#2260242 (10Anomie) >>! In T134262#2260131, @valhallasw wrote: > Visiting en.wikipedia.org/wiki/Special:Login with links shows > > ``` > Username _____________________ > Passwo... [15:09:34] ottomata: that grafana is some kind of demo no one uses [15:09:37] we need to kil it [15:09:44] ah ok [15:09:50] people have been using the normal prod grafana w/ labs graphite as a source [15:10:02] I dont' know exactly how to do it :) but halfak got it working last week or so [15:10:09] maybe check out a few of their graphs sources [15:28:02] 06Labs, 10Tool-Labs: No API login with login token possible at tools-bastion-02 only - https://phabricator.wikimedia.org/T134262#2260342 (10Luke081515) Hm, ok, but why we get the "WrongPass" only at tools-bastion-02, while a login with the same script at tools-bastion-03 works? (Not only my PHP-Framework, for... [16:15:04] (03PS7) 10Lokal Profil: [NOT FULLY TESTED] Better support for sister projects [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) [16:15:12] (03CR) 10Lokal Profil: "Meanwhile I used this as an excuse to upgrade my machine to 5.6... now I can't downgrade it and so it's hard to check what works and what " [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) (owner: 10Lokal Profil) [16:25:50] Question: Is there a wiki where I can develop and test OAuth without having to go through a full approval process? [16:26:23] good question, there is a test wiki I hink, bd808?^ YuviPanda any idea? [16:26:32] Matthew_: yes, all of them. [16:26:40] Matthew_: OAuth is auto approved for the account that proposed it immediately. so if you propose a consumer as User:Matthew_ then User:Matthew_ can use it without getting approved [16:26:47] Matthew_: as long as you use the same user account as the one you requested the consumer with [16:26:48] so you only need approval for *other* people to use it [16:26:50] ^ what YuviPanda said [16:26:51] heh [16:27:01] Oh, I didn't know that. Thank you! [16:27:11] I assumed he meant "as a bot" [16:27:29] which I thought came w/ it's own restrictions but I guess...it depends on waht the activity is and oauth testing is no biggie [16:27:46] 06Labs: Update custom sink handlers for Designate Mitaka - https://phabricator.wikimedia.org/T134280#2260689 (10Andrew) [16:27:46] Yeah, the bot use case is really simple since you don't need to act as other users generally [16:28:07] so you just create a grant request as the bot account and you are ready to roll [16:29:16] chasemp: the bot restrictions are on-wiki based on community rules. For instance you will get slapped pretty quickly on enwiki for running a bot that does significant editing if you haven't asked for permission to do so [16:29:23] other wikis are more lax [16:29:35] but oauth doesn't enforce that at all [16:33:59] yeah [16:34:09] we ran a bunch on tawikisource, and since the only admin was the one running it... [16:35:46] I've only run a bot on enwiki so my view is skewed :) [16:36:18] chasemp, ? [16:37:25] CP678: backscroll you missed was a question about oauth and bots. See the logs for more [16:37:43] !logs [16:37:43] raw text: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs/ cute html: http://tools.wmflabs.org/wm-bot/logs/index.php?display=%23wikimedia-labs [16:38:02] * bd808 is busy "bragging" about his FY2015/16 accomplishments in self-review [16:40:30] Matthew_, one thing you need to know, but before I can clarify, is this for your bot, or for a tool? [16:40:50] 06Labs, 10Tool-Labs: No API login with login token possible at tools-bastion-02 only - https://phabricator.wikimedia.org/T134262#2260719 (10doctaxon) I also noticed, that jobs running under prompt are much faster at bastion-03 than at bastion-02. I suggest some more issues at bastion-02. [16:42:31] CP678: It's for a tool. [16:42:43] Ah okay. [16:42:59] Then you would create an OAuth consumer with your primary account. [16:43:14] OK. [16:43:53] Bot consumers need to be created with the bot account, as it generates the access tokens on the spot for owner-only consumers. [16:46:28] That makes sense. [17:11:17] (03PS1) 10Elukey: Add fake secret for statistics::web role. [labs/private] - 10https://gerrit.wikimedia.org/r/286687 [17:12:22] 06Labs, 10DBA, 13Patch-For-Review: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2260888 (10Andrew) [17:12:50] (03CR) 10Dzahn: [C: 031] Add fake secret for statistics::web role. [labs/private] - 10https://gerrit.wikimedia.org/r/286687 (owner: 10Elukey) [17:13:39] (03CR) 10Elukey: [C: 032 V: 032] Add fake secret for statistics::web role. [labs/private] - 10https://gerrit.wikimedia.org/r/286687 (owner: 10Elukey) [17:13:47] (03CR) 10Dzahn: "this should fix the compiler runs on stat1001/1004 things, yea" [labs/private] - 10https://gerrit.wikimedia.org/r/286687 (owner: 10Elukey) [17:16:54] <_joe_> hey heads up: I am going to merge a security change that will add some deny policy to imagemagick on the toollabs exec nodes [17:17:12] <_joe_> the same change has been applied succesfully in production FWIW [17:17:41] <_joe_> https://gerrit.wikimedia.org/r/#/c/286679/ FTR [17:18:44] <_joe_> YuviPanda, valhallasw`cloud I don't expect any breakage, but in case you know which patch to revert [17:19:27] <_joe_> I strongly recommend to do that only if extremely important, this patch is security-related [17:20:55] PROBLEM - Host tools-worker-1011 is DOWN: PING CRITICAL - Packet loss = 100% [17:26:18] _joe_: thanks! [17:30:01] 06Labs, 10Tool-Labs, 06Security-Team: Procure *.tools.wmflabs.org certificate - https://phabricator.wikimedia.org/T130649#2260977 (10RobH) [17:45:00] 10Tool-Labs-tools-Other, 06Community-Tech, 07I18n: Add i18n JavaScript API to Pageviews Analysis - https://phabricator.wikimedia.org/T133766#2242145 (10Nikerabbit) What is the problem? Plural parsing in JavaScript is a solved problem and available in at least two ways: jquery.i18n and MediaWiki's jqueryMsg. [17:48:05] 06Labs, 06Team-Practices, 07Privacy: http://hatjitsu.wmflabs.org loads resources from numerous 3rd party sites - https://phabricator.wikimedia.org/T134288#2261019 (10bd808) [17:48:23] 06Labs, 06Team-Practices, 07Privacy: http://hatjitsu.wmflabs.org loads resources from numerous 3rd party sites - https://phabricator.wikimedia.org/T134288#2261032 (10bd808) [17:48:25] 06Labs, 06WMF-Legal, 07Epic, 07Privacy: [EPIC] Protect end-user privacy by restricting non-consentual third-party browser interactions - https://phabricator.wikimedia.org/T133919#2261031 (10bd808) [17:50:00] 10Tool-Labs-tools-Other, 06Community-Tech, 07I18n: Add i18n JavaScript API to Pageviews Analysis - https://phabricator.wikimedia.org/T133766#2261035 (10MusikAnimal) @Nikerabbit but do those hook up to translatewiki or do we need a new set of messages? [17:52:22] 10Tool-Labs-tools-Other, 06Community-Tech, 07I18n: Add i18n JavaScript API to Pageviews Analysis - https://phabricator.wikimedia.org/T133766#2261042 (10Nikerabbit) What do you mean with //hook up//? You get the strings in JSON files from translatewiki.net, and then you can use these libraries to parse them b... [17:55:10] 06Labs, 06Team-Practices, 07Privacy: http://hatjitsu.wmflabs.org loads resources from numerous 3rd party sites - https://phabricator.wikimedia.org/T134288#2261047 (10bd808) This is hosted on the jitsu.eqiad.wmflabs instance in the mobile project. [18:09:35] 06Labs, 10Tool-Labs: No API login with login token possible at tools-bastion-02 only - https://phabricator.wikimedia.org/T134262#2261146 (10Anomie) >>! In T134262#2260342, @Luke081515 wrote: > Hm, ok, but why we get the "WrongPass" only at tools-bastion-02, while a login with the same script at tools-bastion-0... [18:10:13] 10Tool-Labs-tools-Other, 06Community-Tech, 07I18n: Add i18n JavaScript API to Pageviews Analysis - https://phabricator.wikimedia.org/T133766#2261160 (10MusikAnimal) Got it! The issue here is we need to evaluate magic words, which I for some reason thought was limited to the Intuition library (PHP). I figured... [18:12:19] 06Labs, 06Team-Practices, 07Privacy: http://hatjitsu.wmflabs.org loads resources from numerous 3rd party sites - https://phabricator.wikimedia.org/T134288#2261174 (10bd808) All of this may come in via the https://github.com/niftylettuce/express-cdn module that is required by https://github.com/richarcher/Hat... [18:12:24] 10Tool-Labs-tools-Other, 06Community-Tech, 07I18n: Add jQuery i18n to Pageviews Analysis - https://phabricator.wikimedia.org/T133766#2261176 (10MusikAnimal) [18:24:18] 06Labs, 06Team-Practices, 07Privacy: http://hatjitsu.wmflabs.org loads resources from numerous 3rd party sites - https://phabricator.wikimedia.org/T134288#2261228 (10MaxSem) Hatjitsu is abandonware and has no updates since it was installed. Furthermore, my current team is not using it, so I'm washing my hand... [18:25:31] 06Labs, 10Tool-Labs: No API login with login token possible at tools-bastion-02 only - https://phabricator.wikimedia.org/T134262#2261233 (10doctaxon) >>! In T134262#2261146, @Anomie wrote: > Because ConfirmEdit decides whether a captcha is needed based on IP and/or username. Someone must have done something fr... [18:30:37] 06Labs, 06Team-Practices, 07Privacy: http://hatjitsu.wmflabs.org loads resources from numerous 3rd party sites - https://phabricator.wikimedia.org/T134288#2261236 (10bd808) >>! In T134288#2261228, @MaxSem wrote: > Hatjitsu is abandonware and has no updates since it was installed. Furthermore, my current team... [18:32:38] bd808, https://www.google.com/search?q=online+planning+poker&oq=online+planning+poker&aqs=chrome..69i57j0l5.7189j0j7&sourceid=chrome&ie=UTF-8 [18:33:12] I don't think it makes sense to do anything with jitsu as people can just switch to something different [18:33:40] Ripping the CDN out of it was actually pretty easy. I know that I've worked with 3-4 teams that are using the instance today [18:34:14] I was going to try running my fork as a tool and if that works fine then tell everybody to switch [18:39:47] 06Labs, 10Labs-Infrastructure: Install xml2 on labs - https://phabricator.wikimedia.org/T134146#2261273 (10Kelson) @scfc I need it in a tool-labs instance. In Labs instances I can indeed install whatever I need. [18:51:53] 06Labs, 06Discovery, 06Discovery-Search-Backlog, 06Operations, 10hardware-requests: eqiad: (2) Relevance forge servers - https://phabricator.wikimedia.org/T131184#2261280 (10chasemp) >>! In T131184#2257101, @EBernhardson wrote: > This project does replace nobelium. Nobelium will be decommissioned and re... [18:54:22] can anyone clue me in about why labs projects have PAM configured to deny login unless an account is explicitly whitelisted in /etc/security/access.conf? [18:55:17] twentyafterfour: access.conf is the mechanism used to restrict users to that ldap group only [18:55:28] i.e. when you try to login to a project not yours that's the level it is denied at [18:55:37] or conversely granted [18:55:52] I believe it whitelists groups actually tho [18:56:59] chasemp: context: I have locally defined users in beta cluster (is there a way to define a user via ldap that doesn't belong to an actual person?) [18:57:18] so my local system users have to each be whitelisted in access.conf [18:58:35] the short answer is it can be done but we haven't solidifed / aren't using a way to do it, combined w/ I'm not sure of the future looking state of "service groups" which is basically this in tools [18:58:39] but in thinking about it [18:59:04] maybe we could do something like a line in access.conf that looks at a local group to allow local nix level service accounts instead of ldap [18:59:08] is there a ticket? [18:59:23] it's a more complex question than it seems only because of how integrated all teh pieces of this are [19:02:02] i can't find a ticket but "LDAP vs. puppet systemusers" is a reoccuring issue [19:02:25] sure this is a different context I think that the existing discussions re: service users [19:02:52] maybe https://phabricator.wikimedia.org/T121721 [19:03:35] it's possible to override access.conf (we do that for tools system hosts, that are admin-only) [19:03:51] mutante: ha that's not even teh edge case I was thinking of [19:03:54] so it should be possible to allow another group that way [19:04:32] we need a unified theory of operation here, too many people wanting a flavor of the same thing from different angles [19:05:11] I'm not sure I understand the issu e/ https://phabricator.wikimedia.org/T121721 [19:05:15] and https://github.com/wikimedia/operations-puppet/blob/623d8760cccca15f7798c1e9c22956507cba1676/modules/beta/manifests/deployaccess.pp#L5 [19:05:28] this would be as close as I can tell teh correct mechannism atm [19:06:15] maybe the description of the issue doesn't match the configuration seen to me [19:09:12] chasemp: I think it mostly boils down to people not wanting beta cluster and prod to work differently with respect to the puppet setup needed for scap3 deployments [19:09:51] so yes there is a pretty easy way to add the puppet code that enables the user to ssh, but that config is not needed in prod and it makes people grumpy [19:09:57] sure I get that, I just am also starting from the premise that it's not possible [19:10:19] there's always a hack :) [19:10:22] the part I was/am confused on [19:10:25] gets around this by adding exceptions for individual system users that need to be able to deploy. [19:10:44] but the exception is actually for a group? https://github.com/wikimedia/operations-puppet/blob/623d8760cccca15f7798c1e9c22956507cba1676/modules/beta/manifests/deployaccess.pp#L5 [19:10:48] or is mwdeploy the user I guess [19:11:22] my thinking was to make a generic extra-project localservices group and then only hack that is then labs specific is to add local users to that but then again I've given it all of 30s thought here [19:11:30] I'm not sure access.conf is the right solution to begin w/ [19:11:38] I just know we can't get rid of it w/o a lot of thought into something else [19:12:56] chasemp: that one line you are looking at in https://github.com/wikimedia/operations-puppet/blob/623d8760cccca15f7798c1e9c22956507cba1676/modules/beta/manifests/deployaccess.pp#L5 grants access to for 2 user accounts, deploy-service and mwdeploy [19:13:06] ah yes [19:14:04] at the moment this is literally teh right solution, although yeah I would have made a generic group and then managed membership to it [19:14:54] The security::access:config module is a clean up Corne made for the more gross way that it used to get handled -- https://gerrit.wikimedia.org/r/#/c/256693/ [19:15:30] This was the gross older hack -- https://gerrit.wikimedia.org/r/#/c/256693/12/modules/beta/manifests/deployaccess.pp,unified [19:15:34] yeah I'm familiar mostly becaue we of all the bad fallout we had [19:15:51] but I meant more intra-beta I would use that and manage the exception differently [19:16:01] I was the first to abuse this outside of the labs roots exception [19:16:51] I feel like I should say it very softly but I would rather beta wasn't tied to ldap at all [19:17:01] but that's a long ways and a lot of arguments off [19:17:22] I would rather see beta cluster on bare meta and managed by actual roots, but ... yeah [19:18:03] only out of curiousity but what difference do you think bare metal would make? [19:18:29] we could do perf testing or at least perf regression measurements [19:19:10] ah I don't know that it will ever happen in that way, it's possible we have a staging (beta) + per site deployments w/ metered user traffic for it [19:19:33] that's the more common model I have seen only because of cost [19:20:16] by the time you reduce variables to account for perf differences that make regression measurement useful [19:20:22] *nod* I was spoiled by our lack of size at $DAYJOB-1 where I had a staging cluster that was an exact hardware mach for a prod DC. [19:20:26] you are building third site [19:20:52] did you have two prod dc's? [19:20:57] yeah. [19:21:03] nice [19:21:18] but we were tiny [19:21:27] 4 racks per DC [19:22:03] I mean if done right it's pretyt possible to have a scaled down prod [19:22:39] etc but usually perf regression is relative and not absolute in most cases you'll catch so a staging on virt is still useful in 95% of cases [19:22:55] anyways, yeah [19:23:09] access.conf defined types in puppet are the only sane method atm twentyafterfour that I know of [19:23:24] I would chagne it up and manage it within beta differently to ease your lives [19:23:31] ok [19:24:18] if you put something up and tag me I'll try to review it :) [19:25:50] chasemp: thanks. I thought service groups was the right thing but then I got told not to use those. then I just set up a local user and spent 20 hours debugging why it couldn't log in (blaming keyholder the whole time, debugged that, eventually ruled every possibility out and started digging into PAM specifics before finding the access.conf stuff) [19:27:30] sorry that happened, I've actually been there yeah. 'service groups' as in ldap ones...atm I would stay away from. from my perspective teh access.conf .d style config addition w/ a beta group add seems ok but I it's true it's novel from prod [19:28:11] chasemp: good morning. Is Ganeti under ops-labs responsibility or is that really production? [19:28:28] ganeti is hosting prod VM's only and is greater ops [19:28:40] what I thought thanks !! [19:28:50] somehow VM in my mind == labs :) [19:56:35] 06Labs, 03Scap3: ssh as system users not allowed in labs - https://phabricator.wikimedia.org/T121721#2261463 (10mmodell) p:05Triage>03Normal I'm going to figure out how to manage this from [[ /diffusion/OPUP/browse/production/modules/scap/manifests/target.pp | scap::target ]] [19:56:40] 06Labs, 03Scap3: ssh as system users not allowed in labs - https://phabricator.wikimedia.org/T121721#2261466 (10mmodell) a:03mmodell [20:06:45] ugh... I don't like hard-coding the IP of an individual labs instance into puppet. is there a way to resolve an IP from a name in a puppet manifest? [20:06:47] hmm [20:07:55] oh nice, there is! [20:08:25] ipresolve() function [20:10:44] twentyafterfour: yeah ferm uses it a lot I think [20:12:22] yes, it does. it works fine [20:13:16] you can also specify which nameserver it should use [20:15:58] you might want to double check if there is both , v4 and v6 address, but for labs instances i guess there is no v6 [20:18:10] nope, no v6 there [20:19:41] (apart from the automatic MAC-based addresses -- those can be used) [20:20:51] * valhallasw`cloud can't remember what the right term for those is [20:27:00] "autoconf" ? [20:28:27] Yes! And link-local addressing -- https://en.wikipedia.org/wiki/Link-local_address [20:32:27] https://gerrit.wikimedia.org/r/#/c/286754/ [21:15:25] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Diffusion, 15User-bd808: Create application to manage Diffusion repositories for a Tool Labs project - https://phabricator.wikimedia.org/T133252#2261749 (10mmodell) @bd808: [[ https://secure.phabricator.com/T10748 | Upstream Task T10748: ]] looks releva... [21:35:13] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:37:53] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Diffusion, 15User-bd808: Create application to manage Diffusion repositories for a Tool Labs project - https://phabricator.wikimedia.org/T133252#2261823 (10mmodell) @bd808: From https://secure.phabricator.com/phame/post/view/771/development_notes_2016_w... [21:38:07] labs-morebots: feeling ok? [21:38:07] I am a logbot running on tools-exec-1214. [21:38:07] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [21:38:07] To log a message, type !log . [21:38:11] hmmm [21:39:33] chasemp: or valhallasw`cloud what is the name of the tool that runs http://tools.wmflabs.org/ ? [21:40:07] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 824605 bytes in 4.138 second response time [21:44:48] andrewbogott: back [21:44:54] so what fixed it? [21:45:09] I restarted pdns, which had no immediate effect [21:45:22] and I restarted a tool called 'tools' but I don't actually know if that's the tool that's at tools.wmflabs.org. Is it? [21:45:42] I think it's tool admin or tools [21:45:57] ok, well the 'tools' tool wasn't running and I started it [21:46:02] so probably that's what fixed it :) [21:46:48] That tool' logs are 100% silent though. [21:46:54] I don't see evidence of other services dying [21:46:56] what was the command? [21:47:01] to start [21:47:14] sudo su -; become tools; webservice restart [21:47:34] tools.tools [21:47:42] that's not easy to find when everything is named tools :) [21:47:47] ok, that's the one [21:48:15] So, I think just that one service died and maybe nothing important happened labs- or tools-wise [21:48:22] seems that way to me so far yeah [21:48:36] I really, really want to make this page not depend on this tool [21:49:14] in favor of a simpler canary? [21:50:17] in favor of some robust checks taht spawn and monitor tools etc, that's actually what the catchpoint checks do [21:50:45] but the tools homepage tool is fragile and a not useful indicator anything when it fails tbh [21:51:38] * andrewbogott nods [21:51:48] it tests a lot of things at once, also, which is not a great thing for a test to do [21:51:50] fodder for another day, sorry I was grabbing a soda from teh gas station [21:51:53] so of course it would happen now [21:52:09] yeah, it's bad a check and then unstable as a meta-tool site [22:01:06] 10Quarry, 10AutoWikiBrowser, 07WorkType-Maintenance: Quarry run result in AWB make list - https://phabricator.wikimedia.org/T134141#2261977 (10Reedy) Yeah, dealing with UTF-8 isn't an issue whatsoever [22:07:35] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Diffusion, 15User-bd808: Create application to manage Diffusion repositories for a Tool Labs project - https://phabricator.wikimedia.org/T133252#2261987 (10chasemp) heh @mmodell your relative T#'s got mangled in that quote to obscure bugzilla things :)... [22:08:55] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Diffusion, 15User-bd808: Create application to manage Diffusion repositories for a Tool Labs project - https://phabricator.wikimedia.org/T133252#2261988 (10mmodell) >>! In T133252#2261987, @chasemp wrote: > heh @mmodell your relative T#'s got mangled in... [22:30:21] 06Labs, 13Patch-For-Review, 03Scap3: ssh as system users not allowed in labs - https://phabricator.wikimedia.org/T121721#2262064 (10mmodell) [22:31:17] twentyafterfour: thanks for finding that upstream work on diffusion. Looks like Evan is going to have all the goodies ready for me to use by the time I get that far into the project. :) [22:32:53] I swear not once, or twice, but probably 10 times we talk about somthing wishy [22:33:07] and evan is doing it by some kind of mystical force [22:33:22] the guy sure does turn out a lot of code [22:34:09] he is a productive guy [22:36:06] indeed [22:44:39] 06Labs, 10MediaWiki-extensions-OpenStackManager: WikiPage::something error encountered while adding two users to Tools project at the same time - https://phabricator.wikimedia.org/T133742#2262069 (10Krenair) Honestly I'm tempted to decline this in favour of migrating to Horizon in T91988 instead (if I understa... [22:50:33] 10Tool-Labs: Install xml2 on labs - https://phabricator.wikimedia.org/T134146#2262073 (10scfc) Do you want to use it only interactively or also as part of grid jobs or web services? [23:03:18] (03CR) 10Jean-Frédéric: [C: 031] "I reviewed this, looks good to me. André, do you want this merged now or prefer to work more on it first?" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) (owner: 10Lokal Profil) [23:24:22] 06Labs, 13Patch-For-Review, 03Scap3: ssh as system users not allowed in labs - https://phabricator.wikimedia.org/T121721#2262088 (10chasemp) Close? [23:27:03] 06Labs, 13Patch-For-Review, 03Scap3: ssh as system users not allowed in labs - https://phabricator.wikimedia.org/T121721#2262091 (10mmodell) seems to be working. deployment-tin crashed and burned right around the same time as this patch merged but it seems to be unrelated. [23:27:10] 06Labs, 13Patch-For-Review, 03Scap3: ssh as system users not allowed in labs - https://phabricator.wikimedia.org/T121721#2262092 (10mmodell) 05Open>03Resolved [23:27:23] 06Labs, 13Patch-For-Review, 03Scap3: ssh as system users not allowed in labs - https://phabricator.wikimedia.org/T121721#1886283 (10mmodell) Thanks @chasemp [23:46:21] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Diffusion, 15User-bd808: Create application to manage Diffusion repositories for a Tool Labs project - https://phabricator.wikimedia.org/T133252#2262137 (10mmodell)