[00:00:10] when are you directly interacting with kubectl? [00:00:23] YuviPanda: spell -.- [00:00:38] YuviPanda: back, whenever you are free [00:01:04] legoktm: whenever I'm using kubernetes and I don't need a webservice [00:01:22] I use it to run an IRC bot [00:01:23] oh, I didn't know we could do that [00:01:35] tom29739 source <(kubectl completion bash) [00:01:37] That's why I'm writing up a guide [00:01:51] provides bash completion [00:01:51] :D [00:02:32] legoktm it basically has much more intense power than the webservice command, but most people don't need it if they're happy with what we already ave [00:02:34] *have [00:03:14] At least there's good docs [00:03:21] so you can do things like set custom environment variables, make it execute things before it starts and after it stops, maintain multiple replicas, etc [00:03:30] and mostly that, yeah. the docs for kubectl are maintained by the kubernetes team [00:03:32] so it's a lot more :D [00:03:34] than what we have [00:03:42] Grid engine left a lot to be desired in those aspects [00:03:49] tom29739 did you try the competions? [00:04:11] I'm trying them [00:04:33] it doesn't provide autocomplete for indivudal pod/service names [00:06:22] The auto completes are good [00:06:49] Means I don't have to type "kubectl get deployments" out [00:06:57] 06Labs, 10Tool-Labs, 13Patch-For-Review: support python3 uwsgi apps - https://phabricator.wikimedia.org/T104374#2494347 (10yuvipanda) Hello! I've more direct and better python3 support in the kubernetes backend - I've written up some docs on https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web/Kubernetes#p... [00:07:29] legoktm you can also migrate python2 webservices btw. pretty much excatly the same, instead of 'python' passed to webservice just 'python2' [00:07:33] (yes, python3 is default python) [00:08:12] madhuvishy I added docs on how to install things like numpy as well https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web/Kubernetes#python_.28uwsgi_.2B_python3.4.29 [00:08:27] 06Labs, 10Tool-Labs, 13Patch-For-Review: support python3 uwsgi apps - https://phabricator.wikimedia.org/T104374#2494348 (10yuvipanda) p:05Low>03Normal [00:08:58] 06Labs, 10Tool-Labs, 13Patch-For-Review: support python3 uwsgi apps - https://phabricator.wikimedia.org/T104374#1414842 (10yuvipanda) Once we migrate all the uwsgi-plain webservice to kubernetes, I think we can call this done *and* remove the uwsgi-plain webservice type. [00:10:26] 06Labs, 10Horizon: Proxy corner case: proxy name foo.wmflabs.org == domain name foo.wmflabs.org - https://phabricator.wikimedia.org/T131367#2494351 (10AlexMonk-WMF) >>! In T131367#2168775, @bd808 wrote: >>>! In T131367#2165108, @Andrew wrote: >> For the record, I'm not sure that fixing this is a better soluti... [00:22:57] YuviPanda: cool hanks [00:23:00] thanks [00:23:19] YuviPanda: how can one import data via clipboard on PAWS? ... [00:23:23] this: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_clipboard.html fails with "NoClipboardProgramError: Pyperclip requires the gtk, PyQt4, or PySide module installed, or either the xclip or xsel command." [00:23:49] HaeB you can't. [00:23:55] you have to paste that into a file or somesuch and then import [00:24:12] that expects you to be running pandas on the same machine as your clipboard, but in this case you aren't [00:25:34] YuviPanda: ok (must have misunderstood somthing). yes, it works like that on my local jupyter installation ;) ... [00:25:38] ...how does one import a file? [00:26:02] haeb in your home (where it shows the file browser) there's an 'upload' button on the top right [00:26:30] cool! will try [00:26:49] ok :) [00:30:33] bd808 is the stashbot project still in use? [00:31:36] ah it has been deleted [00:31:39] I shall delete the hiera page no [00:31:41] w [00:33:01] legoktm I'm going to touch the CI puppetmaster now [00:33:33] for https://gerrit.wikimedia.org/r/#/c/301036/ [00:33:42] YuviPanda: there's lots of empty projects around that have no instances. They're not on the purge page. Are you going to be deleting those too? [00:34:05] not sure. probably? [00:34:14] legoktm can you get into integration-puppetmaster? [00:34:38] checking [00:34:46] YuviPanda: yes [00:35:09] legoktm strange, I got in with my root key tho [00:35:12] puppet is disabled, I wonder why [00:35:54] The last Puppet run was at Mon Jul 25 09:25:10 UTC 2016 (909 minutes ago). Puppet is disabled. reason not specified [00:36:46] yeah [00:38:00] 06Labs, 06WMF-Legal: Potential ambiguities in the Labs Terms of Use - https://phabricator.wikimedia.org/T140486#2494436 (10ZhouZ) Additionally, can someone clarify this statement: > A tool that edit on a user's behalf via OAuth automatically disclose the username to MediaWiki revision table, thus publicly and... [00:39:26] legoktm I'll probably let it be [00:41:11] 10Quarry, 05Security, 07Vuln-MissingAuthz: Quarry: Query edit restriction is enforced in UI, not API - https://phabricator.wikimedia.org/T134699#2494444 (10Bawolff) 05Open>03Resolved a:03Bawolff per yuvi on irc, this can be public now [00:41:30] 10Quarry, 05Security, 07Vuln-MissingAuthz: Quarry: Query edit restriction is enforced in UI, not API - https://phabricator.wikimedia.org/T134699#2494447 (10Bawolff) [00:41:45] 06Labs, 06WMF-Legal: Potential ambiguities in the Labs Terms of Use - https://phabricator.wikimedia.org/T140486#2494451 (10tom29739) I don't think that's the case. The user using OAuth via a tool will have their username disclosed to the tool though (subject to the usual private info restrictions at present).... [00:41:48] * tom29739 pokes wikibugs [00:42:13] It seems to have lagged behind all week [00:49:56] 06Labs, 06WMF-Legal: Potential ambiguities in the Labs Terms of Use - https://phabricator.wikimedia.org/T140486#2494461 (10ZhouZ) > I don't think that's the case. The user using OAuth via a tool will have their username disclosed to the tool though (subject to the usual private info restrictions at present). >... [00:55:51] 06Labs, 06WMF-Legal: Potential ambiguities in the Labs Terms of Use - https://phabricator.wikimedia.org/T140486#2494468 (10tom29739) >>! In T140486#2494461, @ZhouZ wrote: > Ok in that case, going back to the second question - while a username might not be categorized as private information like IP addresses an... [00:58:01] 06Labs, 06WMF-Legal: Potential ambiguities in the Labs Terms of Use - https://phabricator.wikimedia.org/T140486#2494469 (10tom29739) Perhaps it would make sense to have a separate notice, given that the username isn't as confidential as say, an IP address or similar [00:59:53] YuviPanda: I'll email the QA list [01:00:26] legoktm thanks [01:02:48] andrewbogott: can I open a task about the snapshots in Horizon? I think many would find that feature useful (limited to conserve disk) [01:17:52] tom29739: I don't think we have the storage setup for it. We really need a large storage cluster for something like that [01:18:33] right now instance storage is on the actual execute nodes that host the VMs. [01:18:59] Where does stuff like snapshots go? [01:19:03] and that disk is pretty limited/used for live VMs [01:19:20] it would go there because we have no other storage [01:19:39] in a setup that was using sanpshots they would have gluster/swift/something [01:20:33] which would be a nice thing to have in Labs, but there are a lot of other things that are higher priority to work on for the near future [01:20:39] Labs doesn't have a Swift [01:20:56] We can't use the production one can we M [01:20:59] *? [01:21:03] not for instance storage, no [01:21:05] no [01:21:15] Didn't think so [01:21:17] i heard the new thing is that there is swift in labs [01:21:28] I heard something like that [01:21:45] there is a tiny swift cluster on VMs in deployment-prep [01:22:32] but that's not what we would need. We would need to be using swift to store all the things [01:22:41] similar to Amazon S3 [01:24:13] If you were around for the glusterFS implementation in pmtpa, we basically need that again only with actually reliable setup [01:24:30] I don't think I was [01:24:48] When did labs move out of pmtpa? [01:25:03] The opnestack side is https://wiki.openstack.org/wiki/Cinder and then it needs a backend to actually do the storage [01:25:17] spring of 2014 I think [01:27:03] tom29739: anyhow, nobody can stop you from making a ticket, but I'm pretty sure that it won't be fixed for quite some time if ever [01:27:33] Isn't that the case for a lot of tickets [01:27:49] I can think of a couple cases when that's happened.. [01:28:02] sure, but I discourage adding all the pipe dream ones I can :) [01:28:06] only a couple? [01:28:32] I'd *love* to WONTFIX about 80% of the tickets in phab [01:29:06] Those 2 I filed weren't pipe dream ones [01:29:19] I've had people edit war me for WONTFIX'ing tickets that were explicitly assigned to me about my own tools :/ [01:29:39] (good product managers say no) [01:29:41] bd808: don't go back to the old bugzilla days [01:29:48] Please dear God no [01:30:03] in what way? [01:30:33] actually closing things? or leaving a zillion open tickets that everyone knows will never be worked on? [01:30:42] The workflow [01:30:54] There's a fancy diagram somewhere [01:31:46] predicting the future is hard. next thing you are asked to find "easy" tickets for GSOC or sometthing :p [01:33:30] YuviPanda: ok, it worked out quite nicely (was able to import data and generate a plot) ... is it already possible to publish notebooks? (i.e. make them world visible) [01:33:34] mutante: that stalled one I filed a few months ago would be a perfect fit for something like that [01:34:08] A 5 minutes job if you have the right perms [01:34:22] yea, but it's never 5 minutes in reality [01:34:36] you gotta have the whole context too [01:34:48] and/or explain it to somebody [01:35:06] or create the access [01:35:35] But still, it's a relatively easy task [01:36:20] yes [01:36:31] i tend to think that open tasks dont hurt [01:36:54] every once in a while there are surprises [01:37:02] because somebody look for old tickets [01:37:09] T131561 if anybody is interested in solving it [01:37:10] T131561: Tool Labs users .bashrc file does not exist for tools accounts - https://phabricator.wikimedia.org/T131561 [01:37:13] though.. it's a bit rare [01:39:24] I would prefer we implement something similar to the admin module in production [01:39:44] T102173 [01:39:44] T102173: implement a sane way to share dotfiles across labs instances - https://phabricator.wikimedia.org/T102173 [01:39:58] HaeB yes. go to https://paws-public.wmflabs.org/paws-public/User: [01:40:07] HaeB you can also just take your current notebook url and replace any of the words 'paws' with 'paws-public' and it'll take you to the right place [01:41:05] Krenair: that would be nice, but doesn't fix the issue tom29739 wrote up for tool service accounts [01:41:23] That task doesn't apply to tools [01:41:38] I suppose it might not work for the service accounts :/ [01:43:06] Krenair: the default tools one just includes useful ls aliases for colour and options that I find useful [01:43:40] Bit of a pain to copy them over [01:44:04] YuviPanda: Thanks! huh, so it's already automatically public? (not that i mind, but... it's a bit of surprise ;) [01:44:09] I suspect we probably could make it work for service accounts [01:44:45] HaeB yeah, I need to make that more clear. You can keep things private by putting them in folders that begin with a . [01:50:16] PROBLEM - Puppet run on tools-webgrid-lighttpd-1202 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:50:24] PROBLEM - Puppet run on tools-cron-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:50:42] Krenair: it'd be easy, you'd just need to stop toolwatcher from creating the project dir [01:51:05] PROBLEM - Puppet run on tools-webgrid-lighttpd-1415 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:51:26] Then it'd be automatically created and populated when the user first sudo s in [02:10:53] YuviPanda: is there a chance to restore the files that were truncated a while ago? [02:17:37] gifti probably not I'm afraid, no. [02:17:49] :( [02:18:54] how long ago? [02:19:04] 06Labs, 06WMF-Legal: Potential ambiguities in the Labs Terms of Use - https://phabricator.wikimedia.org/T140486#2494580 (10jayvdb) >>! In T140486#2494436, @ZhouZ wrote: > Additionally, can someone clarify this statement: > >> A tool that edit on a user's behalf via OAuth automatically disclose the username to... [02:20:22] it was when you needed space, i guess it was between 20160515 and 20160601 [02:20:58] because from 20160601 on the data is still there [02:21:19] yeah, definitely gone. sorry! [02:22:39] #$|~* [02:23:21] could you spare files that are clearly not .out files next time? like .out-yyyymmdd [02:23:37] or should i make sure that they are named nothing like anything you might want to delete? [02:24:56] can you file a bug explaining what the files were and what the pattern of them is? [02:25:05] we can make sure to have a systematic cleaning out setup from there. [02:25:14] I'm not sure how this was cleaned up or when/why [02:25:14] hm [02:26:13] YuviPanda: what's the difference between the "base" and "web" images in the tools docker registry? [02:26:28] tom29739 the 'web' ones have webservices added on [02:26:34] so lighttpd for php, uwsgi for python, etc [02:26:37] and the toollabs-webservice package [02:27:06] So if I don't need those, then I can use the base image [02:27:13] yeah [02:28:35] What's the toollabs-webservice package for? [02:28:57] it sets up the lighttpd conf / uwsgi config properly and starts them [02:29:04] the 'webservice' tool is also from this package [02:30:16] RECOVERY - Puppet run on tools-webgrid-lighttpd-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [02:30:22] RECOVERY - Puppet run on tools-cron-01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:31:05] RECOVERY - Puppet run on tools-webgrid-lighttpd-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [02:32:43] 06Labs, 10Tool-Labs: Do not truncate my precious data - https://phabricator.wikimedia.org/T141331#2494582 (10Giftpflanze) [02:34:18] 06Labs, 10Tool-Labs: Do not truncate my precious data - https://phabricator.wikimedia.org/T141331#2494596 (10Giftpflanze) [02:41:06] 10Tool-Labs-tools-Pageviews: Query page move log to get pageviews of an article at its older locations - https://phabricator.wikimedia.org/T141332#2494602 (10MusikAnimal) [02:48:34] gifti: brilliant expression of mind in that bug :D (T141331) [02:48:34] T141331: Do not truncate my precious data - https://phabricator.wikimedia.org/T141331 [02:49:18] ok ^^ [04:29:45] 10Gerrit-Patch-Uploader: Gerrit patch uploader redirects to wrong URL - https://phabricator.wikimedia.org/T141333#2494668 (10Fomafix) [04:45:09] hi! bastion-03 is very slow and stucking [04:46:32] hi doctaxon [04:46:35] let me see [04:46:43] bastion-02 too [04:49:25] doctaxon how does -03 feel now? [04:49:49] very slow [04:49:53] still slow yeah [04:50:26] but now I got a prompt at least [04:52:10] I killed a couple more tools [04:52:38] what tools? [04:53:15] err, I mean, processes [04:53:16] not tools [04:53:31] tools-giftbot: I try to grep state-backup * [04:53:42] but it hangs [04:54:07] doctaxon: o.o [04:54:11] hi [04:54:14] maybe just *.tcl? [04:54:23] but i doubt it's anywhere in the code [04:54:36] i doubt too [04:54:44] ok, you are causing the hang :) [04:54:49] maybe that's why it hangs >.> [04:54:53] it's eating waytoo much of the NFS bandwidth on the machine [04:54:54] ahm [04:55:19] I'm just killing your processes now. please use the grid :) [04:55:28] and I should go sleep, intensely long day and it's 10pm :| [04:55:44] good night [04:55:45] good night [04:55:56] um, you killed mine too? [04:57:35] no, he killed the grep process [04:58:24] well, i was testing something and then there was "Killed" [04:59:00] wie meint Yuvi das, wie soll man grep using the grid ausführen? [04:59:30] ja, aber wenn du es auf *.tcl beschränkst gehts auch ohne [05:47:50] 06Labs, 10Labs-Infrastructure, 06Discovery, 06Maps, and 2 others: Update coastline data in OSM postgres db (osmdb.eqiad.wmnet) - https://phabricator.wikimedia.org/T140296#2494695 (10Dzahn) @dschwen Is it fixed now? [05:54:05] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2494696 (10Dzahn) @andrew @yuvipanda as list admins of labs-l and labs-announce, how about the remaining checkbox " Add as mod to labs-l/labs-announce"... [06:40:07] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Dachary was created, changed by Dachary link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Dachary edit summary: Created page with "{{Tools Access Request |Justification=I would like to develop tools to help maintain wikidata items with a focus on the [https://www.wikidata.org/wiki/Wikidata:WikiProject_Inf..." [06:54:27] 10Tool-Labs-tools-Pageviews, 07I18n: [[Wikimedia:Pageviews-api-incomplete-data/en]] i18n issue - https://phabricator.wikimedia.org/T135817#2494730 (10Nemo_bis) >>! In T135817#2493413, @MusikAnimal wrote: > I could add another parameter `$2` that represents the number of dates being passed in with `$1`, then yo... [07:20:03] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2494769 (10MoritzMuehlenhoff) >>! In T140422#2493224, @madhuvishy wrote: > @MoritzMuehlenhoff Done! http://keys.gnupg.net/pks/lookup?op=get&search=0xA4D... [07:57:43] 06Labs, 06Performance-Team, 10Thumbor, 07Tracking: Request creation of thumbor labs project - https://phabricator.wikimedia.org/T141341#2494847 (10Gilles) [07:58:10] 06Labs, 06Performance-Team, 10Thumbor, 07Tracking: Request creation of thumbor labs project - https://phabricator.wikimedia.org/T141341#2494847 (10Gilles) [08:16:59] 06Labs, 10Labs-Infrastructure, 06Operations: Investigate failover failure of LDAP servers - https://phabricator.wikimedia.org/T141277#2494926 (10MoritzMuehlenhoff) >>! In T141277#2493220, @chasemp wrote: > @andrew and I were discussing whether LVS would make sense in front of LDAP with the ability to more in... [08:21:40] 06Labs, 10Labs-Infrastructure, 06Operations: Investigate failover failure of LDAP servers - https://phabricator.wikimedia.org/T141277#2492736 (10Joe) If what happened the other day was that one VM was overloaded and stopped answering ldap queries while still accepting connections (which given the described p... [08:22:42] (03PS2) 10Lokal Profil: Add Albania and Kosovo to the database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/300233 (https://phabricator.wikimedia.org/T140488) [08:29:57] (03CR) 10Jean-Frédéric: [C: 031] "Looks good to me − although I trust our automated tests more than me to review these ;)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/300233 (https://phabricator.wikimedia.org/T140488) (owner: 10Lokal Profil) [08:30:19] !log tools.heritage Add sq.wikipedia to pywikibot user_config [08:30:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [08:38:53] 10Gerrit-Patch-Uploader, 10Gerrit: Gerrit-patch-uploader fails under git 1.9 - https://phabricator.wikimedia.org/T86304#2494958 (10Aklapper) Question is which JGit version our Gerrit instance runs on. If it's 3.5+ this bug is fixed, AFAIU. [08:41:34] (03CR) 10Lokal Profil: [C: 032] "I've added sq.wiki to user_config so merging this should be safe.... fingers crossed" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/300233 (https://phabricator.wikimedia.org/T140488) (owner: 10Lokal Profil) [08:42:08] (03Merged) 10jenkins-bot: Add Albania and Kosovo to the database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/300233 (https://phabricator.wikimedia.org/T140488) (owner: 10Lokal Profil) [08:52:04] 06Labs, 07Graphite, 13Patch-For-Review: Setup "official labs grafana" instance - https://phabricator.wikimedia.org/T120295#2494998 (10fgiunchedi) >>! In T120295#2492682, @yuvipanda wrote: > @hashar no, I meant the opposite - that the grafana at grafana-labs.wikimedia.org also has access to prod graphite - so... [08:54:41] 06Labs, 10Labs-Infrastructure, 06Discovery, 06Maps, and 2 others: Update coastline data in OSM postgres db (osmdb.eqiad.wmnet) - https://phabricator.wikimedia.org/T140296#2494999 (10dschwen) 05Open>03Resolved Yes! Many thanks! [09:36:51] 06Labs, 06Performance-Team, 10Thumbor, 07Tracking: Request creation of thumbor labs project - https://phabricator.wikimedia.org/T141341#2495087 (10Gilles) [09:55:41] 06Labs: Request creation of thumbor labs project - https://phabricator.wikimedia.org/T141341#2495130 (10Danny_B) [10:19:14] 06Labs, 07Graphite, 13Patch-For-Review: Setup "official labs grafana" instance - https://phabricator.wikimedia.org/T120295#2495188 (10akosiaris) >>! In T120295#2494998, @fgiunchedi wrote: >>>! In T120295#2492682, @yuvipanda wrote: >> @hashar no, I meant the opposite - that the grafana at grafana-labs.wikimed... [11:00:38] 06Labs, 06Performance-Team, 10Thumbor: Request creation of thumbor labs project - https://phabricator.wikimedia.org/T141341#2495261 (10Gilles) These tags are appropriate, please don't prevent me from tracking this task on relevant boards/projects. [13:42:48] !log tools.heritage Deployed latest from Git: 0c848e6 (T140488) [13:42:50] T140488: Add Albanian Wikipedia lists to database - https://phabricator.wikimedia.org/T140488 [13:42:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [13:59:27] How would I connect webservice to use an existing k8s yaml? [14:12:49] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/ZaYen was created, changed by ZaYen link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/ZaYen edit summary: Created page with "{{Tools Access Request |Justification=Making bots for wikimedia products |Completed=false |User Name=ZaYen }}" [14:19:08] 10Wikibugs: Inconsistent versioning? - https://phabricator.wikimedia.org/T141364#2495667 (10Danny_B) [14:19:23] 10Wikibugs: Inconsistent versioning? - https://phabricator.wikimedia.org/T141364#2495679 (10Danny_B) p:05Triage>03Low [14:57:24] 10Gerrit-Patch-Uploader, 10Gerrit: Gerrit-patch-uploader fails under git 1.9 - https://phabricator.wikimedia.org/T86304#2495775 (10demon) 05Open>03Resolved a:03demon We're running 4.1.2.201602141800-r. I'm assuming this is solved based on what @valhallasw said, but it'd be nice if someone could confirm t... [15:42:15] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#2495934 (10bd808) [15:42:17] 06Labs, 06Performance-Team, 10Thumbor, 15User-bd808: Request creation of thumbor labs project - https://phabricator.wikimedia.org/T141341#2495930 (10bd808) 05Open>03Resolved a:03bd808 https://wikitech.wikimedia.org/wiki/Nova_Resource:Thumbor [16:09:54] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 13Patch-For-Review: Kubernetes worker nodes hanging - https://phabricator.wikimedia.org/T141017#2496005 (10yuvipanda) @mark pointed out a couple of things: 1. iowait was at 100% on couple of CPUs (0 and 2) 2. There's `atop` which we can use to look at historical dat... [16:15:16] YuviPanda, there's a problem with the jdk8 image [16:15:35] go on [16:15:57] When trying to use a generic webservice with webservice-runner, it crashes with "No such file /etc/active-proxy" [16:16:14] I catted that file on the bastion [16:16:17] It reads: tools-proxy-01 [16:16:23] ah [16:16:32] Maybe you need to put it in the container? [16:16:45] webservice-runner needs it for some reason [16:16:47] what's the exact command you are using? [16:17:45] it thinks it is running on gridengine [16:18:11] /usr/bin/webservice-runner --type generic --port 8000 --register-proxy --extra_args /data/project/loltools/loltools-build/bin/loltools [16:18:23] yup, don't pass --register-proxy [16:18:26] that'll work [16:18:27] Oh [16:19:19] Thanks [16:20:16] I ran into the missing /etc/active-proxy bit when working on https://tools.wmflabs.org/admin-beta/ too. I ended up just hard coding tools-proxy-01 [16:20:21] https://phabricator.wikimedia.org/diffusion/1922/browse/master/src/Tools.php;47d9fa0b1c82d83308588c6d634116f6b13b2854$94 [16:20:37] bd808 how did you trigger it? [16:21:24] aaah [16:21:28] the ?list page fetches all the proxies from tools-proxy-01:8018/list [16:21:55] right [16:21:58] which could be fixed with some haproxy service name or something I guess [16:22:11] a kubernetes headless service [16:23:51] YuviPanda: maybe when I'm in SF next week we can pair for a bit and figure out how to finish T140254 [16:23:52] T140254: Modernize the admin tool's codebase - https://phabricator.wikimedia.org/T140254 [16:24:02] yup! [16:40:59] 06Labs, 10Striker, 15User-bd808: Request creation of Striker labs project - https://phabricator.wikimedia.org/T141375#2496081 (10bd808) [16:41:20] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#1518985 (10bd808) [16:41:22] 06Labs, 10Striker, 15User-bd808: Request creation of Striker labs project - https://phabricator.wikimedia.org/T141375#2496097 (10bd808) 05Open>03Resolved https://wikitech.wikimedia.org/wiki/Nova_Resource:Striker [17:25:50] PROBLEM - Puppet run on tools-webgrid-lighttpd-1404 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:27:50] PROBLEM - Puppet run on tools-exec-1405 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:28:08] PROBLEM - Puppet run on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:31:45] this is just a race I think [17:45:50] RECOVERY - Puppet run on tools-webgrid-lighttpd-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [17:50:53] 10Tool-Labs-tools-Pageviews: Investigate adding OpenSearch to Pageviews Analysis - https://phabricator.wikimedia.org/T141385#2496405 (10MusikAnimal) [17:53:27] 10Tool-Labs-tools-Pageviews, 06Community-Tech: Add option to pageviews tool to show stats for all redirects to a page as well - https://phabricator.wikimedia.org/T128495#2496422 (10MusikAnimal) [17:54:50] tom29739 your jdk8 experiment going well? [17:55:04] YuviPanda, yep [17:55:13] I'm mainly just trying out Java [17:55:14] good to hear :) [17:55:43] * YuviPanda nods [17:55:47] did you get a webservice running? [17:56:40] I think it's working [17:58:28] 10Tool-Labs-tools-Pageviews, 06Community-Tech: Add option to pageviews tool to show stats for all redirects to a page as well - https://phabricator.wikimedia.org/T128495#2496431 (10MusikAnimal) 05Open>03declined Closing as declined since we've collectively decided querying for //all// redirects isn't going... [17:58:33] 10Tool-Labs-tools-Other, 10Analytics, 06Community-Tech, 10Pageviews-API, and 2 others: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2496435 (10MusikAnimal) [18:03:09] RECOVERY - Puppet run on tools-webgrid-lighttpd-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [18:07:48] RECOVERY - Puppet run on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [18:26:05] hey YuviPanda, I was working with tom29739 on the jdk stuff [18:26:16] However, the tools page still says there is no web service [18:26:24] (using this yaml https://dpaste.de/a3eZ) [18:28:09] aaah [18:28:15] okkk [18:28:16] so [18:28:19] moment, switching connections [18:28:27] (I have a solution for you) [18:29:36] :O [18:29:46] OH- tom29739 mind if I mess with your tool a bit? [18:29:58] go ahead [18:30:03] YuviPanda, it's not actually my tool [18:30:14] right ok [18:30:15] It's OH-'s, I've just been helping [18:38:46] OH tom29739 ok! can you look at the error in the file 'error-file' and tell me what seems wrong? [18:38:54] (I'm trying to make this work with the 'webservice' command and fixing things as I go) [18:39:28] I see you have a secret specified. Is that absolutely required? [18:39:35] can you make it work without the secret object? [18:40:01] I think that's the problem [18:41:20] I can try [18:41:41] It's just a string that Play uses to sign requests or something [18:41:46] right [18:41:48] I just hacked that in just now [18:43:38] I'm getting [18:43:38] This application is already running (Or delete /data/project/loltools/loltools-build/RUNNING_PID file). [18:43:52] I've just modified the yaml to use a testing secret without using the secret object [18:44:11] wow that's a complex bash script [18:44:14] when Play crashes it leaves RUNNING_PID behind. I just deleted it [18:44:54] yeah but doesn't seem to be putting logs anywhere [18:45:14] ah [18:45:20] I see loltools-build/logs [18:47:15] Well, it's 502 bad gateway instead of no webservice now. so I guess that's an improvement [18:47:17] OH-, YuviPanda, that secret can be specified as a command line argument [18:47:32] I just hacked in a env variable just now [18:47:40] it's still crashing and I'm not entirely sure why [18:47:41] it isn't leaving behind useful logs [18:47:48] any way I can force it to be more verbose? [18:49:22] also need to get rid of the PID thing :) [18:52:24] tom29739 OH- boom! http://tools.wmflabs.org/loltools/ [18:52:31] the problems found so far are: [18:52:40] webservice generic doesn't pass extra_args through properly [18:52:51] and this PID thing is causing it to not start basicall [18:52:52] y [18:52:58] YuviPanda, you can change the PID location [18:53:03] so I now have a loop running that keeps deleting the PID file [18:53:07] -Dpidfile.path=/var/run/play.pid [18:53:15] As a command line arg [18:53:36] tom29739 the PID stuff makes no difference because there won't be two of them in the same container I guess [18:53:46] As long as it doesn't crash the PID file shouldn't cause any problems [18:53:46] Maybe you could change it to /dev/null [18:54:03] I think it'll fail all the time [18:54:07] yeah [18:54:08] I had a problem with an application with a PID file [18:54:14] you can't put it in /var/run because non roots can't write there by default [18:54:14] Caused a massive pain [18:54:14] maybe in /tmp [18:54:44] I ended up putting the command in a starter script and an rm command at the end of the script [18:54:52] Works very well [18:55:32] YuviPanda, it says how to disable PID creation [18:55:44] YuviPanda, thanks for the help. Is the webservice generic thing fixed throughout now? [18:55:44] "To prevent Play from creating it’s own PID, you can set the path to /dev/null in your application.conf file:" [18:55:51] pidfile.path = "/dev/null" [18:55:52] where? [18:56:00] OH- nope, I'll fix that soon [18:56:18] OH- in the meantime, you can restart your application with kubectl delete pod $podname [18:56:32] OH- and you can get a shell to build it in with 'webservice --backend=kubernetes jdk8 shell' [18:57:17] great, thanks again :) [18:57:26] and I can start up the webservice directly using kubectl create? [18:57:38] OH- yw! [18:57:55] OH- the webservice is already up - I've custom hacked it atm. [18:58:18] in the future (once I fix the generic thing) you would have been able to start it with [18:58:24] webservice --backend=kubernetes jdk8 ~/loltools-build/bin/loltools [18:59:04] What if I have to set up variables etc. that I currently have in loltools.yaml? [18:59:33] OH-, maybe put them in an application.conf file [18:59:33] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Add a easy way to run a ruby webservice on tools - https://phabricator.wikimedia.org/T141388#2496577 (10yuvipanda) [19:00:22] I mean, like the application secret string [19:00:38] previously, kubectl get pods didn't show any errors when I had a secret object, but it was the webservice that wasn't working [19:01:05] right, but that was because you had a custom deployment [19:01:17] (and the reason the webservice wasn't working was because you need a service object as well) [19:01:29] but I want to be able to allow people to use play with just the webservice command, which is why I was going through this :) [19:01:38] this will allow others to easily run play as well without having to understand all of k8s [19:01:44] ah, ok [19:02:10] YuviPanda, it'd be good if a command was made that just let you run a command on k8s [19:02:12] OH- can you write up the steps you did somewhere for me? [19:02:31] tom29739 yeah, I agree. I was thinking of maybe just adding that to jsub [19:02:34] you mean setting up everything up to this point? [19:02:40] OH- yeah. [19:02:53] YuviPanda, I could create a python script that does that [19:03:10] tom29739 jsub is a python script :D [19:03:48] I just built the project and scp'd it onto tool labs [19:04:01] OH- ah, so you built it locally [19:04:09] YuviPanda, I thought it was perl [19:05:05] tom29739 bd808 rewrote it :D [19:05:54] YuviPanda, how do you get patches and stuff accepted in that repo? [19:05:55] my thought would be to have a --backend parameter to it similar to webservice [19:06:44] tom29739 it's the labs/toollabs repo I think. [19:07:08] https://gerrit.wikimedia.org/r/#/admin/projects/labs/toollabs [19:07:28] so add me and bd808 and valhallasw`cloud as reviewers ;) [19:08:40] I'm building loltools inside of a kubernetes shell just to see if it works [19:09:02] what I did was the version of sbt on tool labs I think was too old (?) or something, but it just didn't work for my project [19:09:18] I downloaded the latest sbt binaries into ~/ and I use that [19:09:35] I think that's the only non-standard thing I did [19:09:43] YuviPanda, it's this file? https://phabricator.wikimedia.org/diffusion/LTOL/browse/master/jobutils/bin/jsub [19:09:50] OH-, that isn't non-standard [19:10:06] It's standard practice on tools to install programs into your userspace [19:10:37] OH- yeah, I think that's totally ok [19:10:45] I meant non-standard as in places where I have to deviate from the normal instructions [19:10:57] ah right [19:11:52] I'll just go ahead and write all of this down [19:12:30] Hi would someone be able to restart the gerrit reviewer bot (https://www.mediawiki.org/wiki/Git/Reviewers) [19:12:43] seems to not be adding reviewers automatically in gerrit since [19:12:49] we updated [19:12:57] OH- tyvm! [19:12:57]