[06:05:45] hedonil: Longer lists? [06:06:06] a930913: yep [06:06:52] hedonil: Buffered output? Large output? [06:08:35] a930913: https://tools.wmflabs.org/wikiviewstats/?datefrom=0000-00-00&autofilter&lang=*&project=*&type=totals [06:09:00] stops for me at list line ~701 [06:09:49] aaaand pointless to say: worked before ^ [06:11:24] Mmm. [06:12:46] Coren: YuviPanda|brb: Any changes to webproxy setting since yesterday? [06:14:24] first google, first hit: http://mailman.nginx.org/pipermail/nginx/2007-July/001143.html [06:15:03] hedonil: It's not just PHP. [06:16:52] a930913: I'd suggest to put this nginx back to the cradle and test it until it's mature for prod [06:18:27] * a930913 hands hedonil an HRT. [06:20:26] * hedonil hands proxy admins an ITIL change management manual [06:22:07] hedonil: With a hardware reconfiguration tool, you can fix the problem yourself ;) [06:22:45] Hmm, should be a BHRT, B for blunt. [07:40:12] I have the impression that there are problems again [07:40:27] http://tools.wmflabs.org/reasonator/?&q=3630705 does not load [08:19:57] GerardM-: I see "loading...." [08:20:18] petan... that should provide a screen with a lot of information [08:20:29] it does not [08:20:32] likely a problem with SQL server I guess? or how does the webpage retrieve the info [08:20:43] the http server works for sure [08:20:56] it does not use SQL [08:21:05] so it uses api or what? [08:21:12] it uses WDQ [08:21:24] what is that [08:21:37] Wikidata Query [08:21:40] Lydia_WMDE: hi [08:21:43] one of Magnus's tools [08:21:44] mhm [08:21:53] petan: hey [08:22:07] Hoi Lydia_WMDE good morning [08:22:14] working on a list for you [08:22:20] oh? [08:22:23] morning [08:22:47] the list with new Wikidata languages (based on activity in the Incubator) [08:23:05] ah cool [08:23:29] Lydia_WMDE: regarding the wikidata and languages, I am just wondering, how would you get from 1 language version to another? would there be "translation" key for every language? [08:24:12] so that basically getting a translation for "apple" to some language would be just about looking up "translation" key for that language of item "apple"? [08:25:08] I am wondring if I was to create wikidata based online dictionary (for translation) how would I do that :P [08:25:36] petan: you can make a very crude one [08:25:42] but for a good one we need more functionaily [08:25:55] as lined out in the proposal for wiktionary support [08:26:00] for beginning translation of single words would do [08:27:22] how would you know which apple you mean? the fruit or the company? ;-) [08:27:49] but anyway you can get the list of all labels and then filter for language code [08:28:19] petan, translator already use Wikidata for names [08:28:42] yes, but if someone just said "translate apple" you wouldn't know the context anyway so you would have to provide all possible versions [08:28:53] they use it through Wikipedia ... did not get the fact that Wikidata is getting real [08:29:00] which "translator" [08:29:13] professional translators I know [08:29:21] ah [08:29:27] you mean people :) [08:29:35] I was thinking it's some service [08:29:47] we could make it that [08:29:52] it is not that hard [08:31:36] basicaly what I am missing now, is a really free and open (in a wiki way, so that anyone can contribute and improve it) service that would be able to do even most simple translations, right now wikimedia has none such a service, which IMHO is a shame. And I really can't find any working open source project that would be matching all these criteria (not just be open source but also the content would need to be free and open for others to contribute o [08:32:17] wikidata could be used a base for this for sure I guess [08:32:21] * as a [08:33:42] petan: did you already check http://translatewiki.net/ for any kind of support? api, files? [08:33:58] translatewiki has some interesting interface but... [08:34:22] it's used as interface for translators to translate some data that already exist [08:34:41] what I am talking is more like an open service similar to google translate, so that other people can use it to translate anything they need in real time [08:35:33] but yes wikidata seems to be slowly doing progress on this, I can see for example that https://www.wikidata.org/wiki/Q89 links to 3 translations already [08:36:37] if every single existing word had a network of translations to all possible languages, then it could be used for this purpose [08:38:21] GerardM-: your problem with http://tools.wmflabs.org/reasonator/?&q=3630705 seems to be a problem with this item, as eg. this one works fine http://tools.wmflabs.org/reasonator/?q=2513 [08:39:39] hedonil, I know that some items, the easy ones with not much information work [08:40:03] try searching for Nelson Mandela... I bet it will not load [08:40:42] petan: afair in #mediawiki-i18n, there are sometimes messages like "...auto translation service xyz currently not available" (yahoo?) [08:41:02] hedonil: yes, but none of these are free nor open [08:41:10] well, they are "free" to a point [08:41:11] petan: ok [08:42:22] GerardM-: if this problem is related to large outputs, I have similar problems atm. probably proxy stuff again [08:42:52] it is large output [08:43:28] are the admins aware ? [08:43:50] I put something on labs-l [08:43:59] thanks [08:44:05] it is a daily occurence [08:44:19] is it not possible to test for this ? [10:31:37] applications on Labs are not working [10:32:02] ^^ indicates a possible problem with proxies ... HELP [13:00:47] https://pythonhosted.org/oursql/ claims that it supports python3, but if I try to import oursql in python3 it says no such module... any ideas? [13:02:47] Coren... as far as I can see we still suffer from broken apps [13:03:49] Nikerabbit: I don't know whether we installed both versions at that time. (Cave: My knowledge of Python 2 vs. 3 is pretty much non-existing.) [13:05:57] I remember having one such problem recently but I don't remember with what [13:06:05] @translate en de Hello, this is a test [13:06:58] If only code.google.com had activity history [13:09:58] @translate en de Hello, this is a test [13:10:04] here we go [13:12:32] To nit-pick, "test" should be written "Test" in that case :-). [13:18:15] petan: hell, yeah /Yandex/ was the name in #i18 channel, I was looking for :P. Seems it works fine for your bot [13:18:39] yes byt it has a number of limitations [13:18:55] for example max 2000 translations a day [13:19:07] and I must display "powered by Yandex" in every message [13:19:27] that is what their TOS say [13:19:30] well, this should be ok for a bot [13:19:50] maybe but... open & free alternative would be so much better [13:19:59] for example now when it translate wrong, I can't even fix it [13:20:10] if it was "wiki-like" I could do that [13:22:22] hey Coren. i have a problem which looks similar to this: https://tickets.puppetlabs.com/browse/PUP-1271 [13:23:12] Coren: i've sent you the exact puppet output by mail because i wasn't sure whether i can post it online. can you reset the certificate on the master as suggested by puppet? [13:24:10] petan: you can correct Google Translate, if you wish to make Google richer [13:24:39] Nemo_bis: google translate is even worse, it doesn't even provide these api's for free [13:24:56] so it's absolutely unusable [13:25:26] What makes you think I could not know? :) https://translatewiki.net/wiki/Technology [13:25:39] petan: you'd probably need a legion of translators / fixers (vocabulary, context..) [13:25:54] complex business [13:26:15] And a few billions dollars ;) [13:26:34] if the recipient understands the meaning, who cares about 100% coorectness? [13:26:34] hedonil: I am pretty sure people were telling this to J. Wales when he started wikipedia.... [13:26:34] petan: .-) [13:27:15] petan: from this point, you're absolutely right - never give up trying to make things better :-) [13:27:39] @translate en de nobody cares! | scfc_de [13:27:56] haha I love it :P [13:28:17] @translate fr en webproxy est une chausette pd [13:28:19] @translate en de that was regarding your "nit-pick" thing | scfc_de [13:28:39] *grin* [13:28:57] hedonil: you see? it suck. and you can't fix it, because it's not open like wikipedia is [13:29:35] I am pretty sure if there was open source translation system, people would already contribute so much information to it, that it would 20 times better than google translate [13:30:38] petan: http://wiki.apertium.org/wiki/Contributing_to_an_existing_pair awaits your eager help [13:30:45] @translate en fr hey man how are you? :) | hashar [13:31:12] apertium provides yandex-like apis? [13:31:16] I doubt [13:31:42] No idea what you mean by that :) they have API, sure [13:35:56] Nemo_bis: why translate wiki don't use these then? there is only microsoft and yandex [13:36:05] if apertium is free, twn should support it too [13:36:21] that would make it more popular and visible [13:36:25] !log deployment-prep restarting elasticsearch on deployment-elastic01 to pick up some gc setting recommended by elasticsearch team [13:36:27] Logged the message, Master [13:36:38] I agree, hence I'm trying to use you as tester ;) [13:37:03] I can't test things that aren't existing :( there is no apertium in TWN now [13:44:01] Apertium exists, twn is free software as well, your bot was just created and could have used apertium from the start [13:44:13] Whenever you have some insight to share, please do: https://translatewiki.net/wiki/Thread:Support/Apertium [13:46:07] Nemo_bis: I didn't know apertium existed when I started writing that module for a bot [13:46:27] andrewbogott ... it was suggested that Reasonator is not working well because of a problem with a proxy [13:46:39] other apps have a same issue [13:46:42] wut [13:47:00] GerardM-: what are you seeing? [13:47:06] translatewiki.net would gladly start using Apertium again if someone put up a stable web service for it [13:47:11] (YuviPanda|brb, this is probably of interest) [13:48:49] petan: so you don't even read replies to your own threads? http://markmail.org/message/v4jvwvlfyhdnwugr [13:49:29] TBH I was sitting in a pub yesteday when reading these replies :P [13:49:39] I sent the first mail right when I finished at office [13:50:00] Ok, so you don't. Helpful. [13:50:48] I think that link didn't work yesterday [13:51:20] Nikerabbit: what about hosting it at labs? [13:51:53] Do it! [13:53:45] petan: we are likely going to do that soon, and hopefully move it to production then, for content translation [13:54:04] ok that mean "don't do it" I guess [13:54:21] on this one I get an illegal invocation http://tools.wmflabs.org/toolscript/index.html?pastebin=guXqaGQE [13:54:26] it works well normally [13:54:30] nginx si broken, it close connection after 20-30 K [13:54:30] http://tools.wmflabs.org/reasonator/?&q=3630705 indicates its loading and does not [13:54:41] GerardM-: when you say 'normally' can you tell me when it did or didn't work? [13:54:54] yesterday.. last week [13:55:12] whenever Labs is fully functional4 [13:55:30] as a consequence my workflow is broken [13:55:40] Sorry, you're this errors only just appeared? Like, a few hours ago? [13:58:46] GerardM-: I ask because the most recent change to that code was on the 20th. [14:01:57] andrewbogott: something happened like 10 - 20 hours ago. since then: broken [14:02:06] hm, ok [14:03:08] downloading something from a tools hangs around 28K, then after 15s the connection is closed [14:04:00] wget workaround that by using GET byte range multiple time to get data by chunk of 28K [14:04:05] phe, can you give me an example url? [14:04:24] andrewbogott, try that: wget --server-response http://tools.wmflabs.org/phetools/graphs/Wikisource_-_pr_texts.png [14:05:42] !tools-status is http://icinga.wmflabs.org/cgi-bin/icinga/status.cgi?hostgroup=tools&style=detail&nostatusheader [14:06:10] andrewbogott all day (in EUrope) [14:07:01] phe, any better now? [14:07:48] andrewbogott now it works for me as well [14:07:55] can you tell us what you did [14:08:04] works for me as well [14:08:17] JohannesK_WMDE: Sure. Give me a few. [14:08:25] better with wget but I've trouble when dl with my brower [14:08:46] the /var/log partition was full so lots of services were unhappy. I'm going to repartition /var/log but then I may need to reboot the proxy [14:09:36] JohannesK_WMDE: Done on the master; all you need to do now is axe it on the instance and rerun puppet. [14:09:46] andrewbogott is it possible to monitor for such things [14:10:04] a tool like Reasonator loses its public demonstrably in this way [14:10:14] !log tools applying role::labs::lvm::biglogs on tools-webproxy because /var/log was full and causing errors [14:10:16] Logged the message, dummy [14:10:42] GerardM-: Doing proper monitoring for labs is one of my primary objectives for this quarter. [14:11:14] and YuviPanda|brb is working on monitoring for the web proxies. [14:11:21] andrewbogott, works fine now [14:11:45] Coren, having broken /var/log, I'd like to reboot tools-webproxy. Safe or unsafe? [14:11:53] andrewbogott: In this particular case, though, do we really want to give the room for the logs to grow unbounded rather than limit them? [14:12:11] Coren: I believe they are limited, just not enough. [14:12:16] Or, at least, rotated. I'll verify. [14:12:21] But there was /very/ little space before. [14:12:27] * Coren nods. [14:13:18] andrewbogott: It should be okay to reboot; the redis on it should have disk store. The only "really" important part is that redis comes back in the same state before reboot. [14:14:12] !log tools rebooting tools-webproxy so that services start logging again [14:14:13] Logged the message, dummy [14:15:10] Coren: thanks... but there still seems something wrong: [14:15:25] jkroll@sylvester:~> sudo rm -f /var/lib/puppet/client/ssl/certs/[...].pem [14:15:32] jkroll@sylvester:~> sudo puppet agent -tv [14:15:33] Exiting; no certificate found and waitforcert is disabled [14:15:33] jkroll@sylvester:~> sudo puppet agent -tv -w 60 [14:15:33] notice: Did not receive certificate [14:15:36] Coren: looks like we already only save logs for 2 days. (or 3? I can never remember if it's 0 or 1 indexed) [14:15:37] [...] [14:16:13] phe, GerardM-, hedonil, check the proxy one more time for me please? [14:16:15] andrewbogott: 1 indexed by default. Yeah, we'll need to grow /var/log regardless to guard against traffic spikes. [14:16:31] 503 Service Temporarily Unavailable [14:16:36] JohannesK_WMDE: I may need to sign your new cert. Gimme a minute. [14:17:18] JohannesK_WMDE: Try again? [14:17:39] welp, now the proxy says [14:17:40] *12562 attempt to send data on a closed socket: u:00000000013503F8, c:0000000000000000, ft:1 eof:0, client: 218.20.36.19, server: , request: "GET /widar/index.php?callback=jQuery110105301851099357009_1400849767311&botmode=1&action=set_claims&ids=Q15885588&prop=P31&target=Q4167410&_=1400849768799 HTTP/1.1", host: "tools.wmflabs.org", referrer: "http://tools.wmflabs.org/wikidata-todo/autolist2.php" [14:17:43] the heck's that about? [14:17:47] Getting 503 [14:17:54] Coren: pm'ing you with output... [14:18:16] GerardM-: 503 normally means that the proxy can't find the other end in redis. [14:18:21] andrewbogott: Did you reboot webproxy? [14:18:24] Coren, I'm getting 503 [14:18:31] Coren: I did, yes. [14:18:37] well, no, just tools-webproxy [14:18:37] Nothing wrong on my end it seems [14:19:10] Cyberpower678: still? [14:19:19] yes as well [14:19:26] Yes [14:19:35] andrewbogott, ^ [14:19:58] Crap. It looks like redis lost its cache. I have to restart the webservices. [14:20:03] * Coren goes do just that. [14:20:43] ~200-something jobs rescheduled. This will take a while. [14:20:51] :p [14:21:09] Coren, aren't you glad it wasn't me that caused it? :D [14:21:18] At least the system was designed to make that possible, if annoying. [14:21:55] YuviPanda|brb: Isn't the redis setup on tool-webproxy supposed to back its store on disk so that it survives reboots? [14:22:36] Reasonator is back [14:22:44] Up now [14:22:45] Coren: I don't think that's right... [14:23:03] The problem that I saw was that redis couldn't find its logdir (which was my fault) [14:23:06] and hence wouldn't start [14:23:12] andrewbogott: Oh. [14:23:18] there are a few more services with that same problem, which I'm trying to mop up [14:23:38] But things should work tolerably in the meantime [14:24:46] andrewbogott: Well, the global restart is harmless anyways. [14:24:49] Man, would it kill folks to ensure that a log dir exists before writing to it? [14:25:39] "t has been 0 days since the last incident" Haha... what is the record? [14:28:36] 3Wikimedia Labs: Hindi Beta Wikipedia appears to be broken - 10https://bugzilla.wikimedia.org/65677 (10Amir E. Aharoni) 3NEW p:3Unprio s:3normal a:3None If I go to http://hi.wikipedia.beta.wmflabs.org/ , I get the following error in the browser: MediaWiki internal error. Original exception: [6ccdf071... [14:29:38] hedonil, hi [14:30:52] Coren, I can't think of a good way to get a full list of all services that are upset because /var/log/ is missing. I've fixed the ones that puppet was explicitly trying to start... [14:30:54] any thoughts? [14:33:02] andrewbogott: The contents of /var/log underneath the mount should be able to help; that's why I normally mv it out of the way first. There's a workaround: [14:33:15] mount -o bind /var /mnt [14:33:22] Then look in /mnt/log [14:33:47] What does -o mean? [14:33:54] 'option' [14:34:16] -o bind makes it a bind mount, which doesn't traverse filesystems (so you'll see "under" the mounted /var/log) [14:34:22] oh, now I follow, I misready before [14:37:43] Ah! Cool to know. [14:39:52] ok… so, hedonil, phe, things are working now? (If so I'm going to have breakfast) [14:41:11] andrewbogott: Cleared cache and stuff is working now. [14:42:26] great, thanks [14:43:51] andrewbogott, working now [14:58:44] andrewbogott: yep. thanks. enjoy breakfast [16:31:02] Coren: Isn't 1800 on a Friday the way to maximise tool downtimes? [16:32:12] a930913: I saw it as a way to maximise maintainer availability rather. [16:32:42] Many are mostly avaliable in and around weekends. [16:32:45] !log deployment-prep Upgraded elasticsearch to 1.1.0 on deployment-logstash1 [16:32:47] Logged the message, Master [16:37:01] Coren: yes, it does persist to disk [16:37:05] reading backlog now [16:38:37] Coren: andrewbogott what happened? Redis died because of some FS issueds? [17:16:50] YuviPanda: I emailed labs-l… far as I can tell it was just that /var was full [17:19:00] andrewbogott: ah, hmm. It's the second time in a few days where we've had something that's a part of tools crash because of out of disk-space [17:25:33] 3Wikimedia Labs / 3deployment-prep (beta): beta cluster: deployment-cache-upload02 does not seem to recieve htcp purges properly - 10https://bugzilla.wikimedia.org/65683 (10Bawolff (Brian Wolff)) [17:30:48] 3Wikimedia Labs / 3deployment-prep (beta): beta cluster: deployment-cache-upload02 does not seem to recieve htcp purges properly - 10https://bugzilla.wikimedia.org/65683#c4 (10Jean-Fred) > *Does not appear to be a gwtoolset problem, but a beta cluster configuration > problem. This issue will happen on the l... [17:38:33] 3Wikimedia Labs / 3deployment-prep (beta): shell wrapper to connect to databases - 10https://bugzilla.wikimedia.org/45706#c3 (10Bryan Davis) (In reply to Antoine "hashar" Musso from comment #2) > We should use the production sql script. It does not work right now because: > - wikiadmin_pass belongs to the w... [17:44:03] 3Wikimedia Labs / 3deployment-prep (beta): [OPS] udp2log prevents udp2log-mw from starting - 10https://bugzilla.wikimedia.org/38995#c2 (10Bryan Davis) 5NEW>3RES/WOR udp2log now runs on deployment-bastion and seems to start with no issues. [17:46:33] 3Wikimedia Labs / 3deployment-prep (beta): MediaWiki code update sometime leave deleted files - 10https://bugzilla.wikimedia.org/52316#c3 (10Bryan Davis) Antoine and Greg, is this still valid? I haven't heard about this issue since we moved into eqiad. [17:47:33] 3Wikimedia Labs / 3deployment-prep (beta): beta cluster: deployment-cache-upload02 does not seem to recieve htcp purges properly - 10https://bugzilla.wikimedia.org/65683 (10Greg Grossmeier) s:5major>3normal [17:48:33] 3Wikimedia Labs / 3deployment-prep (beta): MediaWiki code update sometime leave deleted files - 10https://bugzilla.wikimedia.org/52316#c4 (10Antoine "hashar" Musso) 5NEW>3RES/FIX That was most certainly some issue with git/ pmtpa NFS server. For some reason some files would be deleted. I haven't spot th... [17:57:48] 3Wikimedia Labs / 3deployment-prep (beta): beta cluster: deployment-cache-upload02 does not seem to recieve htcp purges properly - 10https://bugzilla.wikimedia.org/65683#c5 (10Antoine "hashar" Musso) The purge destination is configured in operations/mediawiki-config.git in wmf-config/squid-labs.php : $wgHTC... [18:11:33] 3Wikimedia Labs / 3deployment-prep (beta): beta cluster: deployment-cache-upload02 does not seem to recieve htcp purges properly - 10https://bugzilla.wikimedia.org/65683#c6 (10Bawolff (Brian Wolff)) [mid-air collision] Hmm, vhtcpd seems to at least be getting some packets: http://ganglia.wmflabs.org/latest/g... [18:17:33] 3Wikimedia Labs / 3deployment-prep (beta): beta cluster: deployment-cache-upload02 does not seem to recieve htcp purges properly - 10https://bugzilla.wikimedia.org/65683#c7 (10Bawolff (Brian Wolff)) > On varnish side: > > deployment-cache-upload02.eqiad.wmflabs 25 2014-05-23T17:56:19 0.000068188 > 127.0.0.... [18:55:03] 3Wikimedia Labs / 3tools: /var is full on tools-webgrid-01 due to me spamming /var/log/auth.log with sudo - 10https://bugzilla.wikimedia.org/64683#c2 (10Tim Landscheidt) 5ASS>3RES/FIX I've now moved auth.log.4.gz to auth.log.5.gz and /data/project/admin/auth.log.scfc.bz2 (re-compressed) to auth.log.4.gz. [18:59:23] petan: what's the current status of the 'bots' project? Is it largely defunct, or are folks still using it as a dev platform? [19:30:19] 'mostly' defunct [19:30:48] 3Wikimedia Labs / 3deployment-prep (beta): beta cluster: deployment-cache-upload02 does not seem to recieve htcp purges properly - 10https://bugzilla.wikimedia.org/65683#c8 (10Antoine "hashar" Musso) Purging the upload.beta.wmflabs.org URLs definitely send PURGE requests to varnish. I double checked it via... [19:30:53] Damianz: Yep, I read the docs finally, and it is clearly labeled as such. [19:31:14] Damianz: mostly I'm wondering if there is currently a catch-all project for random tool devs who want to administrate their own machines. [19:31:22] I think 'bots' was that catchall, for a while at least. [19:32:02] bots was that, tbh right now I'm not sure - I don't think bots is 'officially' used anymore [19:32:21] so maybe we need a new project like 'tools-wildcat' [19:32:43] tbh, for misc stuff... just 'wildcat' would do... even if it's a project with 1 instance [19:32:58] Otherwise you're into sudo groups and all that pain with admins above admins, which was bots [19:34:11] hm... [19:34:58] Hmm, I really need to finish my tools ansible stuff... reminds me, why is 'become' shitty when it comes to ttys? would be nice to be able to do ssh tools-login 'become bot soemthing' [19:35:14] * andrewbogott knows nothing [19:38:09] heh [19:38:36] Know what happens if you point 2 tomcat instances to 1 webapp? [19:38:42] * Damianz thinks this could be interesting [19:39:04] Damianz: any reason you prefer 'wildcat' to 'tools-wildcat'? [19:39:09] * andrewbogott is in the mood to bikeshed apparently [19:40:04] andrewbogott: Coren: Looks like the crontab that keeps ecmabot-wm running disappeared. ...DATA.crontab was empty. I've fixed it (it was still doing */5 sh submit.sh, where submit.sh did jstart) [19:40:17] I fixed it by using jsub with the relevant parameters instead. [19:40:27] just fyi in case there's something more to it that shouldn't have happened. [19:40:46] Krinkle: do you think it vanished recently, or could it have been a few weeks ago? [19:40:49] tools.ecmabot [19:40:58] The bot was kept running until a few days ago [19:41:11] andrewbogott: Not really, just project per tool over general catch all. tools-xxx might get confusing for people... not really sure it needs a standard defining tbh [19:42:05] anybody know how to get a coloured terminal (shell for tools-lab).. [19:42:34] andrewbogott: I think petan uses Bots only for wm-bot; everything else is gone. I'd be quite hesitant for (tools-)?wildcat. I'd much rather speed up the process of greenlighting projects than having one project with no clear purpose. [19:43:00] Creating/deleting projects should be simple and that would solve so many problems [19:44:30] scfc_de: I started out agreeing with you an hour ago. But I'm torn… sometimes a user wants a single box to run multiple (unrelated, but all useful) tools. And setting up a project-per-tiny-tool seems a waste of resources. [19:45:39] should people be encouraged to make 'mini tools' setups rather than just add features to tools for that use case though? [19:45:44] Does anybody know what the bug is that causes wikitech notification emails to be essentially empty? [19:45:57] that's a bug? [19:46:00] Damainz: You mean like additional tools-like projects ? [19:46:08] bd808: There's a Bugzilla item for that ... [19:46:48] bd808: https://bugzilla.wikimedia.org/show_bug.cgi?id=53778 [19:46:49] andrewbogott: Yeah - Like hosting multiple projects on an instance is very mini-tools. Specific requirements should get a project each as required imo [19:47:13] Though that depends on how easy getting changes into tools/testing changes for tools is [19:47:20] scfc_de: Thanks. Maybe I'll be bored enough this weekend to poke at it. [19:47:31] Damianz: getting changes into tools isn't too hard... [19:47:31] Damianz: I think part of my mindset is not wanting to squash the spirit of play… if a user wants free reign to screw around with a server I hate to say no. [19:48:00] And there's overhead associated with creating a new project per user [19:48:41] andrewbogott: We should be able to remove new project overhead and make easy deletions possible though... maybe have a scratch project + a script to kill instances after 48hours for 'playing'? [19:48:53] I agree sometimes it is useufl to just throw 1 class on a box and test something [19:48:59] andrewbogott: What additional resources does a project consume? Problem a) is multiple roots on one instance not coordinating, b) people setting up something unmaintainable and c) vanity ("my tool needs its own instance!!!eleven!"). [19:49:16] Y'all are treading very close to saying "Instead of why not just " [19:49:18] YuviPanda: Not tried recently tbh... experience says it's not hard as long as you can find someone willing to merge [19:49:34] yeah, and usually pinging andrewbogott or Coren gets it merged easily [19:49:44] When they're not flying for 3 weeks ;) [19:49:44] andrewbogott: No, I'm saying let's find the problem first before we set up a solution :-). [19:50:03] Yes well, if Danny_B would like to chime in here about 'the problem'... [19:50:33] (meanwhile I have to go eat a sandwich, my brain has stopped working) [19:50:40] Bon appetit! [19:50:49] my brain stopped working a long time ago, now I just operate on auto pilot [19:57:43] I guess by 'the problem' you probably meant the problem of project overhead... [19:57:44] hm [20:01:09] i hear my name [20:01:14] what's up? [20:03:20] rain [20:03:46] same here [20:03:46] what else? ;-) [20:04:07] andrewbogott's misc tool project problem definition [20:07:47] * Danny_B dives in backlog [20:08:30] I think he's finally found his sandwich so logs are probably the best idea [20:09:33] Damianz: re the become and ttys - google for ttyallow, it may help you if that's your case [20:10:37] Yeah - I wored around it by just directly using sudo.. need to look into that again some more [20:15:49] hmm, i am not any smarter from the conversation above since it uses bunch of idioms and other stuff i am not familiar with :-/ [20:16:19] let me describe the thing in my own way [20:17:03] 3Wikimedia Labs: [Regression] wikitech.wikimedia.org is sending empty Echo notification emails - 10https://bugzilla.wikimedia.org/53778#c5 (10Bryan Davis) It looks to me like OpenStackManager needs to implement the onBeforeCreateEchoEvent hook [0] to provide the proper messages for the email subject and body.... [20:17:43] when we discussed the idea of wmflabs in haifa, i have been told that any user can have his own vm(s) to do pretty much whatever he wants to on them (as long as it it useful and related to wikimedia world) [20:18:07] that it will be much more freedome than toolserver was [20:18:39] now toollabs is much less freedom than toolserver was and it is obviously hard or even impossible to get the vm [20:19:15] so from this point of view i see it as a move towards worse than better [20:19:33] the key idea was independence on admins [20:19:56] anybody can administrate his own vm(s) and do whatever needs/wants to [20:20:03] (modulo security issues of course) [20:21:45] on toolserver users had their own home where they had various tools bots etc. some in development stage, some semi private (aka for certain scope of users), etc. [20:22:56] on toollabs this is not possible (i am not questioning that particular fact) thus it should be possible to easy get own vm to develop and run tools and stuff which is not possible to run on toollabs [20:23:33] from various reasons (interactive stuff is not allowed, dependent sw is not installed and won't be etc.) [20:23:51] sw? [20:23:57] software [20:24:00] oh [20:24:36] So in the sense that tools (project) provides 'toolserver like setup in labs', imo that should be managed as a shared platform as is... in regards to creating instances to do with what you want - that can be done in 2 ways, shared project (sudo groups limiting root access) or project per thing (restricted to admins, can be shared if wanted). [20:24:56] Only issue atm is project creation/deletion is manual [20:25:57] Damianz: tools is no longer 'toolserver like setup in labs' unfortunately. in certain cases is more restrictive than toolserver was [20:27:36] I'm not totally familar with toolserver tbh - aside for license restrictions and the more open (none private bots) and shitting crontab, is there anything else majorly different? [20:27:37] anyway, i'm simply looking for my own vm where i can as a root set up things as i need them and run whatever i need (again, modulo obvious issues like illegal, irrelevant or insecure) [20:27:54] Tbh I'm not totally up for re-creating toolserver, but defining what people need/want and meeting that [20:28:18] 3Wikimedia Labs / 3deployment-prep (beta): beta cluster: deployment-cache-upload02 does not seem to purge content when getting PURGE - 10https://bugzilla.wikimedia.org/65683#c9 (10Bawolff (Brian Wolff)) Changing title to reflect the HTCP part of purging is working fine, its what varnish does with the purges... [20:28:20] Damianz: i can not run "interactive tools" on toollabs for instance [20:30:36] ok [20:31:31] I still think the best case is seperate projects, but that has associated admin overhead. Doing a generic project with admins then sudo groups is potentially more overhead hmm, but what you want is simple overall. [20:32:14] whois FastLizard4 [20:32:39] Damianz: i never spoke about sudo groups [20:33:28] also again, there are tools you can not run on toollabs, while it was possible to run them on toolserver [20:33:47] and obviously it would not be good to loose such tools [20:35:22] so let's get to the point - how can i obtain vm? [20:47:06] hey andrewbogott , Flow engineer mlitn created aft5.eqiad.wmflabs with role::mediawiki-install::labs , but it's not set up [20:47:10] Damianz: so, you see, the question that I'm trying to answer with Danny_B is 'where'? [20:47:27] `sudo puppetd -tv` fails with "err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class role::mediawiki-install::labs for i-00000379.eqiad.wmflabs on node i-00000379.eqiad.wmflabs" [20:48:01] not at all urgent, but I'm not sure what went wrong. Should I file a bug? [20:48:37] spagewmf: Sounds like that role does not exist. Lemme see if I can find what to use instead. [20:49:01] I created an ee-flow-mlitn instance with role::labs::vagrant for him and that worked fine [20:49:13] Danny_B: What tools can't you run on Tools? [20:55:51] Coren: https://gerrit.wikimedia.org/r/#/c/132605/ ? [20:56:30] Ah, I guess I see what happened... [20:56:35] collateral damage :( [20:56:46] sounds like a movie... [20:57:25] Danny_B: What tools can't you run on Tools? [20:58:56] scfc_de: i honestly do not want to run into the same discussion loop again... so in a nutshell: been told that on tools you are not allowed to run interactive tools (one example: irssi) [20:59:42] the other thing is, that on tools not everything is willing to be installed by its admins [21:00:03] What not? [21:00:13] why asking me, ask them [21:00:28] besides this all is not the core [21:00:42] Danny_B: I think that unless this conversation happens in an email thread or bugzilla you may wind up having it lots of times :( [21:00:49] the core is - the original idea of labs was anybody can have its own vm [21:01:17] andrewbogott: enough for me that i've discussed with labs admins [21:01:27] Danny_B: I am one of them :-). That's why I ask. [21:01:46] scfc_de: eh, then púerhaps the last one who hasn't heard yet ;-) [21:01:55] anyway [21:01:59] spagewmf: sorry, that role was completely wiped out, I guess by accident? I'm replacing it and frowning. [21:02:32] Danny_B: right, but the advantage of email and/or bugzilla is that the conversation persists :) [21:02:38] (And in general, I certainly would object to the idea that Labs should provide a VM for everyone. It should help produce free content, and an interactive IRC client isn't necessarily the best use case for that.) [21:02:46] Who runs the wikisense project? [21:02:55] currently in the amount of restrictions it is toollabs > toolserver > own vm on labs [21:03:22] or vice versa, amount of freedom is vm < toolserver < toollabs [21:03:49] but getting the vm is obviously hard if not impossible [21:04:19] which completely changes the original aim and purpose of labs [21:04:25] andrewbogott thx. How does Special:NovaInstance know what roles to offer in its checkboxes? Interestingly, the [?] link for the role shows something, unlike the labs-vagrant role [21:04:30] scfc_de: I'm convinced that Danny's use of irssi is labs-appropriate. But since we currently organize projects by use rather than by user it's unclear where it would go. [21:04:37] which were supposed to be much more flexible than toolserver [21:04:42]