[02:34:03] 3Tool Labs tools / 3[other]: enwp10 tool is down - 10https://bugzilla.wikimedia.org/66565#c4 (10Mike Linksvayer) 5RESO/FIX>3REOP It is down again. [04:13:46] 3Tool Labs tools / 3[other]: geohack.php - Internal Error - "camera location" links on commons broken - 10https://bugzilla.wikimedia.org/67785#c3 (10Daniel Zahn) 5UNCO>3RESO/FIX it did not when i reported it, now it works again. ok,well..resolving [04:14:48] 3Tool Labs tools / 3[other]: wiwosm - broken links from commons to open street map - 10https://bugzilla.wikimedia.org/67786#c2 (10Daniel Zahn) 5UNCO>3RESO/FIX did not when i reported it, now it works again..resolving [10:08:48] 3Tool Labs tools / 3[other]: enwp10 tool is down - 10https://bugzilla.wikimedia.org/66565#c5 (10Tim Landscheidt) 5REOP>3RESO/FIX I've restarted it again. [11:50:24] hi there! still having the same problem as yesterday, when my bot is running under SGE, it keeps sending an error with tools-db [11:50:24] (2005, "Unknown MySQL server host 'tools-db' (11)") [11:50:32] 3Wikimedia Labs / 3tools: libvips-tools, libtiff etc install - 10https://bugzilla.wikimedia.org/52717#c18 (10Tim Landscheidt) Filed RT #7852. [11:52:13] Toto_Azero: Do you know which host your bot is executed on or could you log that as well for debugging (environment variable INSTANCENAME)? [11:52:52] let me check [11:53:12] Toto_Azero: you can also check this for old jobs with qacct -j [11:53:32] Merlissimo: great thanks :) that was actually the command I was looking for :) [11:53:35] (Thing is: You're (so far :-)) the only one reporting the issue, and if it was a systematic error, I would expect more complaints :-).) [11:54:31] but hat the same problem two days ago on 06. bei my bot framework automatically makes a 2nd try after 1 minute, which was successful [11:54:58] scfc_de: from tools-exec-11.eqiad.wmflabs it just got an error less than 10 minutes ago [11:55:33] (and maybe I’m doing something wrong, though I didn’t change anything :) ) [11:55:41] Toto_Azero: Let's see. [11:56:16] Ah! Okay, "mysql --defaults-file=replica.my.cnf -htools-db" fails there for me as well (works on -06, though). [11:56:25] I’m looking if there’s other hosts implicated [11:57:05] So apparently the tools-db alias is only in /etc/hosts which isn't puppetized yet. I'll sync them manually. Hang on. [11:57:22] tools-exec-13.eqiad.wmflabs seems to have the same problem too [11:57:48] tools-exec-08.eqiad.wmflabs looks fine [11:58:13] tools-exec-09.eqiad.wmflabs fine too [11:59:12] !log tools tools-exec-11, tools-exec-12, tools-exec-13: cp -f /data/project/.system/hosts /etc/hosts [11:59:14] Logged the message, Master [12:00:19] scfc_de: is it ok ? :) [12:00:36] Toto_Azero: So the problem was with the alias not being set up in the newly added exec nodes. It /should/ work now, if you see any more signs of that, just ping me please. [12:01:15] no problem :) thanks ! [13:11:55] Hm.. labs has 100s of dead redirects from instance renames that were then subsequently deleted [13:11:58] left behind by 127.0.0.1 [13:12:46] Krinkle: You mean wikitech? [13:13:31] Yeah [13:13:33] https://gist.github.com/Krinkle/5cfdf40900091a508185 [13:13:41] https://wikitech.wikimedia.org/wiki/Special:Log/Krinkle [13:13:51] Yay for 10-line bots in javascript from within the browser [13:27:15] Krinkle: Wikitech, as an interface, is an unmitigated mess. This is why reviewing the entire Openstack UI is on our radar for the year. IMO, trying to implement this with a wiki was... ill-advised to begin with. [13:28:39] Coren: any plans on getting the bugzilla db available on labs? [13:29:12] Coren: Having it be inside wikitech (documentation and project coordination for wmf production) also seemed odd to me. I never got why we merged the two. [13:29:24] Having labsconsole.wikimedia.org back as a non-wiki would be interesting [13:30:14] We now have production and project documentation, labs management and individual tools all in one happy namespace :) [13:30:30] Betacommand: It's not a 'major project' so isn't on the planning thing; but there is no technical obstable to the concept. I seem to remember Andre having a few reservation because security bugs, but it's probably doable (with his involvement) [13:31:51] Krinkle: Managing Openstack through a mw extension dates from before my arrival. It always seemed to me to be a silly way of doing it. :-) So yeah, review this year with alternatives ahead. [13:32:12] I know, I was there when Ryan built it :) [13:32:40] I'm not sure why it made sense. It made sense to everyone, including myself. [13:32:59] Not that my questioning it would've made a difference though. [13:33:13] Ryan sold the idea well :) [13:34:02] I guess one motivation would be mediawiki as a web app framework that we all know (database abstraction, output management, configuration etc.) [13:34:38] as well as available abstraction for loading and bundling front-end resources, using jquery.ui etc. [13:35:09] The part where it generated wikitext pages (instead of like Special:Instance/) was weird though [13:35:46] but made sense becayse that's the only built-in mediawiki has for versioning any data [13:36:18] And being able to see when what happened and being able to diff between them is useful I guess :) [13:40:28] I guess once Phabricator is up and running, it might be interesting to move it there. [13:50:39] Krinkle: I would have expected the code review extension would have been a good lesson learned that trying to use mediawiki as the front end to non-wiki stuff is not always a good idea. :-) [13:51:13] scfc_de: I doubt it would fit in Phabricator [13:51:31] (or should) [14:23:48] Coren: we have little other choice at the time :-D [14:24:06] Coren: Brion wrote CodeReview in 3 or 4 days so that was a cheap investment [14:27:00] Yeah, OpenStackManager predated Horizon so it wasn't that weird of a choice originally. [14:27:30] ryan blogged about it iirc [14:28:37] ah [14:28:39] Coren: andrewbogott Krinkle http://ryandlane.com/blog/2011/04/09/why-i-chose-mediawiki-for-my-openstack-manager-project/ [14:29:03] that is from April 2011 [14:29:52] And actually some of the data tracking and SMW integration is kind of cool. I don't immediately know how we'll do without that if we move to horizon. [14:29:53] 8 days after I was hired [14:29:56] (officially) [14:30:03] I mean, it doesn't work that well, but it's cool when it does :/ [14:30:11] April 1st, 2011 [14:30:15] :D [14:31:26] will want to evaluate horizon [14:31:42] maybe on a test openstack build on top labs? :D [14:32:30] Probably. I'd like to set it up as an alternative/simultaneous option to OSM, but a bunch of things would get out of sync... [14:32:54] switching to Horizon proper will involve rewriting a bunch of OSM stuff in the form of horizon plugins. Which sounds like fun to me but isn't necessarily the best use of our time [14:37:04] Yeah, I don't think Ryan was insane - he clearly wasn't - but it's just that the best solution at the time may not be a good solution. [14:37:26] hihihi [14:50:16] ah Coren ! [14:50:29] Coren: will you be around week of July 21st - 25th ? [14:50:40] Coren: would need to talk with you about jenkins isolation / openstack :D [14:51:52] hashar: Yeah, I'm there that week. I think the only set thing on my schedule is a meeting or two. [14:52:21] Coren: what time works best for you? [14:52:29] I guess we have nice overlap in your morning which are my afternoons [14:54:25] hashar: Best time to grab me is between 13:00 and 16:00 UTC; I'm around later than that but that'd be annoying for you. :-) [14:54:56] Monday 21st @ 13:00 UTC? [14:55:43] hashar: Fine with me. Send an invite? [14:55:49] doing so [14:55:57] if I figure out how gmail calendar works [14:59:34] done! [16:17:02] 3Wikimedia Labs / 3tools: Include pagecounts dumps in datasets - 10https://bugzilla.wikimedia.org/48894#c22 (10Marc A. Pelletier) There will be a new server allocated for dumps and pagecounts which will give us several times more space than we need, and will return that space from labstore1001 back to the us... [17:47:11] !ping [17:47:11] !pong [17:51:02] 3Wikimedia Labs / 3Infrastructure: "The specified resource does not exist" when you try to configure an instance and are not a projectadmin - 10https://bugzilla.wikimedia.org/65379#c1 (10Matthew Flaschen) Testing with bastion (since I'm not a projectadmin, and it has instances), I can't see the list at https... [19:28:21] scfc_de: nobody's /var seems to be blowing up :) http://tools.wmflabs.org/giraffe/index.html#dashboard=ToolLabs+Basics&timeFrame=1h [19:28:21] yay [20:05:34] 3Wikimedia Labs / 3tools: Useful graphite metrics to be tracked for Tool labs (tracking) - 10https://bugzilla.wikimedia.org/67879 (10Yuvi Panda) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier Tracking bugs for metrics that should be collected from tools specific infrastructure. [20:08:03] 3Wikimedia Labs / 3tools: Track 5xx error stats on Graphite - 10https://bugzilla.wikimedia.org/67880 (10Yuvi Panda) [20:08:03] 3Wikimedia Labs / 3tools: Useful graphite metrics to be tracked for Tool labs (tracking) - 10https://bugzilla.wikimedia.org/67879 (10Yuvi Panda) [20:08:06] 3Wikimedia Labs / 3tools: Track 5xx error stats on Graphite - 10https://bugzilla.wikimedia.org/67880 (10Yuvi Panda) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier To increase reliability of toollabs as a whole. Potentially in the future track them *per tool*. Should probably use logster from modules/... [20:09:19] 3Wikimedia Labs / 3tools: Track gridengine stats on Graphite - 10https://bugzilla.wikimedia.org/67881 (10Yuvi Panda) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier Which stats exactly need to be determined. [20:12:04] 3Wikimedia Labs / 3deployment-prep (beta): Set up graphite monitoring for the beta cluster - 10https://bugzilla.wikimedia.org/52357#c3 (10Yuvi Panda) 5NEW>3RESO/FIX Graphite stats from deployment-prep are now sent to graphite.wmflabs.org. Labs Graphite is soon going to get 'real hardware' and then the da... [20:19:18] 3Wikimedia Labs / 3tools: Track labsdb stats on Labs Graphite - 10https://bugzilla.wikimedia.org/67884 (10Yuvi Panda) [20:19:19] 3Wikimedia Labs / 3tools: Track labsdb stats on Labs Graphite - 10https://bugzilla.wikimedia.org/67884 (10Yuvi Panda) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier Would probably contain things like: 1. Replag 2. General system stats 3. Long running queries (in some form? Average query exec length m... [20:19:31] 3Wikimedia Labs / 3tools: Useful graphite metrics to be tracked for Tool labs (tracking) - 10https://bugzilla.wikimedia.org/67879 (10Yuvi Panda) [20:20:03] 3Wikimedia Labs / 3tools: Track gridengine stats on Graphite - 10https://bugzilla.wikimedia.org/67881 (10Yuvi Panda) [20:20:03] 3Wikimedia Labs / 3tools: Useful graphite metrics to be tracked for Tool labs (tracking) - 10https://bugzilla.wikimedia.org/67879 (10Yuvi Panda) [20:20:18] 3Wikimedia Labs / 3tools: Useful graphite metrics to be tracked for Tool labs (tracking) - 10https://bugzilla.wikimedia.org/67879 (10Yuvi Panda) a:5Marc A. Pelletier>3Yuvi Panda [20:20:33] Coren: scfc_de https://bugzilla.wikimedia.org/show_bug.cgi?id=67879 tracking bug for useful stats to be tracked that are tool specific. do add / expand [20:25:47] YuviPanda: Thanks! I'll connect "my" bugs later. [20:26:07] scfc_de: :D cool. I don't know enough about grid engine to figure out what to track, so feel free to add those as well [20:26:26] scfc_de: also feel free to assign them all to me :) I'll be using diamond for everything, and already have a few in progress pull requests there [20:32:38] hey, eh, i gotta graceful wikitech [20:34:07] done, updated the SSL cipher list [20:34:20] in the moment apache becomes 2.4 it will support pFS [20:34:35] STS has been enabled earlier today [21:10:30] where do i find apache logs on deployment-bastion again? memory failing me [21:10:37] /srv/something/something/logs [21:11:48] ebernhardson: /data/project/logs [21:12:10] bd808: thats why i couldn't find it :) thanks [21:12:40] Oh, apache access logs? [21:12:45] hmm.. [21:13:19] They may not got anywhere other than /var/log/apache2 on each apache host [21:13:50] nah not access, i meant the exception/fatal/etc [21:13:59] this is right ones :) [21:14:06] excellent [21:15:49] is there a procedure for requesting access to the beta cluster project? [21:18:10] jackmcbarn: Not sure about "official" but asking here or in #-qa is usually good enough. [21:18:19] YOu need to have a signed NDA with WFM [21:18:22] WMF [21:18:31] argh [21:18:39] That's because there are logs in beta that have real ips in them [21:18:48] Which is lame I agree [21:19:06] okay, never mind then. thanks for your help though [21:19:21] <^d> bd808: OH NO NOT IP ADDRESSES!?!?!?!?!!! [21:19:22] hey, but we have a wiki page about _how_ to sign that NDA and how it works [21:19:27] and that is soooo much better than before [21:19:29] * ^d thinks it's incredibly f'ing lame [21:19:30] heh [21:19:34] ^d: Yeah I didn't dream the rule up [21:19:39] <^d> Not blaming you ;-) [21:20:06] https://wikitech.wikimedia.org/wiki/Volunteer_NDA [21:20:32] I know many villagers will come after me with pitchforks, but IPv4/IPv6 addresses are NOT private information [21:20:46] If you want to hide, use tor [21:20:50] the judges ruled otherwise over in europe [21:21:00] but they were wrong [21:21:07] judges are wrong all the time [21:21:08] bd808, umm.... [21:21:28] if we'd let people use WP over tor :) [21:21:30] I have access to that as a volunteer. [21:22:08] bd808: oh, that's a different thing than the identifying requirement for things like OTRS [21:22:20] Oh. Am I totally wrong? I would be happy to be wrong. [21:22:21] <^d> mutante: We could remove TorBlock from beta and then we could use it there :p [21:22:42] that NDA would also give you things like graphite,RT,analytics data... [21:22:49] bd808: no, i meant i thought you meant something else [21:22:54] that's not as bad as what i thought it was [21:22:57] ^d: :) [21:23:29] mutante: Ideally. I don't know that we have solved the "users with NDA" ldap problem yet [21:23:47] for graphite, logstash, etc [21:23:53] bd808, well I've definitely had access before I dealt with any kind of WMF NDA etc. [21:24:18] bd808: afaict it's just about making a new group or adding them to wmf .. [21:24:20] https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Deployment-prep&diff=prev&oldid=70998 [21:24:43] Is it just that you need root be an admin? [21:24:49] s/root/nda/ [21:25:11] "Just" :-). [21:25:37] should be handled by legalpad now, heh [21:25:39] jackmcbarn, we don't require proper identification for OTRS either. [21:25:49] (in general) [21:25:54] Krenair: don't you have to send something in to prove you're over 18? [21:26:04] greg-g: Does a volunteer need to have an NDA to be added as a member of the beta labs project? [21:26:17] No, because a) the limit is not 18, I joined earlier, and b) I just stated my age, not proved it [21:26:44] US contract law :/ [21:27:24] bd808: I thought we decided yes? [21:27:36] Well that's awkward. You're kicking me out then? [21:27:43] how about not logging on beta instead? [21:27:50] 3Wikimedia Labs / 3deployment-prep (beta): deploy websocket feed - 10https://bugzilla.wikimedia.org/67888 (10Peter Bena) 3NEW p:3Unprio s:3normal a:3None It was announced IRC feed is going to be shut down it would be nice to have a test place for devs to implement websocket listeners to utilities. Th... [21:27:59] mutante: That would be ... not very useful [21:28:17] * bd808 is kicking no one out [21:28:22] we really need the IPs ..for ? [21:28:28] just stripping IPs from logs on beta? [21:28:56] Krenair: https://wikimediafoundation.org/wiki/Resolution:Access_to_nonpublic_data [21:29:00] mutante: I'd be all for stripping ips if we can do it in a way that makes whoever made the rule happy [21:29:22] jackmcbarn, what's that got to do with OTRS? [21:29:31] Krenair: doesn't OTRS link there? [21:29:38] bd808: seems it was the board:) [21:29:45] OTRS does not require identification and it doesn't have an age limit of 18. [21:29:46] I wonder if we could just put beta behind yuvi proxy? [21:29:48] I just told you this. Seriously. [21:31:04] LogFormat "0.0.0.0 - - %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %T %V" noip [21:31:07] CustomLog /var/log/apache/what_you_want.log noip [21:31:12] https://docs.indymedia.org/Sysadmin/ApacheLogsWithoutIPs#Micro_howto_how_to_get_rid_of_IP [21:31:29] Krenair: oh, i see, it just says you have to be willing "if necessary" [21:31:35] could have sworn it was a hard requirement though... [21:32:02] 3Wikimedia Labs / 3deployment-prep (beta): deploy websocket feed - 10https://bugzilla.wikimedia.org/67888 (10Peter Bena) [21:32:20] jackmcbarn, Meh. I'd just tell them to remove my account - so they wouldn't [21:32:23] <^d> bd808: Maybe we could tell the board that IP addresses aren't private and should be stricken from that resolution :p [21:32:23] Problem is that users on the machines still see the IP addresses. [21:32:31] mutante: It would have to be done in all the MW logs too. [21:33:02] doesn't tools do some magic to hide IPs? could beta do the same, or would it be too leaky? [21:33:26] <^d> Plus, every random wfDebug() call that might have a random $IP or $USERNAME thrown in. [21:33:30] * ^d cries a little [21:34:08] I think the easier solution is to either pass out membership freely for NDAs or adapt the Beta TOS to say: "No privacy here". [21:34:37] Yeah, let's start kicking people out who suddenly don't meet the new requirements! Great idea. [21:34:46] The funny thing about ips being private is that it only applies to logged in users (see current and past email threads) [21:35:28] * bd808 goes back to doing something productive [21:39:21] Krenair: that's what would be called "grandfathered in" or something [21:41:13] "under 18 year olds who got access before ..grandfathered in" :p [21:43:01] 3Wikimedia Labs / 3deployment-prep (beta): deploy websocket feed - 10https://bugzilla.wikimedia.org/67888#c1 (10Bryan Davis) It should be setup and running at stream.wmflabs.org. That is the external ip for the deployment-stream instance in the beta project. [21:43:11] Heh. [21:58:31] 3Wikimedia Labs / 3deployment-prep (beta): Setup rcstream for beta.wmflabs.org wikis - 10https://bugzilla.wikimedia.org/67888 (10Krinkle) [22:02:40] mutante: NDA doesn't automatically give you graphite/etc, there are still judgement calls if applicable [22:02:51] just like it doesn't give you root [22:04:19] ok, i dont know anything except tickets are open [22:07:51] mutante: "tickets are open" meaning? [22:08:44] the ones asking for graphite access and an NDA for that [22:08:56] maybe it's not the same one.. [22:10:35] mwalker: you around? [22:11:19] I have one (1) user testing the citoid server and for some reason it just isn't working. failure is getting called for the ajax calls to the server [22:11:19] yepyep [22:11:34] I have no idea what's going on. [22:11:44] ah; my guess is that it's a CORS violation [22:12:18] we're both accessing it from en.wiki :( [22:12:25] are you using chrome? [22:12:27] he's literally got my script in his common.js [22:12:32] yeah, and he's using firefoz [22:12:43] that's interesting; usually it's the other way around... [22:12:47] hokay [22:13:03] what's the script you're adding? [22:14:12] oh, yup, it doesn't work for me in FF either. haha, probably jumped the gun on panicking. I always panic at 11pm. [22:15:03] https://en.wikipedia.org/wiki/User:Mvolz/veCiteFromURL.js it's a mess. :P [22:15:38] I can probably figure it out tomorrow. [22:16:38] uh; really dumb question; how do I use it? [22:16:48] I've added it to my common.js [22:16:51] and now I'm in VE [22:17:03] oh you actually have to add the loader [22:17:04] sorry [22:17:29] https://en.wikipedia.org/wiki/User:Mvolz/veCiteFromURLLoader.js [22:17:48] add this line: importScript('User:Mvolz/veCiteFromURLLoader.js'); to your common.js [22:17:59] and then it's in the "cite" menu, cite from url [22:19:01] also, while we're here, I'll accept general tips on actually keeping citoid running [22:19:49] and keeping better logs. i made it slightly more stable yesterday but it still went down today and took translation-server with it, or v.v., and my logs are... not useful [22:20:17] 3Wikimedia Labs / 3Infrastructure: "The specified resource does not exist" when you try to configure an instance and are not a projectadmin - 10https://bugzilla.wikimedia.org/65379#c2 (10Andrew Bogott) I've investigated this some -- it turns out that with the default nova policy, if you aren't a project admi... [22:20:27] I am clearly not doing something correctly... I'm not seeing it load in either browser [22:20:35] but ok -- keeping things running [22:20:41] how do you start citoid currently? [22:20:47] just by hand and you keep a shell open? [22:20:51] I'm using forever right now [22:20:58] which worked like for a day, yay [22:21:13] but then everything broke and i actually had to reboot. [22:21:34] :( [22:21:57] hmm; that should work... [22:22:07] I can give you an upstart script if you want? [22:22:23] basically; that's how we've daemonized parsoid and ocg [22:22:28] ok [22:22:54] mwalker, mvolz: better to use an init script IMO [22:23:16] although for production upstart might be easier [22:23:24] damn those inconsistencies.. [22:23:29] I know right! [22:23:32] mvolz, https://git.wikimedia.org/blob/operations%2Fpuppet/c5fae9e7fc493957be643276c8e0020bc283aad1/modules%2Focg%2Ftemplates%2Focg.upstart.conf.erb [22:23:53] place that in /etc/init (modify it for citoid) [22:24:02] * gwicke wished Ubuntu was using systemd already [22:24:35] ok thanks will give it a try [22:24:54] ah; sorry; place it in /etc/init.d [22:25:13] and then run `initctl reload-configuration` [22:25:16] gwicke, agreed [22:27:04] mvolz, right now you're just logging out to stdout? [22:27:12] I would recommend something like node-syslog [22:27:17] (or winston) [22:27:28] bd808: i hate to bother you, but your the only name mentioned in hashars 'i'll be away email' ... zuul has been running a single parser test for >20 minutes, and everything else is just queued. Do you know the magical poke necessary? https://integration.wikimedia.org/ci/job/mediawiki-core-phpunit-parser/28070/ [22:27:48] Krinkle: ^^ ?? [22:28:03] mvolz, actually; use winston with winston-unix-syslog [22:28:29] mvolz, and then are you just wanting to know how to write logs to know when things fail? basically it's just a matter of adding log lines everywhere and catching exceptions [22:28:40] you can configure winston to log on uncaught exceptions [22:29:04] I would also recommend logging every request (with GET/POST) [22:29:51] ebernhardson: bd808: indeed, hashar coud've mentioned who those other people who have the same knowledge, are. bd808 is mentioned but about beta stuff, not zuul. [22:29:54] ebernhardson: what's up? [22:30:21] Krinkle: well, i +2'd a patch 20 minutes ago and it still hasn't merged. Looking at https://integration.wikimedia.org/zuul/ the queue just keeps growing [22:30:22] mwalker: tnx [22:30:57] Krinkle: the only thing running is the parser test for one, which is now at >24 minutes. I'm guessing the other stuff is waiting for that to finish? [22:31:01] ebernhardson: Right, this happens once a week. [22:31:07] the gallium slave is locked. [22:31:09] will fix [22:31:11] thanks [22:33:33] mwalker: night [22:34:17] mvolz, email me if you need anything; otherwise I see if I can catch you monday