[01:14:03] where are the logs for this chan. Topic doesn't have it [01:15:36] http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs/ [01:16:03] how does wm-bot handle logging? [01:16:25] Krenair: thanks [03:56:55] 6Labs, 10wikitech.wikimedia.org: Build a simple tool to query which instances have which roles / puppet variables - https://phabricator.wikimedia.org/T103995#1407951 (10yuvipanda) https://tools.wmflabs.org/watroles/role/role::puppet::self is a simple version I knocked up in a couple of hours which should do th... [04:04:45] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/MaxSem was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=168184 edit summary: [05:54:25] Hi everyone. :) [05:54:56] I just realised that the blockcalc tool at https://tools.wmflabs.org/blockcalc/ doesn't support IPv6 addresses, and I'm thinking of updating it. [05:55:11] Does anyone know where I can grab the source code? [07:12:19] Nevermind, I've found the source code - although I'm surprised it doesn't seem to be in a git repository [07:19:44] MrStradivarius: o/ tool maintainers are generally free to publish it however they want, the only requirement is that it is licensed under an open source license [08:20:20] 6Labs, 10Tool-Labs, 6Engineering-Community, 6WMF-Legal: Set up process / criteria for taking over abandoned tools - https://phabricator.wikimedia.org/T87730#1408031 (10Qgil) Is the #Labs team considering this task as a July-September quarterly goal, at a team or individual level? Or is there anybody else w... [10:53:09] hi guys! I'm having a problem with mysql using python MySQLdb and tool labs databases. anyone can help me today? :) [10:53:22] i'm having a messag: _mysql_exceptions.OperationalError: (2006, 'MySQL server has gone away'). and I suspect it is because I take to much between queries. But what can I do...if I am processing data in between? [10:53:37] i tried: mysql_con.ping(True) [10:53:45] marmick: disconnect/reconnect [10:54:26] ping just checks if the server has gone away, so you can reconnect [10:54:46] although the mysql man page says "Auto-reconnect is enabled by default before MySQL 5.0.3 and enabled from 5.0.3 on" [10:56:32] how can I do the disconnect/reconnect in that key moment? [10:56:47] my code structure is: [10:56:56] a) create the connection with the db I want. [10:57:12] b) read an arg to choose a method to proceed. [10:57:16] c) use the method till the end, which retrieves info and processes it. [10:57:36] i have the connection outside the method. [10:58:19] *i meant functions (instead of methods) [11:17:38] catch the exception, reconnect? I guess [11:46:30] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Elee was created, changed by Elee link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Elee edit summary: Created page with "{{Tools Access Request |Justification=Bots and Tools to better Wikipedia. |Completed=false |User Name=Elee }}" [12:25:44] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Elee was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=168203 edit summary: [12:25:53] elee: have fun! :-) [12:26:12] elee, oh good, I can add you to xTools now. :p [12:26:20] <3 valhallasw cheers, I'll may need to rotate ssh keys again, ugh [12:26:23] marmick: in the code that runs the query [12:26:25] Hey, morning Cyberpower678 [12:26:52] mutante: if you get a 'server has gone away' exception, reconnect and try again? [12:27:08] FYI, there's someone on IRC lurking around with my acroynym C678. Please note that it is not me. [12:27:41] elee, ready to set up the xTools environment? [12:27:59] Cyberpower678: yeah, let me play with labs first and familiarize myself =p [12:28:05] Cyberpower678: hah interesting, do tell. [12:28:36] elee, are you familiar with puppet? [12:28:39] valhallasw: I'm going to ask you a stupid question. What does Nova Resource stand for/relate to labs? [12:28:55] Cyberpower678: I'm more of a chef shop but know my way around puppet. [12:29:05] elee: openstack nova is the virtualizaiton software that runs labs [12:29:25] Oh good. We need somebody to handle puppet on our environment. None of us know how to use puppet. [12:29:47] valhallasw: wait, I thought Labs was on Mesos? [12:30:27] elee: no, but I think there are some experiments with Mesos in prod, and plans to use Mesos to provide docker-based virtualization on labs [12:30:36] ++ [12:30:54] got a seperate semi related question - difference between Bastion and tool's login? [12:31:24] mostly just a different login host [12:37:23] elee, you now have access to all xTools environments [12:37:34] valhallasw: I'm unable to Create a New Tool - is this expected behavior? [12:37:46] elee: try logging out and in again [12:37:52] Cyberpower678: yeup, just saw the notif on tools.wmflabs - reading things now =] [12:37:59] 'stuff randomly not working until you re-login' is sort-of expected on wikitech [12:38:33] "authentication protocols" [12:38:35] =p [12:39:02] Yeah, still unable to do so. I'm at https://wikitech.wikimedia.org/w/index.php?title=Special:NovaServiceGroup&action=addservicegroup&projectname=tools [12:39:16] and I'm getting "Your account is not in the project tools." [12:40:43] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103: Recover some sql queries - https://phabricator.wikimedia.org/T104134#1408310 (10Multichill) 3NEW a:3coren [12:41:25] elee: :/ [12:41:40] Cyberpower678: is the instances list up to date on NR:Xtools? [12:41:44] Just making sure. ^_^ [12:41:57] elee, probably not. [12:42:04] Yeah valhallasw literally "mediawiki get your shit together on authentication damnit" [12:42:21] elee: ??? I'm not sure how that is related [12:42:52] because this falls completely outside of regular mediawiki's scope [12:43:11] elee: removed and re-added you. please log out, log in again, and try again [12:43:26] valhallasw: its an inside joke from an older mediawiki dev - he once told me "if mediawiki is even remotely involved, auth and user rights probably don't work" [12:43:45] YuviPanda: Not sure if you are around http://wdq.wmflabs.org/stats seems to have stopped updating [12:44:05] ah yeup that did the trick thanks for the bit reset valhallasw [12:44:40] You can now ssh into login.wmflabs.org, webservice.wmflabs.org, webservice2.wmflabs.org, exec-light.wmflabs.org, exec-heavy.wmflabs.org [12:44:44] elee, ^ [12:45:06] They all belong to xtools.wmflabs.org [12:48:30] oh wait... I get it now [12:48:42] Cyberpower678: busy? [12:53:43] Cyberpower678: eh, can you give them a more descriptive name? [12:54:24] valhallasw: I just realized [12:54:30] these are ec2 style instances... [12:54:32] valhallasw, why? :p [12:54:52] Cyberpower678, better question is how? [12:56:22] Cyberpower678: because 'login.eqiad.wmflabs' doesn't indicate what project it belongs to /and/ is awfully generic? [12:56:40] ah damnit I just realized I didn't need access to tool labs all this time [12:56:47] although those DNS names will disappear in the future, I guess [12:56:48] * elee slaps forehead [12:57:06] okay lets deal with this [12:57:50] Cyberpower678: is anything running at the moment on these machines? [12:57:58] Cyberpower678: it's hard to change after the fact, though, so it's probably best to leave it like this for now. But for new hosts, please name them xtools-login, for instance [12:58:33] valhallasw, elee: There's nothing important on them yet. [12:58:51] Cyberpower678: okay - would there be any problem with me asking for them to be nuked? [12:58:55] We can go ahead and delete them, provided I can spawn them with precise again. [12:59:02] yeup that's what I'd want [13:00:17] precise should still be available, and will be for the forseeable future [13:01:02] Cyberpower678: do you know how much traffic we get? [13:01:20] Not at current [13:01:31] I think it'd be reasonable to just have an m1.xlarge and name it xtools-legacy [13:01:37] have it run what we currently have [13:01:44] I'll continue to keep it happy [13:01:53] and we'll work on getting the refresh up and running [13:04:06] valhallasw: for the sake of time - what should I read to request new instances? [13:04:32] (also, how does Wikitech differ from Labs?) [13:04:39] elee: make sure you're and admin in the xtools project, then use https://wikitech.wikimedia.org/wiki/Special:NovaInstance [13:04:45] wikitech is the management interface for labs [13:04:50] valhallasw, he is [13:04:58] and last but not least, does Labs provide load balancing services or is this something we deploy ourselves? [13:05:13] load balancing in which sense? [13:05:27] elee: also https://wikitech.wikimedia.org/wiki/Help:Instances [13:05:43] elee: there's some balancing of hosts between virtualization hosts, I think, but otherwise none [13:07:17] I'd probably deploy memcached or something valhallasw [13:07:24] reading both now thanks [13:08:46] valhallasw: naming scheme wise $service.xtools is reasonable, correct? [13:08:55] elee: xtools-$service [13:09:21] elee: currently all hosts are reachable over $host.$project.eqiad.wmflabs /and/ $host.eqiad.wmflabs [13:09:34] elee: so for now, it's good to have the project name in the host name as well [13:09:36] ah okay [13:09:45] by that you mean from external correct? [13:09:47] (as well as $host directly from any labs hosts) [13:09:55] or do I need to request for machines to be punched through? [13:09:58] no, internally. Hosts are not reachable from the outside by default [13:10:13] roger. How can I get outside setup? [13:10:15] there's a web proxy for web services, for other things you can tunnel through bastion [13:10:27] https://wikitech.wikimedia.org/wiki/Help:Proxy [13:11:23] ah lovely [13:11:56] ah lovely [13:11:59] hrm. [13:12:06] how difficult is obtaining public IPs? [13:13:15] elee: if you really need one, you can request one [13:13:19] ah damnit I need to deal with security groups [13:13:29] yeah this would be for a few months until I get used to WMF [13:13:33] s/WMF/lagbs [13:14:01] elee: tunneling through bastion is the thing to do [13:14:16] valhallasw: yeah this would be what I want to do [13:14:25] uh - any preferences on security group naming? [13:14:26] elee, valhallasw: I'm proceeding to delete the existing instances and replacing them with renamed ones. [13:14:31] otherwise I'm just going to name it legacy [13:15:07] elee: they are per-project, so you can name them how you want. Tools has e.g. 'default', 'grid master', 'webservice' and 'redis' [13:16:38] security group rules are.... whitelist? [13:17:24] yep [13:18:43] okay because of time, I'm going to ask for an IP for xtools-legacy-1 - I'll make sure to do it right next time [13:19:05] hrm failed creates [13:19:56] elee: what do you need the IP for...? [13:20:16] valhallasw: is there a log I can peek at for why my creates are failing? [13:20:17] elee, alright I deleted everything, and respawned instances using precise with names that make more sense. [13:20:37] elee: not immediately, but could be a quota issue [13:20:40] valhallasw: for xtools, which needs to be back up since ages ago [13:20:45] xtools-webservice1, xtools-webservice2, and xtools-webproxy [13:20:50] elee: yes, but whta's the IP for? [13:20:53] ah so you were messing with me =] [13:21:02] valhallasw: to punch through to external net [13:21:11] I also created a proxy and pointed it to webproxy [13:21:12] after I get things running today, I'll deal with actually proxying [13:21:17] oh wait, what? [13:21:31] elee: wut? that's specifically what the webproxy is for [13:21:38] * elee trouts self [13:21:56] I'm not entirely sure how to check quota. as https://wikitech.wikimedia.org/w/index.php?title=Special:NovaProject&action=displayquotas&projectname=xtools seems to be empty [13:21:58] * ToAruShiroiNeko makes trout jerky [13:22:14] valhallasw, now you need to logout and login. [13:22:19] :D maybe [13:22:21] you poke and jerk it until its steaming [13:23:43] okay when I get these autoinstalled instances, is there a default ssh keypair? [13:24:55] ? [13:25:12] I'm in bastion [13:25:17] or wait I"m in tools-bastion, nvm. [13:26:04] elee: to ssh through to the xtools host, you need to either use ProxyCommand or agent forwarding [13:26:55] wait... what? [13:27:09] oh wait [13:27:17] how is access determined for these machines? [13:27:30] SSH [13:27:39] You need a public key on Wikitech [13:28:02] ah okay - so theres an ACL composed of a project's admins, and any instances under that project has their keys [13:28:04] is that correct? [13:29:44] elee: every instance has every key, but only people in the project can login [13:29:56] valhallasw: and this is to the root instance I assume [13:30:08] root instance? [13:30:23] do you mean the root user? you can't login as that. Login as your own user, then use sudo [13:38:59] ugh [13:39:06] this is why kerberos auth is a lot better [13:39:07] done [13:41:25] Cyberpower678: I'm trying to wrack my head from the source [13:41:32] does xtools need mysql? [13:44:06] Yes. [13:44:32] valhallasw, what file contains all the hosts to connect to the correct dbs? [13:45:59] wait, does xtools connect to a sql server not locally? [13:46:29] valhallasw, as in the file that allows the command sql enwiki.labsdb to work. [13:47:11] elee, yes. It accesses the replication DB. [13:47:24] ah okay I'll look for it soon [13:47:36] you know off the top of your head which php modules/libraries it needs? [13:48:04] okay I've got a basic lampstack sans mysql running and php modules on xtools-webserver1 [13:48:15] next thing would be to see what xtools needs [13:48:19] Everything stored on xTools. It has too many dependencies we can't control, so is a rewrite is a must. [13:48:26] blegh yeah [13:48:58] Try not to install anything newer than PHP 5.3 [13:50:14] I tossed 5.3.10 on it [13:50:22] let me know if it needs to go lower [13:50:24] SHould be ok [13:50:30] sounds good [13:50:48] okay so this next step I'm unsure about [13:50:53] since I don't know best practices for wmflabs [13:51:38] With our own instances, we can practically do anything we want. [13:51:40] but long story short "how do we punch webservice into the world?" [13:52:25] oh okay [13:52:26] web proxies [13:52:27] hrm [13:52:35] I used wikitech to set up some sort of webproxy that points to xtools-webroxy [13:52:36] Cyberpower678: /etc/hosts. There's a puppet manifest for it [13:53:05] Cyberpower678: yeup I see it - I'm going to point it right at webserver1 for the time being [13:53:06] elee, ^ contains the mySQL hosts [13:53:22] valhallasw: hrm okay - anything I can read? [13:54:04] hm, not really a manifest [13:54:05] https://github.com/wikimedia/operations-puppet/blob/61841b150b8e37cbefceb5ae0983d259a4a9082b/modules/toollabs/manifests/init.pp#L196 [13:54:12] elee, webproxy should load balance the request load between webservice1 and webservice2. It should used memcached as you suggested. [13:54:32] Cyberpower678: sounds like a plan [13:54:40] I want to see xtools actually working first =p [13:54:58] I want a properly setup environment first. :p [13:55:21] actually working > properly working =] [13:55:52] twentyafterfour: your logo on phab-03 looks really nice [13:56:01] wait actually it'd be reasonable to have xtools-dev.wmflabs.org [13:56:04] I'll do that instead [13:56:35] You can point that straight to webservice [13:56:41] I'll go set it up [13:57:04] sounds good [13:57:09] Done [13:57:10] I'm going to see why I'm failing to add a security rule [13:57:25] Pointing to webservice1 [13:57:33] you can't add security groups post-creation [13:57:52] but you said rule so I bet you're doing it right [13:57:57] ah there we go [13:58:06] I was accidentally also incluiding a source group [13:58:23] all machines in the legacy secgroup have 80 available to g0.0.0.0/0 [13:59:02] Cyberpower678: thanks for the proxy setup it looks correcty [13:59:10] I'm going to delete the one for xtools and repoint it to webproxy [13:59:51] all set [13:59:58] ?? [14:00:14] okay so we definitely see the machine at http://xtools-dev.wmflabs.org/ [14:00:46] :DDDDDD [14:01:44] Cyberpower678: [14:01:49] go to xtools-dev again [14:01:50] elee: [14:02:03] we've got 5.3.10, and I think everything else looks good [14:02:04] Nothing's changed [14:02:09] Cyberpower678: how busy are you right now? [14:02:13] force refresh [14:02:32] I'm remotely trying to set up my dad's new computer I built him. :p [14:02:32] can you paw through the xtools repo and see what php modules/libraries we need? [14:02:36] Cyberpower678: kek [14:03:27] :DDDD [14:03:54] okay what sort of links should I be seeing with xtools? [14:04:00] as in xtools.wmflabs.org/what? [14:04:53] okay so for notes - we don't need mysql as part of the stack [14:06:20] Same as on toollabs except xtools takes the place of tools. So tools.wmflabs.org/xtools/ec -> xtools.wmflabs.org/ec [14:06:37] got an example link? [14:06:47] time to turn debug on and woe over why php sucks =p [14:06:54] "debugging - what is that?" said php [14:07:17] I just gave you one [14:07:53] oh wait I've got one for reference [14:07:53] https://tools.wmflabs.org/xtools/ec/?user=Elee&project=metawiki [14:07:57] so that'd be [14:08:03] Copy out /data/project/xtools and test it. [14:08:11] http://xtools.wmflabs.org/ec/?user=Elee&project=metawiki [14:08:18] or rather [14:08:23] http://xtools-dev.wmflabs.org/ec/?user=Elee&project=metawiki [14:08:35] /data/project/xtools? [14:08:51] We'll need to a shitload of pathway changes. [14:09:03] To get it to initally work. [14:10:06] holy shit why are so many things hardcoded in [14:10:07] jesus christ [14:10:19] elee, ask hedonil [14:10:22] lol [14:10:37] uh what other devs are up and willing to wrench the source into shape? [14:10:42] The rewrite will use a config file and more flexible pathway handling. [14:10:52] valhallasw: i could solve it creating a new function which makes a try-excp with reconnect. :) [14:10:53] thanks. [14:11:35] elee, Technical 13 will help with the rewrite'\ [14:12:12] I thought he asked for a self ban due to that whole sockpuppet ArbCom fiasco? [14:12:26] He's still active on xTools. [14:12:39] sounds good - he around right now? [14:12:43] also [14:12:44] http://www.spacex.com/webcast/ [14:12:51] Only on email. [14:12:52] 8min 30 sec left [14:12:55] such a nerd hardon [14:12:55] ? [14:13:22] 6Labs, 10wikitech.wikimedia.org: Build a simple tool to query which instances have which roles / puppet variables - https://phabricator.wikimedia.org/T103995#1408352 (10Krenair) >>! In T103995#1407951, @yuvipanda wrote: > Code at github.com/wikimedia/watroles 404 [14:14:45] Cyberpower678: making sure - we don't need a mysql server locally correct? [14:14:53] we're just using mysqli to use lab's [14:15:18] If you want to set one up locally, I have no objections. [14:15:32] theres no need, is there? [14:15:54] Not to my knowledge [14:15:59] righto [14:16:05] 5 minutes left guys [14:17:47] 4 [14:17:53] ? [14:18:04] rockets man [14:19:24] 2 [14:20:28] 1 minute - http://www.spacex.com/webcast/ go watch! [14:21:11] 30 seconds guys@ [14:21:18] elee: lets hope it goes better this time [14:21:23] yeah Negative24 [14:21:56] ugh rockets give me a hardon [14:22:11] looking good so far [14:24:25] uh [14:24:33] doesn't seem to look to good anymore ... [14:24:51] was that supposed to happen? [14:25:37] its silent on my end [14:26:44] my feed is delayed [14:26:52] wtf just happened did it disintegrate? [14:27:08] looks like it [14:27:31] it wasn't that "big" per se [14:29:24] yeah [14:29:25] holy shit [14:29:28] no they just lost the rocket :P [14:29:29] so much money wasted [14:29:39] cough cough indonesia [14:29:39] I think its still there [14:29:41] too soon? [14:30:49] elee, what's going on? Does this have something to do with xTools? [14:31:16] yeah, just like the rocket, the current codebase is an explosion =p [14:40:54] Database problem [14:42:12] elee, make sure to remove the redirects in the script index file of articleinfo and ec [14:42:33] elee, and what file path are these located in? [14:56:58] Cyberpower678: okay I'm going to setup a lb with haproxy [14:57:03] its how we're going to KISS [14:59:28] ?????????????????? [14:59:31] What? [15:00:09] Cyberpower678: instead of memcached we're going to use haproxy roundrobin instead [15:00:46] Ok. But Kss? [15:00:49] *kiss [15:00:52] ... [15:00:55] go use urban dictionary [15:01:57] Oh [15:06:35] does labs provide a shared file system? [15:06:42] =p [15:08:39] elee, so where are our files located? [15:08:56] unsure - I was going to have them locally on these instances [15:09:03] but if we have access to NFS, that's useful too [15:09:28] NFS fails consistently. We recently had a catastrophic faiure [15:09:35] that's true. [15:09:42] Local is better, that way it stays up when they go down? [15:09:52] We'll do glusterfs or something between the hosts [15:11:07] lol glusterfs [15:11:35] (we had glusterfs on labs before NFS and it was significantly worse) [15:11:37] but having some sort of local-level shared filesystem might not be a bad idea [15:11:51] yeah which is what I'm thinking [15:12:01] alternatively, something mumble something git hooks auto pull [15:12:09] add this to the end [15:12:12] whgoops [15:22:59] heh [15:23:47] alright great [15:23:50] got load balancing up [15:23:59] Cyberpower678: its up to the others now [15:24:13] what mailing list should I use to keep everyone updated? [15:25:27] Recall everyone's email and list them in the to field. :p [15:25:40] ugh I'll just make an MIT mailing list [15:25:57] Where's the filesystem? [15:26:29] elee, we could convince the WMF to grant us one. [15:26:29] okay so understand [15:26:32] there are no hard files [15:27:07] so we could just have a git hook invoke something to make the servers pull [15:27:10] probably remctl or something [15:27:16] so everything stays synced [15:27:36] We still need a place to put the PHP files. [15:27:41] yes [15:27:45] that's what we do [15:27:47] symlinks [15:28:01] you can easily ask for xtools@lists.wikimedia.org, or you can use xtools@tools.wmflabs.org for now [15:28:23] the latter will just email everyone in the xtools service group on tool labs [15:28:31] valhallasw: does xtools@tools.wmflabs send to all current service group members? [15:28:32] We need to know where the files are going to be pointing [15:28:34] oh there we go [15:28:46] Cyberpower678: how old are you? [15:28:50] relax niqqa [15:29:12] I haven't had to invoke my inner black man in a long time. [15:29:23] I like the mailing list one. Then we can have users subscribe to the list, and at the same time also communicate with each other. [15:29:34] why would users have to sub to a list? [15:29:56] Just a thought. Then they won't consistently hammer the talk pages. :p [15:30:06] no that's what those are for =p [15:31:06] elee, so riddle me this. How do I recode the xTools filepaths on the PHP files, if I don't know where to point them? [15:31:26] [11:29:52] relax niqqa [15:31:28] let me send mail [15:31:43] elee: yes [15:31:59] elee, please don't "invoke your inner black side" [15:32:02] elee: also, please watch your choice of words [15:41:45] Okay Cyberpower678 mail sent. Apologies valhallasw, I didn't realize wm-labs was such an uptight ship ;D [15:41:51] Cyberpower678: bah mail bounced, hold on. [15:42:12] I didn't get a thing. [15:42:14] elee: it has nothing to do with 'uptight', and everything to do with inclusivity. [15:42:16] yeah that address isn't reasonable. I"ll just make a personal mailing list. [15:42:39] Or make on the foundation [15:42:45] valhallasw: standby [15:42:48] I like that idea better. [15:42:51] Cyberpower678: who needs the mail? [15:43:06] You, T13, musik animal, nakon? [15:43:08] MusikAnimal, Nakon, Technical 13 [15:43:23] elee, I prefer a foundation mailing list. [15:43:31] Cyberpower678: gotta send this now [15:44:16] Cyberpower678: I've got you, t13, musik, and nakon. Am I missing anyone? [15:45:17] Cyberpower678: screw it sent. [15:46:38] valhallasw: that's reasonable, although I'm not good with symantics =] [15:51:01] Cyberpower678: are we going to have to deploy ssl at some point? [15:51:31] oh wait nevermind the web proxy handles it =p [15:51:39] https://xtools.wmflabs.org/ that's pretty [15:55:03] Negative24: "Vehicle failure after max Q. NASA says air force terminated flight, Elon says upper stage lox tank over pressurized. More info on Twitter, press conference to come, probably around 12:30 EST." [15:56:39] elee, lol [15:56:47] elee, you sure you're not a robot. [15:56:49]