[07:28:23] so on a labs instance i set it up with a script called crasher.py that crashes every 10-15 seconds or so [07:29:01] and configured email alerting pointing at my wikimedia address [07:29:04] :D [07:29:11] i stopped the supervisord service but left the config file in place, so it got picked up again when i restarted the instance [07:29:12] good times all around [07:29:12] hahaha [07:29:19] ori-l: hey. hey. hey. hey. fix me. hey. you. [07:29:23] overly attached application [07:29:41] http://i.imgur.com/Qb5YFEF.png [07:29:42] hahaha [07:29:42] <3 [07:29:55] Ryan_Lane: the proxy access to the instances ? [07:30:02] Wikinaut: what about it? [07:30:02] is it broken? [07:30:04] yes [07:30:06] ok. let me look [07:30:12] (checking the apaches) [07:30:20] apaches are running [07:30:35] hm [07:30:44] which project has the proxy? [07:31:07] Are there any plans for OpenID as a relying party rather than as a provider? [07:31:20] eventually, yes [07:31:22] but not now [07:31:31] it's easier to introduce things over time [07:32:02] ok. rebooting the instance proxy [07:32:57] It would be nice. It is kind of annoying having to hit the "forgot password" every time one tries to log in after a period of inactivity... [07:34:08] forget password? [07:34:11] where? [07:34:20] you definitely shouldn't be getting that on wikitech [07:35:30] There is a "forgotten your login details" link on the log-in page [07:35:54] yeah, but inactivity shouldn't require a password change [07:36:23] inactivity in the sense of me actually forgetting the password after not needing to remember it for a while [07:36:31] ah [07:36:32] heh [07:36:35] use a password manager [07:36:38] like keepass x [07:37:07] Or convince people to actually implement log-in with OpenID... ;-) [07:37:19] that's going to be pretty hard on wikitech [07:37:45] we need openid+oauth [07:38:14] and we need keystone to support it too [07:38:31] that's my ideal future path, though ;) [07:39:26] :-) [07:40:28] Ryan_Lane: *.instance-proxy.wmflabs.org/wiki still not available [07:40:47] seems I typed in the wrong name [07:41:12] ok, typed the right thing this time [07:41:40] Ryan_Lane: go to bed [07:41:44] ;-) [07:41:47] heh [07:41:49] I should [07:42:58] ok. it's up now [07:43:17] yep [07:43:19] +1 [07:46:46] Ryan_Lane: PM pls [07:56:40] Ryan_Lane: one more PM [08:20:13] addshore I tuned bsql01 let me know when I can restart it [08:21:08] there is some alter running now [08:21:25] I just need an outage of 1 minute or so, when it finish let me know [08:22:09] I think this tuning will improve performance of temp tables and sort performance a lot [08:23:33] !ping [08:23:33] pong [08:23:45] petan: the wm-bot documentation is slightly off [08:23:52] Ryan_Lane did you restart wm-bot [08:23:55] I had to [08:23:56] yes it suck [08:23:59] heh [08:24:02] I will make it better today [08:24:05] thanks [08:24:12] I figured it out [08:24:15] problem is that development is too quick [08:24:19] * Ryan_Lane nods [08:24:20] things changes too fast [08:24:38] but I think current documentation of bot startup is not so obsolete [08:24:50] you just need to start bouncer in nohup and restart.sh in nohup and that's it [08:24:59] ofc you need to kill previous processes [08:25:07] I also needed to nohup mono wmib.ex [08:25:09] *exe [08:25:13] nope [08:25:17] you shouldn't need to do that [08:25:20] restart.sh does that [08:25:29] I couldn't find restart.sh [08:25:33] btw [08:25:37] if you run: mii-tool [08:25:51] anyway I will try to make a init shell but it's too complicated for this case [08:25:56] on an instance, and it shows 100baseTx-FD, if you reboot it, it'll get a 1Gb nic using virtio [08:25:59] maybe I could create two scripts [08:26:11] ok I will reboot it then [08:26:20] bots-1 should already be 1Gb [08:26:25] ah ok [08:26:32] I need to reboot sql server on bots [08:26:44] but addshore has some running jobs there [08:26:51] if you get "SIOCGMIIPHY on 'eth0' failed: Operation not permitted" [08:26:52] it's virtio [08:26:56] aha [08:27:26] I can't figure out how to get the damn link speed when using virtio [08:27:36] it's much, much faster, though [08:27:51] btw restart .sh is in same folder as exe files [08:27:59] dunno why you couldn't find it :/ [08:28:01] hm [08:28:05] I checked for it [08:28:09] not sure why i didn't see it [08:28:13] it's only executable in that folder [08:28:20] doesn't matter... [08:59:01] It looks like virtio probably doesn't have a link speed. It seems to be indeed as fast as the CPU can handle data transfers between VMs on the same host. [10:20:35] Ryan_Lane did you follow this: https://wikitech.wikimedia.org/wiki/Nova_Resource:Bots/Documentation/wm-bot [10:20:49] wmib $HOME is now different, however it should work [10:20:56] you just need to run these 2 commands there [10:21:04] yes, the home location was the issue [10:21:09] ah [10:21:24] if I could change it I would do it :/ but that's the problem when user is in ldap now [10:21:33] just change the docs [10:21:37] mhm [10:21:49] or add the location to the psth [10:21:51] *path [10:21:58] and make the scripts executable [10:22:54] or make an init script [10:23:32] they are executable [10:23:40] only executable in there is restart.sh :P [10:23:46] init script is complicated for this case... [10:23:56] why complicated? [10:24:01] init scripts are simply [10:24:04] *simple [10:24:09] because it needs to start multiple processes with dependencies [10:24:13] and would automatically restart the bot on a restart [10:24:23] I don't mean it's not suitable for init script [10:24:26] it's hard to write one :D [10:24:31] no it's not ;) [10:24:46] well, it needs to [10:24:48] write a wrapper that starts the children [10:24:59] 1. check if bouncer is running [10:24:59] 2. if not start it [10:24:59] 3. check again [10:25:13] nah, it'll write a pid file out [10:25:16] 4. if it's running start the bot and keep watching it [10:25:23] 5. if bot crashes it needs to start it again [10:25:31] if you use upstart it'll do these things, too [10:25:45] 6. if bouncer crashes both bot and bouncer needs to be restarted in proper order [10:25:53] write a wrapper [10:25:58] 7. if you type restart / rehash it must only restart core [10:26:01] have it manage the children [10:26:14] then have the init script manage the wrapper [10:26:25] mhm... I know it's possible just not simple to make [10:26:32] a wrapper would be pretty simply [10:26:44] it starts the children and listens for signals [10:27:02] *simple [10:27:04] it's late :( [10:27:13] mhm I am wondering how you start a process and retrieve its pid [10:27:28] not sure how in mono [10:27:35] I know how in mono [10:27:41] but I don't know how in shell script :D [10:27:43] then use mono ;) [10:27:53] I wouldn't recommend bash for that [10:28:23] I will need to think of that, but IMHO most simple would be to make 2 init scripts [10:28:27] one for bouncer and one for core [10:28:31] that would work too [10:28:39] but they need to start/stop in order, right [10:28:40] ? [10:28:44] yes [10:28:46] that's doable with upstart, though [10:28:52] core would check if bouncer is up [10:29:09] but if they need to restart, then it's likely better to have a single executable that does this [10:29:24] you almost never need to restart bouncer [10:29:24] otherwise they may not restart in the proper order [10:29:29] but you need to restart core often [10:29:50] it's only important for bouncer to run before you start core [10:29:56] ok. I need to go to bed.... [10:29:57] night! [10:30:00] cya [11:58:35] petan: you want to reboot bsql? [12:00:23] * addshore pokes petan with a stick :P [12:04:49] petan ping me when you get back :) [13:01:33] Instance i-00000322 ( signwriting-ase10 ) is not accessible through a browser or ssh ( No route to host ). Problem started last night. Labs says that instance is active. Any ideas? [13:15:15] addshore back [13:15:17] :o [13:15:18] mhm? [13:15:26] addshore @notify? [13:15:27] :P [13:15:28] that work [13:15:30] s [13:15:33] @notify addshore [13:15:33] This user is now online in #huggle so I will let you know when they show some activity (talk etc) [13:17:42] ping MaxSem [13:17:52] pong [13:17:55] :) [13:18:09] I am still fighting with Solr und Solarium in puppet [13:18:25] The problem seems to be the order of things happening [13:18:26] und o.O [13:18:28] :D [13:18:55] do you know how to set order in puppet? [13:20:01] petan! [13:20:02] MaxSem: I told puppet to get solr once WIkibaseSolr and Solarium extensions are there. Nevertheless puppet keeps searching for the schema file right when I start it. Do you have any idea [13:20:14] im going to kill my query [13:20:18] as it is only 25% done >.< [13:20:27] and hope that it runs fast after you reboot :D [13:20:43] MaxSem: that is I required those extensions when including the solr class [13:20:54] Silke_WMDE, commit your work in progress and I'll take a look [13:21:12] ok thanks a lot! [13:22:21] petan: query stopped, ping me when bsql01 is back? :) [13:22:31] addshore can I restart it now? [13:22:35] yep :) [13:26:26] addshore it's abck [13:27:48] on 1gb ethernet [13:28:00] addshore we should reboot other boxes as well to get 1gb [13:28:27] thats fine with me, I'll log into bsql01 and run the query form there :) [13:28:41] mhm I think I will write email before [13:28:47] so that I don't break stuff [13:30:07] * addshore wonder how much petan has optimised bsql01 [13:33:28] 0.7% in a about 2 mins! So Im guessing 5 hours instead of 30? :O [13:34:05] we will see :D [14:34:15] MaxSem: Can I bother you again? [14:34:30] of course) [14:35:49] Yesterday when I had puppet install Solarium via composer it worked with Wikidata items. Now with the Solarium extension the update script fails to create some of the Wikibase specific db tables. Any idea why? [14:45:04] addshore I increased buffers that are needed for your alters [14:45:09] I suppose [14:45:11] :D [14:45:18] I am rather oracle guy more than this :P [14:45:28] * addshore is only just starting to touch oracle stuff [14:45:35] these alters in mysql are using temp tables [14:45:41] yep :/ [14:45:46] they actually recreate whole table from scratch [14:45:51] the copy to the temp table is what takes a long time [14:45:52] piece by piece [14:45:59] and then rename :P [14:46:03] so now these buffers are 10 times bigger [14:46:06] some even more [14:46:06] :D [14:46:09] like 20 times [14:46:15] maybe it will be faster [14:46:31] hmm, *just went and chekced* and its on 7.77% xD [14:46:35] frigging table [14:46:40] heh [14:46:49] you can keep it running over night :P [14:46:57] problem is that ddl usually lock out the table [14:47:02] not everytime [14:47:06] but in this case probably yes [14:47:11] so you can't use your bot now [14:47:25] that's why you should have many small tables rather than 1 huge [14:47:49] Silke_WMDE, first of all, WikibaseSolr doesn't need to mess with Solarium autoloader, ext:Solarium does it itself [14:48:04] but that's not related to updater failures [14:48:58] well, it might be related to what I put into my LocalSettings [14:50:07] do you mean I don't have to specify something like $wgWBSSolariumAutoloader = "$IP/extensions/Solarium/lib/Autoloader.php"; [14:52:12] yes, and all the cruft related to that variable is not needed [14:53:52] I should have my colleague who wrote WikibaseSolr talk to you when he's back from vacation. [14:54:25] addshore but actually the server is quite loaded, iotop shows some 5mb/s writes, given the crappy IO of labs that's a lot [14:54:41] HAHA, thats whats slowing it down :< [14:56:42] mysql doesn't produce many logs :/ [14:56:56] there are 2 log files both 0 size [14:58:04] sure logging is turned on? :P [14:58:31] idk [14:58:36] I suppose it'd by default [14:59:59] it logs to syslog [15:00:01] :D [15:28:22] @notify Beetrstra [15:28:22] I will notify you, when I see Beetrstra around here [15:28:29] @notify Beetstra [15:28:29] I will notify you, when I see Beetstra around here [15:29:13] Damianz ping [15:29:32] can you move cluebot db's to bsql01 [15:32:09] !log removing bots-sql1 [15:32:10] removing is not a valid project. [15:32:20] !log bots permanently removing bots-sql1 [15:32:22] Logged the message, Master [15:37:56] benestar are you "bene"? [15:39:58] petan: yes [15:40:07] so bene* on wiki and bene on labs [15:41:00] benestar ok, you are running bot on bots-4 right? are you using some sql db? [15:41:59] it's not really a problem, but it would be probably better for you to run it on bots-bnr1 which is much larger and faster and more stable [15:42:18] rschen7754 same for you :) [15:42:45] * addshore likes the way this is goin ;p [15:42:53] I don't :D [15:43:00] we really should have some scheduling system [15:43:31] but I would still prefer a system where people can either use scheduling system or just pick an instance by hand, or eventually use some instance with root [15:43:38] Coren's version is too restrictive [15:43:40] petan: I don't need any sql db [15:43:52] benestar okay no problem, but I meant the application :) [15:43:55] maybe later for other tasks... [15:44:05] application is running on bots-4 [15:44:09] which is rather testing instance [15:44:16] :O [15:44:21] it's ok for you to run it there, BUT that instance isn't so stable [15:44:30] well, It wasn't so stable the last time, you're right [15:44:37] also it is very slow compared to bnr1 [15:44:58] how can I move the bot to bnr1? [15:45:28] petan: 10% done ;p [15:45:30] LOL [15:45:33] addshore [15:45:41] howcome there is 13gb of ram used on bnr1 [15:45:46] and 5gb swapping [15:45:53] * addshore checks [15:45:56] benestar: it's simple if you have it in $HOME [15:46:00] * addshore doesnt think he has anything running there [15:46:33] benestar: you just switch off the current bot, and start it again on bots-bnr1, that's all... [15:46:37] petan: do I need any futher rights for that? [15:46:56] benestar: I don't think so, are you running the bot as yourself or are you using some application user? [15:46:57] ahh petan lego is scanning a db dump ;p [15:48:08] petan: using screen [15:48:11] I think it may be a bit memory hungry xD [15:48:18] benestar screen is on bnr1 :) [15:48:23] :) [15:48:30] mono? [15:48:39] ewww, i dont know, i dont use it ;p [15:49:37] benestar yes mono is there I use mono too [15:49:51] nano all the way ;p [15:50:00] legoktm ping!! [15:50:01] * addshore slaps self because it sounds the same ;p [15:50:06] wtf is your bot doingh [15:50:15] it eats too much ram [15:50:40] I think it runs on wikidata, too :P [15:50:42] only 87% ;p [15:51:15] petan: its not a bot, its python xmlrevisionscanner.py , scanning a db dump :P [15:52:29] legoktm: sorry I am killing your job, it's too heavy and it's not even a bot... that's not supposed to be on bots project :/ [15:52:47] but its for a bot? :/ [15:53:22] !log bots petrb: kill 5261, legoktm 5261 5242 81 11:39 pts/4 03:25:25 python xmlrevisionscanner.py [15:53:24] Logged the message, Master [15:53:55] lol [15:54:33] addshore still, not very well optimized [15:54:41] that script was eating more than 10gb of ram [15:54:58] I am pretty sure it is some kind of bug [15:55:06] or maybe not, it's python [15:55:08] who knows lol [15:57:26] Ryan_Lane did you reboot bots-bnr1? [15:57:43] please use !log [16:07:13] I like that php female.php process :> [16:07:15] whatever it does [16:07:31] is it generating porn? :D [16:07:35] lol [16:08:12] we have in iberocoop a thread titled: "Second contest of Iberoamerican women" [16:08:39] petan: :P [16:09:09] as sexist as it may sound, it's not a beauty contest, but one to make wikipedia articles about relevant iberoamerican women [16:09:16] * Silke_WMDE is setting up her feminist watchdog [16:09:31] Silke_WMDE are you a woman? [16:09:38] ah [16:09:40] lol [16:09:42] * petan hides [16:09:42] xD [16:10:27] it can be hard to guess the gender of foreign names [16:10:47] ...and then you have people like River or MZMcBride [16:11:08] hrm, I thought everyone has a basic idea who's working on wikidata [16:11:09] ok and people who say "her feminist" who clearly are women :D [16:11:25] MaxSem I have no idea who is working on wikidata [16:11:35] * MaxSem boos at petan [16:11:36] I just not it's not me [16:11:41] * know [16:11:42] MaxSem, I would need to look up for a list of people adscribed there [16:12:10] is "a bunch of people hired by WM-DE" a complete-enough answer? :) [16:12:48] ok, I think this list should be complete: https://gerrit.wikimedia.org/r/#/admin/groups/32,members [17:09:13] Ryan_Lane: Incidentally, how do these sprints tend to work with peeps split by several megameters? screen -x? :-) [17:20:53] petan: yeah i'm back on bnr-1 [17:24:16] [= [18:08:38] rschen7754 oh if you purposefuly switched the instance it's fine, I just though you didn't know about it [18:08:55] addshore let me know about it :) [18:09:00] aha [18:15:07] * addshore waves [18:22:38] !log bots given admin to addshore [18:22:40] Logged the message, Master [18:38:56] Coren: actually, we set this up so that sprints could be done remotely [18:39:06] Coren: no need to share screens for sprints [18:39:13] just need to be able to break up tasks [18:39:30] and either work together on irc, or via hangout or something [18:39:40] but the idea is to allocate a block of time and do a specific task together [18:39:48] Ryan_Lane: I was mostly jesting. :-) I'm currently reading the code for the extension and catching up on OpenStack docs. [18:39:54] cool [18:40:10] there's a number of features we don't currently have enabled [18:40:15] also, we need to upgrade to folsom [18:45:15] Ryan_Lane: Do we have the list of enabled features somewhere or some way to enquire it? [18:45:55] addshore how's your query? [18:46:23] addshore you can't split the table to multiple? [18:48:10] * addshore logs in to look [18:48:31] 16% [18:48:59] it acctualy make me want to crryyy! [18:49:12] but its fine as I dont need to use the table until WD rolls out [18:49:25] wd? [18:49:28] and I could split iot up but at this stage that would take longer than waiting for this final alter [18:49:33] WD = Wikidata [18:49:37] oh [18:49:51] Damianz ping [18:49:58] @notify Damianz [18:49:58] This user is now online in #wikimedia-tech so I will let you know when they show some activity (talk etc) [18:50:07] here we ho [18:51:15] and petan, as soon as WD rolls out the table is going to be sheding a few hundred rows a minute [18:51:25] maybe even a few thousand xD [18:51:36] lol are you cloning wikidata in that or what [18:51:52] no, but it has details of every page on every project currently with interwiki links :P [18:52:28] bot taks is to remove said links [18:52:31] *task [18:54:49] ttl exceeded [18:55:40] Coren: whatever is in the wikitech interface is what's enabled :) [18:55:46] so, not much [18:55:47] Damianz: on WD? [18:56:06] we don't have resize, snapshots, cloning, or live migration enabled [18:56:25] addshore: no, to petan [18:56:25] * addshore shhhs [18:56:34] ahh! :P [18:56:35] Ryan_Lane: resize would be very much useful. [18:56:59] Coren: totally agree [18:57:02] I'd like to enable that [18:57:11] it requires live migration to be enabled [18:57:16] it uses the same code to do resizes [18:57:20] (which yes, is absurd) [18:57:40] and it needs to be properly tested, as it may actually cause instances to have a different i-xxxx name [18:57:47] which would be problematic [18:58:35] ... that's just dumb. You mean they implement no difference between a resize of a down instance and that of a live one? [18:58:43] * Coren boggles a little. [19:01:14] petan: I think this sql query means another trip to the pub for me ;p [19:01:36] Damianz u here? [19:01:44] !log deployment-prep Log [19:01:46] Logged the message, Master [19:03:49] @notify Damianz [19:03:49] This user is now online in #wikimedia-tech so I will let you know when they show some activity (talk etc) [19:03:49] sort of - in the middle of something [19:03:50] aha lol [19:03:50] !ping [19:03:50] I think you just crashed wm-bot [19:03:50] anyway [19:03:51] pong [19:03:51] Damianz can you move sql of cluebot to bsql [19:03:51] it will be so fast :D [19:03:52] or just make me admin of cluebot and I will happily manage it :D [19:03:53] yeah - probably not today though as it takes a 'while' to dump/import the db [19:03:53] no problem [19:03:54] if you make me admin of cluebot I might do that :D [19:04:00] it survived :D [19:04:06] !ping [19:04:06] pong [19:04:13] but probably deadlocked like hell [19:05:00] legoktm ping [19:05:04] @notify legoktm [19:05:04] This user is now online in #wikimedia-dev so I will let you know when they show some activity (talk etc) [19:05:50] petan if you want to know if lego has any dbs I dont think he does [19:05:59] I think he was just planning on creating one of bsql01 today [19:06:18] no I want to know why he was running 12gb of ram eating python on bnr1 [19:06:57] cause he likes to eat ram too? :D [19:07:16] maybe :D [19:07:28] I want to know if it was important or if I broke stuff by killing :P [19:07:48] I was thinking of that and we could have let it running [19:07:56] in worst case I could increase swap [19:08:16] but if he wanted to run it next time I will create bnr2 lol [19:08:32] btw addshore when ur admin now there are tasks to do [19:08:41] there is file init in my home [19:08:46] Well its a process to parse xml db dumps I think, but I thought he told me he had already finished en wiki, which is the only dump which would have used that much mem [19:08:49] it needs to be puppetized [19:08:53] it initialise application instance [19:09:08] patch it, install all packages, motd fix shares and public_htmk [19:09:12] and stuff like that [19:09:37] on other hand when Coren is working on his project now I am not so sure if we need to do that [19:10:09] but I am still kind of sceptic about his version :o [19:10:34] skeptic lol [19:10:37] petan: :-P [19:10:48] for what reason english is using c somewhere and somewhere k [19:10:56] can't you just use the same letter for same sound [19:11:14] in czech it's actually skeptik [19:11:26] because c and k is pronounciated same in this case [19:11:42] Yeah, but sceptic has a ... different meaning. :-) [19:11:50] what is that [19:11:54] is it a word? :D [19:12:06] btw how you recognize it [19:12:19] it should be pronunciated same [19:13:12] petan is our fav little european [19:13:23] according to http://www.thefreedictionary.com/sceptic is same as skeptic [19:13:28] just archaical version [19:13:33] :D [19:13:36] petan: I will have a look tomorow :) /me is now away [19:13:43] Damianz is our little fav monkey [19:13:54] addpub ahaha [19:14:01] I like how everyone run when I say puppet [19:14:10] * addpub no like puppet [19:14:14] :D [19:14:14] * addpub much prefer pub ;p [19:14:34] Ryan_Lane ^ we should replace puppet with pub [19:14:42] pub? [19:14:48] people like it more [19:14:57] I like all kinds of stuff more [19:15:03] but it's unlikely to happen ;) [19:15:10] you know beer and fun > puppet [19:15:16] mhm [19:15:17] pub = A tavern or bar (not quite sure it does the same as puppet) ;p [19:15:28] it doesn't [19:15:28] but people like it [19:15:32] that's what matter [19:16:57] * Damianz addpub [19:18:02] ? :O [19:20:27] petan: Ah, no, I was thinking 'septic'. :-) [19:20:59] LOL [19:28:35] petan: hi [19:28:38] the server of bots is down [19:28:39] petan: http://bots.wmflabs.org/ [19:52:18] needs rebooting, probably due to Ryan_Lane's changes [19:52:37] hm, I think I rebooted all unresponsive instances [19:52:43] this is apache01? [19:53:24] yeah, it's unresponsive [19:54:06] !log bots rebooting bots-apache01 [19:54:08] Logged the message, Master [19:55:17] oh... nagios doesn't talk here anymore [19:55:22] yeah [19:55:28] also I think the bot needs to be restarted [19:55:37] probably [19:55:42] it was netspliteve the other day [19:55:45] bleh. what's it named? [19:55:51] nagios-main [19:55:53] ah [19:55:54] thanks [19:56:01] reboobed [19:56:10] restarted ircecho [19:56:20] !log nagios restarted ircecho on nagios-main [19:56:22] Logged the message, Master [19:56:48] 64 down? [19:56:52] Sure you got them all? [19:57:04] should have [19:57:04] I muted all the 'been down for long time' instances in the build script =\ [19:57:47] some of these are suspended [19:57:51] because they were owned [19:57:53] like 12 [19:58:18] another 10 or so are down because they were corrupted ages ago and never deleted [19:58:25] ah... good point [19:58:59] snmptt seems broken again, beh [20:00:06] * Damianz thinks we should just use peer runs to notify of puppet runs/failures [20:00:57] peer runs? [20:01:36] salt? [20:01:57] bots apache is back up [20:03:25] mhm [20:03:42] maybe I'll delete all the instances that should be gone [20:03:45] I'll reboot the rest [20:03:55] New patchset: DamianZaremba; "Adding suspended instances" [labs/nagios-builder] (master) - https://gerrit.wikimedia.org/r/52269 [20:04:25] Change merged: DamianZaremba; [labs/nagios-builder] (master) - https://gerrit.wikimedia.org/r/52269 [20:05:35] That should get rid of a few... there wasn't that many though [20:07:11] 15 are either down or suspended [20:07:19] the rest need reboots [20:07:23] what's up [20:07:25] 13 are suspended from your email [20:07:25] someone pinged [20:07:44] Ryan_Lane did you reboot bots-bnr1? [20:07:49] I didn't see anything in sal... [20:07:50] petan: last night, yes [20:07:54] oh right [20:07:55] it was unresponsive [20:08:00] I'll figure the other 2 when you reboot stuff and add them to teh list [20:08:02] didn't know if it crashed or wha [20:08:03] t [20:08:07] bots-1 was OOM'd [20:08:13] * Damianz damns petan for icinga being in french [20:08:13] really? [20:08:15] weird [20:08:27] Damianz damn that person who packaged it like that [20:08:39] I installed it from wikimedia repo [20:08:42] I didn't change a bit [20:08:55] Damianz actually I don't see it in french [20:08:56] heh [20:09:01] just the header say it's french [20:09:05] Google thinks it's in french [20:09:11] google suck [20:09:14] don't believe it [20:09:16] :D [20:09:22] I think hashar broke it [20:09:26] because he is only french around here [20:09:33] he surely inserted that code in that [20:09:47] he secretly attempts to convert us to speak his language [20:09:54] I blame Coren, he secretly wants to be french [20:10:00] mhm [20:10:03] that's it! [20:10:13] Damianz: Clearly, you know nothing of Québecois. :-) [20:10:25] Coren ur canadian? [20:10:42] petan: Clearly, you know *nothing* of Québecois. :-) [20:10:45] Québecois are like canadians that secretly want to be americans [20:10:50] LOL [20:11:04] it sounds like mexicanois [20:11:09] :D [20:11:16] Nah - no one wants to be mexican [20:11:21] hehe [20:11:22] Damianz: No, that's the canadians. A.k.a.: US Lite. :-) [20:11:34] Damianz why not [20:11:43] I want to move somewhere where is warm weather [20:11:45] like mexico [20:11:46] Tacos [20:12:00] how is weather in San Francisco [20:12:07] cold [20:12:09] is it warmer than in Germany? [20:12:10] :D [20:12:14] I was thinking of moving more like Sanfran or Seattle than Mex [20:12:16] it's always cold here [20:12:20] howcome [20:12:23] it's more on south [20:12:55] weird climate [20:13:06] It was 15C here today - that was warm. [20:13:16] damn it's in California that's country which is close to... eh texas and that means the weather needs to be hot :D [20:13:38] because all movies from california or texas are full of deserts and such [20:13:41] :D [20:13:53] haha [20:13:53] 15 C? here is 0 [20:13:54] :D [20:13:56] As much as Cali is full of pretentious people I still want to live out there for a while =\ weird place [20:13:56] California does have deserts [20:14:01] damn that's a good weather for this month [20:14:06] RoanKattouw in movies it does [20:14:06] But not out here near the coast [20:14:30] in some movies it's frozen solid [20:14:37] Also, the coastal weather is very different from the weather even just 100 km inland [20:14:38] like in Russia or something [20:14:39] :D [20:14:42] heh [20:14:55] I wouldn't know, I haven't been to the desert here [20:15:01] mhm ok if I ever worked for foundation i will work remotely from mexico then [20:15:10] :) [20:15:14] haha [20:15:14] that's a weather I like [20:15:27] I even have sombrero here [20:15:32] BTW, because you mentioned Seattle, obligatory link: http://theoatmeal.com/blog/seattle_weather [20:15:34] From our favourite encyclopedia: "California's climate varies from Mediterranean to subarctic." :-) [20:15:37] just it looks weird wearing it in all this snow we have here [20:15:38] Does mexico even have internet? :P [20:15:49] Damianz no but they have hot chicks [20:15:56] in movies I mean [20:16:00] Germany has hot chicks [20:16:10] I don't know I never been in America :( [20:16:18] Damianz really? where [20:16:32] actually there was one beatifull girl last year in Berlin :D [20:16:41] some german girl.... [20:16:46] petan: South - our senior analyst over there is drop dead model like [20:17:03] Damianz huh? [20:17:12] oh lol [20:17:15] got it [20:17:29] you are working with german people? [20:17:36] I am working for german telefonica :D [20:17:55] Telefonica - meh you guys just sold out in the uk [20:17:57] they are quite cool, but I am working with Italians too and that is crazy [20:18:21] I work with people all over the world - we have more offices than I can remember... the german R&D/App support is in Willich. [20:18:25] Damianz don't blame me I am just keeping servers and oracle alive :D [20:18:42] heh yes, development in India :D [20:18:57] everytime we have a conference call I am totally confused by Indians [20:19:02] they speak so weird english [20:19:10] so fast like if it was compressed by 7zip [20:19:26] they say 2 senteces faster than italians say 2 words [20:19:36] Actually we don't have anyone in India I don't think.... most our 'development' is done in the UK/Germany and manufacturing in China since our factory/warehousing is out there. [20:19:44] Though I wouldn't mind a trip to the Japan office :D [20:19:49] lol petan [20:20:22] Platonides are you Italian? :DD [20:20:26] hehe [20:20:43] it's funny how when you send them email they respond in Italian [20:21:19] petan, no [20:21:19] I'm from Spain [20:21:28] Itallians make nice coffee [20:21:29] you know that telefonica comes from Spain, right? [20:21:32] Spain we work with these too lol cuz they own telefonica [20:21:34] of course [20:23:03] Damianz Japan... [20:23:04] mmm [20:23:17] could be interesting [20:23:29] I like exotic destinations just it's expensive to travel there [20:23:34] :/ [20:24:49] if the destination was near, you wouldn't consider it exotic [20:25:00] probably [20:25:01] :) [20:25:05] I quite like europe tbh - I've very nearly taken a job in Sweeden/Norway before. [20:25:07] japans probably think that you live in an exotic country :) [20:25:10] ukraine is exotic [20:25:13] it's close though [20:27:06] ukraine.... hmm, then you're into like soviet russia though [20:28:26] :D [20:35:30] http://securityreactions.tumblr.com/post/42749826944/when-the-customer-accepts-the-risk-of-a-critical-sql rofl [21:16:22] cannot connect via web to http://openid-wiki.instance-proxy.wmflabs.org/wiki/Main_Page [21:16:33] server can be contacted by ssh [21:16:38] Ryan_Lane: [21:16:54] 504 Gateway Time-out [21:18:56] restarted apache now [21:18:57] works [21:19:00] :-) [21:22:32] Is it a symptom of the recent network changes that I can't SSH into orgchart.pmtpa.wmflabs? [21:23:13] "ssh_exchange_identification: Connection closed by remote host" but without the usual "NC_ADDR" stuff that tells me I typed in the wrong name. [21:28:30] Rebooting to see if that helps. [21:41:23] xyzram: hey there :-] [21:41:43] hi antoine [21:41:58] so I got two instances in labs that have lucene running :] [21:42:02] thought that is not working yet hehe [21:42:23] turns out the lsearch parse InitialiseSettings.php to figure out the wiki domain [21:42:25] great. what part is not working ? [21:42:28] like enwiki => en.wikipedia.org [21:42:42] so the OAIHarvester turns out doing queries on production :-] [21:42:55] Interesting ! [21:43:18] xyzram: I will fill a bug report about it [21:43:25] do you have access to the labs instance ? [21:43:36] I can't remember if we have been around that last week [21:43:58] Haven't tried it -- what are the instances ? [21:44:22] ssh deployment-search01.pmtpa.wmflabs [21:44:23] marktraceur: If that's orgcharts-dev then yes [21:44:25] that is the search box [21:44:39] xyzram: indexer is at ssh deployment-searchidx01.pmtpa.wmflabs [21:44:40] [21:44:51] hashar: YAY [21:45:09] Damianz: orgcharts-dev isn't in use anymore...I rebooted orgchart and it's working again now. [21:45:16] Coren: Wrong channel? [21:45:34] xyzram: /a/search/log/log or /a/search/log/log-all shows the issue :-] [21:45:44] !log tools rejiggered the webproxy config to be smarter about paths not leading to specific tools [21:45:45] Logged the message, Master [21:45:48] Bah. [21:45:54] Silly bot. DWIM! [21:47:37] That bot needs imploding into hell [21:47:38] hashar: can't get to deployment-search01.pmtpa.wmflabs [21:47:41] :( [21:48:02] Strangely, can't reach bastion either, this used to work before. [21:48:14] If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [21:48:14] Permission denied (publickey). [21:48:23] That's the error I get. [21:48:27] ssh key not loaded ? [21:48:36] or maybe you are not part of the project ahah [21:48:38] let me check [21:49:22] Ryan_Lane: The intent would be to have any LDAP change done through the extension? [21:49:24] xyzram: you there as "ram" [21:49:35] xyzram: some ssh key not loaded in your agent maybe ? [21:49:46] how do I check that ? [21:50:14] xyzram: If you are using a command-line ssh, usually 'ssh-add -l' will do [21:50:29] xyzram: If you're using pagent, double click on the systray icon [21:50:39] It shows the key loaded. [21:51:06] ahh, need to get my power adaptor upstrair. will brb [21:51:40] hashar: I need to break for a few minutes before our meeting; later ... [21:55:07] Damianz btw regarding [21:55:08] new sql [21:55:21] you can sudo su, then mysql to login there [21:55:30] i didn't create any login for you there [21:55:41] and no idea how to use ldap in mysql [21:55:53] mysql in ldap is bad - makes people leave their pass hanging around [21:56:02] aha [21:56:03] though awesomeness, I'll just make a new user [21:56:05] I never used it [21:58:01] !petan [21:58:01] Petr Bena - http://enwp.org/User:Petrb [21:58:12] !petan del [21:58:13] Successfully removed petan [21:59:38] !petan is Petr Bena - http://enwp.org/User:Petrb - he is a bit septic about Coren's project :) [21:59:38] Key was added [22:00:22] Heh. [22:00:27] !coren [22:00:33] I like teh word :D [22:00:43] Boo. I don't exist. [22:00:53] !coren is Quebecian [22:00:53] Key was added [22:00:58] !Coren [22:01:00] mhm [22:01:06] !coren del [22:01:06] Successfully removed coren [22:01:07] !coren [22:01:11] :-P [22:01:38] !! [22:01:38] petan needs a new hobby :P [22:01:41] :) [22:01:47] wtf is that [22:01:50] :D [22:02:37] !coren is The toolmeister: http://www.mediawiki.org/wiki/User:MPelletier_(WMF) [22:02:37] Key was added [22:02:39] my new hobby is actually the development of some new amazing things :D like clipboard shared across multiple devices :D [22:03:11] I figured out when i was doing stuff on 2 pc's next to each other that it suck you can't copy paste across them [22:03:39] so I started working on cross platform tool that let you share the clipboard across unlimited number of devices [22:03:47] :D :D [22:03:48] Nope - but sharing clipboard is a little insecure maybe [22:03:53] yes I know [22:03:59] it's going to be somehow protected [22:04:02] using ssl or that [22:04:10] magic and unicorns [22:04:13] yes [22:04:14] :D [22:04:36] I'm working on a real time distributed monitoring system atm which is really harder than you'd think [22:04:44] really [22:04:56] is it speaking french if not it's crap [22:05:16] petan: I can see doing stuff on two pcs, but why would you need to use more than one X server for that? [22:05:17] :)) [22:05:33] Coren lol [22:05:33] maybe because on one you have windows [22:06:13] petan: rdesktop supports the selection and cut buffers. :_) [22:06:24] yes but that's too hard [22:06:33] I want to just copy paste without having to do complicated stuff [22:06:34] :D [22:06:43] what about ldap and mysql? [22:06:46] it would be cool if it supported copy pasting of files and that :D [22:06:58] Ryan_Lane we wanted to have that on bots but it's not really important [22:07:10] Ryan_Lane because I don't know how long we are going to use bots project [22:07:12] I really don't want that [22:07:24] I don't want people sticking their ldap passwords into files [22:07:26] Ryan_Lane I have no problem with not having that, but it's annoying for people [22:07:32] cuz they need to request access per db [22:07:36] yeah [22:07:42] we want to set up some automated means for this [22:07:47] mhm [22:07:57] I could just create some script [22:08:03] Coren: maybe we can actually have an interface that's per-person for this [22:08:20] and then another one for project admins, for the project users [22:08:27] local-project users [22:08:28] like it would use getent to retrieve list of ldap users, create and lock all of them on sql server, then people would just run a script which would have suid and reset their pw if they were them [22:08:43] petan: yeah, I was likely to do that [22:09:05] it's difficult to get people's passwords to them, though [22:09:18] so, it's also possible we'll do this via wikitech [22:09:25] in fact it doesn't need pw at all, if only maria supported OS auth [22:09:38] like microsoft sql [22:09:39] we don't want people to have shell access on the db server [22:09:54] true, that probably doesn't work cross server [22:09:59] Ryan_Lane: Hm. But then we need it flexible enough to support use cases for different projects. Something tools-specific is easy to whip up; something generic/configurable enough is an order of magnitude of complexity. [22:10:04] well, on msql it does in domain [22:10:15] Coren: well, that's why to have it per user and per project [22:10:27] the user interface would show their credentials on all projects they are a member of [22:10:41] Ryan_Lane shell access on sql server is cool though :P you can run top and so [22:10:46] then the per-project interface would show credentials for the tools in projects they are a member of [22:10:46] to see how loaded it is [22:11:02] petan: I'd rather ganglia show that ;) [22:11:02] not that ganglia wasn't good enough :D [22:11:14] heh [22:11:15] but unix people like more terminal than fancy graphs [22:11:19] :D [22:11:24] I really wish I had the project name in instance dns names [22:11:36] Ryan_Lane: Hm. I think I see where you're going. So it'd look a little like the manage instance/manage project split? [22:11:44] Coren: yeah [22:12:00] I guess I should make mockups [22:12:16] it's easier to discuss ideas with mockups [22:15:12] Ryan_Lane: Another question; do we create user databases preemptively or only on request? [22:15:17] petan: Script to pull users, add them to mysql if they don't exist, write password out to their home dir's .my.cnf give rights to create dbs #done [22:15:26] Coren: I'd like to do so on request [22:15:28] That way the standard is, read the details out of ~/.my.cnf [22:16:06] We could have mediawiki manage mysql directly [22:16:14] I wanted salt to do this, with salt-api [22:16:31] but we may not have that ready in time [22:16:40] mw directly is just evil [22:16:41] ;P [22:16:44] agreed [22:16:56] I'd very much like salt to do this via a runner and modules [22:17:06] simples, just time consuming a little [22:17:42] Damianz not a bad idea [22:18:00] Ryan_Lane what is simplest way to read all users of project in shell [22:18:09] ldap query [22:18:11] writing to the home directory works inside a single project [22:18:15] look at how the public_html dirs are created [22:18:16] but not across projects [22:18:25] Damianz for some reason ls /home seems easier to me :o [22:18:36] meh [22:18:37] Damianz can you write that query [22:18:41] that won't work with pam_makedir [22:18:48] or w/e it's called [22:18:49] that too :) [22:19:13] petan: Which query? [22:19:23] to retrieve all users of project [22:19:31] @labs-project-members [22:19:37] @labs-project-members bots [22:19:46] @labs-project-users bots [22:19:46] Following users are in this project (displaying 19 of 60 total): Alejrb, Andrew Bogott, Aude, Beetstra, DamianZaremba, DeltaQuad, Dzahn, Fastily, Hashar, Hydriz, Hyperon, Jasonspriggs, Jeremyb, Johnduhart, Kaldari, Krinkle, Ryan Lane, Lcarr, Madman, [22:19:49] like this [22:19:52] :D [22:19:55] but in shell [22:20:02] ploink [22:20:06] you can actually find this out via getent [22:20:15] getent group project- [22:20:18] megaping :) [22:20:20] aha [22:20:27] aude I can do worse :P [22:20:31] heh [22:20:32] since all project members are in the posix group [22:20:37] I still think using ldap bindings for scripting is cleaner :P [22:20:43] okay [22:20:43] Damianz: yes, it is [22:20:48] however..... [22:21:02] that's going to give back DNS [22:21:04] err [22:21:06] DNs [22:21:10] which you then need to look up [22:21:29] and it won't be cached [22:21:38] getent group will cache positive and negative [22:21:40] cache? pussy [22:21:43] :) [22:24:14] I need to clean up my repos of stuff =\ [22:28:12] Ryan_Lane: We totally need @wmflabs.org forwards for awesomes [22:29:29] yeah, we need a repo pretty badly [22:29:31] err [22:29:33] relay [22:35:16] Coren: Will the -webserver's "AllowOverride None" be relaxed in the future? I'd like to use mod_env's SetEnv directive (in .htaccess) if possible. [22:35:50] scfc_de: I see no reason why not. 'None' is my default placeholder-until-I-know-better. [22:36:25] I assumed that, just wanted to know whether to wait or work around :-). [22:37:02] hi, dns seems to be acting up again... do we know when bots-sql1 will be visible again? [22:37:27] Coren: Probably just needs some thought whether it can be used to fool suphp. [22:40:43] scfc_de: hence the default. I plan on AllowOverride of AuthConfig and some Options. [22:41:36] FileInfo I need to investigate. [22:44:42] scfc_de: I priori, I don't think it's an issue, you might be able to confuse your own code by messing up your environment, but that shouldn't allow you to escalate or break someone else's [22:47:15] gribeco: dns? [22:47:34] Ryan_Lane: ping: unknown host bots-sql1 [22:47:44] o.O [22:47:47] let me look into that [22:48:07] thanks [22:48:34] that should never happen [22:48:49] I agree with that ;-) [22:49:13] there is no bots-sql1 [22:49:38] there used to be... maybe a discontinued alias? [22:49:40] did you mean bots-bsql01 ? [22:49:55] maybe petan deleted it? [22:50:06] * Coren goes for food. [22:50:13] gribeco hi [22:50:17] gribeco were you using that? [22:50:19] Petrb deleted instance 'bots-sql1' in project https://wikitech.wikimedia.org/wiki/Nova_Resource:Bots [22:50:20] yep [22:50:21] :) [22:50:27] it;s in SAL [22:50:28] !sal [22:50:28] petan: yes, but I can change the name if necessary [22:50:28] https://labsconsole.wikimedia.org/wiki/Server_Admin_Log see it and you will know all you need [22:50:42] gribeco all databases from there were exported to bots-bsql01 [22:50:43] you guys had me worried for a sec ;) [22:50:50] petan: ok, let me point to that [22:50:53] gribeco what database you were using? [22:51:16] petan: salebot [22:51:32] looks like I need new credentials? [22:51:48] DBI connect('database=salebot;host=bots-bsql01','gribeco',...) failed: Access denied for user 'gribeco'@'bots-salebot.pmtpa.wmflabs' (using password: YES) at lib/User.pm line 221 [22:56:15] petan: Ryan_Lane: Do you know if there is anything in beta right now that enforces the state of the deployed mediawiki? e.g. can I deploy an a change pending in gerrit and check it out for a few hours? [22:56:37] I've spent the last 30 minutes trying to get a more representative environment locally but it is just not doing it for me [22:56:39] petan: do I need to do something on my own to connect to mysql on bots-bsql01? The passwd I used for bots-sql1 no longer works [22:56:51] Krinkle: I don't know [22:56:53] I need more crappy templates and insane extensions that make it more horrifying [22:57:15] Krinkle: as far as I know it sets itself to master every 3 minutes [22:57:22] I'm working fixing the issues in modern browsers related to the high number of repaints resoruceloader is causing each time a module arrives from mw.loader [22:57:27] Ryan_Lane: OK [22:58:01] But I need more modules to be loaded and a more production like environment. [22:58:12] Ryan_Lane: Is it trivial to disable that? [22:58:19] no clue [22:58:24] hashar: ^^ ? [22:58:26] This is basically what labs is for (among other things) to be able to test things in a real environment. [22:58:36] Krinkle: yes, but I don't manage beta ;) [22:58:39] sure sure [22:58:44] hashar: :) [22:59:12] Everyone look at hashar [22:59:25] me [22:59:27] vv [22:59:30] * hashar hides [23:00:01] Krinkle: try out on beta ? [23:00:06] Krinkle: what do you want to do? [23:00:24] gribeco you should be able to connect without password [23:00:27] like since 10 seconds [23:00:29] :D [23:00:34] ;-D [23:00:36] I just ran sync script [23:00:41] hashar: mw.loader is appending to