[00:22:38] Ryan_Lane, andrewbogott_afk: Once again I find myself trying to deal with https://gerrit.wikimedia.org/r/#/c/91906/ only to find that every wiki installed on nova-precise2 has uncommitted changes in the OSM directory [03:29:08] Is there something down in the system [03:29:33] I have a bot suddenly not doing what it should do, building up a queue [03:30:01] possibly since 4 hours ago [03:30:13] It is runniong normally, seems to be waiting for something [03:30:36] since about 4 hours [03:37:11] Beetstra: Not that I am aware of. What does your bot say? [03:37:24] well .. nothing [03:37:35] It is hanging in a module, apparently waiting for something [03:37:44] I can't see why [03:37:52] As if it can't store data [03:38:36] the rest of the bot is responsive [03:39:58] Have you tested you can write data? [03:42:59] eh .. that 'hangs' [03:43:29] so it sounds like a file system issue [03:43:34] I manually logged into MySQL, adding a rubbish value to a table - still busy after .. a minute [03:43:51] show processlist; [03:43:52] Beetstra: Which DB server? [03:43:56] See what's happening? [03:46:41] yeah, a lot of queries .. [03:47:21] Not sure what it is waiting for [03:48:06] Beetstra: Which DB server? [03:48:32] scfc_de: good point, these are personal tables [03:48:59] Beetstra: tools-db? [03:49:12] yes [03:49:21] Wrote a script to log me in, had to look [03:52:54] the oldest query has as time '15500' (but I restarted that part of the bot half an hour ago). COIBot got his last request about 4 hours ago [06:09:28] Krenair: I'm pretty sure those changes are mine and can be wiped [06:09:47] they are all in gerrit changes now [07:35:10] I have hanging queries on tools-db .. killing them puts them in 'killed' state, but no movement further. Any ideas how I can resolve this? [08:04:13] * Beetstra does not understand if it is my 'fault' that I can not insert anything in tools-db .. or if there are other problems [08:05:55] I'll have a look this evening [09:39:55] Krenair: I have been actively developing in /w. /w2 and /w3 are a bit crufty but you're welcome to clean them up and do your work there. [09:52:17] Change on 12mediawiki a page OAuth/For Developers was modified, changed by Jalexander link https://www.mediawiki.org/w/index.php?diff=885056 edit summary: -index.php [10:31:46] paravoid: your instance 'faidon-test' has the biggest footprint on virt8. Would you like me to migrate it (with attending downtime), pare down some log files, or both? [10:32:26] Number two offender is… wikidata-dev-9. aude, you around? [10:33:14] This is going to be my new thing: Announcing things in the dead middle of the night and then taking action when no one responds. [10:33:29] andrewbogott, go to bed:) [10:33:50] MaxSem: I changed TZs, it's not the middle of the night for me. [10:34:05] Hence my temptation to act the BOFH [10:34:49] andrewbogott: where are you? [10:35:10] YuviPanda: Singapore, just up the street from you :) Unless you're in the US [10:35:15] ah! [10:35:18] no, I am in India [10:35:19] nice! [10:35:27] I was going to say Hong Kong... [10:35:34] I'm across the street from a Tamil movie theater. Can catch up on my cinema. [10:36:33] andrewbogott: nice! [10:36:44] although there is nothing worth seeing that's newly released yet... [10:36:53] andrewbogott: s'pore has a high tamil population, it hink [10:36:55] *I think [10:37:36] Yep, especially in my neighborhood. Out my window is a bunch of contruction sites, and then 'Little India' [10:37:59] So it's a double dose. (As I understand it the construction crews here are mostly South Indian migrants.) [10:39:29] YuviPanda: Do you know if people ever travel overland from India to SE Asia? I'm not clear on if there are trains through Burma from India. [10:40:05] andrewbogott: I'm yet to hear of people doing that, so I don't iknow [10:40:15] andrewbogott: yeah, a lot of them are [10:40:30] Yeah, the only thing I hear about is through Tibet which is waaaaay out of the way [10:40:49] Although I guess it'd be cool to see that weird railway that you need O2 tanks to ride [10:40:53] andrewbogott: there are ways, apparently http://travel.stackexchange.com/questions/16305/overland-route-from-southeast-asia-malaysia-to-south-asia-india-nepal-or-pa [10:41:00] haha! [10:41:03] weird... and dangerous [10:42:03] I think it's built on permafrost, so should probably go see it before everything melts. [10:42:11] * andrewbogott is going totally on rumor here [10:42:29] andrewbogott, beware of indian movies! [10:42:36] they can blow your mind! [10:42:38] Everytime I'm here I daydream about traveling overland to Europe, then flake out and fly back through Japan instead. [10:42:59] http://www.collegehumor.com/video/6082319/epic-tractor-fight-scene [10:43:04] MaxSem, that one with the giant robot made out of a bunch of smaller robots, each with giant sideburns? I /loved/ that movie. [10:43:16] that was a Tamil movie [10:43:19] so... cringeworthy [10:43:47] andrewbogott: http://www.youtube.com/watch?v=8xlZ64xrsRo is what you are thinking of :) [10:43:58] (http://en.wikipedia.org/wiki/Enthiran) [10:44:00] I legitimately liked Robot, except for the part where everyone agreed that a lady was better off dead once the neighbors had seen her naked. [10:44:42] Huh, since it was called 'Robot' when I saw it, maybe it was dubbed in Hindi w/English subtitles. Maybe more hindi speakers in MN? [10:44:59] andrewbogott: yeah, the hindi version was Robot [10:45:07] and it was dubbed in Hindi with English subtitles [10:45:17] YuviPanda: What's not to like? At the end he learns a lesson about love. [10:45:30] And, I guess, female modesty :( [10:45:40] well, almost all tamil movies have an inherent sexist basis in them [10:45:51] you don't see it when you start but once you start seeing it yo see it everywhere [10:46:16] not the typical 'oh dear, the girl needs rescuing again', but a bit worse than that [10:46:36] Were Krish/Koi mil gaya also Tamil? [10:46:43] no, that was Hindi [10:46:46] * andrewbogott names all the Indian movies he can remember [10:47:22] andrewbogott: :D [10:47:25] is that all? [10:47:39] Um… apparently I only remember the sci-fi stuff. [10:47:42] * mutante creates project:bollywood [10:47:44] andrewbogott: if you can, I suggest watching http://en.wikipedia.org/wiki/Anbe_Sivam. Wonderful movie, I think. [10:48:10] ok, noted! [10:48:36] :) [10:48:47] andrewbogott: any idea when the eqiad migration of tools is going to happen? [10:48:54] I need to complete the proxy for tools before that [10:49:20] YuviPanda: Hoping to start test migrations in a couple of weeks. [10:49:24] So, tools, maybe a week after that. [10:49:27] Not sure if that's overly optimistic. [10:49:41] heh [10:49:46] andrewbogott: what's a worst case estimate? [10:49:57] If by 'worst' you mean 'soonest' then yes. [10:50:00] had fun with his private commonsapp fork, messing with strings.xml and insulting my friends instead of a friendly "couldn't login" message [10:50:13] no, I mean, worst as in 'it can possibly not not happen in 3 months' [10:50:23] mutante: that's always fun! :) [10:50:44] mutante: btw, the wikipedia app can now do things like edit :) logging in coming next week... [10:50:52] YuviPanda: woot! [10:51:00] mutante: and has anon editing too! [10:51:01] edit source or edit visual?:) [10:51:08] (i want oldschool:) [10:51:08] mutante: just source for now :) [10:51:11] YuviPanda: We're going to leave a big window for people to self-migrate. But tools itself will definitely be moving over in… let's say 'less than two months' [10:51:24] YuviPanda: fine with me:) i still keep deactivating VE to be honest [10:51:24] mutante: VE in about, say, a year and a half, maybe? :) [10:52:02] andrewbogott: ah, sweet. [10:52:11] it's like plain text vs. HTML in RT :p [10:52:18] plain text i know what i get [10:52:33] heh [10:52:46] mutante: yeah, this will be plaintext, but might have a bar or something like WikiEd has. [10:52:47] unsure yet [10:52:53] right now it's just a large textbox with monospaced text [10:53:24] YuviPanda: so people _will_ edit with voice commands depending on their phone features! [10:53:35] mutante: theoretically, they can, yes... [10:53:38] theoretically.... [10:53:38] heh, imagine the editing by telling siri [10:53:53] pfft, Apple will go 'you do not want to edit that' [10:53:58] or commons patrollers watching RC and then "delete that" [10:54:07] I find Siri weird and creepy in a way that Google Now isn't [10:54:15] yes [10:54:19] mutante: heh, I was thinking of kocking out a small prototype that does just that [10:54:23] patrol with voice [10:55:10] mutante: afd with voice! [10:55:15] or NPP with voice.. [10:55:21] hmm [10:55:21] YuviPanda: go one step beyond that, use Google Glass to measure pupil dilation / pupillary response to watching an image (google will do for ads anyways). then you already know if that imagine is "obscene to the average viewer" from stats [10:55:26] :o [10:55:46] so instead of 'block' you can say 'OFF WITH HIS HEAD!' [10:55:52] but then you might get an arbcom case on you... [10:56:13] "this image has been auto-patrolled because we measured that too many people were shocked by it" [10:56:31] well, then dewiki will make it their main page image... [10:56:35] google will do it for ads.. creeepy too [10:56:51] they patented that, and big reason to push glass [10:56:52] seen that german video about searching about religions with siri? with islam being not available without geolocation :) [10:57:22] YuviPanda: hahaa, @ dewiki [10:57:56] YuviPanda: you don't think that's a good way to pick the image for the main page? [10:58:05] so they can charge the advertiser per eyeball for real [10:58:09] DanielK_WMDE_: I definitely think it's far superior to enwiki's. [10:58:11] that's what they want [10:58:16] hehe... [10:58:24] mutante: I do have a google glass... :) [10:58:47] YuviPanda: so they make money from you just _seeing_ the ad, first time in history they dont have to guess [10:59:12] mutante: Won't that result in filtering images of, like, ice cream? And big piles of money? [10:59:15] and they guess if you liked it too [10:59:53] andrewbogott: heh, good point, but it seems they can see the difference between positive and negative reaction [11:00:24] Oh, and you're thinking that 'obscene' = negative reaction? You might not be doing it right. [11:00:48] yeah [11:00:54] andrewbogott: well, but they want to define "obscene" by "what the _average_ viewer finds obscene" [11:00:57] you would need things that trigger negative reactions in, say, Texas [11:01:02] it's a redundant definition [11:01:03] and positive ones in, say, Japan, Germany [11:01:22] so just mine all the data and have geoip ?:P [11:01:36] * andrewbogott presumes that the people who are the most outraged by a given image are the same people who are the most, um… dilated. [11:01:52] But it would great to have data about that! [11:02:07] http://www.ibtimes.com/google-glasses-eye-tracking-glass-patent-may-introduce-pay-gaze-advertising-1393055 [11:03:26] andrewbogott: haha [11:03:28] the irony :) [11:03:54] I don't even remember what purpose it serves [11:04:39] I'll kill it [11:04:49] paravoid: well, that's simple enough. Thank you! [11:05:03] You just narrowly avoided receiving a form letter. [11:05:25] Another one of the big ones was created by peter Y. Not sure what to do about that... [11:05:37] kill them [11:05:41] what project was faidon-test under? [11:06:14] 'puppet' [11:06:38] done [11:06:39] thank you [11:09:09] paravoid: Peter's instance is called 'build-precise1' in the testlabs project. [11:09:13] So, probably disposable. [11:09:29] Still, it pains me to just delete things without knowing what I'm deleting :( [11:09:32] just kill them :) [11:10:06] how do i make labs use charset=utf8 in http responses for plain html files? [11:10:17] some .htaccess magic? [11:11:17] "testlabs" probably has more old stuff because it was the very first project when labs was new and everybody started a personal test instance to see how it works [11:11:34] DanielK_WMDE_: http://www.w3.org/International/questions/qa-htaccess-charset.en [11:11:49] should work, I think [11:12:03] !log testlabs deleting build-precise1 because it's huge and seemingly abandoned. [11:12:04] Logged the message, dummy [11:12:17] Man, deleting is /way/ easier than migrating! [11:12:29] YuviPanda: in public_html/.htaccess ? [11:12:34] yeah [11:12:37] should work [11:12:41] ok, trying [11:12:55] * YuviPanda goes afk for abit [11:12:59] YuviPanda: yep, works. thanks! [11:13:04] DanielK_WMDE_: :D [11:13:05] yay [11:13:23] this should be the default though [11:13:31] true [11:15:27] Hm… anyone have an email for Slevinski? [11:16:06] nm, found [11:24:30] mutante: Are you associated with commons-dev? [11:24:53] The instance 'lynwood' is GIGANTIC, wondering if it can be reduced. [11:25:06] andrewbogott: i dunno, i think YuviPanda would [11:25:13] i just built commons-app on my own laptop [11:25:15] not in labs [11:25:50] ah, that's probably other commons dev, not commons-app-dev? [11:26:15] project commons-dev, the only members are you and Dereckson [11:26:30] in labs [11:33:07] andrewbogott: the reason i'm on it is likely just because i wanted to do support or check a bug report [11:33:18] does it have any project logs? [11:43:27] nope, no logs. [11:44:16] hrmm, i don't know better than asking Dereckson then, sry [11:44:24] np [11:44:32] Looks like you created the project, that's why you're in it. It was a while ago. [11:44:48] yea, i think that is it [11:44:59] just confirming when people said they didnt get shell access on new project [11:45:05] and i wanted to see if key is created etc [12:43:54] andrewbogott: it can probably be deleted.... soon at least but i need to first make sure we have everything off it we need [12:44:09] when is wikidata project being migrated? [12:44:48] aude: It'll be a few weeks at least. [12:44:57] ok, it should be deleted for sure [12:45:01] by then [12:45:09] Ah, this isn't an issue with migration... [12:45:13] the host it's running on is full /now/ [12:45:13] oh [12:45:28] yikes [12:45:30] I can move it to a different host if you don't mind the downtime. [12:45:40] go ahead [12:45:52] ok. Should be just a few minutes, but I haven't done this before so we'll see :) [12:46:06] ok, and if it dies, not a huge problem [12:46:23] 'k thanks. I'll try right now. [12:46:26] k [12:52:52] andrewbogott: could you add User:Tobias_Gritschacher to the bastion members of labs please? :) [12:54:33] addshore: ok, done. [12:54:55] cheers andrewbogott :) [13:25:25] aude: done. Look ok still? [13:36:27] I'm doing clones of mediawiki/extensions on to NFS, hopefully that doesn't cause it to die [13:36:36] andrewbogott: http://wikidata-dev-repo.wikimedia.de/ still works so i assume it's good [13:36:58] great, thank you. [13:37:51] soon we'll replace it with wikidata-dev-repo.wmflabs.org and delete the old instance [13:38:02] once new one does everything [13:59:12] HELP [13:59:20] I can't access labs... [14:01:57] http://s15.postimg.org/hibclxe3t/Capture.png [14:03:46] Are there any database issues? Because some programs hung, I killed all threads and all sql connections, but five connections are still hanging in "killed" state for over an hour now... this is very strange. I thought about some internal rollback, but this should be faster than one hour, I think... [14:04:57] apper: Last time something like this happened, it was because the filesystem was full. [14:05:09] Coren: Is the filesystem on tools-db full again? [14:05:30] anomie: that's possible, Ryan_Lane was talking about something like that a while ago [14:05:35] I go ccheck. [14:05:39] I am from zhwy that need to have a welcome bot.. =.= [14:06:07] * anomie has lots of queries stuck in the "query end" state again [14:06:11] And I can't *wait* for the move to eqiad. We have a real physical DB there. [14:06:22] With, like, disk space. [14:06:38] Is there a public timeline on the move? [14:06:46] Coren: when should this happen? I remember something like "january" [14:07:34] apper: The contractor that does part of the work has been ill, so we're pushed back a couple weeks; but we are hoping to start moving labs projects this month or early next. [14:07:46] Coren: okay [14:08:10] Yeah, tools-db is definitely trashing. Will fix shortly. [14:08:29] Coren: thanks [14:08:57] hmm, nice. only 40mins for a full checkout of mediawiki/extensions.git [14:17:51] tools-db is trashing so hard I have trouble logging in at all. [14:42:19] Can someone help me unblocking access to tools-db (or explain me what is the problem) [14:42:32] My insert queries seem to hang completely there [14:43:40] Coren: tools-webserver-03 down since ~ 4hrs, tools homepage http 500 [14:44:10] funny enough, select-queries run normally on tools-db [14:44:14] Beetstra: tools-db is currently trashing. I'm in the process of fixing it. [14:44:22] Beetstra: Yes, read queries should work okay. [14:44:34] OK [14:44:49] hedonil: Will look into it imidiately afterwards, but is almost certainly related to tools-db being ill. [14:44:54] the most important bot there is at the moment not inserting but backing up .. [14:45:04] Coren: 'k [14:47:48] anomie: what's the process to get an approval as OAuth consumer? [14:49:57] hedonil: I'm not sure. At some point in the hopefully-near future, it will be [[meta:User:DGarry (WMF)/OAuth consumer guidelines]]. But since that's still in userspace, it's probably not active yet. [14:51:13] anomie: I want to make my tool fly w/ OAuth and need to be approved. Who can do this right now? [14:52:21] * Beetstra is confused: you have gone away, but .. I am inside you ;-) [14:55:02] anomie: I filed a bz for the (near) future, but I need an "approver" . best would be /now/ ;) https://bugzilla.wikimedia.org/show_bug.cgi?id=60125 [14:55:22] hedonil: What does your consumer do with the "View your watchlist" grant? [14:56:03] anomie: read the watchlist and show all cooresponding webcounts [14:56:59] anomie: if you have super-rights, you can check it with dev-version https://tools.wmflabs.org/wikiviewstats/dev/index.php [14:57:40] anomie: menu My wachlists - but first login ;) [14:58:58] hedonil: I approved your one request. Why does the "Test" one want to edit the watchlist? [14:59:29] anomie: not needed right now [15:00:03] hedonil: It should be removed from the requested grants then, if you don't need it. [15:01:04] anomie: ok. I make a new reauest for the dev version. In prod it's already not requested [15:05:54] anomie: ok. Could you remove/delete the Wiki ViewStats Test [1.0] consumer and approve the Wiki ViewStats Test [2.0] one? [15:09:07] anomie: great! thx a lot. I'm going to test it right now. [15:13:11] andrewbogott: is that status of virt1000 known? [15:13:17] ldap is down [15:13:23] for two weeks [15:13:34] production services use ldap too, so it kinda sucks not having redundancy [15:14:28] I don't immediately know why it's down -- no chance it's related to the busted cert? [15:27:45] andrewbogott/Coren: can you merge & babysit https://gerrit.wikimedia.org/r/#/c/98307/ ? [15:27:52] well, review merge & babysit [15:29:47] paravoid: I'll have a go at doing the rebase. I won't be up late enough to babysit just now. [15:32:08] Ah. I see what broke things. tools-db was still gluster [15:37:42] hedonil: you can always test OAuth stuff as yourself [15:38:00] also before the consumer is approved [15:38:56] valhallasw: yeah. I did all tests with my account and was surprised when testing it with a brand new one [15:39:03] paravoid, Coren, the ferm patch can't rebase because of conflicts with this: https://gerrit.wikimedia.org/r/#/c/102052/1/manifests/openstack.pp [15:39:48] Which, it'll take me more than a few minutes to understand ferm enough to get that change in there. Hoping one of you will tell me not to bother :) [15:40:44] valhallasw: there are certain surprises in "real" mode. e.g my account works on testwiki, brand new one doesn't [15:41:00] Strange. [15:41:24] hi Coren/andrewbogott, can I get loginviashell for charles-salvia? [15:41:25] Failed to add charles-salvia to analytics. This needs user charles-salvia to have the "loginviashell" right. [15:42:54] (he's a new team member, just joined) [15:43:17] milimetric, one moment... [15:43:23] no prob, thanks! [15:44:42] ok, all set [15:45:00] thanks a lot andrewbogott [15:47:54] * andrewbogott requires sleep [16:01:12] * hedonil rubs his hands - hehe works like charm [16:08:02] valhallasw: near reality tests beat assumptions. If one registers an new user account in just one wiki it's not automatically registered in any other wiki [16:16:15] anomie|away: The fix was disruptive, but everything should be okay now. [16:16:34] hedonil: What URL do you see having the issue, so I can figure out if that was the same underlying cause? [16:17:45] Coren: You mean the http 500 thing? It was https://tools.wmflabs.org/ but works fine now [16:17:55] Coren, andrewbogott_afk: I wrote a Python script to assess the readiness of a labs instance to be migrated from pmtpa to eqiad. I would like to put it in a Gerrit repo under labs/ [16:18:17] drdee: It should probably live in operations/software I'd think. [16:18:20] the package is called labs-migration-assistant, is that a good name? [16:18:34] Sounds good to me. [16:18:58] not under labs/ -- the intended enduser are labs folks [16:19:01] hedonil: Yeah, I expected as much; the webserver was stuck with things piling up waiting on the DB [16:19:30] drdee: Oh, wait, it's a tool to let people evaluate their own instances rather than for us to sweep them? [16:19:35] right [16:19:52] * Coren ponders. [16:20:08] What is it constructed as? Is it a single script? [16:21:20] a small sweet python package, including a setup.py, requirements.txt and two pythonfiles [16:21:21] Either way, I guess labs/migration-assistant would make sense. [16:21:43] so that someone can just git co or git clone it. [16:21:51] indeed [16:22:03] okay i will try to create the repo [16:22:03] thx [16:22:56] should it inherit rights from a particular parent repo? i don't see a labs parent repo [16:23:19] There isn't one, and unless you plan on needing a group for commit rights, it's not necessary. [16:23:34] how about 'All-Projects' ? [16:23:48] Yeah, that's the "don't need one" one. :-) [16:23:55] :D [16:24:17] awesome, repo created [16:55:48] Coren, tools-db apparently went down. [16:56:04] Cyberpower678: It did. It's back up since though. [16:56:27] Or not. [16:56:37] Stoopid startup script. [16:57:18] I supposed I should have sait "It should be back up since though". [16:59:11] Ah, I see what's happening. It's in recovery. [17:00:40] So is spambot, but it's failing to connect to tools-db, so it's kind of stalled right now from starting up [17:01:04] I know, the mysql on tools-db is in recovery mode, replaying logs. [17:01:25] Bot started up. Tools-db seems tohave gone back up. [17:01:43] Ah, that was faster than I feared. [17:02:07] mysql recovery times are sometimes... unpredictable. [17:02:58] What caused it to go down? [17:04:05] Out of disk space for binlogs. Again. [17:04:15] * Coren eyes the new eqiad DB with obvious longing. [17:04:19] Soon... [17:04:33] :p [17:05:01] I bet everything will be much faster when that move happens. Latency is apparently .124 seconds. [17:05:59] Coren, when do you suppose eqiad is going to happen? [17:07:06] Cyberpower678: The contractor doing the new OpenStack install was out sick, but we're talking weeks not months. [17:07:12] The additional data validation I added to spambot's recovery function seems to be working beautifully. [17:07:38] Blank tags should be a thing of the past now. [17:07:50] :DDDDD [17:08:32] Awesome. Can't wait to see how SUL Info performs. I optimized it to get around latency but it still takes a few seconds with all those queries. [17:11:09] !log tools tools-exec-09: "iptables-restore /data/project/.system/iptables.conf" after reboot [17:11:11] Logged the message, Master [17:14:43] Cyberpower678: Hi, what's the actual url of new speedy SUL info? [17:16:36] * anomie decides, since all AnomieBOT's other processes died, to also qdel and resubmit AnomieBOT-6 to get it off of tools-exec-04 [17:18:55] anomie: Is your bot's start-up cost so big that you really need your special technique for reloads? [17:20:01] scfc_de: No. I just don't see the point in deleting and restarting jobs all over the place (and waiting for sge to schedule them) when I could just have the process re-exec itself [17:20:07] Cyberpower678: Or is there even an API for the new fast and furious SUL info tool? I would appreciate that ;) [17:20:50] Actually, ima reboot -04 now. It's about as unoccupied as possible and those sick processes just make things wonky re scheduling. [17:22:37] anomie: Keeping it simple, less moving parts, ... :-) [17:23:04] hey, do people need to be added to the bastion project in order to be able to ssh into labs instances? [17:25:45] milimetric: IIRC if they use bastion{,2,3}.wmflabs.org; do you have your own bastion in the project? [17:26:19] we don't have our own bastion, no [17:26:31] ok, so could someone that has admin on bastion add charles-salvia? [17:27:07] I'm assuming that's Coren or andrewbogott_afk, but if I'm annoying the crap out of those people, feel free to create new targets :) [17:27:57] milimetric: No, but that's done automagically anyways. [17:28:11] milimetric: (By granting shell right which /is/ required) [17:28:31] hm, so it is, I see he's on the project [17:28:40] interesting then, he can't ssh into bastion [17:28:43] he's uploaded his key [17:28:55] What's his username? I go check da logs. [17:29:01] charles-salvia: [17:29:16] Make sure he's logging in as labsuser@bastion if his username is different on his own machine. [17:29:58] I'm not seeing that username in the logs at all. scfc_de is probably putting his finger on the problem. :-) [17:30:12] hm, yeah, he's definitely using Username charles-salvia in the ssh config [17:30:18] k, we'll troubleshoot on our end though [17:30:23] thanks! [17:40:30] crosspost from #wikimedia-dev : Who broke my watchlist on dewiki.beta.wmflabs? ;) (PHP fatal error) Is someone aware of this issue? [17:52:30] Coren: tools-exec-04 seems to be stuck now with "ssh_exchange_identification: Connection closed by remote host"; you're working on that? [17:52:51] Yeah, it's not wanting to reboot gently. [17:53:00] (Those same stuck processes) [17:53:54] Is there any way besides a reboot that can kill such processes? [17:57:27] No, pretty much anything that can survive a kill -9 is gone for good. It's actually pretty darn hard to end up in that state, normally, hardware going away is one of the only ways. [18:00:39] And that VM is actually wedged pretty hard. [18:04:11] Coren: andrewbogott was shuffling VMs between virt*; don't know if tools-exec-04 would be another candidate for that if we have downtime on that anyhow. [18:04:40] It's on virt9 which doesn't seem to be the one having trouble. [18:06:22] Ah! I *finally* managed to reboot it. Had to actually kick the KVM down. [18:14:50] ... and all five continuous jobs have been rescheduled. Nice. [18:21:40] scfc_de: That's how things are supposed to work. It's even nicer when the jobs understand SIGUSR1 [18:23:18] Coren: Yes, but it's always very reaffirming if something that should work does indeed work :-). [18:28:53] Bots are running like normally again, Coren [18:29:10] Beetstra: Victory! [18:29:12] couple of diffs of dataloss at most, nothing really bad [18:29:14] Man over magine again. [18:29:36] (We're really straining the capacity of poor pmtpa labs) [18:30:02] Maybe one spammer got lucky and his diff got unnoticed by the bots [18:30:43] Thanks for the hard work, the anti-spam team depends on it [18:37:13] The uptake of labs in general and tool labs in particular has been much stronger than we had expected; hence the growing pains. But the new setup is pretty much twice as big (more, when you count storage and DB space). [18:39:19] ~73T of storage, plus some 10T for dumps and 8T or so of scratch space. [18:39:46] woah [18:39:54] Coren: and ssd databases? [18:40:11] YuviPanda: Those are already in eqiad. They just get "closer" now. :-) [18:40:18] ah, of course! [18:40:58] Coren: completely different NFS machines too, I think? [18:41:05] The new tools-db is on physical hardware, and shares ~12T of DB space with the OSM postgres setup. [18:41:16] YuviPanda: Yeah. [18:41:20] nicc! [18:41:29]