[00:14:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [00:39:28] Anyone from the deployment cluster here? [00:39:40] *Anyone from the deployment cluster project here? [00:44:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [00:50:53] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 18% free memory [01:00:45] hey there [01:01:21] how r u doing? [01:02:19] im new to irc [01:02:36] HELLLOO????!!!! [01:04:18] Hello dehunter. [01:04:38] hey [01:05:01] how you been doing? [01:05:13] Very busy [01:05:36] ive been busy programming [01:05:49] i made a key logger in visual Basic 6 [01:06:46] do you know any programming languages? [01:06:51] I don't know that language [01:06:52] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 178 processes [01:07:08] I know Pascal, Python and PHP to some extent [01:07:24] it a common programming language [01:07:32] Well...I can get along with references :) [01:07:40] dehunter, I know. [01:07:59] do you know c++ [01:09:37] uhhh.... HELLO!!??? [01:10:04] Hazard-SJ, where are you? [01:11:55] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 102 processes [01:12:23] helll where are you [01:12:26] hi [01:12:26] hi [01:12:27] hi [01:12:27] hi [01:12:28] hi [01:12:28] hi [01:12:28] hi [01:12:29] hi [01:12:29] hi [01:12:29] hi [01:12:30] hi [01:12:30] hi [01:12:31] hi [01:12:31] hi [01:14:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:15:45] dehunter, please stop it [01:16:31] you wouldnt respond!! [01:16:39] me? [01:16:49] You were talking to Hazard-SJ [01:17:01] That doesn't make it acceptable to flood a public channel [01:17:33] ok, i am very sorry [01:17:56] i dont know better i am new to irc [01:18:43] you do know what wikimedia labs is, don't you dehunter? [01:18:58] yea i do [01:21:56] Hello? [01:22:25] I am still here [01:22:30] ok [01:23:32] lets talk about c++ [01:26:46] Hello? [01:27:08] be patient [01:27:34] i cant be patient i am bad at that [01:27:54] bad place for you then [01:28:29] soory [01:29:39] irc is the worst type of chat EVERRR [01:29:58] i disagree ;) [01:30:49] OH MY GOD, why is it taking so long for everybody to respond, did they die or something? [01:35:15] ok you all must of died, becouse u wont sy nuthing [01:36:12] most are away [01:36:21] and as i said, you must be patient [01:36:27] answers may come hours later [01:37:32] woah dude you sound like the god of irc...... [01:38:42] irc sucks, im outta here, im goin to a real chat room [01:39:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.83, 3.98, 4.88 [01:41:12] hehe [01:44:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:46:17] Has the beta wmflabs lost its CSS? Example: http://deployment.wikimedia.beta.wmflabs.org/wiki/User_talk:Hazard-SJ [01:46:33] But http://deployment.wikimedia.beta.wmflabs.org/wiki redirects to the "Real" Meta-Wiki? [01:47:24] huh: Very strange [01:47:46] I've emailed labs-l about the first issue earlier, though [01:49:00] I've seen the second issue and haven't reacted (...yet?). [01:49:18] I've also seen errors like http://en.wikipedia.beta.wmflabs.org/wiki/Special:RecentChanges and that was in my email [01:57:33] RECOVERY Current Users is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [01:57:39] I saw that one too [01:58:22] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [02:11:46] For the record, http://lists.wikimedia.org/pipermail/labs-l/2012-October/000488.html [02:14:44] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:37:42] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 20% free memory [02:40:56] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 40% free memory [02:44:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:50:43] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 14% free memory [03:16:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:41:24] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 153 processes [03:42:48] So I'm running a bot on bots-1 [03:42:57] And I keep getting errors that there is no space left on the device [03:43:03] "IOError: [Errno 28] No space left on device" [03:45:36] PROBLEM Current Users is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS WARNING - 6 users currently logged in [03:45:55] Damianz: You there? [03:46:04] It said to contact you in the login message :P [03:46:14] yes [03:46:37] I'm having an error which says there is no space on the device on bots-1 [03:46:51] And it's trying to read/write to a file on /data/projects/legoktm [03:47:02] hmm [03:47:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:47:23] Sure you're writing to the project data? [03:47:27] Seems nfs is full [03:47:31] (I raised a bug about this last week) [03:48:40] Yeah [03:49:04] Pywikipediabot writes to a file as the "throttle" method. [03:49:10] So I can't really run any of my bots until that gets fixed -.- [03:49:32] [bz] (8NEW - created by: 2Damian Z, priority: 4Unprioritized - 6normal) [Bug 41008] Clear out NFS - https://bugzilla.wikimedia.org/show_bug.cgi?id=41008 [03:50:19] /data/project is only 35% used [03:50:49] Huh? So then how is it full? [03:51:08] It's not, but homedirs are :) [03:51:28] whats the commant to start the bot/what user are you running it as? [03:52:33] "/data/project/legoktm/py2/bin/python /data/project/legoktm/pywp/file_has_rationale_yes.py -pt:3" [03:52:53] I had it running in a screen so I could see the output/error message [03:52:56] since it's a long running bot [03:54:18] urgh, we really need to work on stack support rather than custom compiles of stuff [03:54:34] also it seems to be running fine now [03:54:38] Damianz: can we fix the long time that it takes to connect to an instance from bastion? [03:54:51] I just cleared 15mb of code that I'll have to re-download at some point [03:55:00] define 'long' [03:55:20] longer than usually [03:55:21] damian@bots-1:~$ logout [03:55:21] Connection to bots-1 closed. [03:55:21] [03:55:21] real 0m1.353s [03:55:21] user 0m0.010s [03:55:23] sys 0m0.000s [03:56:13] If it's being slow use -vvv and find out where it's hanging. I generally get under a second logins. [03:57:52] giftpflanze: logging into bots stuff is fine for me. I'm using the ssh forward thing though [03:59:13] seems to be the -q in my config for forwarding [04:00:04] weird, now it is gone [04:00:27] i'm thinking of a invisible control character [04:00:29] *an [04:00:52] Damianz: I'm not using a custom compiled version, its a virtualenv version [04:00:59] And thanks for starting it [04:01:30] In the future if this happens is it something I can do? Or would you/petan need to do it? [04:01:34] Oh in that case yay for virtualenv [04:01:44] Also I stopped it again, was only running to see what it did with files [04:02:02] Which was redundant as I magiced space until Ryan appears. [04:02:08] Ah ok [04:05:53] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f.pmtpa.wmflabs output: Warning: 15% free memory [04:17:05] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:30:52] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f.pmtpa.wmflabs output: Critical: 4% free memory [04:40:52] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f.pmtpa.wmflabs output: OK: 95% free memory [04:49:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:19:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:49:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:19:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:23:13] RECOVERY Disk Space is now: OK on conventionextension-trial i-000003bf.pmtpa.wmflabs output: DISK OK [06:31:12] PROBLEM Disk Space is now: WARNING on conventionextension-trial i-000003bf.pmtpa.wmflabs output: DISK WARNING - free space: / 74 MB (5% inode=51%): [06:38:22] RECOVERY Disk Space is now: OK on labs-nfs1 i-0000005d.pmtpa.wmflabs output: DISK OK [06:49:25] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:51:13] RECOVERY Disk Space is now: OK on conventionextension-trial i-000003bf.pmtpa.wmflabs output: DISK OK [06:56:46] RECOVERY Disk Space is now: OK on testing-arky i-0000033b.pmtpa.wmflabs output: DISK OK [06:57:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 5.02, 6.22, 5.42 [07:07:54] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.64, 4.70, 4.93 [07:19:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:49:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:19:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:49:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:19:34] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:49:36] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:57:55] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 6.17, 5.83, 5.21 [10:19:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:22:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 4.57, 4.67, 4.99 [10:49:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:19:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:49:37] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:05:12] PROBLEM Disk Space is now: WARNING on conventionextension-trial i-000003bf.pmtpa.wmflabs output: DISK WARNING - free space: / 78 MB (5% inode=51%): [12:06:42] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 19% free memory [12:19:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:41:42] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 20% free memory [12:49:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:49:43] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 18% free memory [12:55:35] PROBLEM Disk Space is now: WARNING on testing-arky i-0000033b.pmtpa.wmflabs output: DISK WARNING - free space: / 78 MB (5% inode=51%): [13:05:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [13:10:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 97 processes [13:19:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:49:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:49:43] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 26% free memory [14:21:53] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:36:17] \msg Reedy Please unban me from wikimedia-dev PLZZ! [14:36:27] Why? [14:36:55] because i like irc and i was so dumb to flood that channel [14:37:06] Not what you said yesterday. [14:37:06] i apologize. [14:38:44] i now realise that what i did was wrong and am SO very sorry [14:44:49] deon_26, why did you flood #wikimedia-operations as well? [14:45:25] i dont what i was thinking, but i promise not to let i happen agiain. [14:45:31] And -tech [14:45:40] They see me trolling, They be hatin [14:46:16] and e [14:46:32] and dev [14:48:49] I'm sorry about last night's flood. I was really dumb to do that,and i wanted to feel like a hacker. That doesn't excuse how I acted, and I promise I won't let it happen again. Next time, I'll let you know when I'm doing something instead of just going off like that. You're a great friend who deserves better, and I hope you can forgive me. [14:49:21] and unban me [14:52:05] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:04:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [15:22:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:39:34] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 97 processes [15:40:10] re long time to connect to instances: it occurs only from time to time, ssh -vvv gives no hint [15:54:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [16:11:32] Hello. Does anyone here have access to fix http://lists.wikimedia.org/pipermail/labs-l/2012-October/000489.html ? [16:13:33] yes, I don't know if I should though [16:13:37] it's a hashar thing [16:24:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [16:31:21] Damianz: will it "fix itself" eventually? [16:32:26] Don't think so, I'm not sure how the maintaince stuff works tbh. Currently the code is just being pulle dstraight from the repo without migration/i8tn stuff being run AFAIK. [16:49:57] Change on 12mediawiki a page Developer access was modified, changed by Hello71 link https://www.mediawiki.org/w/index.php?diff=596023 edit summary: [16:54:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [16:56:18] Someone might want to remove Cupco from labs seeing how he's banned as a anti-GNAA sock puppet... [17:00:09] There's such thing as that? [17:01:56] legoktm_v2, he's blocked on enwiki [17:01:57] not locked [17:02:43] Yeah, I said blocked, and pretty much de-facto banned. [17:03:56] I just don't think its a smart idea to give people access to a server/web space who are not welcome in the community [17:04:36] A bit enwiki centric, but I don't think this is the time to agf. [17:04:49] ಠ_ಠ [17:04:59] He's blocked on enwiki [17:05:09] As part of a major sock farm. [17:05:16] He "is still welcome" (can still edit) on every other wm wiki [17:06:12] I think we do have a potential problem with idiots though *unrelated* [17:07:49] Krenair: Well he hasn't disrupted any other wikis yet, but I don't think it's still safe to give a banned sock puppeteer access to the labs cluster. [17:08:17] Damianz: Yeah…I was a bit surprised when I found out that labs had no real vetting process. [17:08:48] You need a vauge justifications which we could easily check [17:09:21] As far as I can see he only has access to bastion and the bots project [17:10:34] Damianz: Yeah, but compared to the toolserver its pretty lax [17:10:54] Krenair: Tbh I really don't care what happens, I just didn't think it was a good idea so I thought I'd let someone else know about it [17:10:59] We also allow a lot more than the toolserver though. [17:11:35] Yup. [17:13:44] but not proprietary software :p [17:14:11] *cough* joins on the database *cough* [17:14:19] joins on the database is silly really [17:14:28] and proprietary software is a nightmare legally [17:15:08] i'm having some trouble getting into labsconsole. and it's being pretty slow too. can someone confirm whether or not it's possible to log in to labsconsole? [17:16:32] seems really slow to me [17:16:48] riiiight. but is it loginable? ;) [17:16:49] ha! so i'm not the only one [17:17:00] > Labs uses cookies to log in users. You have cookies disabled. Please enable them and try again. [17:17:19] nope [17:17:22] * jeremyb waves at a Ryan_Lane [17:17:30] It was doing this yesterday as well [17:17:37] orly [17:24:15] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [17:29:13] bbl [17:30:42] yes, I'm seeing the same problem [17:42:50] https://fbcdn-sphotos-c-a.akamaihd.net/hphotos-ak-ash3/532726_371403876281033_1070380402_n.jpg [17:51:21] jeremyb: seems the salt-master has a memory leak [17:51:29] I restarted it, memcache, and apache [17:51:33] things are back to normal [17:51:47] you'll need to log out/in, though [17:54:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [18:14:51] Ryan_Lane: Talking of leaks, nfs needs stabbing. [18:15:04] yeah [18:24:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [18:44:01] hi, how do I mount projectstorage.pmtpa.wmnet:/bots-project on /data/project in my instance? [18:44:56] gribeco: is it not mounted automatically when you try to access it? [18:45:16] oh/ [18:45:25] that's nice =) [18:45:27] I really wish we just changed these to real mounts... though that might suck when we drop network randomly to gluster [18:45:38] I just just looking at /data and there was nothing in it... [18:45:39] Though 99% of the time they're perm mounted anyway [18:45:40] I may [18:54:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [19:03:34] 'C++ is Java’s cousin. Similar to Java in many ways, the main difference being she grew up in a more innocent time and doesn’t believe in condoms/automatic memory management, so be cautious.' LOL [19:04:13] http://i.saac.me/post/in-laymans-terms-major-programming-languages-used-for/ is pretty awesome. [19:07:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [19:20:32] RECOVERY Current Users is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [19:21:26] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [19:24:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [19:54:34] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [20:02:32] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 283 processes [20:12:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 196 processes [20:24:34] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [20:54:47] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [21:22:37] Ryan_Lane: can't log out if i'm already logged out ;) /me tests [21:26:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [21:39:14] !log deployment-prep Running php multiversion/MWScript.php rebuildLocalisationCache.php --wiki afwiki [21:39:57] * Damianz gives Platonides a cookie [21:49:28] if giving the script a cookie would make it faster... :P [21:50:19] probably only takes like an hour or so =D [21:50:33] it's in zh-* now [21:50:39] and it still has more... [21:51:02] 10 so far... how many are there?? [21:51:20] that seems to have worked [21:51:53] zh, zh-classical, zh-cn, zh-hans, zh-hant, zh-hk, zh-min-nan, zh-mo, zh-my, zh-sg, zh-tw, zh-yue, zu [21:52:02] 370 languages rebuilt out of 370 [21:52:02] real 85m59.293s [21:52:02] user 5m42.620s [21:52:02] sys 3m37.320s [21:52:18] which -n ? [21:52:29] http://en.wikipedia.beta.wmflabs.org/wiki/Special:RecentChanges still shows a backtrace [21:52:38] none given [21:52:49] I don't think this virtual instance is multicore [21:54:56] night [21:55:03] nacht [21:55:08] gah [21:55:44] * Damianz looks at german [21:57:04] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [21:57:12] where? [21:57:49] nacht [21:58:10] ja [21:58:31] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=596066 edit summary: /* User:Hello71 */ done [22:02:35] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 283 processes [22:08:00] http://en.wikipedia.beta.wmflabs.org/w/index.php?title=Special:UserLogin gets "Error: invalid magic word 'ISONLINE'" , then it follows you around all pages [22:10:15] #0 /data/project/apache/common-local/php-master/includes/MagicWord.php(236): MagicWord->load('ISONLINE')... And thereafter I get the error on every page [22:12:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [22:27:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [22:37:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 102 processes [22:59:17] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [23:29:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [23:59:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)