[00:01:01] Question: Do I need to separately request bastion project access if I got a labs account at http://www.mediawiki.org/wiki/Developer_access? https://labsconsole.wikimedia.org/wiki/Help:Access says it may happen automatically and Help:Getting_Started says that bastion access is automatic but I don't see my name when I check the members at https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bastion [00:05:43] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [00:09:18] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:21:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:35:43] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [00:39:15] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:52:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:05:52] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [01:06:53] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 174 processes [01:09:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:11:53] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:23:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:24:13] nobody game for answering my question ^^ [01:29:39] BewareofDoug: where do you want to connect to [01:30:46] mutante, I'm not sure, I'm just trying to learn the system, the instructions were a bit convoluted (different pages gave different instructions or skipped things) so I wasn't sure if I had done everything I needed to. [01:31:41] BewareofDoug: are you in a project already? [01:32:22] well, depends if you think you need shell access somewhere [01:32:27] No, the instructions said I'd have to ask but implied that bastion was automatic when you get a labs account [01:32:40] i dont think it really is automatic [01:32:45] some users just use git [01:32:52] and never really want or need shell to instances [01:32:58] some want to actually work on instances [01:34:08] mutante, I'm not sure I really understand instances [01:34:22] BewareofDoug: virtual machines [01:34:37] if you just want to commit code to mediwiki or other projects you dont need bastion [01:34:44] yes, I mean I'm not sure I understand how they are used in this context [01:34:52] if you want to test your code on virtual machines , you do [01:35:11] oh, I see [01:35:29] BewareofDoug: check out the list of projects, every project can have one or more "instances" [01:35:54] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [01:36:04] if you are member of a project and you want to SSH to one of them, you need to jump through bastion [01:36:19] if you are not in a project anyways, then bastion wont do much for you [01:36:56] Do I need to be in a project to work on fixing things related to that project? [01:36:56] it's not a problem if you really need it, just explaining [01:37:20] BewareofDoug: depends :) if you just fix code and others review and merge it, you dont [01:37:41] you can clone all the code without it and suggest changes [01:37:56] you just cant look at the results on a machine [01:41:41] OK, so it sounds like there's no immediate reason for me to have it [01:41:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:42:22] BewareofDoug: if that changes, feel free to ping [01:42:26] But if I join a project I should probably get it then? [01:42:32] yea [01:42:55] OK, are there any other benefits to being part of a project, or is that it? [01:44:30] it's having shell access to the machines within that project [01:44:42] and/or being able to configure instances and apply other puppet classes [01:45:05] for reading any code and submitting changes to gerrit, you dont need to be a specific member [01:45:34] but people usually like to, because then they can see the next puppet run and check/test [01:46:28] OK, so maybe somewhere down the road I should join the Wikisource-tools or Wikisource-dev project then, I suppose, since those are most likely the areas where I'll work. [01:47:53] we can do that [01:47:56] hold on [01:48:36] There is no user by the name "BewareofDoug". Check your spelling. [01:48:48] Sorry, my username is Doug [01:48:52] just "Doug" [01:52:33] mutante, i've got to go for a bit, bbs [01:52:40] BewareofDoug: i added you to the 2 wikisource- projects as regular members and gave you the shell permission, i can't add you to bastion but i think that is outdated [01:52:47] ok, np, me as well [01:53:25] don't I need bastion? [01:54:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:54:29] BewareofDoug: i am getting an error currently, i am not 100% sure that is expected, i need to find out [01:54:33] thanks for adding me mutante, hopefully I can figure out how to make it useful [01:54:50] OK, thanks for the help and I"ll check in later on that part [01:54:53] yeah, np, i also need to go [01:54:59] we can check later, yeah [01:55:04] cu [01:55:13] bye [01:55:28] you should likely see a bot message here in a while [01:55:34] saying how it creates a home dir for you in that project [01:55:43] bye [02:07:32] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [02:11:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:24:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:30:49] Hello, anyone here? I requested that we test the Global AbuseFilter on beta.wmflabs.org http://deployment.wikimedia.beta.wmflabs.org/wiki/Global_requests#Set_.24wgAbuseFilterIsCentral_on_deployment.wikimedia quite a while ago (May!) [02:31:09] Can someone please tell me how to get this implemented on beta.wmflabs? [02:37:35] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [02:41:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:45:13] PiRSquared17: it takes time to get attention in here, possibly sumanah can advise, she entered just after you posted. Unfortunately, I'm no real help. [02:45:34] I'll look at the log [02:45:42] !htmllogs [02:45:43] experimental: http://bots.wmflabs.org/~wm-bot/html/%23wikimedia-labs [02:46:23] BewareofDoug: It's probably going to need extensive testing before being implemented in https://meta.wikimedia.org/wiki/Global_AbuseFilter [02:46:33] andrewbogott_afk: paravoid - do you have a moment? [02:46:47] "Hello, anyone here? I requested that we test the Global AbuseFilter on beta.wmflabs.org .... Can someone please tell me how to get this implemented on beta.wmflabs?" [02:47:00] Looking at https://www.mediawiki.org/wiki/Beta_cluster . [02:47:17] PiRSquared17: take a look at https://labsconsole.wikimedia.org/wiki/Deployment/Overview [02:47:38] sumanah: that's all Greek to me :P [02:48:01] PiRSquared17: ok. So, "beta labs" is another term for the "beta cluster" which "aims to be a functional, production-like environment in Wikimedia Labs suitable for final-stage testing of new features." [02:48:32] PiRSquared17: tell me where in https://labsconsole.wikimedia.org/wiki/Deployment/Overview you got confused and I'll see what I can do to explain :) [02:49:33] PiRSquared17: the main people who know how to add something to the beta cluster are Chris McMahon, Antoine Musso, and Brad Jorsch. All of them are evidently done with their workdays but if you email the labs-l mailing list they would respond tomorrow, I think [02:49:56] sumanah: Static assets are served by varnish on the deployment-cache-bits instance which serves the bits.beta.wmflabs.org domain might as well be written in https://en.wikipedia.org/wiki/Linear_A script [02:50:18] PiRSquared17: ha, understandable [02:50:26] I bet https://labsconsole.wikimedia.org/wiki/Help:Terminology doesn't help too much, either [02:50:27] also some babbling about squids [02:50:35] cephalopods are scary [02:50:36] and smart [02:51:34] * PiRSquared17 remembers watching a documentary on cephalopod camouflage/intelligence [02:52:11] PiRSquared17: remind me, have you ever submitted a change in Gerrit? [02:52:11] thanks BewareofDoug [02:52:11] I think https://labsconsole.wikimedia.org/wiki/Deployment/Overview#MediaWiki_config is key [02:52:19] * sumanah waits for lag to subside [02:53:10] PiRSquared17: "You will mostly want to add your configuration change in wmf-config/InitialiseSettings-wmflabs.php or wmf-config/CommonSettings-wmflabs.php and it will need to be reviewed via Gerrit." if you don't know how to do that, I suggest you look at this example: https://gerrit.wikimedia.org/r/#/c/26611/ [02:53:27] sumanah: I don't know PHP, so I can't really help much [02:53:41] and I don't have a gerrit account, whatever that is :S [02:54:05] PiRSquared17: if you can log into labsconsole.wikimedia.org then you can also log into Gerrit, which is a code review tool [02:54:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:54:43] sumanah: I tried logging in on labsconsole but my SUL account doesn't work there [02:55:09] No, it doesn't, it's a shame. If you get https://www.mediawiki.org/wiki/Developer_access then that'll give you Labs & Gerrit access [02:55:22] PiRSquared17: we'll turn it around fast [02:55:30] sumanah: I have recently learned that the instructions for getting going with a labs account and gerrit are almost as difficult to understand as the stuff above about cephalopods - worse in some ways because they actually appear to make sense but then don't work because things are missing, etc. I may make some changes based on what I found and ask you to take a look as I frequently misuse techspeak [02:55:53] BewareofDoug: thanks for telling us of bugs in the docs [02:56:21] my sympathies on the problems -- happy to help. emailing labs-l would be good because then other smart people can get to fix it before I even wake up, sometimes [02:56:52] part of the problem is the instructions at mediawiki skip things and presuppose other things that only make any sense after you read the instructions at labs, which one isn't likely to read until one has an account [02:56:54] Is there a way to just submit a request without knowing how to write PHP or Git/Gerrit/Bugzilla? [02:57:05] * PiRSquared17 never understands this mediawiki stuff [02:57:52] PiRSquared17: yes, you can ask at https://www.mediawiki.org/wiki/User_talk:Cmcmahon%28WMF%29 and also send me a note (I am Sharihareswara (WMF) on mediawiki.org) so I can followup [02:57:53] sound good? [02:58:07] ok [02:58:42] thank you for working to improve our sites [02:58:51] sorry that stuff is Under Construction [03:00:51] https://www.mediawiki.org/wiki/User_talk:Cmcmahon%28WMF%29#Global_AbuseFilter_on_beta.wmflabs.org is this good? [03:01:42] sure. in the morning I will file a Bugzilla ticket so we can track this properly [03:01:51] for now, offline I go [03:01:51] thanks again [03:07:38] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [03:11:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:24:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:36:26] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 155 processes [03:37:33] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [03:41:56] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:41:56] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: WARNING - load average: 16.46, 13.71, 8.77 [03:43:55] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 14.22, 12.56, 8.18 [03:46:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 10.64, 10.09, 6.51 [03:48:33] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 10.94, 9.81, 6.28 [03:48:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 10.26, 9.23, 6.30 [03:50:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 9.33, 9.62, 6.76 [03:55:24] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:03:52] PROBLEM Current Load is now: CRITICAL on accounts-database i-00000505.pmtpa.wmflabs output: Connection refused by host [04:04:32] PROBLEM Current Users is now: CRITICAL on accounts-database i-00000505.pmtpa.wmflabs output: Connection refused by host [04:05:14] PROBLEM Disk Space is now: CRITICAL on accounts-database i-00000505.pmtpa.wmflabs output: Connection refused by host [04:05:52] PROBLEM Free ram is now: CRITICAL on accounts-database i-00000505.pmtpa.wmflabs output: Connection refused by host [04:07:22] PROBLEM Total processes is now: CRITICAL on accounts-database i-00000505.pmtpa.wmflabs output: Connection refused by host [04:08:26] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [04:08:57] RECOVERY Current Load is now: OK on accounts-database i-00000505.pmtpa.wmflabs output: OK - load average: 0.93, 1.16, 0.74 [04:09:32] RECOVERY Current Users is now: OK on accounts-database i-00000505.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [04:10:13] RECOVERY Disk Space is now: OK on accounts-database i-00000505.pmtpa.wmflabs output: DISK OK [04:10:56] RECOVERY Free ram is now: OK on accounts-database i-00000505.pmtpa.wmflabs output: OK: 1054% free memory [04:11:57] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:12:28] RECOVERY Total processes is now: OK on accounts-database i-00000505.pmtpa.wmflabs output: PROCS OK: 87 processes [04:23:18] BewareofDoug: what does "shell username" mean? [04:23:34] specifically the preferred type [04:26:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:38:54] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [04:42:04] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:56:57] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:07:52] PROBLEM Free ram is now: WARNING on dumps-bot2 i-000003f4.pmtpa.wmflabs output: Warning: 19% free memory [05:10:22] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [05:12:04] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:27:45] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:40:48] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [05:42:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:58:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:10:53] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 4.45, 3.95, 4.95 [06:10:53] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [06:13:12] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:16:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.59, 3.80, 4.88 [06:18:33] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 2.56, 3.53, 4.62 [06:28:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:32:22] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:39:54] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 9.71, 8.27, 6.28 [06:41:13] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [06:41:33] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 9.92, 8.56, 6.44 [06:43:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:43:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 9.02, 8.49, 6.44 [06:51:23] RECOVERY dpkg-check is now: OK on centralauth-daemons i-000004aa.pmtpa.wmflabs output: All packages OK [06:57:22] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [06:59:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:10:53] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [07:13:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:28:56] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 3.73, 3.67, 4.69 [07:29:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:31:26] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [07:34:56] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.32, 3.86, 4.90 [07:36:32] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 2.41, 3.22, 4.75 [07:42:20] Change on 12mediawiki a page Wikimedia Labs was modified, changed by 1.232.232.98 link https://www.mediawiki.org/w/index.php?diff=601544 edit summary: 20000 [07:42:32] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [07:43:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.83, 3.89, 4.90 [07:44:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:44:07] Change on 12mediawiki a page Wikimedia Labs was modified, changed by Jasper Deng link https://www.mediawiki.org/w/index.php?diff=601545 edit summary: Undo revision 601544 by [[Special:Contributions/1.232.232.98|1.232.232.98]] ([[User talk:1.232.232.98|talk]]) rvt [07:53:51] Change on 12mediawiki a page Developer access was modified, changed by Sayak Sarkar link https://www.mediawiki.org/w/index.php?diff=601546 edit summary: [07:54:58] Change on 12mediawiki a page Developer access was modified, changed by Sayak Sarkar link https://www.mediawiki.org/w/index.php?diff=601559 edit summary: /* User:Sayak Sarkar */ [07:55:21] Change on 12mediawiki a page Developer access was modified, changed by Sayak Sarkar link https://www.mediawiki.org/w/index.php?diff=601565 edit summary: /* User:Sayak Sarkar */ [07:59:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:12:33] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [08:14:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:23:58] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.51, 4.50, 4.96 [08:29:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:42:36] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [08:44:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:44:44] !log manually running the beta auto updater from -bastion instance [08:44:45] manually is not a valid project. [08:45:03] grmblblbl [08:45:10] !log deployment-prep manually running the beta auto updater from -bastion instance [08:45:13] Logged the message, Master [08:59:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:06:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 13.20, 9.82, 6.93 [09:13:22] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [09:14:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:14:35] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 10.29, 9.32, 6.17 [09:16:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 9.76, 9.15, 6.48 [09:16:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 10.26, 9.21, 6.18 [09:17:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 11.42, 10.69, 7.52 [09:30:35] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:43:22] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [09:44:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:00:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:13:22] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [10:14:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:14:50] !log deployment-prep made mwdeploy gitconfig file to support color + added the 'git lg' and 'git lg2' aliases which gives a nice + concise log [10:14:50] Logged the message, Master [10:30:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:34:48] Change on 12mediawiki a page Developer access was modified, changed by Rajiff link https://www.mediawiki.org/w/index.php?diff=601610 edit summary: [10:36:52] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 4.27, 4.10, 4.83 [10:43:53] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [10:44:33] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 4.29, 4.33, 4.86 [10:45:58] !beta stashing change in /h/w/c [10:45:58] !log deployment-prep stashing change in /h/w/c [10:46:00] Logged the message, Master [10:46:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:52:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.97, 4.19, 4.98 [10:56:53] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 2.87, 3.63, 4.71 [11:01:06] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:15:13] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [11:16:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:31:14] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:45:23] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [11:46:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:01:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:15:23] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [12:16:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:31:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:37:20] Hi Jan_Luca! I am fighting with the puppet changes Andrew made to mediawiki.pp How exactly would you include a $role_requires( bla )??? [12:38:13] What I want to include is not inside the mediawiki/wikidata files but in the puppet template... How can I require_once them? [12:45:42] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [12:46:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:01:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:05:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [13:10:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 107 processes [13:15:42] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [13:16:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:31:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:45:43] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [13:46:55] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:48:55] Silke_WMDE_: Do you need the help anymore? [13:56:33] beta shows blank pages again [13:56:34] :/ [13:57:15] Silke_WMDE_: You can use $role_config_lines of mediawiki::singlenode for raw php lines [13:58:45] when I write sth like class wikidata::singlenode (blabla [13:58:45] $role_requires = [ "$IP/extensions/Diff/Diff.php" ], blabla [13:59:06] it is not inserted into my LocalSettings file [14:02:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:03:02] hashar_: https://www.mediawiki.org/wiki/User_talk:Cmcmahon%28WMF%29#Global_AbuseFilter_on_beta.wmflabs.org do you have a moment to look at this beta cluster request? [14:04:36] ahhh if only people used bugzilla instead of talk pages [14:05:02] hashar_: this particular person seemed quite frightened/bewildered by BZ, Gerrit, etc [14:05:06] sumanah: I will let chris answer to that. [14:05:07] hashar_: I'm happy to file the bug [14:05:29] I guess it will land in beta if it get scheduled to be added in production [14:05:33] else that should be setup in a labs instance :-] [14:08:04] hashar_: am I right in presuming that the way to add it to beta labs is to add it to wmf-config/CommonSettings-wmflabs.php ? https://gerrit.wikimedia.org/r/#/c/26611/ as an example? [14:08:33] not really [14:08:39] the first question is : is that extension going to be installed on the production cluster [14:08:56] if not, that has nothing to do on beta which aims to be as close to production as possible [14:09:17] if it were to be installed in production, we would indeed use the wmf-config/CommonSettings-wmflabs.php to configure it [14:09:25] to make sure it does not interact badly with production files [14:10:06] hashar_: https://labsconsole.wikimedia.org/wiki/Deployment/Overview#MediaWiki_config and https://www.mediawiki.org/wiki/Beta_cluster are what I'm using as process docs here [14:12:01] !log deployment-prep configured apaches to send their errors log in /home/wikipedia/logs (conf file is /data/project/apache/conf/wmflabs-logging.conf ) [14:12:01] Logged the message, Master [14:13:50] mlitn: the wikipedia.org docroot has been deleted on beta :/ [14:14:30] wow :o [14:14:45] oh no [14:14:51] I have read the wrong error message [14:14:52] so hmm [14:14:53] basically [14:15:15] I have NO IDEA what is going on [14:15:18] curl -i --proxy http://deployment-apache32:80/ http://en.wikipedia.beta.wmflabs.org/ [14:15:23] that gives me a 500 error [14:15:39] but the apache box does not seem to be report any error :/ [14:15:44] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [14:16:44] mlitn: the apache error logs in production are sent over syslog [14:16:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:17:05] hashar_: en.wikivoyage.beta.wmflabs.org also produces 500 [14:17:13] yeah I guess everything does [14:17:30] well, http://ee-prototype.wmflabs.org/ still servers something :p [14:17:36] serves* [14:18:35] ErrorLog /home/wikipedia/logs/apache-error.log [14:18:36] LogLevel warn [14:18:42] that is the apache conf I added [14:18:43] does not seem that helpful [14:18:49] maybe PHP is set report nothing [14:24:42] PROBLEM Current Users is now: WARNING on deployment-bastion i-00000390.pmtpa.wmflabs output: USERS WARNING - 6 users currently logged in [14:26:24] PROBLEM dpkg-check is now: CRITICAL on deployment-bastion i-00000390.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [14:28:22] Change on 12mediawiki a page Developer access was modified, changed by Mponnada link https://www.mediawiki.org/w/index.php?diff=601628 edit summary: /* Requesting Developer access */ new section [14:32:47] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:33:11] and now the bastion instance is dead [14:33:17] :/ [14:33:40] lost my internet connection during an apt-get upgrade :( [14:33:47] should have used screen [14:33:47] grbmblbl [14:34:16] hashar: bastion is ok [14:34:16] to me [14:34:21] ohh [14:34:26] so that is just me :-) [14:34:31] sec [14:34:31] Permission denied (publickey). [14:34:32] :/ [14:34:45] RECOVERY Current Users is now: OK on deployment-bastion i-00000390.pmtpa.wmflabs output: USERS OK - 5 users currently logged in [14:34:47] yeah [14:34:49] it's you [14:34:53] :/ [14:35:00] are you talking about !bastion :P [14:35:03] !bastion [14:35:03] http://en.wikipedia.org/wiki/Bastion_host; lab's specific bastion host is: bastion.wmflabs.org which should resolve to 208.80.153.194; see !access [14:36:14] in again :-] [14:40:22] 11/06/2012 - 14:40:20 - Updating keys for hashar at /export/keys/hashar [14:41:23] RECOVERY dpkg-check is now: OK on deployment-bastion i-00000390.pmtpa.wmflabs output: All packages OK [14:41:24] yet another issue with wikidiff :-D [14:42:43] PROBLEM Current Users is now: WARNING on deployment-bastion i-00000390.pmtpa.wmflabs output: USERS WARNING - 6 users currently logged in [14:45:52] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [14:47:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:47:47] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 13% free memory [14:48:41] Silke_WMDE_: Did you rebase your change before add the new class [14:50:35] because your change is not rebased (in Gerrit) after Andrew add his class: [14:51:34] !log deployment-prep blank pages on beta are caused by the EventLogging extensions being required although it is not pulled. [14:51:34] Logged the message, Master [14:51:39] !log wikidata-dev wikidata-dev-2: restarted apache because of the ages you had to wait on test-client [14:51:41] Logged the message, Master [14:52:43] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 32% free memory [14:57:06] Silke_WMDE_: I will be away in few minutes, when you have another question, I will be back 18.00 - 19.00 CET or you have to wait for Andrew [14:57:37] Ok, thx. I'm in touch with Andrew. See you later. [15:02:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:04:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [15:11:13] !log deployment Reseted all extensions to their latest master version... [15:11:13] deployment is not a valid project. [15:12:47] !log deployment-prep Resetted all extensions to their latest master version... [15:12:49] Logged the message, Master [15:14:18] https://labsconsole.wikimedia.org/wiki/Special:NovaProject says "There were no Nova credentials found for your user account. Please ask a Nova administrator to create credentials for you." [15:14:19] Could somebody do that please? I'd like to become a member of https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bugtracker . My account is Aklapper [15:14:47] !credentials [15:14:47] when you see No Nova credentials found for your account just relog to wiki and should be ok [15:15:02] andre__ ^ [15:15:52] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [15:17:04] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:18:53] !log deployment-prep fixed up beta by repelling EventLogging extension which was in a weiiiiird stat [15:24:12] sumanah: Hello, should I report that change on Bugzilla? (to enable global abusefilter on beta) [15:24:17] !htmllogs [15:24:17] experimental: http://bots.wmflabs.org/~wm-bot/html/%23wikimedia-labs [15:24:43] PiRSquared: hashar and chrismcmahon and anomie are better placed to help you out on this -- the short answer is, beta may not be the best place to test it [15:24:48] I have a meeting in a few minutes, sorry :/ [15:25:16] If beta isn't a good place, what is? [15:25:16] Labs, but not beta [15:25:52] petan: Thanks! As I get "Login error: Labs uses cookies to log in users. You have cookies disabled." in two browsers, and this is definitely not true, my issue must be somewhere else. I'll play around a bit (password reset etc). [15:26:15] restart browser :P [15:27:12] that already was a fresh instance :) [15:29:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 107 processes [15:30:15] andre__: This is silly, but -- when I get that cookies error it always seems to go away if I check the 'remember me for 30 days' box. [15:30:32] I'll give it a shot [15:31:01] Well, crap, looks like labsconsole is also offline. [15:31:32] andrewbogott: memcached died on virt0 [15:31:36] yep, give me a moment. [15:31:47] That's happened twice this month now :( [15:31:55] a bit more according to my records :-] [15:32:02] that is more like twice per week [15:32:37] \O/ [15:32:48] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:32:48] I guess this is the second time that I've noticed before someone else fixed it. [15:32:55] hashar: Back up, right? [15:33:04] should [15:33:19] works for me -] [15:33:28] andrewbogott: since you get interrupted, would you mind merging a change for me please ? https://gerrit.wikimedia.org/r/#/c/32047/ that is a shell script to automatically update beta ;) [15:33:49] not going to cause any harm to production, it is only running in labs [15:34:34] Error: Cannot open log file '/var/log/nagios/archives/nagios-11-06-2012-00.log' for reading! [15:34:34] oh no :( [15:35:04] You're on thin ice asking me to review something that relies on 'git pull' [15:35:11] well chad reviewed the change [15:35:19] that follow up a discussion I had with him [15:35:35] I guess it is merely inspecting the change making sure it is not going to provide us with root access on the cluster :-] [15:36:14] yeah, it looks fine to me. [15:38:09] [bz] (8NEW - created by: 2Antoine "hashar" Musso, priority: 4Unprioritized - 6normal) [Bug 41825] beta autoupdater does not update new extensions - https://bugzilla.wikimedia.org/show_bug.cgi?id=41825 [15:38:46] andrewbogott: yeah maybe we should rebase [15:39:10] andrewbogott: though I don't think anyone is doing any commit directly on the beta cluster [15:39:12] thanks for the merge :-] [15:39:47] I was at a table full of non-WMF devs yesterday all of whom agreed that 'git pull' should be expunged from the git user docs :) [15:42:17] ahh [15:42:22] the change is working. [15:42:25] thanks andrewbogott !!!!!!!! [15:45:20] !log deployment-prep Fixed up the WMF beta auto updater so it nows take care care of new extensions {{gerrit|32047}} (thanks Chad). That produced blank page on the site for most of the afternoon since a new extension as been added in mediawiki-config but was not updated by the script. [15:45:22] come on bot [15:45:23] ping [15:45:43] !log deployment-prep Fixed up the WMF beta auto updater so it nows take care care of new extensions That produced blank page on the site for most of the afternoon since a new extension as been added in mediawiki-config but was not updated by the script [15:45:50] !log foo [15:45:52] ;- [15:45:52] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [15:45:59] petan: da bot is dead again [15:46:12] mmm [15:46:23] it is really unreliable :( [15:46:50]