[00:05:33] deployment-db1 seems to be stuck in 'query end' [00:06:51] bd808: does that strike you as funny? should we ping springle? [00:07:37] ori: If he has some time to look into the beta dbs that would be cool [00:07:43] They seem to be slow all the time to me [00:08:01] importing dumps always takes far longer than it seems like it should [00:08:37] * bd808 disavows any knowledge of how to run a mariadb instance [00:09:08] it's very simple, really [00:09:10] "ask springle" :) [00:17:22] 3Wikimedia Labs / 3(other): (Tracking) Database replication services - 10https://bugzilla.wikimedia.org/48930 (10Kunal Mehta (Legoktm)) [00:17:23] 3Wikimedia Labs: Replicate centralauth.renameuser_status table to labs - 10https://bugzilla.wikimedia.org/68356 (10Kunal Mehta (Legoktm)) 3NEW p:3Unprio s:3normal a:3None Doesn't have any private information since all renames are publicly logged. Wanted so I can add some basic monitoring of global rena... [00:21:51] 3Wikimedia Labs: Replicate centralauth.renameuser_status table to labs - 10https://bugzilla.wikimedia.org/68356 (10Kunal Mehta (Legoktm)) [00:22:08] 3Tool Labs tools / 3Global user contributions: Support sorting results chronologically - 10https://bugzilla.wikimedia.org/68358 (10Waldir) 3NEW p:3Unprio s:3normal a:3Luxo Krinkle's MoreContributions tool in the Toolserver was quite handy in that it provided a global contributions listing that actual... [00:22:16] Coren: who do I need to bribe to get ^ to happen? :) [00:22:33] The centralauth one, not global user contribs :P [00:22:52] 3Tool Labs tools / 3[other]: Migrate to Tool Labs: https://toolserver.org/~krinkle/MoreContributions/input.php - 10https://bugzilla.wikimedia.org/61036#c4 (10Waldir) For me the most useful feature was the chronological sorting of the results. I reported that as bug 68358. [00:23:05] 3Tool Labs tools / 3Global user contributions: Support sorting results chronologically - 10https://bugzilla.wikimedia.org/68358 (10Krinkle) p:5Unprio>3Normal s:5normal>3enhanc [00:23:37] 3Tool Labs tools / 3Global user contributions: Global user contributions: Support sorting results chronologically - 10https://bugzilla.wikimedia.org/68358 (10Krinkle) [00:23:37] 3Tool Labs tools / 3Global user contributions: Global user contributions: Support wildcard in username - 10https://bugzilla.wikimedia.org/64499 (10Krinkle) [00:24:24] 3Tool Labs tools / 3Global user contributions: Global user contributions: Support wildcard in username - 10https://bugzilla.wikimedia.org/64499 (10Krinkle) [00:24:24] 3Tool Labs tools / 3[other]: Migrate to Tool Labs: https://toolserver.org/~krinkle/MoreContributions/input.php - 10https://bugzilla.wikimedia.org/61036 (10Krinkle) [00:24:24] 3Tool Labs tools / 3Global user contributions: Global user contributions: Support wildcard in username - 10https://bugzilla.wikimedia.org/64499 (10Krinkle) [00:24:24] 3Tool Labs tools / 3Global user contributions: Global user contributions: Support sorting results chronologically - 10https://bugzilla.wikimedia.org/68358 (10Krinkle) [00:24:37] 3Tool Labs tools / 3[other]: Migrate to Tool Labs: https://toolserver.org/~krinkle/MoreContributions/input.php - 10https://bugzilla.wikimedia.org/61036 (10Krinkle) [00:30:52] 3Wikimedia Labs / 3deployment-prep (beta): The current db schema change upgrade is taking far too long - 10https://bugzilla.wikimedia.org/68349#c8 (10James Forrester) Assuming it's running them in order (which seems likely), as of this comment it's taken 106 minutes to do 27k of BL-Wikidata's ~30k pages, so... [00:36:12] legoktm: Should be relatively simple, but we're in the middle of the MariaDB 10 upgrade. [00:36:23] hmm [00:36:39] aka: after s5 [00:36:47] how long will that take? [00:37:19] if it's going to be more than a few days, I'll start looking into other solutions [00:37:20] Couple days I expect; I really really want this to be done by the time I fly to London since many dewiki tools depend on it. [00:37:43] heh [00:37:53] But springle should be able to give a more precise ETA since he's the one currently doing the tokudb replication bootstrap. [00:38:09] ok, I'll ask him when he gets online [00:38:32] thanks [00:38:45] Said upgrade to MariaDB 10 has more useful features than you can shake a stick at. Getting rid of effing federation is the biggest one. [00:41:20] 3Wikimedia Labs / 3deployment-prep (beta): The current db schema change upgrade is taking far too long - 10https://bugzilla.wikimedia.org/68349#c9 (10Bawolff (Brian Wolff)) So right now the update script does a lot of little queries (Batch sizes of 200, each batch involves one select query for all the page_i... [00:42:38] is prod also upgrading to 10? or just labs? [00:50:37] legoktm: Bits of prod are now using 10 [00:51:04] legoktm: Analytics certainly. I think the upgrade to 10 for all prod is on the roadmap but I dunno what the timeline for that looks like. [00:51:14] ah, neat [00:53:50] 3Wikimedia Labs / 3deployment-prep (beta): The current db schema change upgrade is taking far too long - 10https://bugzilla.wikimedia.org/68349#c11 (10Bawolff (Brian Wolff)) (In reply to Gerrit Notification Bot from comment #10) > Change 148296 had a related patch set uploaded by Brian Wolff: > Reduce batch... [00:54:34] !log integration puppet somehow stalled on integration-slave instances. Had to delete /var/lib/puppet/state/agent_catalog_run.lock [00:54:36] Logged the message, Master [00:56:17] !log integration restarting diamond on integration-slave1001 - 1003 Related to {{bug|68254}} [00:56:20] Logged the message, Master [01:04:50] 3Wikimedia Labs / 3deployment-prep (beta): populateBacklinkNamespace script causing massive slave lag on beta - 10https://bugzilla.wikimedia.org/68349 (10Bawolff (Brian Wolff)) [03:07:27] is there a maximum number of threads allowed for a tool? [03:09:03] I've been trying to set up a tool that makes lots of requests to stats.grok.se, and when I try it with lots of threads in my python script (like 40), root kills the webservice. [03:09:19] it seems to be okay with 10, though. [03:09:47] (40 works fine on my laptop, and stats.grok.se seems to be able to handle it.) [03:12:16] ori: you rock [08:24:37] 3Wikimedia Labs / 3deployment-prep (beta): Set up graphite monitoring for the beta cluster - 10https://bugzilla.wikimedia.org/52357#c4 (10Antoine "hashar" Musso) 5RESO/FIX>3REOP Reopening. Yuvi made diamond send host metrics to a central graphite.wmflabs.org instance. Now we need a dashboard on top of... [08:27:50] 3Wikimedia Labs / 3deployment-prep (beta): Beta should not use productions interwiki.cdb - 10https://bugzilla.wikimedia.org/67931#c3 (10Antoine "hashar" Musso) I have absolute no idea how interwikis are generated nor how they are cached. Maybe the interwiki.cdb file generated on beta uses the list of beta... [08:30:38] 3Wikimedia Labs / 3deployment-prep (beta): db error on beta labs "centralauth.renameuser_status' doesn't exist" - 10https://bugzilla.wikimedia.org/67485#c6 (10Antoine "hashar" Musso) Thank you Kunal =) [08:51:20] 3Wikimedia Labs / 3deployment-prep (beta): Beta should not use productions interwiki.cdb - 10https://bugzilla.wikimedia.org/67931#c4 (10Antoine "hashar" Musso) (In reply to Greg Grossmeier from comment #2) > Antoine: would this break any of the auto-fancy stuff like where we fetch an > image from prod common... [08:52:50] 3Wikimedia Labs / 3deployment-prep (beta): Beta should not use productions interwiki.cdb - 10https://bugzilla.wikimedia.org/67931#c5 (10Marius Hoch) (In reply to Antoine "hashar" Musso from comment #3) > I have absolute no idea how interwikis are generated nor how they are > cached. Maybe the interwiki.cdb... [09:28:10] !log deployment-prep Removing role::beta::natfix that is now handled by labs DNS and the class is removed with {{gerrit|146091}} [09:28:13] Logged the message, Master [10:26:54] 3Wikimedia Labs / 3deployment-prep (beta): Setup rcstream for beta.wmflabs.org wikis - 10https://bugzilla.wikimedia.org/67888#c3 (10Antoine "hashar" Musso) 5NEW>3RESO/FIX Removing unrealted see also. Beta cluster RC stream is publicly available at http://stream.wmflabs.org/ per Bryan comment #1. [11:32:32] andrewbogott_afk, Coren, petan, YuviPanda: Sorry for the grid-ganglia-report spam. Didn't thought about that it would take two Puppet runs to kick in, so when my network connection dropped and I hadn't seen any bad effects, I went to bed with a clear conscience :-(. [11:36:38] !log toolsbeta Removed andrewbogott_afk, Coren, petan, YuviPanda from service group admin to prevent further spamming :-) [11:36:40] Logged the message, Master [12:25:38] 3Wikimedia Labs / 3deployment-prep (beta): beta.wmflabs.org failing with various error messages - 10https://bugzilla.wikimedia.org/68373 (10Željko Filipin) 3NEW p:3Unprio s:3normal a:3None Error messages: - Service Temporarily Unavailable - Database locked Noticed while investigating failed ULS Je... [13:35:32] !log deployment-prep apt-get upgrade on deployment-cache-bits01 + varnish upgrade [13:35:35] Logged the message, Master [13:43:26] !log deployment-prep rebased puppetmaster repo. Rebase got broken after ''0317463 - beta: New script to restart apaches'' got merged in. [13:43:28] Logged the message, Master [13:51:43] !log deployment-prep rebooting bits varnish cache [13:51:45] Logged the message, Master [14:02:30] !log deployment-prep rebooting deployment-cache-upload02 varnish not happy with memory mapping [14:02:32] Logged the message, Master [14:17:30] andrewbogott: hi, mind creating a new project called openmeetings ? [14:17:35] same setup as jitsi [14:17:44] matanya: ok, stay tuned [14:17:49] should I delete jitsi? [14:17:49] can reuse the public IP of jitsi [14:17:56] yes, it won't work [14:19:46] matanya: ok, done [14:20:40] thanks! i'll try to make this one work [14:22:22] !log deployment-prep upgrading varnish on deployment-cache-text02 [14:22:24] Logged the message, Master [14:26:36] !log deployment-prep upgrading varnish on deployment-cache-mobile03 [14:26:39] Logged the message, Master [14:32:09] andrewbogott: does labs have sun java 6 ? [14:32:42] I don't know what you mean by 'have' :) Can't you just install it? [14:34:58] I don't see it in apt [14:35:11] i can download from oracle of course [14:35:30] but wonder if it is available in apt.wikimedia.org [14:35:39] Yeah, that may be what you need to do if there isn't a standard ubuntu package. [14:35:51] If apt can't see it then it isn't in our repo [14:36:03] more compilations ... :) [14:36:27] i'll become an expert in weird dependencies after this project [14:42:50] 3Wikimedia Labs / 3deployment-prep (beta): populateBacklinkNamespace script causing massive slave lag on beta - 10https://bugzilla.wikimedia.org/68349 (10Chris McMahon) [14:43:05] 3Wikimedia Labs / 3deployment-prep (beta): beta labs mysteriously goes read-only overnight - 10https://bugzilla.wikimedia.org/65486 (10Chris McMahon) [14:57:07] 3Wikimedia Labs / 3deployment-prep (beta): populateBacklinkNamespace script causing massive slave lag on beta - 10https://bugzilla.wikimedia.org/68349#c12 (10Antoine "hashar" Musso) *** Bug 68373 has been marked as a duplicate of this bug. *** [14:57:07] 3Wikimedia Labs / 3deployment-prep (beta): beta.wmflabs.org failing with various error messages - 10https://bugzilla.wikimedia.org/68373#c1 (10Antoine "hashar" Musso) 5NEW>3RESO/DUP Caused by a massive database update *** This bug has been marked as a duplicate of bug 68349 *** [15:00:07] 3Wikimedia Labs / 3deployment-prep (beta): populateBacklinkNamespace script causing massive slave lag on beta - 10https://bugzilla.wikimedia.org/68349#c13 (10Antoine "hashar" Musso) Created attachment 15999 --> https://bugzilla.wikimedia.org/attachment.cgi?id=15999&action=edit log of update.php for the bet... [15:00:35] 3Wikimedia Labs / 3deployment-prep (beta): populateBacklinkNamespace script causing massive slave lag on beta - 10https://bugzilla.wikimedia.org/68349#c14 (10Antoine "hashar" Musso) Aaron Schulz might be interested in this bug report. [15:53:09] Hoi any news for new storage for the dumps ? [16:13:39] GerardM-: The hardware is being setup. [16:13:56] GerardM-: I should have it available for install very shortly. [16:15:27] :) [16:16:08] Please let the happy news be known widely [16:16:20] (when it is online and functional) [16:18:24] 3Wikimedia Labs / 3tools: Failed to set group members for local-oclc-reference - 10https://bugzilla.wikimedia.org/65534#c1 (10Andrew Bogott) Sorry for the delay in responding. If this is still happening, can you tell me what member you are adding? [16:50:36] 3Wikimedia Labs / 3tools: Install perl module CGI::Fast (libcgi-fast-perl) - 10https://bugzilla.wikimedia.org/68269 (10Tim Landscheidt) 5PATC>3RESO/FIX [16:59:59] Hey all, is there still an issue with DNS or is the SSL issue on beta labs not related? [17:00:29] marktraceur: when was there a DNS issue? [17:00:39] It think it's unrelated, hasn't https been broken on beta for ages? [17:00:43] andrewbogott: Dunno, the topic says so. [17:00:50] oops [17:00:52] andrewbogott: Anyway, https://en.wikipedia.beta.wmflabs.org/wiki/Lightbox_demo doesn't work [17:02:05] marktraceur: yeah, I think ssl has been broken for a while. I don't know who you would bug about that… hashar maybe? [17:02:56] Hm, I think he's on vacances? [17:03:09] YuviPanda: OI. Maybe you can halp [17:03:26] yeah, ssl on betalabs, uh, never worked? [17:03:27] IIRC? [17:03:36] marktraceur: and hashar is back from vacation [17:03:43] Huh, why do we have links to SSL versions then [17:03:47] Stupid computers [17:04:10] yeah, there were problems with getting valid certs, and I don't recall them being ever resolved [17:04:42] I'll pass the word on. [17:05:07] I thought that we had valid certs at some point, because the mobile people needed that for their test suites? [17:05:16] But if so YuviPanda would know about that it seems :/ [17:05:41] "mobile people" is a big group [17:05:44] heh [17:06:33] Anyway -- I think it can be made to work, but it not working is not a surprise (or presumably a crisis) [17:11:37] andrewbogott: Fair enough. Fabrice reports that his bookmark (to HTTPS) was working until yesterday. [17:12:02] He's also saying that Chrome is redirecting him to HTTPS for an unknown reason (no HTTPSE) [17:12:32] marktraceur: hm. [17:12:44] That's what I said. [17:12:48] Well… this is blind-leading-blind at this point :) We need to get someone who actually manages beta involved [17:12:55] Yeah [17:21:07] hi all, I am trying to create an account but theres a section when creating the account to include: Instance shell account name. how do I get this? [17:23:54] andrewbogott: marktraceur we [17:24:33] andrewbogott: marktraceur we've never had valid certs on beta labs. however, for a while we did enforce HTTPS connections, which was a nightmare. we no longer enforce HTTPS connections [17:24:36] mazza: Are you trying to create an account on wikitech.wikimedia.org? [17:24:55] chrismcmahon: So maybe his browser has cached the redirect to HTTPS? [17:24:58] mazza: You will have an on-wiki name and also a login name on virtual machines. 'Instance shell accont name' is the login name you will use on labs VMs. [17:25:01] yes [17:25:10] andrewbogott: marktraceur but HTTPS is available, you just have to manually tell your browser it's OK. [17:25:33] chrismcmahon: Not based on the HTTPS links we've been following... [17:26:30] marktraceur: aaaand, trying it I'm finding no HTTPS at all right now on beta [17:26:37] Heh. [17:26:46] wtf [17:28:53] I dont have a on-wiki name or a login name for the virtual machines. Is there a step I am missing? [17:29:17] and yes, I am trying to create an account on wikitech.wikimedia.org [17:36:38] 3Wikimedia Labs / 3deployment-prep (beta): beta labs no longer listens for HTTPS - 10https://bugzilla.wikimedia.org/68387 (10Chris McMahon) 3NEW p:3Unprio s:3normal a:3None https://en.wikipedia.beta.wmflabs.org/ no longer responds at all. While we have never had a valid cert for beta, we did in the... [18:12:05] mazza: If you are logged in on wikitech.wikimedia.org, you can see your instance shell account name under "Preferences". [18:16:51] I hate labs security groups [18:54:17] hiya, andrewbogott, yt? [18:54:27] ottomata: yep, just give me a few [18:54:35] k, answer this whenever, no hurry: [18:54:44] how do I officially add puppet group classes and variables to the labs interface? [18:54:51] i want the hadoop stuff available for all projects, not just analytics [18:54:55] so that I can document usage [18:55:01] for whoever wants to spawn up a hadoop cluster [18:55:12] right now we've got manually added puppet classes and variables [18:57:43] ottomata: I can answer you: it takes on of us to add a class to the defaults. [18:58:01] are the defaults in puppet somewhere? [18:58:04] ottomata: You can cause that effect to occur indirectly by opening a bugzilla. :-) [18:58:14] aww,i wanted to know how? [18:58:16] ottomata: No, they are managed through Wikitech. [18:58:16] is that in ldap? [18:58:18] oh! [18:58:19] interseting. [18:58:22] ha [18:58:23] inter-setting. [18:58:26] hmm, ok [18:58:29] bugzilla comin atcha [18:59:01] ottomata: IIRC, it requires the cloudadmin right to change them. [18:59:03] ottomata: want to help with a videoconf test? [18:59:06] sure [18:59:15] link is in #wikimedia-operations [19:00:49] joining...i think [19:00:52] Coren, hm [19:01:01] it is possible for things to change as these classes get more use [19:01:07] 3Wikimedia Labs / 3deployment-prep (beta): beta labs no longer listens for HTTPS - 10https://bugzilla.wikimedia.org/68387#c1 (10Antoine "hashar" Musso) HTTPS is handled using nginx on the varnish server by applying role::protoproxy::ssl::beta Looking at the puppet run of deployment-cache-text02.eqiad.wmflab... [19:01:22] how annoying is it to change them if/when they change? [19:08:07] 3Wikimedia Labs / 3deployment-prep (beta): beta labs no longer listens for HTTPS - 10https://bugzilla.wikimedia.org/68387 (10Greg Grossmeier) p:5Unprio>3Normal [19:08:53] 3Wikimedia Labs: Create "next page" function for tools.wmflabs.org - 10https://bugzilla.wikimedia.org/68390 (10Steinsplitter) 3NEW p:3Unprio s:3enhanc a:3None tools.wmflabs.org is very large, something like this should be crated imho: Page 1/3 (Show maximal: 20 | 50 | 100 | 250 | 500) [19:10:24] 3Wikimedia Labs / 3deployment-prep (beta): beta labs no longer listens for HTTPS - 10https://bugzilla.wikimedia.org/68387#c2 (10Antoine "hashar" Musso) I attempted to start it manually: # service nginx start Starting nginx: nginx: [emerg] SSL_CTX_use_PrivateKey_file("/etc/ssl/private/star.wmflabs.org.key")... [19:10:26] 3Wikimedia Labs / 3wikitech-interface: Include role::analytics::hadoop roles in default list of labs puppet groups - 10https://bugzilla.wikimedia.org/68391 (10Andrew Otto) 3NEW p:3Unprio s:3normal a:3None I'd like it if anyone could use the analytics roles to spawn up Hadoop clusters in any labs proj... [19:10:51] 3Wikimedia Labs / 3wikitech-interface: Include role::analytics::hadoop roles in default list of labs puppet groups - 10https://bugzilla.wikimedia.org/68391#c1 (10Andrew Otto) Ah, oops, amended list, I don't want to do Kafka or Zookeeper right now: Classes: role::analytics::clients role::analytics::hadoop::c... [19:10:54] ah cool! [19:10:57] Coren: ^ [19:19:08] 3Wikimedia Labs / 3deployment-prep (beta): beta labs no longer listens for HTTPS - 10https://bugzilla.wikimedia.org/68387#c3 (10Antoine "hashar" Musso) Apparently broken since April 11 :/ [19:48:52] 3Wikimedia Labs / 3deployment-prep (beta): beta labs no longer listens for HTTPS - 10https://bugzilla.wikimedia.org/68387#c4 (10Bryan Davis) This has been broken as long as we have been in eqiad as far as I know. role::protoproxy::ssl::beta is used to setup the nginx ssl terminators in front of *.beta.wmflab... [19:55:21] 3Wikimedia Labs / 3deployment-prep (beta): populateBacklinkNamespace script causing massive slave lag on beta - 10https://bugzilla.wikimedia.org/68349#c15 (10Bawolff (Brian Wolff)) FWIW, the update.php job finished successfully. lag on deployment-db2 seems to be holding at about 3 hours and 20 minutes for no... [20:04:56] Why does 'centralauth' have an entry in the 'wiki' table in meta_p? [20:05:03] beta labs index.php and api.php aren't responding, 503 Service Temporarily Unavilable. load.php works [20:07:51] 3Wikimedia Labs / 3deployment-prep (beta): beta labs no longer listens for HTTPS - 10https://bugzilla.wikimedia.org/68387#c5 (10spage) (In reply to Bryan Davis from comment #4) > This has been broken as long as we have been in eqiad as far as I know. FWIW I'm about 90% sure that https to beta labs worked in... [20:16:28] Krenair: I think there's a bug for that. [20:16:48] I ended up just checking that url was not null [20:25:43] Hmm... I can't seem to connect to dickson.wikimedia.org:6667 from tools? [20:55:23] Krenair: Do you happen to know from which host (tools-exec-xx) you are trying to connect? [20:55:45] I was actually just testing something from tools-login [20:57:00] I could use chat.freenode.net but not dickson.freenode.net/dickson.wikimedia.org, weirdly [20:59:55] Krenair: I was asking because some exec nodes use the shared Labs IPs; but tools-login should have its own. I can connect to dickson from external, but not from tools-login or other Labs instances (same as you). [21:01:28] Don't remember: Do we need to run identds on hosts bots want to do IRC from? [21:04:22] yes [21:06:12] scfc_de: Some of the newer nodes didn't have public IPs; I fixed that yesterday. [21:09:54] Coren: Yep, but the issue is with tools-login & Co. as well, and just tested: identd answers on that on 113. Could there be an issue with dickson/network to there? [21:11:19] Shouldn't be, really. Especially from -login where I wouldn't expect any IRC clients to exist normally. [21:11:44] Coren: traceroute seems to stop at second hop ae2-1118.cr2-eqiad.wikimedia.org (so I don't really know what I'm looking at there). [21:12:36] Ah, interesting. Yeah, there's some networking badness there. [21:12:53] 3Wikimedia Labs / 3wikitech-interface: sudo 'allow running as' gui slightly broken - 10https://bugzilla.wikimedia.org/61129#c1 (10Andrew Bogott) 5NEW>3RESO/INV I'm poking at this now and I can't detect any ill-effects. Since I logged this in the first place I'm closing it until I (or someone else) compl... [21:18:40] 3Wikimedia Labs / 3Infrastructure: LAMP instance becomes 404 a few hours after spawn (reproducible) - 10https://bugzilla.wikimedia.org/54059 (10Andrew Bogott) 5RESO/FIX>3RESO/WOR [21:18:41] 3Wikimedia Labs / 3Infrastructure: LAMP instance becomes 404 a few hours after spawn (reproducible) - 10https://bugzilla.wikimedia.org/54059 (10Andrew Bogott) 5UNCO>3RESO/FIX [21:25:05] 3Wikimedia Labs / 3Infrastructure: Puppet does not run on new instances: err: Could not retrieve catalog from remote server: Error 400 on SERVER: Must pass gmond_port to Class[Ganglia_new::Monitor::Config] - 10https://bugzilla.wikimedia.org/47773#c5 (10Andrew Bogott) 5NEW>3RESO/FIX That half-assed soluti... [21:52:11] Coren, bd808, chrismcmahon : Is anyone investigating "beta labs index.php and api.php aren't responding, 503 Service Temporarily Unavailable. load.php works." Should I file a bug? [21:52:57] spagewmf: consistent or intermittant? [21:52:58] spagewmf: file a bug please, that is pretty new [21:53:13] I just loaded http://en.wikipedia.beta.wmflabs.org/wiki/Special:Version [21:53:36] My first guess would be something to do with hhvm fcgi tuning [21:53:57] I've seen intermittant 503s in mw-vagrant with hhvm turned on [21:54:10] spagewmf: oh, actually all of a sudden I get a response, it's intermittent now for me [21:54:14] good to know. I've been trying since 10:40 local time, just got another 503. api.php just completed after a long delay [21:55:43] ori: ^^ ideas for tuning the fcgi container in beta? [21:56:05] ori: /data/project/logs/apache-error.log in beta is full of [proxy_fcgi:error] lines [22:00:31] chrismcmahon, bd808 I filed https://bugzilla.wikimedia.org/show_bug.cgi?id=68407 [22:00:37] 3Wikimedia Labs / 3deployment-prep (beta): beta labs getting 503 Service unavailable or slow - 10https://bugzilla.wikimedia.org/68407 (10spage) 3NEW p:3Unprio s:3major a:3None Since around 2014-07-22 20:00 UTC labs URLS like http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page have been failing for me... [22:03:27] http://www.downforeveryoneorjustme.com/en.wikipedia.beta.wmflabs.org says it's just me :) [22:08:05] 3Wikimedia Labs / 3deployment-prep (beta): beta labs no longer listens for HTTPS - 10https://bugzilla.wikimedia.org/68387#c6 (10Mark Holmquist) Especially given that Fabrice reports it only broke for him yesterday, I'm pretty sure this had been working until pretty recently. [22:08:50] 3Wikimedia Labs / 3deployment-prep (beta): beta labs getting 503 Service unavailable or slow - 10https://bugzilla.wikimedia.org/68407#c1 (10Bryan Davis) p:5Unprio>3High The apache error logs (/data/project/logs/apache-error.log) show quite a few errors from proxy_fcgi: [Tue Jul 22 21:59:53.156019 2014]... [22:09:41] bd808: i missed your ping as well [22:09:46] sorry, investigating [22:09:49] spagewmf, chrismcmahon. [22:10:34] thanks! [22:11:08] ori: It's an aside, but we are going to need to figure out how to aggregate those errors in prod and send them to fluorine [22:11:23] and to logstash probably [22:59:46] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools#Server_admin_log - hm, parser error? :/ [22:59:58] ("===UNIQ432ecb373692417b-h-11--QIN...") [23:06:45] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL looks fine, so only when transcluded? [23:39:54] 3Wikimedia Labs / 3deployment-prep (beta): Apaches refuse to start due to hhvm config - 10https://bugzilla.wikimedia.org/66234#c3 (10Ori Livneh) 5NEW>3RESO/INV The HHVM-on-Precise setup has been torn down. [23:40:37] 3Wikimedia Labs / 3deployment-prep (beta): HHVM: crashes with "boost::program_options::invalid_option_value" exception - 10https://bugzilla.wikimedia.org/68413 (10Ori Livneh) 3NEW p:3Unprio s:3normal a:3None HHVM package version: 3.1+20140630-1+wm1 Host: deployment-mediawiki01 ProcessID: 966 ThreadI...