[00:00:08] Seems safest to me but maybe there is something cooler that can be done [00:00:18] right. that would work [00:04:41] 3Wikimedia Labs / 3deployment-prep (beta): Setup a mediawiki03 (or what not) on Beta Cluster that we can direct the security scanning work to - 10https://bugzilla.wikimedia.org/70181#c7 (10Dan Duvall) The host should be the same (en.wikipedia.beta.wmflabs.org). You just need to make sure the requests contain... [00:08:26] 3Wikimedia Labs / 3deployment-prep (beta): Setup a mediawiki03 (or what not) on Beta Cluster that we can direct the security scanning work to - 10https://bugzilla.wikimedia.org/70181#c8 (10Sherif Mansour) Will do [00:09:26] 3Wikimedia Labs / 3deployment-prep (beta): Unable to log in to beta labs using Safari - 10https://bugzilla.wikimedia.org/70145#c7 (10Bryan Davis) I'm getting redirected to the ssl version of https://en.wikipedia.beta.wmflabs.org when I login using Safari and that is broken. Not sure why the https redirect is... [00:10:41] 3Wikimedia Labs / 3deployment-prep (beta): Unable to log in to beta labs using Safari - 10https://bugzilla.wikimedia.org/70145#c8 (10Ryan Kaldari) It's currently working for me from an iPhone. [00:16:42] 3Wikimedia Labs / 3deployment-prep (beta): Unable to log in to beta labs using Safari - 10https://bugzilla.wikimedia.org/70145#c9 (10Chris McMahon) I am guessing that your local browser actually wants to log in securely (beta labs stopped listening for https traffic a while back): https://bugzilla.wikimedia.... [00:29:41] 3Wikimedia Labs / 3deployment-prep (beta): Unable to log in to beta labs using Safari - 10https://bugzilla.wikimedia.org/70145#c10 (10Ryan Kaldari) On a totally reset installation of Safari going to... http://en.m.wikipedia.beta.wmflabs.org/wiki/Special:UserLogin ... and logging in sporadically fails for me... [00:31:56] Nemo_bis: be sure that you didn't confuse url-rewrite with url-redirect [00:32:37] Nemo_bis: see a working example with $ less /data/project/newwebtest/.lighttpd.conf (1st enry) [00:32:41] 3Wikimedia Labs / 3deployment-prep (beta): Unable to log in to beta labs using Safari - 10https://bugzilla.wikimedia.org/70145#c11 (10Ryan Kaldari) I'm seeing the same behavior as Bryan, it's trying to redirect to https://en.m.wikipedia.beta.wmflabs.org/wiki/Main_Page which is failing. [00:33:06] *entry [00:33:43] * hedonil wants to update the help page, but something is wrong with wikitech atm [00:35:01] hedonil: andrewbogott_afk's fault? [00:35:37] a930913: idk. strange css [00:37:13] Does dumping a load of data into a database quickly differ from doing it slower? [00:40:05] a930913: how would you manage it to load it slower? In chunks? [00:40:10] CSS is gone? [00:40:31] hedonil: Yeah. [00:40:44] It started quickly, and is getting slower and slower. [00:41:20] So I wondered if it was waiting to index or something. [00:41:55] And the lack of index or whatever was slowing it down. [00:41:59] Coren: CSS of wikitech is gone it seems [00:42:24] mutante: Reedy and andrewbogott_afk are in the middle of messing with Wikitech (see -ops) [00:42:33] a930913: tools-db has binlogging enabled, that'll give a huge pile of logs ;-) - and indexing of course [00:42:39] no, i reported it there first [00:43:15] a930913: best is always to load first and apply indices later [00:43:30] I didn't say they knew what they broke yet, but they're the ones messing with it. :-) [00:43:58] * hedonil confirms: broken. [00:44:04] hedonil: Yeah, but if it's getting exponentially slower, it will take ages. [00:45:31] a930913: IIRC Kolossos had a problem when he aborted large transactions that the rollback took ages. So I wouldn't load "everything", but don't commit after every row either. [00:46:34] a930913: yep. it's an ongoing transaction, killing will probably not make you happier ;-) [00:46:35] No rollbacks. [00:46:58] It's not a large one, it's many small ones. [01:35:02] a930913: how is it going? [01:36:02] hedonil: Seems to have sped up a bit. [01:36:23] I wonder if the data differs in density. [01:37:04] a930913: all slow users are heading to bed now :P [01:37:16] :p [01:38:09] a930913: and even Coren has now time to play with his $20 SOHO DNS thingy [01:40:37] * hedonil lhao, while drinking another long island iced tea [01:44:43] * hedonil is bitchy atm, will apologize later... [03:26:28] 3Wikimedia Labs / 3tools: Install php5-xdebug on tool labs - 10https://bugzilla.wikimedia.org/70313 (10Marc A. Pelletier) 5PATC>3RESO/FIX [03:51:14] 3Wikimedia Labs / 3tools: Trusty doesn't have "at" installed by default - 10https://bugzilla.wikimedia.org/70324 (10Tim Landscheidt) 3NEW p:3Unprio s:3normal a:3Tim Landscheidt tools-exec-12, the experimental Trusty instance, has no "at" installed. While regular users should use the grid instead, it... [06:40:42] 3Wikimedia Labs / 3tools: Create postgresql user databases on request - 10https://bugzilla.wikimedia.org/63382#c21 (10Alexandros Kosiaris) Quite true. Thanks Tim. I would like to add that due to some changes in the placement of the OSM databases that freed up some hardware and quite some work from Yuvipanda... [07:15:52] (03PS1) 10Legoktm: Add ImageMetrics extension to multimedia channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/158062 [07:17:05] (03CR) 10Legoktm: [C: 032] Add ImageMetrics extension to multimedia channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/158062 (owner: 10Legoktm) [07:17:08] (03Merged) 10jenkins-bot: Add ImageMetrics extension to multimedia channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/158062 (owner: 10Legoktm) [07:18:22] !log tools restarting grrrit-wm for https://gerrit.wikimedia.org/r/158062 [07:19:37] eh, the log bot is down? [07:20:11] (03Abandoned) 10Legoktm: Add ImageMetrics to -multimedia [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/156945 (owner: 10MarkTraceur) [07:22:31] (03CR) 10Alexandros Kosiaris: [C: 032] Added wikitech passwords [labs/private] - 10https://gerrit.wikimedia.org/r/157668 (owner: 10Alexandros Kosiaris) [07:22:39] (03CR) 10Alexandros Kosiaris: [V: 032] Added wikitech passwords [labs/private] - 10https://gerrit.wikimedia.org/r/157668 (owner: 10Alexandros Kosiaris) [10:15:02] <_joe_> !log deployment-prep rolling out new hhvm package [10:17:02] <_joe_> wrong channel, sorry :/ [13:28:05] who touched wikitech last? [13:28:08] css is busted [13:28:50] its like the internet of yore [14:10:34] manybubbles: observe topic [14:13:11] andrewbogott: thanks [14:49:00] !log deployment-prep lgmsgbot test [14:49:08] ! [14:49:08] hello :) [14:49:12] !log [14:49:49] Boo [14:51:26] _joe_: You can log manually at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [14:53:23] Who has the rights to restart morebots? [14:59:00] YuviPanda: Do you know who to poke to get morebots back in this channel? [14:59:38] I know [14:59:46] bd808: apparently petan? :) [14:59:53] :) [15:00:47] My tool seach fu is weak. I can't even find the page that tells what tools are setup. [15:01:03] bd808: tools.wmflabs.org [15:01:14] !log [15:01:14] !log deployment-prep morebots is back thanks to petan [15:01:18] Logged the message, Master [15:02:27] !log deployment-prep _joe_ rolled out a new hhvm package ~5 hours ago [15:02:30] Logged the message, Master [15:02:39] Thanks much petan [15:02:47] np [15:09:38] <_joe_> petan: thanks a ton!! [15:09:50] no prob :o [15:31:56] 3Wikimedia Labs / 3deployment-prep (beta): Unable to log in to beta labs using Safari - 10https://bugzilla.wikimedia.org/70145#c12 (10Greg Grossmeier) Obvious question: SSL Everywhere installed? If so, tried disabling? [15:38:12] 3Wikimedia Labs / 3deployment-prep (beta): Unable to log in to beta labs using Safari - 10https://bugzilla.wikimedia.org/70145#c13 (10Bryan Davis) (In reply to Greg Grossmeier from comment #12) > Obvious question: SSL Everywhere installed? If so, tried disabling? I think my Safari is completely stock. I nev... [15:47:41] 3Wikimedia Labs / 3deployment-prep (beta): Unable to log in to beta labs using Safari - 10https://bugzilla.wikimedia.org/70145#c14 (10Dan Garry) (In reply to Maryana Pinchuk from comment #0) > You'll get taken to a "Safari cannot open page because it could not connect > to the server" page. Makes testing new... [15:54:11] 3Wikimedia Labs / 3deployment-prep (beta): Unable to log in to beta labs using Safari - 10https://bugzilla.wikimedia.org/70145#c15 (10Greg Grossmeier) Chris/Tyler: Any ideas on why Safari would be unable to login on Beta Cluster? [16:50:05] !log integration Restarted jenkins worker on deployment-bastion. [16:50:07] Logged the message, Master [16:54:02] !log integration Restarted jenkins worker on deployment-bastion a second time. Jenkins seems to see its executors again now. [16:54:04] Logged the message, Master [16:59:44] Coren, scfc_de, YuviPanda: If you have a few minutes, please log in to https://virt1000.wikimedia.org and poke around. That's a scap'd replacement for wikitech, I want confirmation that it's functionally equivalent before I replace the former wikitech. [16:59:58] wow, that was quick [16:59:58] Make sure you try things that require auth -- change privs, create instances, etc. [17:00:10] YuviPanda: it didn't feel quick to me :) [17:00:10] andrewbogott: True creds? [17:00:12] knee deep in icinga tho, and then have meetings [17:00:20] Coren: Yep, it shares ldap and db with the real wikitech [17:00:27] Just a different wiki frontend [17:00:53] * bd808 will poke some things there too [17:00:57] thx [17:01:15] So far the only error I've seen is that it said 'failed to delete' when I delete an instance… but then it deletes it anyway. [17:01:21] So, a curiosity but not a deal-breaker. [17:03:00] andrewbogott: Looking at it now. I seem to remember though that the "failed to delete" thing is a known bug. [17:03:11] scfc_de: yeah, there are a couple of situations where it happens. [17:03:39] Including an ldap leak which continues to thwart me [17:05:46] andrewbogott: Probably not an issue but some links to the sidebar lead to wikitech explicitly (i.e.: not relative) [17:06:02] Yeah, that seems fine. [17:06:15] And, at least, is a data problem [17:06:58] andrewbogott: As far as I can tell, things work fine. [17:07:32] Great, thanks for looking! [17:07:37] andrewbogott: Editing https://virt1000.wikimedia.org/wiki/Shell_Request/Yavorstoynov?action=formedit gave me "Fatal error: Cannot access protected property EditPage::$mTokenOk in /usr/local/apache/common-local/php-1.24wmf15/extensions/SemanticForms/includes/SF_AutoeditAPI.php on line 459 [17:07:37] (208.80.154.18)" [17:08:05] scfc_de: that's a good one! [17:08:05] And giving a user shell right: "Sorry! This site is experiencing technical difficulties. (Cannot contact the database server: Access denied for user 'wikiuser'@'208.80.154.18' (using password: YES) (10.64.32.18))" [17:08:18] Reedy, bd808, ^ [17:08:38] wrong db -- 10.64.32.18 [17:08:55] So where does that come from? [17:09:17] external storage [17:09:18] es3 [17:09:26] # es3 [17:09:26] 'cluster25' => array( [17:09:26] '10.64.32.18' => 1, # es1008 [17:09:48] *But* apparently the action itself (giving shell right) succeeded: . So "only" the result page failed? [17:10:32] I guess it logged too? [17:11:30] Talk page doesn't work -- https://virt1000.wikimedia.org/wiki/User_talk:BryanDavis [17:11:33] On https://virt1000.wikimedia.org/wiki/Special:UserRights/Yavorstoynov the "User rights log" is empty, so not? [17:11:44] Which is the same db error [17:11:54] db1059 [17:12:06] which is an s4 slave [17:12:11] s4 being commons [17:12:16] trying to use instantcommons? [17:12:48] well, "instantcommons" (via db, not http) [17:13:12] For this -- https://wikitech.wikimedia.org/wiki/File:Export_hell_seidel_steiner.png [17:15:44] Probably shouldn't load the filebackend.php at all for labswiki [17:15:52] https://noc.wikimedia.org/conf/highlight.php?file=filebackend.php [17:17:27] It sure would be nice to just move it into the cluster proper and fix a lot of these things. That's probably a big crazy step though [17:17:46] there are advantages to having it mostly out of the cluster… failsafe [17:17:55] although we also have -static [17:18:27] one step at a time ;) [17:18:33] we're moving in the right direction though [17:18:47] Reedy: The jobqueue stuff that comes right after filebackend.php is loaded is probably problematic too [17:18:54] yeah... [17:19:07] of course, the issues then come with N apaches connecting to LDAP, the OSM hosts etc [17:19:53] wikitech currently uses a cron for the jobqueue, I was figuring I'd have to reproduce that for the new install [17:20:58] We should be able to setup Aaron's new jobrunner but probably better left for the next steps list [17:21:05] ^ [17:21:26] 3Wikimedia Labs / 3deployment-prep (beta): Unable to log in to beta labs using Safari - 10https://bugzilla.wikimedia.org/70145#c16 (10Ryan Kaldari) My Safari is also stock. It seems that first the user is redirected to Special:CentralLogin/complete and then to Main Page (or wherever they came from). I've see... [17:24:52] andrewbogott: Uh,hah. I test a wiki, and I don't try /editing/ :-) Fails for me in SMW [17:25:29] that's probably the same error… maybe? [17:25:43] I'm hoping that Reedy and/or bd808 is preparing a magic fix-everything patch [17:25:50] andrewbogott: Userrights work fine for me. [17:26:11] Coren: Yeah, it was checking the 'completed' box that failed for scfc_de I think [17:26:40] * Coren mumbles something not quite polite about smw [17:27:55] andrewbogott: Reedy and I have a quarterly review in 30m. :/ [17:28:30] bd808: ok :) Want to regroup later in the day, or are you out for good? [17:28:41] 3Wikimedia Labs / 3deployment-prep (beta): Unable to log in to beta labs using Safari - 10https://bugzilla.wikimedia.org/70145#c17 (10Chris Steipp) (In reply to Ryan Kaldari from comment #10) > On a totally reset installation of Safari going to... > http://en.m.wikipedia.beta.wmflabs.org/wiki/Special:UserLog... [17:28:43] * bd808 wonders about adding some $wmgWhatever feature guards to CommonSettings.php [17:29:31] andrewbogott: I'll be back although I should spend some time working on my "real" project today too :) [17:30:02] bd808: fair enough. Any words of wisdom about what's going on with that other database ref? [17:30:29] andrewbogott: We have guards like "if ( $wmgUsePoolCounter ) {" in CommonSettings. I think maybe we need to invent some more of those to block out things wikitech doesn't want' [17:30:43] ok. [17:30:59] I'll look through commonsettings but I fear it won't be obvious to me what we do or don't want. [17:31:07] The other db things are probably about getting weights right for the various db sections [17:31:07] Anyway, enjoy your marathon meeting! [17:31:29] andrewbogott: All of the getRealmSpecificFilename() things would be suspect [17:31:46] filebackend.php, jobqueue.php especially [17:32:09] ok [17:32:59] Are those things that should be excluded for all fishbowl wikis? [17:33:06] I suppose not or they'd be already [17:34:37] $wgDefaultExternalStore (from db-eqiad.php) needs to be overridden for the error scfc_de saw with ES actions on approving a shell account [17:34:58] That can probably be put in the wikitech config file [17:35:30] there sure is a lot of getRealmSpecificFilename [17:36:00] wikitech is going to be a unique snowflake because it can't talk to prod and prod can't talk to it [17:36:38] so what do I want for $wgDefaultExternalStore? [17:36:40] Which is why at one point I wanted to make a realm for it [17:36:53] just DB://localhost [17:36:54] ? [17:36:59] 3Wikimedia Labs / 3tools: dump directory not updating - 10https://bugzilla.wikimedia.org/70354 (10bgwhite) 3UNCO p:3Unprio s:3normal a:3Marc A. Pelletier /public/dumps stopped updating about a week ago. It didn't even finish syncing up when brought online. [17:37:14] andrewbogott: Whatever you have now? eval.php on the existing site is your friend here. [17:38:58] DefaultSettings.php has "$wgDefaultExternalStore = false;" [17:39:17] Yeah, it's unset [17:50:04] andrewbogott: On https://virt1000.wikimedia.org/wiki/Special:UserRights/Yavorstoynov, the user has the shell right, on https://wikitech.wikimedia.org/wiki/Special:UserRights/Yavorstoynov, he doesn't. Are they supposed to be in sync? [17:50:28] scfc_de: I'd expect them to be, yes. Should be the same db [17:50:33] but, I'm messing with that now [17:52:53] !screen [17:52:53] script /dev/null [17:54:12] 3Wikimedia Labs / 3deployment-prep (beta): Unable to log in to beta labs using Safari - 10https://bugzilla.wikimedia.org/70145#c18 (10Dan Duvall) For me, it's redirecting to https on the final CA login redirect. Authentication seems to complete as I can navigate to an http URL from there and I appear to be l... [18:06:13] 3Wikimedia Labs / 3tools: dump directory not updating - 10https://bugzilla.wikimedia.org/70354#c1 (10Marc A. Pelletier) 5UNCO>3RESO/FIX Oh duh, actually /turning it back on/ might help (it was disabled while the manual copy was in progress). Turned back on, and started a sync. [18:23:27] Nischay Nahata and I are working on an extension that is supposed to include an 'inline diffs' javascript feature. It works on his local machine (and on en.wiki, as the user script we're starting from) but it doesn't work on my local vagrant or the labs-vagrant instance we set up to test it. http://education.wmflabs.org/w/index.php?title=Education_Program:WikiWorks/MediaWiki_Extension_Design_%28Fall_2014%29&action=epcourseactivity [18:23:42] it gives a js error, wgAction not defined. [18:24:19] Anyone have ideas on how we can get it working on the labs instance? [18:24:32] ragesoss: use mw.config.get('wgAction') ? [18:25:33] ragesoss: your wiki appears to have $wgLegacyJavaScriptGlobals = false (a good thing!) [18:25:51] is that different from en.wiki? [18:27:11] (here's the .js it's running, by the way: https://github.com/nischayn22/RecentActivityFeed/blob/master/modules/ext.inlinediff.js ) [18:28:36] ragesoss: As in WikEdDiff? [18:29:29] a930913: yeah, this is something we're working on to replace the current 'course activity' tab on course pages, for the EducationProgram extension. [18:29:50] the inline diff stuff is based on Writ_Keeper's user scripts. [18:30:26] ragesoss: as legoktm says, it needs code changes to not use deprecated wgAction but mw.config.get('wgAction') instead [18:30:32] not a labs problem [18:30:42] cool. :) [18:30:45] enwiki might have $wgLegacyJavaScriptGlobals = true, but it'll be false soooooon [18:30:53] and if you let this go to prod it'll break *then* [18:30:57] easier to have it break now ;) [18:31:01] :D [18:31:05] thanks much. [18:31:17] I'll pass this on to Nischay. [18:31:49] ragesoss: cool :) Also you should check out quarry.wmflabs.org :) [18:32:11] when did this go up? [18:32:18] ragesoss: 3-4 weeks [18:32:22] a week before wikimania [18:32:28] yeah, I will check it out soon! [18:32:31] ragesoss: cool [19:19:12] 3Wikimedia Labs / 3deployment-prep (beta): hhvm creates core file in /tmp/ filling mediawiki02 labs instance root partition - 10https://bugzilla.wikimedia.org/69979 (10Antoine "hashar" Musso) 5PATC>3NEW [19:43:39] andrewbogott_afk: can you rebuild the cirrus search index for labswiki? I tried to rebuild it on terbium but it looks like there was some kind of version mismatch because search is no longer working [19:44:34] andrewbogott_afk: instructions are https://wikitech.wikimedia.org/wiki/Search#In_place_reindex [19:44:37] sorry for the trouble [20:20:26] manybubbles: so I'm just running that script on vir1000? [20:20:45] andrewbogott: whereever you run maintenance scripts, I guess [20:20:53] ok... [20:20:56] but it has that mwscript stuff in it - so you might have to remove that [20:21:06] because its cribbed for the cluster [20:21:53] Hm, I don't actually have an mwscript in my path [20:22:03] so I guess I'll have to edit it in…various ways :/ [20:56:12] 3Wikimedia Labs / 3deployment-prep (beta): bits.beta.wmflabs.org down with 503 - 10https://bugzilla.wikimedia.org/69921#c6 (10Matthew Flaschen) 5REOP>3RESO/FIX Still working for me. If you still get a 503, please re-open, and add the URL and error text you get. [21:11:10] Coren, scfc_de, virt1000 is worth another go now, lots of fixes. I'll be back in a few. [21:12:55] YuviPanda: around? [21:29:28] Good afternoon all, so i'm having issues ssh'ing into our project (utrs-primary) with public key denied. It's worked up until now, but I also notice that our instances' puppet service is stale? Any help rectify the issue? [21:30:53] Izhidez: Lemme to take a look. [21:35:28] thanks Coren [21:41:22] Izhidez: What username are you trying? [21:43:05] Coren: deltaquad, as I do for everything else, and everything else is working still. I also checked with TParis a few nights ago, he can't either [21:44:19] Izhidez: I see (part of) the issue. Something broke the instance's puppet config and that led to LDAP being broken too. [21:45:14] oh boy, that doesn't sound good. [21:45:41] Izhidez: It's actually probably not all that hard to fix, though I'm trying to figure out why it broke first. [21:50:09] ... holey sheets. That instance seems to never have been properly migrated from pmtpa for some reason; it was still trying to fetch things from there. :-) [21:50:35] * Coren is surprised it ran at all. [21:51:34] Izhidez: Won't be too long now; puppet has a lot of catching up to do, but the box should be back to normal afterwards. [21:51:56] sweet [21:52:00] thank you [21:52:27] That's what they pay me the big^H^H^Hadequate bucks for. :-P [21:52:51] Izhidez: Try this? [21:54:31] :P [21:54:34] * Izhidez tries [21:56:03] works, thanks Coren [22:08:09] Coren: more issues...no files whatsoever on the server. [22:08:20] except a replica.my.cnf [22:09:09] Are there anybody knows that who to reset administrative password of mailing list? [22:10:03] (non-hidden that is from normal ls -l [22:10:46] jayanta: this is not the right channel; but query me and I'll find someone/file a task for it :) [22:13:55] Sorry! could you please guide me , where I can talk? [22:14:17] jayanta: I just queries you. [22:14:25] *queried [22:25:34] Coren: disregard, I lied. My home directory is empty 9_9 [22:47:51] trying to setup the math extension via the mw-vagrant roles, but running into two errors: [22:48:26] * ebernhardson just realized it didn't update the repo when enabling, so its months old code... [22:48:29] ignore me :_) [22:53:44] ebernhardson: Best bug report all day :) [22:59:21] Heh [23:01:08] Izhidez: You mean that it's empty but it's supposed to be? [23:01:43] Coren: ya. I forgot the default was $HOME when you first SSH in :P [23:01:58] files I was looking for were not there :P [23:13:11] where can I get publicly available uptime stats? [23:16:41] 3Wikimedia Labs / 3deployment-prep (beta): Unable to log in to beta labs using Safari - 10https://bugzilla.wikimedia.org/70145#c19 (10Dan Duvall) Created attachment 16355 --> https://bugzilla.wikimedia.org/attachment.cgi?id=16355&action=edit tcpflow log of failed login attempt in Safari (redirected to https) [23:17:12] 3Wikimedia Labs / 3deployment-prep (beta): Unable to log in to beta labs using Safari - 10https://bugzilla.wikimedia.org/70145#c20 (10Dan Duvall) Created attachment 16356 --> https://bugzilla.wikimedia.org/attachment.cgi?id=16356&action=edit tcpflow log of successful login attempt in Safari (redirected to... [23:17:56] 3Wikimedia Labs / 3deployment-prep (beta): Unable to log in to beta labs using Safari - 10https://bugzilla.wikimedia.org/70145#c21 (10Dan Duvall) Created attachment 16357 --> https://bugzilla.wikimedia.org/attachment.cgi?id=16357&action=edit tcpflow log of successful login attempt in Safari (no redirect to... [23:23:26] 3Wikimedia Labs / 3deployment-prep (beta): Unable to log in to beta labs using Safari - 10https://bugzilla.wikimedia.org/70145#c22 (10Dan Duvall) Chris, I've attached the tcpflow logs from our troubleshooting session. Let me know if there's anything else that would be helpful. A brief summary of what we obs... [23:38:14] fyi, I'm about to break wikitech again. With luck this will be brief... [23:38:53] yep, agreed [23:39:09] * andrewbogott types in wrong room, agrees with self [23:41:37] andrewbogott: do you know where I see some kind of uptime stats, especially about tools.wmflabs.org? [23:42:02] whym: Yuvi is writing monitoring tools right now. So, soon... [23:42:11] but I don't think there's much to see at the moment [23:45:54] andrewbogott: is there one for the infrastructure? [23:46:23] Yes, but access to the raw stats is somewhat restricted. [23:46:36] I believe that the monthly engineering report generally reports outages if there are any. [23:46:55] But, we recently discussed at our quarterly Operations review that we don't have great uptime stats. [23:47:01] um… I mean -- [23:47:04] we have good uptime [23:47:09] but not good tracking of what exactly our uptime is. [23:51:46] andrewbogott: thanks, someone involved saying "good uptime" is enough for me fow now, I guess. :) [23:52:23] whym: I think for tools in particular, the answer is: there have been issues that caused particular tools or sets of tools to die. But the infrasructure itself has been very stable. [23:52:33] Tools that are submitted to the grid engine can generally weather the bumps. [23:55:29] whym: you should be able to access http://ganglia.wikimedia.org/ [23:56:12] 3Wikimedia Labs / 3deployment-prep (beta): Unable to log in to beta labs using Safari - 10https://bugzilla.wikimedia.org/70145#c23 (10Chris Steipp) (In reply to Dan Duvall from comment #22) > A brief summary of what we observed while troubleshooting (Chris, please > correct me if this is wrong/incomplete): S... [23:58:31] YuviPanda: Yes, I am, but how can I interpret "labs stability" or something close to it from these charts? [23:59:30] more preferably tools, but I don't think that precision is not available in ganglia.