[00:25:07] bd808: merged [00:26:18] cool. [00:27:12] should we call this 1.1.0? [00:27:23] or 1.0.1? [00:28:06] Jeroen's feature seems like it's worth the bigger bump [00:29:04] 1.1.0 [00:38:11] legoktm: {{done}} [00:41:02] woot [02:39:11] so, I got roped into working on the board election scripts by James Alexander, because due to some last-minute miscommunication he and Philippe realized late that they hadn't sent out the "remember to vote" emails. [02:39:47] I have been very careful to stay very far away from any code that has anything remotely to do with CentralAuth in the past, so this is not my area. [02:39:58] I'd appreciate reviews of for that reason. [02:40:05] I'm sure it's awful, but it can be awful; it just has to work. [02:40:10] I already ran it on tin and the output seems sane. [02:40:20] ^ anomie, bd808, AaronSchulz, legoktm [02:41:20] if you do review it, please restrict comments to things that are wrong or broken; I already ran it so I'm not going to bother updating it for aesthetic / coding standard things. [02:42:19] ori: gu_home_db can be null [02:42:38] how do you determine the language in that case? [02:43:27] just default to en and hope for the best? [02:43:33] in general we don't use the "home wiki" for anything outside of deciding who the SUL winner was [02:43:42] that seems reasonable [02:44:10] you could manually call CentralAuthUser::getHomeWiki() [02:44:37] what does that do that the current script doesn't? [02:45:11] getHomeWiki() will calculate and pick a home wiki if one isn't set in the database [02:45:22] ahh, good. [02:45:47] okay, those are both good suggestions, i'll update and rerun [02:45:52] let me know if you spot anything else [02:46:27] so, $ca = new CentralAuthUser( $username); $homeWiki = $ca->getHomeWiki() [02:46:58] right [02:47:57] maybe I missed it, but I don't actually see where Maintenance.php is being loaded? but you said you already ran it... [02:48:58] i have my own wrapper https://github.com/wikimedia/operations-puppet/blob/production/modules/admin/files/home/ori/.binned/mw [02:49:06] that's probably not a great idea for a script that should be reproducible [02:49:47] i'll change that too [02:55:09] oh, and it ignores $nomail [02:57:19] ori: core calls RequestContext::sanitizeLangCode() after getting the user's language out of the db [02:58:56] is there another script that actually sends out the emails? [02:59:06] yeah [02:59:36] https://github.com/wikimedia/mediawiki-extensions-SecurePoll/blob/master/cli/wm-scripts/bv2013/sendMails.php [03:00:47] I think core calls that for another reason [03:00:49] $code = $request->getVal( 'uselang', 'user' ); [03:00:50] if ( $code === 'user' ) { [03:00:50] $code = $user->getOption( 'language' ); [03:00:52] } [03:00:53] $code = self::sanitizeLangCode( $code ); [03:01:01] the sanitizeLangCode is for the $request->getVal case I think [03:01:09] though I suppose it doesn't hurt [03:04:51] i added it [03:07:09] ok, looks sane. I tested the $wgConf language fallback stuff and it seemed to work as expected [03:07:50] cool, thanks very much! [03:26:37] holy fucking fuck! var_dump(in_array('fail', array(0))); [03:30:46] nice [03:30:54] MaxSem: var_dump(in_array('fail', array(0), true)); -- true -- type coercion fail [03:31:08] well [03:31:25] for e, it's human brain fail first of all [03:32:09] var_dump("fail" == 0); [03:32:38] var_dump((int) "fail"); [03:33:29] how about $a=[0, 'fail']; var_dump($a); then? consistency, common sense? fuck it all, we're PHP! [03:34:03] err, not very good example [03:34:17] MaxSem: i'm making you my guinnea pig. do you mind if i send you an email inviting you to vote in the 2015 board election? [03:34:21] $a = [0 => 1, 'foo' => 1]; [03:34:30] go on [03:34:45] I will help google improve their spam filters [03:34:50] :P [03:35:11] it will be in russian, i want to test localisation [03:36:34] ok, coming your way [03:37:13] $ACTIVEPROJECT fail [03:37:22] "You are eligible to vote in the 2015 elections for the Board of Trustees of the Wikimedia Foundation, which operates projects such as EnwikiS." [03:37:39] err, does it support wikitext {{GENDER}}? [03:37:53] cuz we totally have this info in the database [03:38:39] ori, I hope that "RuwikiS" is something that will not be in final emails? [03:39:35] also, since spamming date is kinda overdue, one tense in this email is wrong :] [03:39:40] we're trying leet speak this year [03:39:48] v0t3 plz [03:40:04] the gender thing is epic fail, yes [03:40:19] i am not taking responsibility for that, i got roped in at the last second [03:40:51] overall, it's bearable [03:40:53] as for tense errors https://meta.wikimedia.org/wiki/Wikimedia_Foundation_elections_2015/MassMessages/Voter_e-mail/ru [03:41:00] yeah, too late to fix messages [03:42:40] meh, if I change the tense it will be even worse [03:42:49] whatever [03:43:02] should have just spammed before it starts;) [03:43:12] heh [03:43:57] thanks for looking [03:47:05] ori, is my geshi fork in line with your plans or you wanted to switch to something else? [03:48:02] i'd like to switch to something else, but concrete action (creating a fork and working through the backlog of patches) trumps vague wishful thinking [03:48:37] i don't have any concrete plans to actually take on the job of migrating us to something else, and i don't know that anyone else does, so i think we should go with your fork for now [03:48:43] how much progress have you made? [03:48:57] cleaning up the ugliest places [03:49:44] like support for PHP 4 [03:50:04] cool [03:50:08] let's do it [03:52:42] do you know how I can get the project name from 'enwiki'? [03:53:47] uhh, maybe legoktm knows with his CA chops? [03:54:13] ori: you want "Wikipedia" from "enwiki"? [03:54:18] yes [03:54:50] $wgConf->siteFromDB should do that, but I think it's broken for special wikis [03:55:22] i'll just use "Wikipedia" for special wikis [03:55:29] since the wording does not imply that this is your home wiki [03:56:59] > var_dump( $wgConf->siteFromDB('enwiki')); [03:56:59] array(2) { [03:56:59] [0]=> [03:56:59] string(9) "wikipedia" [03:56:59] [1]=> [03:57:01] string(2) "en" [03:57:03] } [03:57:05] > [03:58:35] * legoktm bbl for dinner [03:59:31] thanks [03:59:34] that's excellent [04:00:40] problem is, "wikipedia" is not localised. not even properly capitalized [04:13:07] ori, did you still need review on that vote thing? [04:27:23] AaronSchulz: I think I'm good, but if you want to take a look and scan for anything obviously wrong, that'd be welcome [05:24:18] ori: emails aren't excluding bots :/ [05:24:24] Dear Legobot, [05:24:24] You are eligible to vote in the 2015 elections for the Board of Trustees of the Wikimedia Foundation [05:25:46] legoktm: sigh. [05:26:19] that's annoying. sorry. [05:36:45] ori, AND SOCKPUPPETS! [05:51:51] MaxSem: :o someone posted it to reddit: https://www.reddit.com/r/PHP/comments/37d0j3/chechil_an_updated_fork_of_geshi_syntax/ [05:53:48] ori, I've received 2 messages already [05:54:09] and I have accounts one very wiki [05:54:28] legoktm, :} [05:57:51] MaxSem: the second e-mail is localized, right? [05:57:59] yup [05:58:11] but not the subject line [05:58:28] heh [06:01:10] well, at least it's not http://www.usnews.com/news/articles/2015/04/23/drexel-university-accidentally-sends-acceptance-letters-to-500-rejected-students ; http://www.bloomberg.com/news/articles/2014-12-17/johns-hopkins-just-accidentally-accepted-almost-300-people-why-do-colleges-keep-doing-this ; http://www.washingtonpost.com/local/education/johns-hopkins-mistakenly-says-yes-to-hundreds-of-rejected-applicants/2014/12/16/20b5f9f4-8575-1 [06:01:10] 1e4-b9b7-b8632ae73d25_story.html ; https://www.insidehighered.com/quicktakes/2015/04/08/texas-state-accidentally-sends-acceptance-letters-450 ; http://www.universityherald.com/articles/16018/20150218/carnegie-mellon-university-accidentally-sends-acceptance-letters-to-800-applicants.htm [06:02:20] i love that: " Why Do Colleges Keep Doing This?" -- from the bloomberg article [06:02:25] e-mail is hard. [13:38:05] * anomie sees election stuff in scrollback and is glad someone else answered [15:03:23] 10MediaWiki-Core-Team, 10MediaWiki-General-or-Unknown, 6Phabricator, 6Project-Creators: Allow to search tasks about MediaWiki core and core only (create MediaWiki umbrella project?) - https://phabricator.wikimedia.org/T76942#1317765 (10Aklapper) a:5Aklapper>3None [15:36:29] anomie: you're wanting a saturday deploy for https://gerrit.wikimedia.org/r/#/c/207769/ and https://gerrit.wikimedia.org/r/#/c/207279/ ? [15:37:26] greg-g: You mean the reverting of them? I just put the earliest date the reverts could be done, feel free to push it back however far you want. [15:38:11] anomie: ah, I see, how about Monday? [15:40:39] greg-g: Monday is fine. The point is just not leaving the config cluttered with no-longer-needed hacks. [15:44:17] anomie: got it [15:47:57] greg-g: is there a plan to retry wmf8? [15:58:40] ori: how did things end yesterday? [15:59:07] with the memcache and 500s [16:00:20] this looks better: [16:00:23] http://graphite.wikimedia.org/render/?width=588&height=311&_salt=1432768192.869&target=MediaWiki.xhprof.ApiQueryFileRepoInfo.execute.calls.rate [16:01:02] the short version is that there was a cookie change that made Varnish very sad/useless [16:01:23] that has been reverted and bblack hacked a fix into our vcl scripts [16:01:52] I *think* it has been reverted anyway. legoktm made patches to do that [17:05:13] greg-g, bd808: it was reverted and I synced it out, but everything was on wmf6 so it didn't matter. bblack had the varnish hack, and I think twentyafterfour was just going to do the train today [17:05:55] he's doing it now [17:12:55] bd808: is that documented anywhere? [17:13:05] i missed that part of the troubleshooting session [17:26:28] ori: I think bblack sent an email about it... [17:29:15] ori: https://wikitech.wikimedia.org/wiki/Incident_documentation/20150527-Cookie [17:29:17] ? [17:35:03] thanks [17:39:17] Hey marktraceur, is there someone dapatrick should ping to get a cloak and into the staff channel? [17:39:56] Cloaks are a community thing [17:40:11] You follow the meta instructions (or the officewiki instructions) and wait. It'll happen within 2-3 weeks. [17:40:18] Getting into the staff channel is my purview. [17:40:45] Done. [17:40:54] Awesome. Thanks! [17:40:58] dapatrick: ^ [17:41:13] yeah, you don't need a cloak to get into channels [17:41:42] marktraceur: Swell. Thanks! [17:41:47] the cloak process run by the WM GCs is really slow [17:41:57] I don't really know the justification [17:42:06] Krenair: Mostly they're just busy folks [17:42:13] So why are they GCs and not other people? [17:42:24] That's just how the freenode group registration stuff works [17:42:33] We need contacts, and these people are it. [18:01:59] Krenair: They also have the power to ban people from all of our channels, so it's a really heavy responsibility [18:49:53] dapatrick: My initial stab at a report template: https://www.mediawiki.org/wiki/Wikimedia_Security_Team/Security_review_template feel free to update if you think of anything else [18:50:29] csteipp: will do. [18:52:17] Links to relavent background reading should be listed someplace. [18:52:29] Either in the initial request for review, or in the resutling report. [18:53:24] A link to a general page explaining what the thing is under reivew, e.g. https://www.mediawiki.org/wiki/Extension:Graph [18:57:08] Yeah, that would be nice [18:57:50] That may be more because I'm new and every project is new to me and I have to go search for background about them. [18:58:12] anomie: your input / review on https://gerrit.wikimedia.org/r/#/c/214404/ would be much appreciated. I have to admit I am a bit out of my depth with that change. [18:58:44] I think it's good :). I do want them to be generally accessible so people can read them and learn from what we found in other projects. So I think it would help. [18:58:50] anomie: I basically implemented TimStarling's suggested solution without entirely understanding why it is the best solution. [18:59:56] Does Phabricator support markdown? [19:00:34] * anomie looks [19:00:35] they have a custom syntax called "remarkup" or something [19:00:44] dapatrick: not exactly markdown but it's similar [19:00:50] it's kinda a hybrid of markdown and wiki markup [19:01:03] Got it. [19:01:12] phabricator DOES support RAEDME.md [19:01:16] in diffusion [19:07:21] ori, the patch looks reasonable to me, one problem is that it needs a unit test to make sure that token injection from wikitext is impossible. breaking anything semantic is a big plus [19:27:00] ori: Reviewed. I didn't see any major issues. As far as I can tell, Tim's solution is that we don't actually need unique strip markers after all since we never use multiple Parsers in a way that we would need them. [19:37:23] is there any way to categorize certain log messages as "info" or "debug" so that they don't get lumped in with "hhvm" or "exception" in kibana? it seems like an ongoing problem that we could solve somehow [19:37:26] bd808: ^ [19:37:51] twentyafterfour: yes'ish [19:38:08] we can tag things in the logstash layer [19:38:40] we actually parse levels from the hhvm messages [19:38:44] https://logstash.wikimedia.org/#/dashboard/elasticsearch/hhvm [19:39:40] twentyafterfour: which dashboard and which messages are you wanting to quiet? Something in fatalmonitor I'm guessing? [19:40:55] That "OutputPage::getModuleStyles" stuff is certainly spammy [19:41:50] yeah fatalmonitor... I was thinking there must be a way that intentional spammy stuff can be separated out of fatalmonitor...by the developers writing the log message rather than the people maintaining fatalmonitor dashboard [19:42:18] something that developers could stick in the message that tells logstash to put it into some other category and not in 'hhvm' [19:42:46] the problem is they are bypassing the logging layer and instead emitting PHP warnings [19:42:50] which is fucking dumb [19:42:55] oh [19:43:10] so whoever did this just needs to be educated and stop being fucking dumb? [19:43:18] :) [19:43:23] :) [19:43:25] yes [19:44:42] running blame now.... [19:45:30] this did it -- https://gerrit.wikimedia.org/r/#/c/206831/ [19:46:11] There is a wfDebugLog() [good] and a wfLogWarning() [yuck] [19:46:34] the wflogwarning is what causes the spew [19:46:42] I wonder .... [19:46:48] looks like Superm401 [19:47:17] hmm [19:47:41] so the reason people do this is that MediaWiki doesn't log by default [19:47:53] so they add noisy things to show up in the apache error logs [19:48:02] we are getting a lot of "OutputPage::getModuleStyles: style module should define its position explicitly" in the logs [19:48:17] I'm wondering if we can "fix" this for WMF prod by making a way to disable that [19:48:18] aude: that's what I'm trying to fix [19:48:18] in geshi stuff [19:48:21] ok [19:48:48] it's just creating a lot of noise and thus difficult to see important things [19:48:48] aude: its a new warning introduced by this patch -- https://gerrit.wikimedia.org/r/#/c/206831 [19:48:56] agreed [19:48:57] ok [19:49:02] bd808: they explicitly put that in there trying to make it noisy so that it'd get fixed [19:49:09] *nod* [19:49:27] they == gilles [19:49:41] yeah ... but noisy is one thing, but these are not fatals [19:49:53] also a lot of jobqueue stuff [19:49:55] and it's not just gilles, or just this one thing, there are constantly things like this popping up [19:50:02] yeah what aude said ^ [19:50:09] *nod* [19:50:25] So let me look at a fix for this in general [19:50:32] that's why I was looking for a more general solution to silencing these things instead of constantly adjusting filters in fatalmonitor [19:51:22] I don't mind seeing the log events as long as they can be categorized and colorized as non-critical issues for the purposes of rolling back branches ;) [19:51:24] wfLogWarning() says "Send a warning as a PHP error and the debug log. This is intended for logging warnings in production." [19:51:47] and it hands off to MWDebug::warning() [19:51:52] we could have a warning.log or different things (as long as people would still look at them) [19:51:53] * bd808 hates MWDebug [19:52:04] if it goes some place and no one looks, then that's also not good [19:52:40] well these messages do need to be seen, I just want critical and warning to be separate [19:52:59] +1 [19:53:14] like I said, I don't mind seeing them, I just want to be able to say "ok, this sync caused actual fatals to go up by this amount" [19:53:25] and not be confused by a lot of noise that isn't really fatal at all [19:54:35] because in general it's not at all straightforward to make a call like "this deployment needs to be rolled back"... and that's a call that needs to be made within seconds not minutes [19:54:59] Agreed. I think I can fix the dashboard query to ignore them [19:55:11] bd808: cool [19:55:30] if we need to make a change to the way developers implement warnings in the code, then I'll evangelize it [19:56:37] twentyafterfour: did you add this query there? -- (type:mediawiki OR type:hhvm) AND message:"OutputPage::getModuleStyles" [19:57:09] bd808: yes [19:57:20] ok. I'm going to nuke it [19:57:25] no problem [19:57:34] a crazy discussion we had last weekend was to craft a small frontend to link search queries filtering out logs and associate them with a task. The idea was to get rid of the known issues [19:57:42] I actually created my own dashboard so I wouldn't mess up the shared one with my changes [19:57:55] I have no idea how it can be done with kibana/logstash/elasticsearch though [19:58:24] easy enough [19:58:33] * aude uses stuff like tail + grep :) [19:58:44] and sometimes logstash [19:59:04] it would be nice if kibana had a priority order on the search queries so that something that matches one query could be automatically excluded from the rest of the following queries, and then a catch-all would catch new issues [20:00:14] potentially Sentry does that, no individuals to implement it thought [20:00:33] that way when you add a known issue you don't have to add another "OR term:" to the AND NOT ( ... OR ) hhvm query [20:00:46] I just filtered out type:hhvm AND level:Warning [20:00:56] yeah sentry might be cool but I'm not convinced, and it's operationally very difficult to deploy [20:01:05] bd808: nice [20:01:27] I'm pretty sure sentry won't scale for us at all without a *lot* of ops attention [20:01:42] +1 [20:01:53] in general, sending log events to an sql database is a really bad idea [20:02:17] and then using django to to correlation queries in real time makes that bad idea worse [20:02:49] at JOB-1 a colleague used Simple Event Correlator (SEC) to process a bunch of syslog http://simple-evcorr.sourceforge.net [20:02:52] but that was really low level [20:03:28] yeah ... at deviantART we built something based on redis and implemented decaying data structures kinda like RRDB ... then used a fuzzy match against the log message so that it only stored one record per unique error message type and then stored counters with decaying precision [20:03:43] for real work we used Netcool which is top notch :-} [20:03:49] but proprietary [20:04:24] SEC looks interesting [20:05:27] the alerting tool we had at deviantART was super cool because you could input expected levels for a message and then it would alert if the frequency of that message changed significantly from normal levels [20:06:14] it allowed us to essentially silence all the known issues but get notified if they got worse all of a sudden [20:06:30] anyway the main issue is creating unique id that properly qualify events [20:07:04] yes, we used simple string similarity I think, looking for patterns of sameness with sections of uniqueness in between ;) [20:07:22] but you could input custom regex patterns [20:07:35] or simplified regex really [20:07:41] just wildcards [20:08:33] but we also upgraded to structured logging around the same time [20:09:06] I made something that sounds similar fro $DAYJOB-1 that used python and regex matches to map log messages to known bugs [20:09:35] so in the code every log event had a unique string identifier that got hashed to make the key for the redis structure that stored everything [20:09:37] it was a daily batch process [20:10:37] twentyafterfour: So for logstash's fatal montitor I'm thinking we should discard notice and warning messages from hhvm [20:10:56] which you can get an idea about from this dashboard -- https://logstash.wikimedia.org/#/dashboard/elasticsearch/hhvm [20:11:26] we would keep Error and Fatal [20:12:13] bd808: sounds good [20:13:08] Warning: OutputPage::getModuleStyles: style module should define its position explicitly: ext.geshi.language.bash ResourceLoaderGeSHiModule [20:13:09] ... [20:13:14] those should be caught by a test really [20:15:17] hashar: all the modules are missing an explicit position, they added the warning to help track down all the call sites so that the code could be upgraded to add an explicit position [20:15:35] ahh [20:15:37] it was intentionally noisy to get developers attention [20:16:12] gilles added the warning, and fixed just about all modules everywhere [20:16:15] bd808: so adding a mustNot filter on "type:hhvm and (level:warning OR level:INFO)" [20:16:25] geshi's are generated on the fly, probably didn't grep [20:16:30] MatmaRex: great :) [20:16:36] MatmaRex: problem is he added the warning before the branch cut and fixed them after the branch cut ;) [20:16:47] twentyafterfour: I've updated the dashboard. Do a heard reload to see the new hotness [20:16:52] https://logstash.wikimedia.org/#/dashboard/elasticsearch/fatalmonitor [20:17:01] *hard reload [20:17:09] now I am just wondering why we need two log() calls but I am a newbie :-} wfDebugLog( 'resourceloader', $warning ); wfLogWarning( $warning ); [20:17:24] now the obivous problem is that the db layer hates us [20:17:37] hashar: damn fine question [20:18:03] my guess is that they were worried that nobody sees the RL logging channel [20:18:25] I would expect: log = $logging->getLogger('ressourceloader' ); log->warning( $message ); , but I have done too much python [20:18:39] that is the future [20:18:42] bd808: any objection to me making the graph row taller? [20:18:53] twentyafterfour: nope. [20:19:11] twentyafterfour: you have to stare at this one, so make it look like you like it :) [20:20:03] We are now diverging from the traditional "fatalmontor" script on fluorine pretty strongly [20:20:04] * twentyafterfour has a 27" apple display ... [20:20:16] bd808: can we have a permalink to whatever search we are looking at ? [20:20:28] in general? [20:20:31] bd808: yeah but fatalmonitor on fluorine is useless due to all the junk logging [20:20:57] hashar: https://logstash.wikimedia.org/#/dashboard/elasticsearch/fatalmonitor [20:21:09] ah I found the tiny icon at the top right to share a view :-} [20:21:16] hashar: it's the "share" icon (square with an arrow; second from right in the upper corner) [20:21:37] those links work for 2 weeks I think [20:22:21] and saving a new dashboard is easy; hit the floppy disk icon, enter a new name and hit the other floppy disk icon in the flyout [20:23:06] there are a lot more saved dashboards than the list shows too. It has a top-N limit [20:23:26] bd808: nice thanks :} [20:23:43] hopefully people will help fixing all those errors [20:26:04] now that the junk is gone I can see that there seem to be some real issues going on [20:26:09] MessageGroupStats::clear 10.64.16.22 1205 Lock wait timeout exceeded; try restarting transaction (10.64.16.22) DELETE FROM `translate_groupstats` WHERE tgs_group = 'page-Affiliations Committee/Resolutions' AND tgs_lang = 'en' [20:31:54] twentyafterfour, lemme guess: they're deleting blindly not checking there's something to delete? [20:32:28] why would that cause a lock timeout? [20:32:39] because mysql is silly [20:32:40] I would think that would fail silently [20:33:09] I had that problem in WikiGrok, fixed by checking first [20:44:10] ori, I said breaking semantic is good! :P [20:44:17] MaxSem: heh [20:50:16] twentyafterfour: bd808: sentry has a sampling feature [20:50:35] !log migrating Wikimedia to Postgres [20:50:43] if there are a million identical errors, it will count them but only actually store a hundred [20:51:13] that shields the DB from most of the load [20:51:51] MaxSem: But won't they just reject ori's patch anyway since it's submitted to gerrit instead of github? [20:52:09] anomie, their problem [20:52:24] hashar: oh thank goodness. I guess you saw the email to wikitech-l about how MySQL doesn't scale. [20:53:07] anomie: SemanticForms is still on gerrit [20:54:38] legoktm: it scales. We are the proof it does :-} [20:54:57] legoktm: it is just that MySQL as we use it is not a RDBMS [20:55:03] tgr: but to sample it needs to search for the error message right? [20:55:08] :P [20:55:11] which is the part I would expect to melt [20:55:30] mind you postgres as a json field and can index the keys in the json ! ;-} [20:55:57] mysql totally doesn't scale. Every oracle sales engineer I've ever talked to has told me so [20:56:23] those facebook guys are crazy to run their enterprise on it ;) [20:57:05] it's free too, can't be trusted [20:57:46] true. a solid rdbms must charge you based on cpu core use [20:59:22] bd808, per year! [20:59:42] Plus a bunch more for the ability to make a backup [20:59:54] +15% for maintenance/support [21:00:15] 25% discount on preprod, 50% on dev [21:00:27] I had an Oracle license at one point that was tied to "power units". One power unit == 1Mhz of CPU [21:00:54] when the server it was on dies I had to buy one via Ebay to get a machine slow enough to use the license [21:01:22] ohhh that is mean [21:01:32] oracle as worldwide unlimited licenses iirc [21:01:54] now you'd just put it in a restricted vm? ;) [21:02:31] Reedy, virtualization flexibility charge: +20% [21:02:50] I got a B in my database class because I refused to use oracle and did the project in mysql against the instructors wishes [21:03:24] it was the best project in the class, otherwise I probably would have gotten even worse grade [21:04:22] twentyafterfour, you should have thanked the instructor for attempting to make you a less worthless enterprise developer! [21:04:41] but this was in the computer science department, not CIS [21:04:51] I wasn't trying to be an enterprise dev [21:05:09] WELL YOU SHOULD HAVE HAD [21:08:29] bd808: yes, in the timeseries store, which is typically redis [21:08:30] cause J2EE / Oracle get you employment! [21:08:35] have fun folks [21:08:46] http://cramer.io/2014/05/12/scaling-sql-with-redis/ has some info about it [21:33:02] MaxSem: one of the best i18n bug reports https://phabricator.wikimedia.org/T100699 :) [23:52:36] ori: in your metrics and monitoring wanderings have you seen something that could alert based on an elasticsearch query? [23:53:09] that nutcracker thing stands right out when you look at the graphs (hosts that have lost connection to the memcached pool) [23:53:27] bd808: well, you can execute an elasticsearch query from the command line via cURL, and writing a bash script that works as a ganglia check is easy too [23:53:41] if this goes to graphite as well, there's a ready-made puppet resource [23:54:12] I've thought some about graphite but I'm not sure exactly what to graph [23:54:21] which graph should i look at? [23:54:26] message rate by channel I guess [23:54:28] * ori confesses to not having good logstash habits yet [23:54:41] ah right. we could send that to graphite [23:54:41] https://logstash.wikimedia.org/#/dashboard/elasticsearch/memcached-serious [23:54:48] rather easily too [23:54:55] zoom out to the 1 hour view to see what I saw [23:55:07] where is all the channel config stuff these days? [23:55:31] log channel, i mean [23:55:37] MediaWiki [23:56:00] they are the mwdebuglog labels [23:56:25] yeah I know, I mean where is the config to specify which log bucket goes where, the sampling, etc. [23:56:32] the stuff I reviewed at some point but now forget :P [23:56:44] logging.php [23:56:45] found it [23:57:00] yeah and stuff in initialzesettings [23:57:16] it's possible to stack log handlers, right? [23:57:38] yeah. that's how we do logstash + udp2log [23:57:58] we can use logstash to feed graphite too [23:58:11] it can act like statsd bascially [23:58:12] we can just add a StatsHandler [23:58:15] that increments a counter [23:58:21] let me submit a proof-of-concept PT to show [23:58:54] https://www.elastic.co/guide/en/logstash/current/plugins-outputs-graphite.html [23:59:46] this could work too [23:59:57] got a preference either way? (i don't)