[00:11:34] 3Librarization, MediaWiki-Core-Team, Deployment-Systems: Have a check for reported security issues in dependencies - https://phabricator.wikimedia.org/T74193#983367 (10csteipp) >>! In T74193#980413, @hashar wrote: > Can we get the job defined with Jenkins Job Builder in `integration/config.git`. Multiple people... [00:23:46] 3MediaWiki-Core-Team: Move purgeExpiredChanges to a job - https://phabricator.wikimedia.org/T87076#983383 (10aaron) 3NEW a:3aaron [00:36:25] bd808: jfdi [00:36:40] ori: :) I did quite a while ago [00:36:47] cool [00:37:03] things are still not happy but whatever [00:37:15] the boxes need more ram [00:37:24] or we need to log a lot less shit [00:37:29] and probably both really [00:37:37] what is the best way to see the problem? [00:37:39] the ganglia graphs? [00:37:46] like, what are you looking at that is telling you it's not happy? [00:37:47] yeah [00:38:09] [14:30:44] legoktm: http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Logstash%20cluster%20eqiad&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2&st=1421447393&g=mem_report&z=large [00:38:17] are the logstash redis instances password protected? [00:38:27] icinga alerts; OOM messages in ES logs; redis queu depth [00:38:46] yes, the redis instances are using ta password [00:38:51] *a password [00:38:58] "the" password I think [00:38:59] 3Librarization, MediaWiki-Core-Team, Deployment-Systems: Have a check for reported security issues in dependencies - https://phabricator.wikimedia.org/T74193#983416 (10Krinkle) Instead of adding the logic to a Jenkins job (whether via Jenkins web interface or via `integration/config.git`), I'd recommend adding t... [00:39:39] LLEN logstash (integer) 824777 [00:39:41] :( [00:39:41] uh, they only have 15 gigs of ram? [00:39:49] yeah the boxes suck [00:39:58] we got a lot more disk but no more ram [00:39:59] oh yeah, more ram [00:40:03] 64gb [00:40:22] yeah 48 or 64G would make a lot of differnece [00:40:24] just file a phab ticket asking for exactly that amount on the logstashes and copy/paste the ganglia graph [00:40:43] ask for 64, human time for dealing with OOMs is more precious than the cost difference between 48 and 64 [00:40:54] https://phabricator.wikimedia.org/T84958 [00:41:21] make a subtask that is more focused [00:41:47] you'd be helping robh and mark out, seriously. it's better for them to not have to think through all aspects of where logstash is at and where it is headed [00:42:03] and jgage hasn't been around long enough to know how to navigate the procurement waters [00:42:27] just "hi upgrade logstash100[1-3] to 64gb ram. see ganglia. kthxbye." [00:42:33] it doesn't need to be more elaborate than that. [00:42:34] :) [00:46:04] https://phabricator.wikimedia.org/T87078 [00:55:36] why in the fuck does hadoop send 500K log events per 15 minutes? [00:59:57] 3MediaWiki-Vendor, Librarization, MediaWiki-Core-Team: Upgrade monolog to 1.12.0 - https://phabricator.wikimedia.org/T85535#983434 (10csteipp) [00:59:59] 3Librarization, MediaWiki-Core-Team, Deployment-Systems: Have a check for reported security issues in dependencies - https://phabricator.wikimedia.org/T74193#983435 (10csteipp) [01:01:03] 3MediaWiki-Vendor, Librarization, MediaWiki-Core-Team: Upgrade monolog to 1.12.0 - https://phabricator.wikimedia.org/T85535#949087 (10csteipp) The test was successful! Hammering out exactly how we define the Jenkins job should be defined shouldn't block this task any more. [01:03:43] beer:30. see you guys later [13:34:09] legoktm: I hate to annoy you with this stuff... but could you merge my 3 patches touching Special:GlobalRenameQueue? [13:34:24] I'd like to move forward there, but I don't feel like merge conflicting myself [18:13:49] hoo: sure, looking now [18:14:05] Awesome :)