[00:40:33] got to commute before the earth eclipses the sun [01:23:10] wikimedia/ores#1184 (example - ea85df5 : Adam Wight): The build passed. https://travis-ci.org/wikimedia/ores/builds/463094969 [06:23:22] 10Scoring-platform-team (Current), 10Operations, 10User-Ladsgroup, 10Wikimedia-Incident: Celery manager implodes horribly if Redis goes down - https://phabricator.wikimedia.org/T181632 (10Joe) [06:23:27] 10Scoring-platform-team, 10Operations, 10Wikimedia-Incident: ORES overload incident, 2017-11-28 - https://phabricator.wikimedia.org/T181538 (10Joe) [06:23:32] 10Scoring-platform-team (Current), 10Operations, 10User-Ladsgroup, 10Wikimedia-Incident: Investigate redis-cluster or other techniques for making Redis not a single point of failure. - https://phabricator.wikimedia.org/T181559 (10Joe) 05Resolved>03Open [06:26:13] 10Scoring-platform-team (Current), 10Operations, 10User-Ladsgroup, 10Wikimedia-Incident: Investigate redis-cluster or other techniques for making Redis not a single point of failure. - https://phabricator.wikimedia.org/T181559 (10Joe) Hi @Ladsgroup can you please elaborate on why you decided to go with sen... [06:29:52] 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Implement sentinel for ORES production Redis - https://phabricator.wikimedia.org/T122676 (10Joe) >>! In T122676#4750103, @Ladsgroup wrote: > I looked at sentinel. It's a little bit complex but easily doable. We probably need to do install redis on... [06:31:05] 10ORES, 10Scoring-platform-team (Current), 10Operations, 10vm-requests: New node request: oresrdb[12]003 - https://phabricator.wikimedia.org/T210582 (10Joe) I think we should pause this request until the choices that generated this ticket have been properly discussed with the SRE team. [06:31:05] El búfer 12 está vacío. [07:10:14] 10ORES, 10Scoring-platform-team, 10Operations, 10Puppet, 10Wikimedia-Incident: Logrotate should restart services when more people are around - https://phabricator.wikimedia.org/T210720 (10akosiaris) 05Open>03Resolved a:03akosiaris I 'll do so, thanks [07:15:41] 10Scoring-platform-team, 10DBA, 10MediaWiki-Database, 10Blocked-on-schema-change, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) [07:24:05] 10Scoring-platform-team, 10DBA, 10MediaWiki-Database, 10Blocked-on-schema-change, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) [08:40:45] 10Scoring-platform-team, 10DBA, 10MediaWiki-Database, 10Blocked-on-schema-change, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) [08:41:11] 10Scoring-platform-team, 10DBA, 10MediaWiki-Database, 10Blocked-on-schema-change, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) s4 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1002 [] db1125 [] db1121 [] db1103 [] db110... [08:41:37] 10Scoring-platform-team, 10DBA, 10MediaWiki-Database, 10Blocked-on-schema-change, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) [10:14:57] 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Implement sentinel for ORES production Redis - https://phabricator.wikimedia.org/T122676 (10Ladsgroup) Hey, - @akosiaris tested twemproxy in prod and it fails because celery issues redis transactions and twemproxy doesn't support redis transaction... [10:52:02] 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Implement sentinel for ORES production Redis - https://phabricator.wikimedia.org/T122676 (10Joe) So, I would separate the needs for the cache (where I guess we can use twemproxy) and celery (where we can't use it). Redis Sentinel is effectively a... [10:53:45] 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Implement sentinel for ORES production Redis - https://phabricator.wikimedia.org/T122676 (10akosiaris) >>! In T122676#4796844, @Ladsgroup wrote: > Hey, > - @akosiaris tested twemproxy in prod and it fails because celery issues redis transactions a... [11:12:34] 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Implement sentinel for ORES production Redis - https://phabricator.wikimedia.org/T122676 (10Ladsgroup) >>! In T122676#4796956, @akosiaris wrote: > > Just for completeness and expanding a bit on the above (which is correct), the celery broker AND t... [15:16:28] p [15:16:32] o/ [15:16:34] :) [16:04:39] (03CR) 10Hoo man: "> The ORES extension used to cause production outages every month or two, because various types of misconfiguration and external failures " [extensions/JADE] - 10https://gerrit.wikimedia.org/r/476446 (owner: 10Awight) [16:07:39] (03CR) 10Hoo man: [C: 04-1] Define a constant for the judgment model (031 comment) [extensions/JADE] - 10https://gerrit.wikimedia.org/r/476993 (owner: 10Awight) [16:09:58] (03CR) 10Hoo man: [C: 032] Rename functions to match the new model name [extensions/JADE] - 10https://gerrit.wikimedia.org/r/477000 (owner: 10Awight) [16:21:14] (03CR) 10Hoo man: [C: 031] "Code looks good at a glance, but I'm still not sure about the indexes" [extensions/JADE] - 10https://gerrit.wikimedia.org/r/475932 (https://phabricator.wikimedia.org/T200297) (owner: 10Awight) [16:32:31] (03CR) 10Hoo man: "Is this about highlighting or selecting such values (more concrete do you have a list of rc ids and you want to highlight some or do you w" [extensions/JADE] - 10https://gerrit.wikimedia.org/r/476447 (https://phabricator.wikimedia.org/T200297) (owner: 10Awight) [16:55:33] PROBLEM - puppet on ORES-web01.Experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [17:23:33] RECOVERY - puppet on ORES-web01.Experimental is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:33:31] o/ hoo [17:33:48] any progress on the wikidata suggested property features? [17:44:05] Amir1, apparently fr-tech also uses redis and might be interested in talking about redis-sentinel. [17:53:41] oh nice [17:53:56] I was talking to Joe about it to see how we can move forward [17:54:35] Seems to me his response was standard Ops: Let's not jump into new technology -- as opposed to "I don't like sentinel specifically." [17:55:15] Is this negotiation in akosiaris' sphere? I.e. has he signed off on the development of sentinel? [17:55:29] *sentinel related code/config [17:56:37] OK Time to pack up. I'm offline until I'm in Berlin [17:56:42] See you soon, Amir1! [17:56:45] Safe travels [17:57:16] BTW, I land tomorrow morning. I'll probably need some downtime but I'll drop you a message. maybe we can grab some lunch and we can co-work a bit before the rest of the team shows up :) [17:59:40] harej, I won't join for the PM monthly social. Please share my kind regards and apologies for not attending recently. [19:20:17] 10JADE, 10Scoring-platform-team, 10Gerrit, 10Patch-For-Review: Rename "JADE" extension to "Jade" - https://phabricator.wikimedia.org/T211046 (10hashar) We do not rename repositories in Gerrit. One would have to create a new one `mediawiki/extensions/Jade` then git push from the old repository to the new on... [19:21:06] 10JADE, 10Scoring-platform-team, 10Gerrit, 10Patch-For-Review: Rename "JADE" extension to "Jade" - https://phabricator.wikimedia.org/T211046 (10Paladox) Note that there is now a gerrit plugin that can rename repos, but it does not support 2.15 yet. [21:15:03] 10Scoring-platform-team, 10Icinga, 10Operations, 10Patch-For-Review: Add ahalfaker to ORES-related icinga contacts - https://phabricator.wikimedia.org/T210742 (10jijiki) 05Open>03Resolved a:03jijiki @Halfak Let us know it everything works as it should :) [21:54:42] 10ORES, 10Scoring-platform-team, 10Operations: Build helm charts for ORES - https://phabricator.wikimedia.org/T210269 (10jijiki) p:05Triage>03Normal