[08:43:52] 10Analytics-Cluster, 06Analytics-Kanban: Provision new Kafka cluster(s) with security features - https://phabricator.wikimedia.org/T152015#3272804 (10Ottomata) We recently submitted two hardware orders to provision beefy aggregate clusters in both eqiad (T161636) and codfw (T161637). After some more in depth... [08:54:23] 10Analytics, 10ChangeProp, 10EventBus, 06Services (later), 15User-mobrovac: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088#3272831 (10Ottomata) Yo dudes! We are ordering more Kafka hardware (see T152015). We've decided not to provision a beefy aggregate... [10:55:55] 10Analytics, 10Analytics-Cluster, 06Operations, 10Traffic: Enable Kafka TLS and secure the kafka traffic with it - https://phabricator.wikimedia.org/T121561#3273006 (10Ottomata) [11:01:42] 10Analytics, 10Analytics-Cluster, 06Operations, 10Traffic: Encrypt Kafka traffic, and restrict access via ACLs - https://phabricator.wikimedia.org/T121561#3273015 (10Ottomata) [11:13:16] 10Analytics: Correct uniques computation to not exclude countries that don't have either underestimates or offset - https://phabricator.wikimedia.org/T165661#3273019 (10JAllemandou) [11:46:36] (03PS1) 10Joal: Correct last uniques oozie jobs (wrong join) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/354214 (https://phabricator.wikimedia.org/T165661) [12:12:44] 06Analytics-Kanban, 13Patch-For-Review: Count global unique devices per top domain (like *.wikipedia.org) - https://phabricator.wikimedia.org/T143928#3273176 (10JAllemandou) Another thing we forgot on core data checking: pageview and redirect_to_pageview should not happen conjointly ``` SELECT is_pageview,... [12:49:38] 10Analytics, 13Patch-For-Review: Correct uniques computation to not exclude countries that don't have either underestimates or offset - https://phabricator.wikimedia.org/T165661#3273248 (10Nuria) [12:52:58] (03CR) 10Nuria: [V: 032 C: 032] Correct last uniques oozie jobs (wrong join) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/354214 (https://phabricator.wikimedia.org/T165661) (owner: 10Joal) [12:53:55] 06Analytics-Kanban, 13Patch-For-Review: Correct uniques computation to not exclude countries that don't have either underestimates or offset - https://phabricator.wikimedia.org/T165661#3273276 (10Nuria) [13:02:10] (03PS1) 10Elukey: Remove logrotate and syslog configuration [analytics/kafkatee] - 10https://gerrit.wikimedia.org/r/354223 (https://phabricator.wikimedia.org/T151748) [13:02:20] ottomata: --^ [13:02:33] (since I need to rebuild kafkatee for the last change that Filippo made) [13:10:16] (03CR) 10Ottomata: [C: 031] Remove logrotate and syslog configuration [analytics/kafkatee] - 10https://gerrit.wikimedia.org/r/354223 (https://phabricator.wikimedia.org/T151748) (owner: 10Elukey) [14:21:13] 10Analytics, 10ChangeProp, 10EventBus, 06Services (later), 15User-mobrovac: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088#2995179 (10GWicke) >>! In T157088#3272831, @Ottomata wrote: > Yo dudes! We are ordering more Kafka hardware (see T152015). We've de... [14:34:43] 10Analytics, 10ChangeProp, 10EventBus, 06Services (later), 15User-mobrovac: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088#3273489 (10Ottomata) > a purely cpu bound service like eventbus could live anywhere; there is no real need to expand the Kafka cluste... [14:47:16] 10Analytics, 10ChangeProp, 10EventBus, 06Services (later), 15User-mobrovac: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088#3273498 (10Pchelolo) > While template edits should only enqueue a constant number of small jobs directly, I suspect that the expansio... [15:52:29] 10Analytics, 10ChangeProp, 10EventBus, 06Services (later), 15User-mobrovac: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088#3273566 (10elukey) >>! In T157088#3273498, @Pchelolo wrote: > How pressing is the issue? Can we have a meeting after the hackaton, di... [16:06:49] elukey: https://phabricator.wikimedia.org/P5457 [16:06:57] :) [16:07:11] 10Analytics, 06Research-and-Data: clientIP needs to be collected as part of the schema or ... - https://phabricator.wikimedia.org/T165678#3273579 (10leila) [16:08:29] urandom: ah! nice :) [16:09:01] elukey: it was a while ago, i guess i forget to ping you when i pushed the new version out [16:09:13] s/forget/forgot/ [16:11:36] urandom: upgraded! [16:11:56] cool! [16:16:29] urandom: about trickle fsync vs dirty_background_bytes - should the former be false when we set the latter or are they independent? didn't get which one I should use (IIUC dirty_background_bytes is better for multi instance) [16:18:08] elukey: i would disable tricke_fsync as part of the same change [16:19:10] elukey: the timing probably isn't critical either way, you guys batch write at a fixed time of day [16:19:30] and i doubt it would create such a difference as to cause major problems [16:19:36] its a write optimization [16:20:43] elukey: also, the value used in dirty_background_bytes was roughly the aggregate of the trickle_fsync for 3 instances, you guys have 2 [16:21:02] i'd be surprised if that matters a whole lot, but it's worth consideringg [16:21:23] urandom: now I am getting the point, reading the cassandra and linux docs [16:21:37] we'd need to have that parameter restbase independent probably [16:21:59] and if it is a multiplier of something, make it explicit in the profile's logic [16:22:02] no? [16:22:02] elukey: you could always do a test, disable puppet, disable trickle_fsync, restart, set dirty_background_bytes, enable puppet [16:22:07] and let it run over night [16:22:32] sure makes sense [16:22:35] will do it next week [16:22:41] elukey: i played with various values at the time and it didn't seem too sensitive [16:23:00] i double anything +/- 10MB would result in a huge diff [16:23:00] urandom: as far as I can ready dirty_background_bytes should be enough no? [16:23:08] yup [16:23:16] s/double/doubt/ [16:23:34] all right, the dirty_background_bytes option seems better (let the kernel do it for you, managing it via sysctl [16:23:46] yeah, it did help some [16:23:59] increase write throughput, lowered write latency [16:24:02] a little [16:24:15] all right will do a test and if good I'll merge the code review [16:24:19] enough to measure, and enough to warrant making good on a config change [16:24:38] I like the idea of re-using the cassandra code for aqs and restbase [16:24:51] yeah, the fewer exceptions the better IMO [16:25:09] the few unnecessary exceptions, anyway :) [16:25:11] fewer [16:25:37] uniformity ftw! [16:38:59] \o/ [16:39:23] all right running errand to Prague, ttl! [17:08:56] 10Analytics, 06Research-and-Data: clientIP needs to be collected as part of the schema or ... - https://phabricator.wikimedia.org/T165678#3273760 (10schana) From the email thread: Here’s a little more flushed-out query for extracting relevant pieces of the event while retaining the webrequest data. The event... [18:22:08] 10Analytics, 06Research-and-Data: clientIP needs to be collected as part of the schema or ... - https://phabricator.wikimedia.org/T165678#3273816 (10leila) @schana per our follow up conversation in IRC and your suggestion: this may be a more accurate way to link the two datasets (EL and webrequest logs) anyway... [23:01:12] 10Quarry: No data after 20170517193000 available via Quarry from tables (recentchanges, revisions, logging) for several Mediawiki databases (svwiki_p, fiwiki_p, nowiki_p, ...) - https://phabricator.wikimedia.org/T165705#3274403 (10Larske) [23:03:34] 10Quarry: No data after 20170517193000 available via Quarry from tables (recentchanges, revision, logging) for several Mediawiki databases (svwiki_p, fiwiki_p, nowiki_p, ...) - https://phabricator.wikimedia.org/T165705#3274415 (10Larske)